[D] What are the benefits of being a reviewer?

Submitted by Signal-Mixture-4046 t3_ykbp1s in MachineLearning

So I just got invited to serve as a reviewer for CVPR'23, but I am quite new to the field. I have only one accepted paper and one under review at top conferences, and I have never been a reviewer before.

Because I understand that being a reviewer (especially for CVPR) is a huge responsibility, I would love to know what benefits I could gain from this experience. For those who have done it before, what makes you voluntarily want to be a reviewer again?

49

Comments

You must log in or register to comment.

DeepGamingAI t1_iusdqfr wrote

Think of it as a GAN, you train generators when you publish and train your discriminator when you review. Do these processes together alternately and you can see what you'll slowly converge to :)

94

curiousshortguy t1_iutwnuz wrote

I think this is a terrible analogy. Reviewing isn't about rejecting. It's about enabling good scholarship and guiding researchers.

Putting the review as the gatekeeper just gives you shitty results, as seen by the last round of reviews at large conferences with uneducated and unqualified reviewers.

21

DeepGamingAI t1_iuv83lt wrote

>It's about enabling good scholarship and guiding researchers.

You just described the role of a discriminator in a gan

​

>uneducated and unqualified reviewers

op got an invite because they published there before, its on merit not a random review request. besides, the question solely focuses on how reviewing benefits the reviewer, it doesnt seem to cover the whole picture surrounding peer review system

4

curiousshortguy t1_iuvswa7 wrote

>You just described the role of a discriminator in a gan

I disagree. The discriminator is just used in a binary fashion and doesn't add a lot of explanatory value.

​

Just to clarify, I am not trying to say that OP is unqualified. But I think just thinking about it in a binary way isn't enough for good peer review and a functioning system.

1

rustyryan t1_ivx9why wrote

The critic provides gradients to the generator, which indicate what aspects of its generation led to the classification.

1

dssevero t1_iuyj0q1 wrote

Horrible analogy. While the discriminator improves the generator, there are no resulting consequences for the real/fake samples. This is completely different.

Besides, you can do this without the obligation of reviewing (which I encourage you to do).

2

IPvIV t1_iusexqq wrote

You get to put it on your CV. Probably see some cool papers before they come out. And train your discriminator as the other comment says

I also did quite a bit of reviewing this year despite being new to ML, I think reviewing 1-2 papers is good for a new person but the marginal benefit decreases sharply when you review more

39

RustBucket03 t1_iussp02 wrote

I've never actually seen people putting conference reviewing info on their CV. For more senior researchers, being on the committee or something sure, but never really seen just putting being a reviewer in. Also, unless someone gets "great reviewer" awards or something, just being a reviewer doesn't really say a lot, does it?

11

IPvIV t1_iuswk33 wrote

I just put it in a line under my “service” section, (e.g Reviewer: conference1 (year), conference2 (year)) most other grad students I know do the same. When you’re at a more senior level it might not matter as much anymore though.

20

RustBucket03 t1_iusxpd2 wrote

Yeah, I never did it (as a current grad student) not because I didn’t think it was an important task. But rather thought that it's just something that everyone does/should do and doesn’t add anything to the CV.

Maybe I should add a line like that as well.

9

stiffitydoodah t1_iusveaa wrote

It says that you understand that the community functions by the service of its members, and that you're willing to carry a share of that load.

9

aepr88 t1_iutje8e wrote

Meh, no one cares if this is on your CV.

3

ArmandDerech t1_iuvuqjp wrote

Not true, it makes a little difference for early grad students applying for competitive external funding

1

fastglow t1_iusllds wrote

I see it mainly as one of my responsibilities as a researcher: I submit papers, so I should also review papers. If not us, then who?

But I think it does benefit me in that I have gotten better at distinguishing good research ideas from bad ones, and good experimental protocols from bad ones, good reporting from bad reporting, etc.

39

Proud_Ad_5895 t1_iuszsmc wrote

>If not us, then who?

The people harvesting the profits should ideally be the ones doing the work!

22

Mefaso t1_iuva45y wrote

>The people harvesting the profits should ideally be the ones doing the work!

Conferences are all nonprofits

4

PeedLearning t1_iuwdj73 wrote

[citation needed]

Some are non-profit, but for example most of the IEEE conferences are for-profit... Another common theme is conference non-profits subcontracting a lot of organizational aspects to for-profits.

3

Mefaso t1_iuymntw wrote

IEEE is also a nonprofit

> Another common theme is conference non-profits subcontracting a lot of organizational aspects to for-profits.

That is true, but it's also a lot of work to organize a conference

1

aepr88 t1_iutk19c wrote

The first few times you are a reviewers are great and wonderful experience. It gets boring quick. One out of 10 papers is a good read. The rest are badly written or unnecessarily complicated. It takes a lot of times and efforts to review a paper. I used to be quite excited to review papers. But now, I decline almost all incoming invites. Just not worth the time.

The only good argument for reviewing is community service. You basically volunteer your time to make sure the community is strong. So, if you are still submitting papers for others to review, you should be doing the reviewing yourself.

9

px05j t1_iut968g wrote

As other said, it does help you gain more perspective.

However, since you mentioned that you are new and never reviewed before :

  • My suggestion would also be to take as much time as you need to review the papers (I took days initially as I would read the paper multiple times and will also read relevant citations).
  • If you really think you didn't get some part in the paper even after you tried reading the background. You can always tell this in reviewer discussion section, this will help meta-reviewer and other reviewers to look at that section.
  • There are blogs and even videos on how to do a good review.
  • Think that someone like you will receive this review, so think if you were in their place will the review you are providing be helpful.
  • Lastly, please go through the guidelines.
6

Seankala t1_iuuxb24 wrote

There's not really any merit. That's why the reviewing process is such a mess. There's virtually no way to incentivize or penalize people for being shitty reviewers.

6

danja t1_iusumf1 wrote

It gives you different perspectives.

I've only been on a handful of program committees but spent a couple of years reviewing tech books for a publisher. You use the word 'responsibility', yeah, hold that. Having to look hard at what people have done, critically, is really challenging but very educational. It may be things that you are not sure about, so you have to get yourself up to speed to do it justice.

It can be a nightmare - borderline cases are painful.

Seeing what works and what doesn't, for the subject matter, but also for the write-up, your main interface.

I'm not in academia so it isn't necessary for me, but I'm pretty sure I'd make a better paper now than before reviewing.

Also CV :) .

5

lack_of_novelty006 t1_iutrpf3 wrote

I have seen many people putting reviewing as a service in their resume, also some conferences allow you to be reviewer if you are one of the co-authors, thus hints at your credibility to be a part of such conferences. Finally reviewing also helps build a better profile for visa and all, if you are in the USA as an immigrant.

5

dssevero t1_iuuzc10 wrote

IMO absolutely nothing.

The truth is that without free reviewing from the community: research would simply not exist (at least peer-reviewing wouldn't). That is why we "must" do it.

We then do mental gymnastics to convince ourselves that there are benefits (many cited here in the thread), when the truth is time would be better spent (at the individual level) on other tasks.

That being said, please put time into reviewing so the community can continue to exist ;)

5

Signal-Mixture-4046 OP t1_iuv2fc4 wrote

Yeah, I think I agree with you the most. I feel like there is no real concrete benefit to being a reviewer, but I want to do it anyway as a way to "give back" to the community.

3

performanceburst t1_iuv3clj wrote

It can help you immigrate for some countries.

2

Signal-Mixture-4046 OP t1_iuv63v9 wrote

Really? I'm applying for PhD in the US this year. Can you explain more about how it helps?

1

CrazyCrab t1_iuvhe4r wrote

The US has a long term visa, I forgot its name. The idea is that it's for exceptional people, including exceptional scientists. And to prove that you're exceptional, there's a list of like 9 items, of which you need to have 3 or more or something like that. One of them is "has published an article in a serious publication venue". Another is "has been a reviewer of the works of others at a serious venue" or something like that. I wonder if other countries have this.

2

DanielSeita t1_iuz9g26 wrote

Yes, as far as I understand, it can help with getting a green card.

Disclaimer: I am not a lawyer (and I'm a US citizen so it wouldn't matter for me).

1

99posse t1_iuv3z54 wrote

For me the main advantage was that reviewing and organizing a conference forced me to read quite a few papers back to back, very carefully, verify results and look at the references. When I read for work, I only care about what I need and skip all the boring stuff (a lot, usually).

Talking in the past as I decided to leave the organization of the conference to younger people, stopped all reviewing, and abandoned the IEEE mafia.

2

oli4100 t1_iuvghh1 wrote

In general I think you should always review at least for those conferences/journals that you publish in yourself too. So, I simply consider it my duty and obligation to the field.

That said, I learn a lot from reviewing which also makes me want to voluntarily do it again. Different perspectives, different approaches, sometimes a peak into interesting new research directions, writing styles. There's a lot to learn from other people's work. Also, since I tend to read mostly literature related to my field, sometimes reviewing a paper that is outside my own narrow scope can be quite refreshing.

Sadly, I have the feeling that new reviewers are not always properly instructed / taught how to review by their mentors/superiors. This can lead to poor quality reviews (short, uninformative, using bad language, etc).

I'd very much advise you to always keep in mind: 'the person at the other side of this paper is just like me, struggling just like me, putting in insane amounts of effort just like me, and hoping to make a small impact just like me'.

2

trnka t1_iuy68yo wrote

There isn't much short-term reward, aside from exposing me to interesting papers now and then. Really it's about helping to create the kind of community I want to be a part of and helping to create the kinds of publications I'd like to read.

It's kinda like doing community cleanup. It's not really an immediate or direct benefit for myself.

2

rustyryan t1_ivxac9h wrote

Beyond the many things others have covered, at Google, an aspect of your performance evaluation is "community contributions". These are anything that makes Google a better place to work that is not directly related to your work responsibilities. Everyone is expected to make community contributions, and sometimes the lack of community contributions can lead to lower performance ratings.

Conference / journal reviewing is traditionally treated as a community contribution in Google Research.

As your level scales, significant participation in the research community (e.g. being an AC, conference organizer, etc.) begin to be required aspects of the Research Scientist job ladder.

No idea if anything similar exists at other companies.

1