I'm a Master's student in Computer Science that is working on research papers in Machine Learning. We submitted a paper to a workshop for a CS research conference, and to my surprise I also received an invitation to review other papers in the same workshop.

I guess my main question is, when is someone "qualified" to peer review other people's work? I ask because I haven't reviewed other researcher's papers before, but have submitted a paper and read many research papers to write a couple others. But the AI field is so vast, I feel like I barely scratched the surface with all the concepts and ideas being thrown around. Some of the abstracts I read to bid on reviewing later have familiar terminology/concepts, but others seem entirely alien to me.

Reviewers have to start somewhere, but is there a baseline competence to properly peer review other papers?

Comments

You must log in or register to comment.

neuralbeans t1_irdc8aw wrote on October 7, 2022 at 4:28 AM

When you review, one of the fields to fill in is how confident you are in your review. If you don't understand your assigned paper then you give yourself a low confidence rating so that the chairperson would give more weighting to other reviews of the same paper. Usually there is a shortage of reviewers so they accept anyone who is willing to do it for free.

Flippynips987 t1_irdr8tq wrote on October 7, 2022 at 7:52 AM

What the others say plus, workshop papers are the lower end of the line. They usually have low novelty and contribution to the field and workshops are for networking and ideation. The whole process is less serious and a workshop publication does not count as major contribution to a PhD.

A good review not only shows weaknesses but also emphasizes strengths and give tips how to overcome weaknesses.

Overall a review is honest and if you feel not really ready for doing it, do it for the practice and tell the chairs your own doubts (there are usually ways to do so). You won't make any progress if you hesitate doing it.

TheDeviousPanda t1_irdcat4 wrote on October 7, 2022 at 4:29 AM

The primary factor in determining the quality of a review is not the maturity of the reviewer but in my opinion the time that is invested into reviewing the paper. AI research papers are not so complex; generally undergraduates that I have worked with are able to fully understand a paper when given sufficient time. By contrast, the average review you can expect to get at a conference will be from a reviewer who has spent maybe 30 minutes on your paper.

I would encourage you to read the papers that you have been assigned in depth and try to give good reviews. Assuming that there is some discussion period where you can see other reviews on the same paper, you will get some indirect signal on the quality of your review.

dupondius t1_irdr8lx wrote on October 7, 2022 at 7:52 AM

I know the feeling! I started working in a research team straight out of my bachelor's and was also intimidated by the first review invitation.

In my (in)experience, the most useful skill to have is being able to read other papers since no one would be familiar with everything else. I think specific instructions like this are helpful for us newer reviewers: https://2020.emnlp.org/blog/2020-05-17-write-good-reviews

Also, one thing that really helped was the conference we submitted to (ACL I think) had a program where first time reviewers would have their review looked over by a senior reviewer before submitting. If this doesn't exist for your venue, maybe do this informally with a more experienced colleague.

LordWetbeard t1_ire3lm2 wrote on October 7, 2022 at 11:04 AM

It depends on the venue, I'd say. Some are happy to have any grad students review regardless of experience. Others expect you to have already completed your PhD or have 1-3 publications in the same venue or one of equivalent stature.

Downess t1_irejm49 wrote on October 7, 2022 at 1:45 PM

Baseline competence? You should have published a few papers in the field and seen the review process from the other end first.

sensei_von_bonzai t1_irf0nv2 wrote on October 7, 2022 at 3:56 PM

From some of the reviews I see, a high school degree or an equivalent seems to be a sufficient condition.