AdMassive9465 t1_j59pyqi wrote on January 21, 2023 at 12:16 PM

Personally, as a reviewer, this year's ICLR reviewing system was a trainwreck.

Reviewers had 12 days to review ~5 papers. Twelve days to review 5 papers submitted at (arguably) one of the best venues of this (complex) domain.
Of the 16 expected reviews (for these 5 papers), only 8 had been delivered before the deadline (5 were mine). 5 were submitted on the day of the deadline. 3 were submitted in the "buffer" period before the opening of the rebuttal phase.
A paper was withdrawn by the authors after the opening of the rebuttal phase. Surprisingly -- for this paper -- a reviewer proposed to reject the paper and, among the motivations, they stated that the paper should "compare" with a paper that had been accepted 1 month before the submission deadline of ICLR'23, and that was supposed to be presented at a conference held in mid 2023.
Of the other 16 reviewers assigned to the other papers I reviewed, only 3 wrote one post (aside from the review). I even explicitly asked the support of other reviewers w.r.t. an obscure point of a paper, and received no response (this paper was rejected).
I championed one paper, which was ultimately rejected by the AC who had great misconceptions of the paper as a whole. None of the reviewers chimed in, and my attempts at changing the AC's opinion were in vain (the AC was not "wrong", but they were just looking at the paper from the wrong perspective). I feel very sorry for the authors since they had very little they could do: they interacted just with myself, and I was pleased by the paper; and now they got a reject due to concerns expressed _after_ the closure of the rebuttal phase (and raised just between myself and the AC).
Out of the 5 papers I reviewed, only one was accepted.

I think that the worst part, however, is that all of the above will be "meaningless" given the ridiculous amount of submissions to ICLR'23.

Frankly speaking, this is discouraging as a reviewer who "tries their best" to help research.

^(Edit: fixed numbers after double-checking)

ukamal6 t1_j59rey7 wrote on January 21, 2023 at 12:33 PM

About the trainwreck review part, I totally agree with you. Even after the deadline, I was getting random (I am saying random because I had almost close to zero expertise in those fields, so I had to decline all those requests) emergency review invitations which made me understand the overall situation. One thing that made me really confused was: why did they take so much time (almost 2 months, after the author discussion period was over) to send out the decision? I was really not sure why the meta-review process deserved this much time (~60 days) when the actual review time was just 12 days! Any thoughts about this?

AdMassive9465 t1_j59s6n6 wrote on January 21, 2023 at 12:41 PM

Agreed. I don't know what happened behind the curtains, and I hope that whatever happened was done for a good reason.

Yet, I have a bad aftertaste: I cannot stop thinking that my effort was useless. Truly, I could have written shit-reviews such as "no novelty, the paper ultimately describes a method related with neural networks, which have been studied for decades" (jk) and forget about it.

spionski t1_j5ac1t4 wrote on January 21, 2023 at 3:30 PM

This is very interesting to read. For the paper you championed, was that discussion between you and the AC public, i.e. visible to authors? I had a similar experience from an author perspective. We had really constructive discussions with all the reviewers, and were fairly optimistic we'd get in, but then the meta-review was pretty disconnected from all of that. Unfortunately no idea what may have happened between AC and reviewers, so it's a bit of a mystery from our perspective. It's frustrating to have all these in-depth discussions, but then no opportunity to clear up even basic misunderstandings in the meta-review, where it actually matters. Having a discussion period with the AC would have made such a difference, and given the timeline it seems it would totally have been feasible.

AdMassive9465 t1_j5aliqj wrote on January 21, 2023 at 4:35 PM

It was not: in theory, it should have been a discussion that involved "only" the reviewers and the AC. However, none of the reviewers (besides me) participated---thereby resulting in a 1:1 discussion with the AC (whose remarks could not have been addressed in any "revision" of the paper, since this occurred in December).

Interestingly, there is even a "gap" between the Reviewer and AC guidelines:

according to the AC guidelines, the authors could "participate in the discussion" occurring between November 18th and December 12th...
...but this was not stated in the reviewer guidelines, which stated that, after November 18th, the only role of the reviewer was to have the "virtual discussion with the AC for borderline papers".

I was unaware that the authors could still "participate" after November 18th, so I did not even think that it was possible to involve them in the discussion (had I known that the other reviewers would have done nothing, I would have certainly involved the authors)