[D] Git Re-Basin Paper Accused of Misinformation

Submitted by fryingnem0 t3_yo5e4v in MachineLearning

I read the paper when it was posted here a couple months ago. I thought it was a pretty interesting work so I was curious to check if the authors submitted to ICLR 2023.

I found out that the the paper received glowing scores of 8, 8, 10 from the reviewers, and another researcher apparently didn't agree with the reviews in the public comment section here. They accused the paper of rehashing prior works, stating exaggerated and deceptive claims of results that are already known. I need to take a closer look at the comment to see if its points are valid.

51

Comments

You must log in or register to comment.

dineNshine t1_ivd4bwd wrote

Can we please avoid using the word misinformation while describing a paper? It is a very loaded word and it makes me cringe.

108

mdibmpmqnt t1_iveqrpw wrote

For those that don't read the link in question, the word misinformation is not OPs

It is with great regret that we need to bother you for a case where we believe misinformation is being seeped into the scientific process.

26

capStop1 t1_ivfw6bv wrote

I think is fairly overused in a context when you don't agree with the other person instead of its real meaning

6

Seankala t1_ivdqt68 wrote

I think the word "misinformation" is a little dangerous to be using here. People can criticize the authors of not providing any actual novelty (which is actually super common) but pointing fingers and saying they're "spreading misinformation" is a little much.

44

netw0rkf10w t1_ived3vs wrote

The paper is accused of being simply a rehash of previous work (which is much stronger than "misleading (presentation of) contributions"). The accuser supported his claim with detailed technical arguments, which I find to be rather convincing, but of course I would prefer to hear from the authors and especially from other experts before drawing any conclusions.

In general I believe that "misleading contributions" should not be tolerated in academic research.

Whatever the results will turn out, I love the openness of ICLR. There is a paper accepted at NeurIPS 2022 that is presented in a quite misleading manner (even though related work had been privately communicated to the authors via email during the review process). I would have loved to post a comment not to accuse of anything but to point out previous work and provide technical clarifications that I think would be beneficial to the readers (including the reviewers). Unfortunately this is not possible.

P/s: Some previous comments question the use of the word "misinformation". I would have used "misleading" (which is more common in academia, but perhaps a bit light if the accusation is true), though I don't feel too much difference when hearing "misinformation" over "misleading" (being a non-native English speaker). According to Oxford Dictionary, they are more or less the same:

>misinformation: the act of giving wrong information about something; the wrong information that is given
>
>misleading: giving the wrong idea or impression and making you believe something that is not true

The point here is that the accuser may not be a native English speaker either, and thus his technical arguments should not be overlooked because of this wording.

27

VeritaSimulacra t1_ivcqown wrote

I’m not up to date with this work, but this sort of rehashing without credit is insanely common, most of the time no one points it out tho

22

jfrankle t1_iveolu5 wrote

Oh come on. The author of the Git Re-Basin paper is a scientist acting in good faith based on the results they believe they have observed. Chill out and let the scientific process play out. Nobody here is acting out of malevolence, except maybe you for stirring the pot and putting that author through unnecessary pain. I've been there as a junior scientist getting beaten up on OpenReview. That was bad enough without a live audience on Reddit commenting on it.

13

t0t0t4t4 t1_ivettuk wrote

ICLR is open to public review and he is posted one supported by technical arguments. Are you aware that this is part of the "scientific process" that is accepted by the conference? The authors still have an opportunity to give a response, and then based on that it is the responsibility of the reviewers and the committee members to have an in-depth discussion on the merits of the paper.

If you want to defend the authors, then maybe consider doing that on the technical aspects?

14

ktpr t1_ivoev3k wrote

The scientific process is typically undertaken within context of scientific venues. The authors can not be realistically expected to respond to ICLR and arm chair ml geniuses on the internet. Scientific venues aim to provide minimally acceptable quality control.

1

juliusadml t1_ivgj8ks wrote

Completely agree with your point. It is hard to read this 'rebuttal' as anything but a deliberate attempt to 'sink' this paper at a time when the authors are responding to the reviewers. Now, they also have to respond to a public comment that says that they deliberately over-claimed and rehashed previous ideas. This kind of grievance should've been handled via email.

While it is easy to think that an author is being malicious. Often the honest truth is just that there are *a lot* of papers out there and one might often miss a reference here and there. When these references are pointed out, they can be easily incorporated and more carefully contextualized.

The public comment even went further and replied to each reviewer. They are essentially saying please reject this paper! This level of interference is insane to me. This said, academic 'brand' battles never cease to surprise me.

2

needlzor t1_ivessgu wrote

Misinformation is not necessarily malevolent, it's just misleading or incorrect. Disinformation is deliberately spread false information.

1

jfrankle t1_ivetqzy wrote

It's science. We have no idea what the truth is, and an inherent part of seeking it is that we'll propose ideas that are incomplete or wrong.

−1

apliens t1_ivggkdd wrote

Agreed, I would argue that no paper is free of "misinformation" in the sense described above.

1

Competitive_Dog_6639 t1_ivfrjh9 wrote

My take: the prior work mentioned doesnt undermine the main claims of the paper, which is that without retraining one can find permutations to map nets to the same basin.

The objection to this point is raise in part C) by the commenter, where an appeal is made to the commenters own paper. I read that and didn't see explicit ideas related to connected modes in the commenters paper. Plus, the commenters paper retains the nets, which is against the main idea of git reason. While the ideas of mode connectivity may be latent, they are not mentioned at all. Why is it the job of the git rebasin authors to dig so deeply into one out of thousands of related paper out there to give the commenter credit for an idea that isn't even explicitly discussed? Would also like to point out that the commenters paper might be missing related references to things like SWA, so maybe nobodys perfect?

Even if many of the methods come from previous work, I dont see anything to undermine the central claim of git rebasin, and for me that's that, it's an original and important idea. Could it use better relations to previous work? Sure.

7

netw0rkf10w t1_ivgnfd3 wrote

Could you comment on part A, B, and D? Let's consider the review in its integrality.

3

akardashian t1_ivguyrx wrote

I have no skin in the game, but this kind of provocation on SPS's part seems so slimy. Like I think it's fair game to leave an open comment on OpenReview, but to respond individually after each reviewer (and linking your own post) just feels like you are not acting in good faith.

7

apliens t1_ivcv1u4 wrote

I get the point that the official comment is trying to make, but this seems like something that should've been handled internally via email. The paper has been public for some time now (2 months, with substational discussion on reddit and twitter). To clarify, it feels as if SPS drafted their comment only after seeing the high scores on openreview. If the goal is to correct the record and present past work in the correct light - why not email the authors months ago?

4

Seankala t1_ivdqnvf wrote

Nah screw that. Everything should be publicly discussed.

19

sleeplessinseattle00 t1_ivdr71m wrote

Well, twitter has also been a big platform now a days to public discussions. (I’m not saying that to not put open review comment, but I’m saying they should’ve discussed this over there as well, given the amount of reaction to this paper)

0

Bot-69912020 t1_ive3h2r wrote

Can we assume SPS has not contacted the authors before? I don't know. Maybe he did and they ignored him. Anyway, I am waiting for the authors' response before making any conclusions.

8