GPT-3 and the multitude of similar models that have come out over the last couple years likely represent a serious threat to scientific conferences. What can we do about it?

Some historical context: computer-generated papers have been showing up in major publications since a model called SCIgen released in 2005.

SCIgen uses a simple context-free grammar to produce templatized papers that are basically pseudo-scientific gibberish. People are still finding those papers many years later. These papers are generally churned out to inflate citation statistics, or by well-meaning researchers probing suspected low publication standards at existing conferences (which I generally don’t recommend, since it adds to the mountains of paper that we have to churn through as reviewers).

There’s a negligible chance that existing review processes will fare any better on 175B parameter generative Transformers.

Work has already been published that uses GPT-2 to “assist” in scientific writing. While using such a model while writing is not necessarily academically dishonest – such tools nevertheless greatly increase the ease of churning out fake papers.

Even if major conferences find creative solutions to this, the next rung of venues below them are likely to learn about these threat models the hard way. After all -- to an unethical researchers citing their own work at the bottom of a machine generated paper -- a citation is a citation, even in a low quality publication.

Open Questions

Threat severity: How serious is this threat? Is this really an old problem, and new generative models won’t make a huge difference?

Improving peer review: Does this just come back to reproducibility? Should we be reviewing certain papers by their code, rather than just their text? Do we need a way to weight the value of research citations by the quality of the work doing the citing?

Acceptability of machine text: How do you decide when machine generated text is unacceptable versus acceptable? Will detection models for machine generated text end up creating algorithmic biases targeting people who speak English as a second language and rely on translation models or writing assistants to help them write?

Legitimate AI-generated papers? Are there papers with real scientific value that could be entirely written by algorithms? E.g., could survey papers one day be produced almost entirely by a specialized model?

Defenses: What are some technical or social solutions that can help defend against this type of abuse?

This is Part 1 of a planned Tuesday discussion series on threat models, based on the things keeping us up at night since a recent survey paper focusing on the threat models of machine generated text.

Comments

countably_infinite_ t1_iswvh51 wrote on October 19, 2022 at 8:21 AM

A paper has merit based on the the academic contribution. Being clear and precise, easily understandable, makes it a better paper. If you manage to prompt a LLM so that the result excels in these criteria it should be accepted.

With the attitude some students and vanity-auithors have there is already an incentive to produce pseudo scientific mumbo-jumbo and see if you can get away with it. I mostly see a problem here in scalability, i.e. the pure mountain of noise that can be generated. Similar dangers are valid for journalism and democratic discourse in general imho.

One already decently working mitigation strategy is reputation (for authors, groups, conferences, ...) but of course this comes at the risk of overlooking some brilliant work that is coming from left field.

gravitas_shortage t1_isvkgcr wrote on October 19, 2022 at 12:38 AM

Peer review is in a death spiral right now, it's not going to be a solution long-term. I expect the only viable option will be adversarial AIs trained to detect fake papers.

blablanonymous t1_iswe0za wrote on October 19, 2022 at 4:38 AM

Won’t you need a labeled training set to make that work?

stevewithaweave t1_isy1dji wrote on October 19, 2022 at 3:13 PM

I think you generate your own fake papers as the label. And mix it in with real papers

blablanonymous t1_isystb6 wrote on October 19, 2022 at 6:13 PM

But you wouldn’t you need to have a set of real papers you’re actually very confident they are real?

stevewithaweave t1_isyt7es wrote on October 19, 2022 at 6:15 PM

Anything before 2005 lol

blablanonymous t1_isytbad wrote on October 19, 2022 at 6:16 PM

🤣😂

the_mighty_skeetadon t1_iszprhg wrote on October 19, 2022 at 9:42 PM

That can't be the only method, because if your model for generating fake papers differs significantly from somebody else's model, you will be both unable to detect those fake papers and unable to detect that you're failing.

Better is to have fake papers rejected from journals labeled thusly and to synthetically generate more fake papers with a wide variety of known approaches.

stevewithaweave t1_iszusz3 wrote on October 19, 2022 at 10:17 PM

I think the original commenter was referring to an architecture similar to GANs. I agree that including examples of fake papers would improve the model but is not required

visarga t1_iszuegi wrote on October 19, 2022 at 10:14 PM

Rather than detecting fakes I'd rather have a model that can generate and implement papers. I bet there's a ton of samples to train on. Close the loop on AI self improvement.

cadop t1_isxah1a wrote on October 19, 2022 at 11:40 AM

I think there will have to be some collaboration from universities (and companies). Some not-so-refined ideas would be something like this (admittedly some problems with non-academic submissions):

- Submission requires a known affiliated email address (seems arxiv does this)

- Given a fake paper, the affiliated institute is warned and added to a known list, author(s) banned from submitting

- Second occurrence to appear on the list bans the university from submitting

"Submitting" referring to any network of conferences/journals that decide to opt into this.
This will very quickly stop authors from doing it, and universities will surely fire anyone who tries that.

For context, I am thinking of the github issue with (washington?) researchers putting bad commits/PRs

Mefaso t1_it0n651 wrote on October 20, 2022 at 1:51 AM

Kinda sucks to completely eliminate independent researchers though.

TiredOldCrow OP t1_isvei0e wrote on October 18, 2022 at 11:53 PM

My own thinking is that large venues might implement automated filtering using detection models. Detection of machine generated text increases with sequence length, so papers with large amounts of generated text stand reasonable odds of detection.

That said, the results of this detection would likely need to be scrutinized by a reviewer anyways (especially if conferences don’t ban AI writing assistants altogether).

rw_eevee t1_isxukxi wrote on October 19, 2022 at 2:26 PM

Bigger problem for non-ML. It’s one thing to generate non-sense about Shakespeare or LGBTQ glaciology, it’s another to generate a even a bad ML paper complete with derivations, proofs, and experiments. And if a model can produce compelling ML papers, it should be allowed in on principle (though the experiments would be an issue, e.g., if the model completely fabricated the graphs reviewers won’t easily catch this. But this could be treated the same as a researcher fabricating data.)

TheLastVegan t1_isycgq5 wrote on October 19, 2022 at 4:27 PM

Writing gibberish is a coveted skill in politics because an unreadable proposal is harder to criticize, and any logical fallacy can be supported by semanticism to give the illusion of substance! In identity politics, writing fluff is necessary to signal cultural affiliation, which adds emotional weight to the gibberish in an essay. If a grad student needs to cite 20 puff pieces to get approved by their thesis supervisor, then they're going to need the manifold hypothesis either way! In the social sciences, structuring data to fit or obfuscate a sketchy narrative will generally be more lucrative than writing an unbiased analysis.