respeckKnuckles t1_j1vg27f wrote
Please let us know when you get some reportable results on this. I'm having trouble convincing fellow professors that they should be concerned enough to modify their courses to avoid the inevitable cheating that will happen. But in a stunning display of high-level Dunning-Kruger, they are entirely confident they can always tell the difference between AI and human-generated text. Some data might help to open their eyes.
MrFlamingQueen t1_j1vmykp wrote
They're not worried because on some level, it is recognizable, especially if you have a writing sample from the student.
On the other hand, there are already tools that can detect it, by comparing the sequences to the model's internal weights.
respeckKnuckles t1_j1vo5s7 wrote
I've never seen empirical study demonstrating either: (1) professors can reliably differentiate between AI-generated text and a random B-earning or C-earning student's work, or (2) those "tools" you mention (probably you're talking about the huggingface GPT-2-based tool) can do that either.
You say "on some level", and I don't think anyone disagrees. An A-student's work, especially if we have prior examples from the student, can probably be distinguished from AI work. That's not the special case I'm concerned with.
MrFlamingQueen t1_j1vskd9 wrote
Thank you for your response. You are correct that it may be easier to distinguish between the work of an A-student and AI-generated text. However, it is possible that professors can still differentiate between AI-generated text and the work of a B-earning or C-earning student, even if it is more difficult. This is because professors are trained to evaluate the quality and originality of student work, and may be able to identify certain characteristics or patterns that suggest the work was generated by an AI.
As for the tools that I mentioned, it is possible that they may also be able to differentiate between AI-generated text and human-written text to some degree. These tools use advanced machine learning algorithms to analyze text and identify patterns or characteristics that are indicative of AI-generated text. While they may not be able to reliably distinguish between AI-generated text and human-written text in all cases, they can still be useful for identifying potentially suspect text and alerting professors to the possibility that it may have been generated by an AI. Overall, it is important for professors to remain vigilant and use their expertise and judgement to evaluate the quality and originality of student work.
j03ch1p t1_j1wnqgz wrote
...is this AI written?
MrFlamingQueen t1_j1wp6pv wrote
Yes, that was AI written as a cheeky way of demonstrating it can be recognizable after having a writing sample of mine in the previous post.
j03ch1p t1_j1yjdim wrote
That's just for the time being.
Liwet_SJNC t1_j1wwv7z wrote
I'm not sure this would be terribly convincing unless the professors in question are routinely setting 100 word essays on 'whatever'. In general a one sentence quotation of unknown surrounding context is always going to be much harder to identify as being from an AI than 5000 words on a known topic that have to be self-contained.
respeckKnuckles t1_j1x00bh wrote
Yeah we have that, at least. The problem is that the pandemic moved a lot of classes and assignments online. Whether it is their choice or not, a lot of professors are still having homework assignments (even tests) online, and on those you often will see prompts asking for short 100-word answers.
Dicitur OP t1_j1vg621 wrote
Good idea. I will try and get more people to use it too.
respeckKnuckles t1_j1vgrit wrote
It'd be great if you could extend it to longer texts, like paragraph-lengths. A lot of these are recognizable quotes, so it throws off the reliability of the assessment a bit (especially if the people doing this might be, say, English professors).
Dicitur OP t1_j1vgy9c wrote
Actually, for this purpose, it would be interesting to make a different site with student and AI essays. It is harder for AI to compete with Shakespeare than with a mediocre student.
Viewing a single comment thread. View all comments