Viewing a single comment thread. View all comments

SkinnyJoshPeck t1_jdvk16j wrote

This is an important thing I've been telling everyone I can about - people talk about how GPT kills education because someone can just ask for a paper and never do the work themselves to learn.

This is a language model, not an encyclopedia, or a quantitative machine, or some other use. It fakes sources; it has no concept of right/wrong or truth vs untruth. It doesn't reason between sources.

The beauty of it is, frankly, it's ability to mimic (at this point) a pseudo-intellectual, haha. Kids are going to turn in papers sourced like they talked to their conspiracy theory uncle, and it will be the "watermark" of AI written papers. It can't reason, it can't generate opinions, thus it can't write a paper. We're long from that (if we could ever get there anyways).


adventuringraw t1_jdw6enx wrote

You're right that there isn't a system yet that has the power of a LLM without the risk of hallucinated 'facts' woven in, but I don't think it's fair to say 'we're a long ways from that'. There's a ton of research going into different ways to approach this problem, approaches involving a tool using LLM seem likely to work even in the relatively short term (production models in the next few years, say) and that's only one approach.

I certainly don't think it's a /given/ that this problem will be solved soon, I wouldn't bet money that you're wrong about it taking a long time to get it perfect. But I also wouldn't bet money that you're right, given all the progress being made on multiple fronts towards solving this, and given the increasingly extreme focus by so many researchers and companies on this problem, and especially given the fact that solutions like this are both promising and seemingly realistic. After all, if there's a sub-system to detect that an arxiv search should be used to verify a reference before giving it, you could at least eliminate halucinated examples in this narrow area. The downside then might just be an incomplete overview of available papers, but it could eliminate any false papers from what the user sees.

All that said, this only fixes formal citations with a somewhat bespoke system. Fixing ALL inaccurate facts probably won't be possible with even dozens of 'tools'... that'll take more what you're thinking I imagine: something more like a truly general learned knowledge graph embedded as a system component. I know there's work on that too, but when THAT's fully solved, (like, TRULY solved, where modular elements of the world can be inferred from raw sensory data, and facts accumulated about their nature from interaction and written content) we'll be a lot closer to something that's arguably AGI, so... yeah. I think you're right about that being a fair ways away at least (hopefully).


Ok-Hunt-5902 t1_jdvm7kp wrote

It’s as much an encyclopedia as any.. outdated/incorrect info is ubiquitous in them. What op shows here is ChatGPTs potential to show more accuracy now and in future iterations.


SkinnyJoshPeck t1_jdvpkge wrote

but as others are saying, who knows if those confidence scores aren’t also just generated to look like confidence scores. we should ask it for a bunch of confidence scores for sources and see what the actual classification metrics are.. it could just be assuming the further a source is from the top, the less likely it is to be a real source. i don’t see how it could possibly have an understanding that isn’t completely binary since it seems to be generating the fake sources itself.

imo, it’s a bit sketchy if it only identifies its own fake sources with anything less than 100% - it implies basically two things: there is secondary models for true v. false that’s detached from its generative stuff (why wouldn’t it have something that says “this isn’t a great response, maybe i should admit that”); and it seems to have the ability to deceive lol


Peleton011 t1_jdvtqq0 wrote

Unless I'm wrong somewhere LLMs work with probabilities, they output the most likely response based on training.

They definitely could be able to show you how likely of a response a given paper is, and given that the real papers would be part of the training set answers it's less sure of are going to statistically be less likely to be true.


RageOnGoneDo t1_jdxm91o wrote

Why are you assuming it's actualyl doing that calculation, though?


Peleton011 t1_jdxolt1 wrote

I mean, i said LLMs definetely could do that, i never intended to convey that that's what's going on in OPs case or that chatgpt specifically is able to do so.


RageOnGoneDo t1_jdxoqxf wrote

How, though? How can an LLM do that kind of statistical analysis?


TotallyNotGunnar t1_jdwbg7n wrote

Students are already doing that with research engines. When I graded lab papers in grad school, I swear 80% of the students wrote down whatever they remembered from class and then back filled their citations using Google Scholar results.


NigroqueSimillima t1_je2l4j3 wrote

It absolutely has a concept of right or wrong. Ask it basic true or false questions and it will get them right most of the time.

In fact I asked it for grammar mistakes in your post and it noticed you used the incorrect for of "its" in your 3rd paragraph, and used "anyways" when it should be "anyway".

Seems like it knows right from wrong.

>It doesn't reason between sources.

It doesn't have access to source, it only has access to its own memory.

This is like if you asked me a question and I answered correctly, then you asked for sources and I tried to remember where I got it from. I could tell you sources that I think are right but are actually wrong due to my own memory degradation. Human memory is also very unreliable, but they're very good at making up things that "sound" like they could be right to them.

People "hallucinate" facts all the time.


gnramires t1_jdvt5u2 wrote

I don't think this is accurate. I think it's clear that truth is an important concept in human conversations, and it seems advanced models can clearly learn and model truth as an abstract concept and probably have an internal representation of reality that aids in its overall "job" of text completion.

Indeed, this does not alone guarantee that text completion tasks will really reflect reality, the true state of the world (again, because text completion can be in any context). However, with good prompts, and with an aid of reinforcement learning, I believe the "neural circuits" and neural representations associated with truth, distinguishing whats real or not, and building internal models of reality, get exercised and prioritized. In this way, a Chat model trained for and encouraged through prompts for truth telling indeed does have a genuine notion of truth and capability to understand reality -- although clearly not perfect by any means yet.