Submitted by Singularian2501 t3_zm22ff in MachineLearning
Purplekeyboard t1_j0a7dwb wrote
Reply to comment by Nameless1995 in [R] Talking About Large Language Models - Murray Shanahan 2022 by Singularian2501
> You can't really explain those phenomena without hypothesizing that LLMs model deeper relational principles underlying the statistics of the data -- which is not necessarily much different from "understanding". > > > > Sure, sure, it won't have the exact sensori-motor-affordance associations with language; and we have to go further for grounding; but I am not sure why we should be drawing a hard line to "understanding" because some of these things are missing.
AI language models have a large amount of information that is baked into them, but they clearly cannot understand any of it in the way that a person does.
You could create a fictional language, call it Mungo, and use an algorithm to churn out tens of thousands of nonsense words. Fritox, purdlip, orp, nunta, bip. Then write another highly complex algorithm to combine these nonsense words into text, and use it to churn out millions of pages of text of these nonsense words. You could make some words much more likely to appear than others, and give it hundreds of thousands of rules to follow regarding what words are likely to follow other words. (You'd want an algorithm to write all those rules as well)
Then take your millions of pages of text in Mungo and train GPT-3 on it. GPT-3 would learn Mungo well enough that it could then churn out large amounts of text that would be very similar to your text. It might reproduce your text so well that you couldn't tell the difference between your pages and the ones GPT-3 came up with.
But it would all be nonsense. And from the perspective of GPT-3, there would be little or no difference between what it was doing producing Mungo text and producing English text. It just knows that certain words tend to follow other words in a highly complex pattern.
So GPT-3 can define democracy, and it also can tell you that zorbot mo woosh woshony (a common phrase in Mongo), but these both mean exactly the same thing to GPT-3.
There is vast amounts of information baked into GPT-3 and other large language models, and you can call it "understanding" if you want, but there can't be anything there which actually understands the world. GPT-3 only knows the text world, it only knows what words tend to follow what other words.
Nameless1995 t1_j0a97yt wrote
> But it would all be nonsense.
Modeling the data generating rules (even if arbitrarily created rules) and relations from data, seems to be close to "understanding". I don't know what would even count as a positive conception of understanding. In our case, the data that we recieve is not just generated by an arbitrarily created algorithm, but by the world - and so the models we create helps us orient better to the world and in that sense "more senseful", but at a functional level not necessarily fundamentally different.
More this applies to any "intelligent agent". If you feed it arbitrary procedurally generated data what it can "understand" will be restricted to that specific domain (and not reach the larger world).
> GPT-3 only knows the text world, it only knows what words tend to follow what other words.
One thing to note that text world is not just something that exists in the air, it is a part of the larget world and created by social interactions. In essence they are "offline" expert demonstrations in virtual worlds (forums, QA, reviews, critics etc.).
However, obviously, GPT3 cannot go beyond that, and cannot comprehend the multimodal associations (images, proprioception, bodily signals etc.) beyond text (it can still associate different sub-modalities within text like programs vs natural texts and so on), and whatever it "understands" would be far alien from what a human understands (having much limited text data, but much richer multimodally embodied data). But that doesn't mean it doesn't have any form of understanding (understood in a functionalist (multiply realizable) sense -- ignoring any matter about "phenomenal consciousness") at all; and moreover, none of these mean somehow "making likely prediction from statistics" is dichotomous with understanding.
Purplekeyboard t1_j0aayw0 wrote
One thing that impresses me about GPT-3 (the best of the language models I've been able to use) is that it is functionally able to synthesize information it has about the world to produce conclusions that aren't in its training material.
I've used a chat bot prompt (and now ChatGPT) to have a conversation with GPT-3 regarding whether it is dangerous for a person to be upstairs in a house if there is a great white shark in the basement. GPT-3, speaking as a chat partner, told me that it is not dangerous because sharks can't climb stairs.
ChatGPT insisted that it was highly unlikely that a great white shark would be in a basement, and after I asked it what would happen if someone filled the basement with water and put the shark there, once again said that sharks lack the ability to move from the basement of a house to the upstairs.
This is not information that is in its training material, there are no conversations on the internet or anywhere about sharks being in basements or unable to climb stairs. This is a novel situation, one that has not been discussed anywhere likely before, and GPT-3 can take what it does know about sharks and use it to conclude that I am safe in the upstairs of my house from the shark in the basement.
So we've managed to create intelligence (text world intelligence) without awareness.
respeckKnuckles t1_j0ajork wrote
> which actually understands the world.
Please define what it means to "actually understand" the world in an operationalizable, non-circular way.
Purplekeyboard t1_j0amzem wrote
I'm referring to two things here. One is having an experience of understanding the world, which of course GPT-3 lacks as it is not having any experience at all. The other is the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.
You could have an AGI machine which had no actual experience, no qualia, nobody is really home, but still understand things as per my second definition above. Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.
respeckKnuckles t1_j0apq5l wrote
I asked for an operationalizable, non-circular definition. These are neither.
> the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.
Can it be measured? Can it be detected in a measurable, objective way? How is this not simply circular: truly understanding is defined as truly knowing, and truly knowing is defined as truly understanding?
> Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.
This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.
Purplekeyboard t1_j0ar5ea wrote
>Can it be measured? Can it be detected in a measurable, objective way?
Yes, we can measure whether someone (or some AI) knows things, can analyze them, take in new information about them, change their mind, and so on. We can observe them and put them in situations which would result in them doing those things and watch to see if they do them.
An AI language model sits there and does nothing until given some words, and then adds more words to the end of the first words which goes with them. This is very different from what an AGI would do, or what a person would do, and the difference is easily recognizable and measurable.
>This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.
I wasn't talking about qualia at all here. You misunderstand what I was saying. I was talking about the difference between an AGI and an AI language model. An AGI wouldn't need to have any qualia at all.
Forms_Deep t1_j0b2vw6 wrote
Sorry to butt in, but I took your statement "Having an experience of understanding the world" as a reference to qualia also.
If it isn't, could you explain what you mean by "experience of understanding" and how it can be measured?
calciumcitrate t1_j0amrhw wrote
But a model is just a model - it learns statistical correlations* within its training data. If you train it on nonsense, then it will learn nonsense patterns. If you train it on real text, it will learn patterns within that, but patterns within real text also correspond to patterns in the real world, albeit in way that's heavily biased toward text. If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.
So, I don't think it makes sense to assign "understanding" based on the architecture as a model is a combination of both its architecture and the data you train it on. Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond to real world patterns.
* This isn't necessarily a direct reply to anything you said, but I feel like people use "correlations" as a way to discount the ability of statistical models to learn meaning. I think people used to say the same thing about models just being "function approximators." Correlations (and models) are just a mathematical lens with which to view the world: everything's a correlation -- it's the mechanism in the model that produces those correlations that's interesting.
Purplekeyboard t1_j0aorp8 wrote
>Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond the real world patterns.
GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.
I think that's the key point here. AI language models are text predictors which functionally contain a model of the world, they contain a vast amount of information, which can make them very good at writing text. But we want to make sure not to anthropomorphize them, which tends to happen when people use them as chatbots. In a chatbot conversation, you are not talking to anything like a conscious being, but instead to a character which the language model is creating.
By the way, minor point:
>If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.
I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.
calciumcitrate t1_j0asr1n wrote
> GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.
What differs GPT-3 from a database of text is that it seems like GPT-3 contains some representations of concepts that make sense outside of a text domain. It's that ability to create generalizable representations of concepts from sensory input that constitutes understanding.
> I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.
Maybe my analogy wasn't clear. The point I was trying to make was that if your argument is:
GPT-3 holds no understanding because you can feed it data with patterns not representative of the world, and it'll learn those incorrect patterns.
Then my counter is:
People being fed incorrect data (i.e. incorrect sensory input) would also learn incorrect patterns. e.g. someone who feels cold things as hot and hot things as cold is being given incorrect sensory patterns (ones that aren't representative of real-world temperature), and forming an incorrect idea of what "hot" and "cold" things are as a result, i.e. not properly understanding the world.
My point being that it's the learned representations that determine understanding, not the architecture itself. Of course, if you gave a model completely random data with no correlations at all, then the model would not train either.
Anti-Queen_Elle t1_j13k3lb wrote
And if you create 6 billion neural networks, all speaking Mungo, and they invent a space ship and fly to the moon, would you still readily call it nonsense?
Viewing a single comment thread. View all comments