Submitted by Singularian2501 t3_zm22ff in MachineLearning
Nameless1995 t1_j09pzkf wrote
> “Here’s a fragment of text. Tell me how this fragment might go on. According to your model of the statistics of human language, what words are likely to come next?”1
> Even if an LLM is fine-tuned, for example using reinforcement learning with human feedback (e.g. to filter out potentially toxic language) (Glaese et al., 2022), the result is still a model of the distribution of tokens in human language, albeit one that has been slightly perturbed.
....I don't see what's the point is.
I have an internal model of a world developed from the statistics of my experiences through which I model mereology (object boundaries, speech segmentation, and such), environmental dynamics, affordances, and the distribution of next events and actions. If the incoming signal is highly divergent from my estimated distribution, I experience "surprise" or "salience". In my imagination, I can use the world model generatively to simulate actions and feedbacks. When I am generating language, I am modeling a distribution of "likely" sequence of words to write down conditioned on a high level plan, style, persona, and other associated aspects of my world model (all of which can be modeled in a NN, and may even be implicitly modeled in LLMs; or can be constrained in different manners (eg. prompting)).
Moreover in neuroscience and cognitive science, there is a rise of predictive coding/predictive error minimization/predictive processing frameworks treating error minimization as a core unifying principle about function of the cortical regions of brains:
https://arxiv.org/pdf/2107.12979.pdf
> Predictive coding theory is an influential theory in computational and cognitive neuroscience, which proposes a potential unifying theory of cortical function (Clark, 2013; K. Friston, 2003, 2005, 2010; Rao & Ballard, 1999; A. K. Seth, 2014) – namely that the core function of the brain is simply to minimize prediction error, where the prediction errors signal mismatches between predicted input and the input actually received
> “Here’s a fragment of text. Tell me how this fragment might go on. According to your model of the statistics of human language, what words are likely to come next?”1
One can argue the semantics of whether LLMs can be understood to be understanding meanings of words if not learning in the exact kind fo live physically embedded active context as humans or not, but I don't see the point of this kind of "it's just statistics" argument -- it seems completely orthogonal. Even if we make a full-blown embodied multi-modal model it will "likely" constitute a world model based on the statistics of environmental-oberservations, providing distributing of "likely" events and actions given some context.
My guess it that these statements makes people think in frequentists terms which feels like "not really understanding" but merely counting frequencies of words/tokens in data. But that's hardly what happens. LLMs can easily generalize to highly novel requests alien to anything occuring in the data (eg. novel math problems, asking about creatively integrating nordvpn advertisement to any random answer and so on - even though nothing as familiar appear in the training data (I guess)). You can't really explain those phenomena without hypothesizing that LLMs model deeper relational principles underlying the statistics of the data -- which is not necessarily much different from "understanding".
Sure, sure, it won't have the exact sensori-motor-affordance associations with language; and we have to go further for grounding; but I am not sure why we should be drawing a hard line to "understanding" because some of these things are missing.
> These examples of what Dennett calls the intentional stance are harmless and useful forms of shorthand for complex processes whose details we don’t know or care about.
The author seems to cherry-pick from Dennett. He is making it sound as if taking an intentional stance is simply about "harmless metaphorical" ascriptions of intentional states to systems; and based on intentional stance we can be licensed to attribute intentional states to LLMs.
But Dennett also argues against the idea that there is some principled difference between "original/true intentionality" vs "as-if metaphorical intentionality". Instead Dennett considers that to be simply a matter of continuum.
> (1) there is no principled (theoretically motivated) way to distinguish ‘original’ intentionality from ‘derived’ intentionality, and
> (2) there is a continuum of cases of legitimate attributions, with no theoretically motivated threshold distinguishing the ‘literal’ from the ‘metaphorical’ or merely
https://ase.tufts.edu/cogstud/dennett/papers/intentionalsystems.pdf
Dennett seems also happy to attribute "true intentionality" to simple robots (and possibly LLMs (I don't see why not; his reasons here also applies to LLMs)):
> The robot poker player that bluffs its makers seems to be guided by internal states that function just as a human poker player’s intentions do, and if that is not original intentionality, it is hard to say why not. Moreover, our ‘original’ intentionality, if it is not a miraculous or God-given property, must have evolved over the eons from ancestors with simpler cognitive equipment, and there is no plausible candidate for an origin of original intentionality that doesn’t run afoul of a problem with the second distinction, between literal and metaphorical attributions. ‘as if’ cases.
The author seems to be trying to do the exact opposite by arguing against the use of intentional ascriptions to LLMs in a "less-than-metaphorical" sense (and even in the metaphorical sense for some unclear sociopolitical reason) despite current LLMs being able to perform bluffing and all kind of complex functionalities.
Purplekeyboard t1_j0a7dwb wrote
> You can't really explain those phenomena without hypothesizing that LLMs model deeper relational principles underlying the statistics of the data -- which is not necessarily much different from "understanding". > > > > Sure, sure, it won't have the exact sensori-motor-affordance associations with language; and we have to go further for grounding; but I am not sure why we should be drawing a hard line to "understanding" because some of these things are missing.
AI language models have a large amount of information that is baked into them, but they clearly cannot understand any of it in the way that a person does.
You could create a fictional language, call it Mungo, and use an algorithm to churn out tens of thousands of nonsense words. Fritox, purdlip, orp, nunta, bip. Then write another highly complex algorithm to combine these nonsense words into text, and use it to churn out millions of pages of text of these nonsense words. You could make some words much more likely to appear than others, and give it hundreds of thousands of rules to follow regarding what words are likely to follow other words. (You'd want an algorithm to write all those rules as well)
Then take your millions of pages of text in Mungo and train GPT-3 on it. GPT-3 would learn Mungo well enough that it could then churn out large amounts of text that would be very similar to your text. It might reproduce your text so well that you couldn't tell the difference between your pages and the ones GPT-3 came up with.
But it would all be nonsense. And from the perspective of GPT-3, there would be little or no difference between what it was doing producing Mungo text and producing English text. It just knows that certain words tend to follow other words in a highly complex pattern.
So GPT-3 can define democracy, and it also can tell you that zorbot mo woosh woshony (a common phrase in Mongo), but these both mean exactly the same thing to GPT-3.
There is vast amounts of information baked into GPT-3 and other large language models, and you can call it "understanding" if you want, but there can't be anything there which actually understands the world. GPT-3 only knows the text world, it only knows what words tend to follow what other words.
Nameless1995 t1_j0a97yt wrote
> But it would all be nonsense.
Modeling the data generating rules (even if arbitrarily created rules) and relations from data, seems to be close to "understanding". I don't know what would even count as a positive conception of understanding. In our case, the data that we recieve is not just generated by an arbitrarily created algorithm, but by the world - and so the models we create helps us orient better to the world and in that sense "more senseful", but at a functional level not necessarily fundamentally different.
More this applies to any "intelligent agent". If you feed it arbitrary procedurally generated data what it can "understand" will be restricted to that specific domain (and not reach the larger world).
> GPT-3 only knows the text world, it only knows what words tend to follow what other words.
One thing to note that text world is not just something that exists in the air, it is a part of the larget world and created by social interactions. In essence they are "offline" expert demonstrations in virtual worlds (forums, QA, reviews, critics etc.).
However, obviously, GPT3 cannot go beyond that, and cannot comprehend the multimodal associations (images, proprioception, bodily signals etc.) beyond text (it can still associate different sub-modalities within text like programs vs natural texts and so on), and whatever it "understands" would be far alien from what a human understands (having much limited text data, but much richer multimodally embodied data). But that doesn't mean it doesn't have any form of understanding (understood in a functionalist (multiply realizable) sense -- ignoring any matter about "phenomenal consciousness") at all; and moreover, none of these mean somehow "making likely prediction from statistics" is dichotomous with understanding.
Purplekeyboard t1_j0aayw0 wrote
One thing that impresses me about GPT-3 (the best of the language models I've been able to use) is that it is functionally able to synthesize information it has about the world to produce conclusions that aren't in its training material.
I've used a chat bot prompt (and now ChatGPT) to have a conversation with GPT-3 regarding whether it is dangerous for a person to be upstairs in a house if there is a great white shark in the basement. GPT-3, speaking as a chat partner, told me that it is not dangerous because sharks can't climb stairs.
ChatGPT insisted that it was highly unlikely that a great white shark would be in a basement, and after I asked it what would happen if someone filled the basement with water and put the shark there, once again said that sharks lack the ability to move from the basement of a house to the upstairs.
This is not information that is in its training material, there are no conversations on the internet or anywhere about sharks being in basements or unable to climb stairs. This is a novel situation, one that has not been discussed anywhere likely before, and GPT-3 can take what it does know about sharks and use it to conclude that I am safe in the upstairs of my house from the shark in the basement.
So we've managed to create intelligence (text world intelligence) without awareness.
respeckKnuckles t1_j0ajork wrote
> which actually understands the world.
Please define what it means to "actually understand" the world in an operationalizable, non-circular way.
Purplekeyboard t1_j0amzem wrote
I'm referring to two things here. One is having an experience of understanding the world, which of course GPT-3 lacks as it is not having any experience at all. The other is the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.
You could have an AGI machine which had no actual experience, no qualia, nobody is really home, but still understand things as per my second definition above. Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.
respeckKnuckles t1_j0apq5l wrote
I asked for an operationalizable, non-circular definition. These are neither.
> the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.
Can it be measured? Can it be detected in a measurable, objective way? How is this not simply circular: truly understanding is defined as truly knowing, and truly knowing is defined as truly understanding?
> Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.
This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.
Purplekeyboard t1_j0ar5ea wrote
>Can it be measured? Can it be detected in a measurable, objective way?
Yes, we can measure whether someone (or some AI) knows things, can analyze them, take in new information about them, change their mind, and so on. We can observe them and put them in situations which would result in them doing those things and watch to see if they do them.
An AI language model sits there and does nothing until given some words, and then adds more words to the end of the first words which goes with them. This is very different from what an AGI would do, or what a person would do, and the difference is easily recognizable and measurable.
>This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.
I wasn't talking about qualia at all here. You misunderstand what I was saying. I was talking about the difference between an AGI and an AI language model. An AGI wouldn't need to have any qualia at all.
Forms_Deep t1_j0b2vw6 wrote
Sorry to butt in, but I took your statement "Having an experience of understanding the world" as a reference to qualia also.
If it isn't, could you explain what you mean by "experience of understanding" and how it can be measured?
calciumcitrate t1_j0amrhw wrote
But a model is just a model - it learns statistical correlations* within its training data. If you train it on nonsense, then it will learn nonsense patterns. If you train it on real text, it will learn patterns within that, but patterns within real text also correspond to patterns in the real world, albeit in way that's heavily biased toward text. If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.
So, I don't think it makes sense to assign "understanding" based on the architecture as a model is a combination of both its architecture and the data you train it on. Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond to real world patterns.
* This isn't necessarily a direct reply to anything you said, but I feel like people use "correlations" as a way to discount the ability of statistical models to learn meaning. I think people used to say the same thing about models just being "function approximators." Correlations (and models) are just a mathematical lens with which to view the world: everything's a correlation -- it's the mechanism in the model that produces those correlations that's interesting.
Purplekeyboard t1_j0aorp8 wrote
>Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond the real world patterns.
GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.
I think that's the key point here. AI language models are text predictors which functionally contain a model of the world, they contain a vast amount of information, which can make them very good at writing text. But we want to make sure not to anthropomorphize them, which tends to happen when people use them as chatbots. In a chatbot conversation, you are not talking to anything like a conscious being, but instead to a character which the language model is creating.
By the way, minor point:
>If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.
I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.
calciumcitrate t1_j0asr1n wrote
> GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.
What differs GPT-3 from a database of text is that it seems like GPT-3 contains some representations of concepts that make sense outside of a text domain. It's that ability to create generalizable representations of concepts from sensory input that constitutes understanding.
> I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.
Maybe my analogy wasn't clear. The point I was trying to make was that if your argument is:
GPT-3 holds no understanding because you can feed it data with patterns not representative of the world, and it'll learn those incorrect patterns.
Then my counter is:
People being fed incorrect data (i.e. incorrect sensory input) would also learn incorrect patterns. e.g. someone who feels cold things as hot and hot things as cold is being given incorrect sensory patterns (ones that aren't representative of real-world temperature), and forming an incorrect idea of what "hot" and "cold" things are as a result, i.e. not properly understanding the world.
My point being that it's the learned representations that determine understanding, not the architecture itself. Of course, if you gave a model completely random data with no correlations at all, then the model would not train either.
Anti-Queen_Elle t1_j13k3lb wrote
And if you create 6 billion neural networks, all speaking Mungo, and they invent a space ship and fly to the moon, would you still readily call it nonsense?
Luftetari t1_j0am47x wrote
GPT or any LLM is just another actor in the production and reproduction of human intelligence. Like a teacher at a school, who tries to explain relativity theory without understanding it fully. Just another node in the vast human network. It just happens to be evolving at faster rates and impacting a larger number of actors. From an actor-centered and action-centered perspective it is an actor that operates intentionally and understands (operates, reflects, produces) its position and assigned role in the network.
Viewing a single comment thread. View all comments