It sounds like you're pretty hard set on there being no ghost in the shell and pretty judgmental of anyone who thinks otherwise. I'm just saying you're far too certain you have the answers, as my example demonstrates. I also never said I believe a living being is jumping into existence because of whatever successful Turing test. I'm actually agnostic on that and think it's a waste of time trying to answer something that will never be answered. It's always going to come down to bickering over definitions and goalpost-shifting ("It can't be sentient if it doesn't model glial cells/recurrent cortical stacks/neurotransmitter densities/electrochemistry/quantum computational effects inside microtubules/the gut microbiome/the embedding within the environment/the entire rest of the universe like us"). I'd much rather play it safe and treat it as though it is conscious.

Maybe I'm misunderstanding you, but it sounds like you're now also being far too dismissive of the representational power tensors/linear operations and especially eigendecompositions can have (I could probably blow your mind with the examples I've seen), and of statistics as a loss function. After all, we as humans are literally no more than statistical mechanical partition functions of Von Neumann density matrices, what would you even use for a loss function instead? MSE, cross-entropy (perplexity), KL, L1/L2 are statistical and used to build every neural net you've heard about. The only difference between us and say a Turing-complete (nonlinear ReLU + attentional) Kalman filter for text like you're making GPT out to be is how the hyperparameters are adjusted. A Kalman filter uses Bayesian inference with either Laplace's method or maximum-likelihoodist rules, whereas we (and ChatGPT) are genetically rewarded for minimizing both cost (resp. perplexity) and nonlinear human feedback. Keep boiling things down and you'll find you're surrounded by philosophical zombies.

Edit: Didn't see the second paragraph you added. I'm not sure what ML orthodoxy you're from, but Chalmers' result is pretty well accepted in CogSci. The setup that you're describing, the Chinese room, is an appeal to common sense, but a lot of what motivates scientific development is trying to understand paradoxes and counter-intuitive results. Sure, it sounds absurd, but so does Schrödinger's cat or black holes, both of which failed to disprove the underlying phenomena. Chalmer's 1995 result came after the Chinese Room thought experiment (by about 15 years in fact) and updated the consensus since on the Chinese Room by stressing the importance of universality. Since your example has humans performing the computation, I would say it could be alive (depending on the complexity of the equations, are they reproducing the action potentials of a human brain?), and case in point I think the internet even before ChatGPT is the most likely and well-known contender for a mass of human scribbles being conscious.

The use of neural nets (ReLU + LayerNorms) layered between each attention step counts as a brain, no? I know the attention mechanism is what gets the most ... attention, but there's still traditional neural nets sandwiched between and in some cases the transformer is just a neck feeding into more traditional modules. ReLU is Turing complete so I can always tune a neural net to have the same response pattern of electrical outputs as any neuron in your brain.

The million dollar question according to David Chalmers is, would you agree that slowly replacing each neuron with a perfect copy one at a time will never cause you to go from completely fine to instantly blacked out? If you answered yes, then it can be shown (sections 3&4) that you must accept that neural nets can be conscious, since by contradiction if there was a gradual phasing out of conscious experience rather than sudden disappearance, that would necessarily require the artificial neurons to at some point begin behaving differently than the original neurons would (we would be aware of the dulling of our sensation).

Considering we lose brain cells all the time and don't get immediately knocked out, I think you can at least agree that most people would find these assumptions reasonable. It would be pretty weird to have such a drastic effect for such a small disturbance.

It always is. If you want to get really freaky with it, just look at how NFTs became demonized at the same time as when Gamestop's pivot to NFT third-party provider was leaked by WSJ. Just the other month people were bashing the author of Terminal Shock and hard sci-fi cyberpunk pioneer Neal Stephenson in his AMA for having a NFT project/tech demo by arguing with someone that knows 1000x more than they do, saying it's just a CO2 emitter and only scam artists use it and were disappointed to see he'd try to do this to his followers. Of course, the tech has evolved and those claims weren't true in his case, but it was literally all in one ear out the other for these people even after he'd defend himself with the actual facts about his green implementation and how it works. They bought an overly general narrative and they're sticking to it!

Interesting that now, with a technology that produces an order of magnitude more pollution (you can actually list models on Hugging Face by the metric tonnes of CO2 equivalent released during training) and producing an epidemic of cheaters in high schools, universities, and the work force, it's all radio silence. God only knows how much scamming and propaganda (which is just scamming but "too big to fail") is waiting in the wings.

I don't think the average person even knows what they would do with such a powerful LLM beyond having entertaining convos with it or having it write articles for them. Of course they see other people doing great things with it and not really any of the other ways it's being misused by degens right now, which goes back to an advantage in corporate propaganda.

Furthermore, if we are to assume that an LLM can be boiled down to nothing more than a statistical word probability engine because that's what its goal is (which is dubious for the same reason we don't think of people with jobs as being only defined as payraise probability engines, what if a client asks a salesman important questions unrelated to the salesman's goal, etc.), this point of view is self-destructive and completely incoherent when you factor in that for ChatGPT in particular, it's also trained using RLHF ("Reinforcement Learning with Human Feedback").

Everytime you leave a Like/Dislike (or take the time to write out longer feedback) on one of ChatGPT's messages, that gets used directly by ChatGPT to train the model through a never-ending process of (simulated) evolution through model competition with permutations of itself. So there are two things to note here, A. It's goals include not only maximizing log-likelihoods of word sequences but also in inferring new goals from whatever vague feedback you've provided it, and B. How can anyone be so sure that such a system couldn't develop sophisticated complexity like sentience or consciousness like humans did through evolution (especially when such a system is capable of creating its own goals/heuristics and we aren't sure how many layers of abstraction with which it's recursively doing so)?

On that second point in particular, we just don't currently have the philosophical tools to make any sort of statements about that, but people are sticking to hard-and-fast black and white statements of the kind we made about even other humans until recent history. We as humans love to have hard answers about others' opinions so I see the motivation for wanting to tamp down the tendency to infer emotion from ChatGPT's responses, but this camp has gone full swing in the other direction with unscientific and self-inconsistent arguments because they've read a buzzfeed or verge article produced by people with skin in the game (long/short msft, it's in everyone's retirement account too).

I think the best reply in general to someone taking the paperclip-maximizer stance while claiming to know better than everyone else the intricacies of an LLM's latent representations of concepts encoded through the linear algebraic matrix multiplication in the V space, the eigenvector (Q,K) embeddings from PCA or BERT-like systems, or embedded in its separate neuromorphic structure ("it's just autocorrect, bro") is to draw the same analogy that they're just a human meat-puppet designed to maximize dopamine and therefore merely a mechanical automaton slave to biological impulses. Obviously this reductionism is in general a fallacious way of rationalizing things (something we "forget" time and again throughout history because this time it's different), but you also can't counter by outright stating that ChatGPT is sentient/conscious/whatever, we don't know for sure whether that's even possible (cf. Chinese room -against, David Chalmers' Brain of Theseus -for, Penrose's contentious Gödelian construction demonstrating human supremacy as Turing machine halt checkers -against).