Also “the structure of language follows from abstracting the world into compact communicable units” is itself a “folk theory” of languages. Many supposedly neuroscientific theories of language are little more than conjecture based on assumptions.

Linguistics is a field of study and analysis, not philosophy. And I am specifically talking about exploring the anthropological and ethnographical study of language which is where you might lose many of your assumptions. The way different languages work, how they change over time, is relevant to anyone working in NLP.

I would argue the number one fallacy of modern LLM design is people disregarding all we have come to know about language in favor of just hoping something interesting will emerge when we throw billions of parameters at it.

Just want to reiterate: if you think language is just syntax, I'd recommend listening to some linguistics lectures or reading a book or two on the subject (i.e.: books on language, not on syntax or grammar). John McWhorter has some very approachable and eye-opening Audible courses that might change your perspective.