It is fine if you disagree and I believe a lot more people will disagree with this philosophical position as it is not very popular these days.

Near-death experiences, out-of-body experiences, contact with 'immaterial entities' and so on hint towards an existence beyond our material reality. Since there is no way one could 'scientifically' test these does not mean these things simply do not exist.

Testimony widely used yet mostly dismissed method of knowledge acquisition establishes all of the above:

A patient being operated on while in a complete medical comma explaining the things happening in clear details in a nearby room after the operation that there is no way they could have known that, one such testimony by a reliable person is sufficient to establish that our current understanding of the world is insufficient. And there are so many of these.

I am not saying u have to change your worldview just because I am saying so. do your research. the world is much bigger than what is out there on the internet. (pun intended)

Hard disagree with Materialism. I know I might get a lot of -ve votes, but this has to be said:

A large portion of the world (especially outside of the west) does not believe in 'consciousness "emerging" from electrical impulses of the brain.' While the west has progressed a lot materially, bringing us to modernity (and now post-modernity), people outside of the west believe in an immaterial soul that cannot be captured by definition by the scientific method and it transcends our material body.

While I believe we will reach general human-level intelligence (and may go beyond this) because intelligence has a purely material component that we can replicate in computers, consciousness will never ever arise in these systems. There are very strong philosophical arguments to support this case.

That is understandable. I am working with that assumption as well. (I have failed too many of such experiments to have a blind faith 🙈)

Thanks for that paper; I came across it a while ago but have not read it yet. Is the limit due to number of model parameters or size of embedding. I suspect size of embedding to be the biggest factor in limiting how big the context can be.

This

I think one just needs to duplicate positional embeddings and we are good to go. Of course, there needs to be more comprehensive empirical analysis on this and I have not come across any of such attempts. I did a basic experiment and it seems to work but will have to wait and see.

Nuance is proportion to context.

Imagine we want to ask the language model to improve a certain module in Linux Kernel.

If I understood them correctly, memory-augmented transformers won't be able to fit together all the pieces to understand what needs to be improved and how because they need to make repeated calls to memory and search/summarize those calls to get a basic understanding and thus miss out on important details.

Compare that to huge context, they have everything they need for the memory in their context and there is no loss of details (in case of full attention).

RIP 💀 scientific progress for "the entire humanity" for the profits of a few. :(

The only way forward is if we as a collective AI community systematically fight against this type of censorship, or we might end up in an AI-dominated Orwellian world.

Ironic that I had read Life 3.0 by Max Teggmark where he was one of the guys raising concerns about the future of AI and trying to build an organization called 'OpenAI' for the benefit of mankind.