I think the field may eventually move in the direction of online unsupervised generative models implementing something akin to the free energy principle and active inference. These are the kinds of models I am developing and they seem to circumvent many of the issues with modern SOTA ML. I figure it will be a while before this happens because it seems that there isn't a lot of interest yet from the typical sources.

Can you reference some works along these lines? "online unsupervised generative models implementing something akin to the free energy principle and active inference"

Here's a fairly accessible free e-book by the principal researcher on the topic, Karl Friston...

https://mitpress.mit.edu/9780262045353/active-inference/

He's got tons of papers. He's one of the most cited scientists alive.

Also, there are lectures and such on YouTube. Just search terms "free energy principle", "active inference", "predictive processing".

Some other good books are "Surfing Uncertainty" by Andy Clark and "The Predictive Mind" by Howhy.

memeGPT

Hey thanks for sharing all this - it's all very fascinating.

I'm interested in what kind of work you've been doing with models that use active inference.

It's a model that "chooses" its input stream from a 2d array of sensor data (cam, mics, and servo encoders) in real time using policies decoded from predictions of the bottom layer. Then, it processes this input up the hierarchy of identical layers. Higher layer predictions are used to modulate attention.

It may qualify as a general intelligence (idk) as any data can be encoded into the format of its input stream. What I mean is that I have a particular way of encoding video, audio, anything really, into a universal format which preserves the salient semantics.

Currently, it is greatly inhibited in what it can learn because I cannot feed it experiences at the rate it could take them. It has far more potential than realized knowledge.

Sry, what's the "active" part here? Is the model actually generative? I'm aware of Karl Friston and the free-energy principle. Is the active part the input stream selection? I thought that the active part refers to learning, in a sense that I get to pick my training data along the way. Sounds like what you're doing is akin to Gato from DeepMind with tokenization and is about multi-modal policies (modulo the hierarchical processing and attention).

Is there a math writeup somewhere?

I meant more like research papers from top conferences in ML (neurips, iclr, icml)

The model is generative. Each layer generates predictions about the patterns of the layers below. The bottom layer generates predictions about the sensory data, some of which is proprioception data.

I have never published anything. I do not have that much time and it would largely be redundant. You can look at Friston, et.al. for the math. I use nearly the same math and logic.

What I'm doing bears only a superficial similarity to Gato in my opinion, but I can't say I've looked into it deeply. I've been far too busy with life. I only have my tiny spare time for this project unfortunately.

Also, I do mean "active" in the ways you describe. The bottom layer actively controls the sensors via servos and a voice coil. The other layers actively modulate their input by masking it (ignoring it non-trivially).

So the active part is the self-predictive part?

I don't think this model is within the realm ML (it's theoretical neuroscience; although there is much overlap) but does qualify as AI which is what was asked about in the post title.

There is an annual symposium called the International Workshop on Active Inference for about 3 years now where research is presented and the papers are linked there.

And of course the dozens of research papers you can find through Google Scholar on the topic.

Edit: I did find where a few active inference papers have been presented at NeurIPS.

Active just means that it directly modifies its input stream. And, yes, it is also predicting what that input will be, so it is reasonable to say that it is, in part, self-predictive.

Crucially, its input stream also includes features that are not itself or have not been changed by itself. The proprioceptive signals help it learn which is which.