blimpyway t1_j4pndcs wrote on January 17, 2023 at 11:27 AM

Reply to comment by blackkettle in [R] The Predictive Forward-Forward Algorithm by radi-cho

One application I can think of is learning on edge. There is an industry fashion to embed AI inference capabilities in the newer ARM chips. The so called NPUs. Which are simplified GPUs optimized only for inference (forward passes). Such an algorithm would enable them to learn using only forward passes, hence without requiring backpropagation.

Another possibility I think is ability to train one layer at a time, which diminishes GPU memory requirements.

And probably more important it opens the gates for all kind of not yet seen network architectures, topologies and training methods that do not require fully differentiable pathways.

edit: regarding the brain inspired part.. well you can dismiss it as AI's reversed cargo cult - if it imitates some properties of the brain it should act like the brain, but I would be cautious to attribute Hinton this kind of thinking. Brains are very different from ANNs and trying to emulate their properties could provide insights on how they work.

currentscurrents t1_j657n2z wrote on January 27, 2023 at 8:12 PM

>The so called NPUs. Which are simplified GPUs optimized only for inference (forward passes). Such an algorithm would enable them to learn using only forward passes, hence without requiring backpropagation.

More importantly, you could build even simpler chips that physically implement a neural network out of analog circuits instead of emulating one with digital math.

This would use orders of magnitude less power, and also let you fit a larger network on the same amount of die space.

amassivek t1_j6owo9f wrote on January 31, 2023 at 9:26 PM

Here is an reversed view, where ANNs provide inspiration for neuroscience to investigate the brain. Forward learning models provide a new perspective on how neurons without "feedback" or "learning" connections have the ability to still learn, a common scenerio. We make note of this and show the conceptual framework for forward learning: https://arxiv.org/abs/2204.01723. This conceptual framework is applicable to neuroscience models, providing an investigative path forward.