Viewing a single comment thread. View all comments

aleph__one t1_izwos7v wrote

Reply to comment by sea-shunned in [D] G. Hinton proposes FF – an alternative to Backprop by mrx-ai

I use a custom SNN variant in production on real use cases, and the way we train those is very similar to the FF proposal. Most people just assume SNNs are impossible to train because SGD isn’t immediately available, when in reality there are dozens of ways to train SNNs to achieve solid performance.

23

captain_arroganto t1_izwufye wrote

> spiking neural networks

Could you give more details. Books, articles, tutorials, application areas etc.

I am curious to explore this area.

Thanks.

8

aleph__one t1_izwxy4q wrote

Unfortunately, beginner literature on this stuff is virtually nonexistent. Your best bet is to read papers and experiment.

8

ChuckSeven t1_j016ugh wrote

Time for you to write a series of beginner tutorials for the community! Now would be a good time.

2

aleph__one t1_j01c1kf wrote

Yea I was thinking the same thing. I teach some of this stuff at the graduate level but it’s tough for newcomers to get used to even in that setting.

1

arhetorical t1_izwxay5 wrote

I mostly hear about surrogate gradient descent, what other methods work well in practice?

4

aleph__one t1_izwyrcf wrote

Yea the surrogate gradient stuff works ok, others that are decent 1) STDP variants, especially dopamine modulated STDP (emulates RL-like reinforcement) 2) for networks < 10M params, evolution strategies and similar zero-order solvers can work well operating directly on the weights 3) variational solvers can work if you structure the net + activations appropriately

12

arhetorical t1_izxbkdf wrote

I see, thanks. Why did you choose to use SNNs for your application instead of conventional ANNs? Are you using a neuromorphic chip?

1

aleph__one t1_izxu46b wrote

No neuromorphic chip. Main reason is interpretability.

2

arhetorical t1_izzryk4 wrote

Oh, I haven't heard about using SNNs for interpretability. I thought they were on the same level as ANNs. Sorry for all the questions, but can you elaborate on how they're more interpretable?

2

2358452 t1_j04t3pw wrote

The spiking events should be much more sparse and therefore probably easier to interpret.

1