Let's fight the fire with gasoline.

"Truth" only exists in the context of verification. You probaly would need some kind of RL to improve "truthfulness"

see previous response

There was a paper about it. There was a find - specific set of positions not encountered or pooply represented during self-play. Fully trained AlphaGo was failing on those positions. However then they were explicitly added to the training set the problem was fixed and AlphaGo was able to play them well. This adversarial traning seems just an automatic way to find those positions.

PS fintess landscape is not convex it separated by hills and valleys. Self-play may have a problem in reaching all important states.

It's a question of training. AlphaGo was not trained agains adversarial attacks. If it was the whole family of attacks wouldn't work, and new adversarial traning would be order of magnitude more difficult. It's a shield and sword again.

The notable diffrence between GA and other random searches is cross-over operator, and in it's theory "building blocks" hypothesis. Neither were confirmed during years (dozens of years) of attemted use of GA.

In my experience many of software engineers forgot most of linear algebra and calculus if they knew them from the start. Some also forgot probailty/statistics. If there was no preliminary requirements for participants course should start from refreshing those areas.

Hessian-free second order will not likely work. There are reasons why everyone using gradient descent. The only working second order method seems K-FAC (disclaimer - I have no first hand experience) but as you will use Julia you will have to implement it from scratch, and it's highly non-trivial (as you can expect from method which work where other failed)

Yes for specific cases and mostly overly strong assumptions. It was talked about a lot several years ago and in this same subreddit too. For example:

https://arxiv.org/abs/1810.02054

https://arxiv.org/abs/1811.03804

https://arxiv.org/abs/1811.03962

https://arxiv.org/abs/1811.08888

This is recurring question, people asking it every year. Some papers should be made sticky :(

> beyond just some toy experiment?

Compress models. See if you can fit 8GB model into 1G, capable to run on mobile, and at what cost.

If both paper have similar results that's acually good IMO. That mean approach is actually works and not some hyperparameters fiddling.

DL is not working well on low-dimentional samples data, data with low correlation between sample elements, and especially bad for time series prediction which is both. Many people put that kind of senseless projects (DL for time series) on their CV and that is instant black mark for candidate, at least for me. They say "but that approach did work!" I ask "did you try anything else?" "No".

3D Localization/Registration/Reconstruction are traditional area of use for regularized Gauss-Newton and all are highly non-convex. The trick is to strat in nearly-convex area, sometimes after several tries, and/or convexify with regularizers and/or sensors fusion.

K-FAC seems stable enough but quite complex in implementation. It's identical to low-dimentional-blocks approximation of Gauss-Newton. Fisher information is only decoration.

For that first AMD had to make normal implementation of OpenCL. People complain all the time - slowdowns, crashes, lack of portability. This going on for 10 years already and it doesn't get better.

Rule of the thumb - bandwidth win.

No, but that doesn't mean that you wouldn't be able to monetarize it somewhat. Put it on github and link it to your CV/linkedin. You would likely get better offers form potential employers, especially if your project get some starts.

overfitting