Also, think a lot of ML research is now highly dependent on GPU parallel computation, which is expensive. My guess is that it will be the academics that collaborate the GCP, AWS, Azure to get near free usage of GPU clusters. Coming up with a paradigm shift means a fair bit of trial and error experimentation. For the time being Cloud providers have been happy to promote ML pipelines to academics. But that may change with the tightening of costs with the layoffs and recession.

The transformer and self-attention progress has been an interesting achievement. I foresee us stuck in this trend until most ML groups have fully explored the avenues of research on self-attention. Without the advances in GPUs I don't think we'd be where we are now.

What's next? I'd love to see more progress on recommender systems and sparse training data.

I feel there are more gains to be had out of the more boring stuff in ML: data rangling design patterns to help non-data scientists choose the best model and customisation to answer their hypothesis questions. Also, the mechanics and infrastructure around ML at scale in an enterprise is not mature.

There are a lot of pain points for ML and big data teams in Enterprise, they need to skill up on a variety of hardcore DevOps tasks. Once it becomes trivial to spin up a ML pipeline with a Cloud infrastructure team supporting that work, then we'll see more commercial successes and collaborations between academia and Tech companies.

ML is at a strange evolutionary stage, there are lots of Tech companies relying on ML models to give them a USP over their competitors. So their willingness to share their breakthroughs is small. Once the barriers to entry are reduced for enterprise scale ML modelling, that's when we'll see more adoption of ML systems across whole sectors. Right now ML commercial research is too expensive for small players to get involved