I have been using these physics-informed dynamics models in controls / RL.

From a CV / NLP perspective, I'm not particularly sure. There was some work from Deepmind on learning latent dynamics from images. Check out "Benchmarking models for learning latent dynamics". However, I'm not sure if this is something you'd be interested in.

Recently there has been growing interest in developing better deep neural network based dynamics models for physical systems, through better inductive biases. Here are some papers that utilize the structure of Lagrangian / Hamiltonian mechanics to learn better dynamics models,

• Deep Lagrangian Networks (DeLaN)
• Hamiltonian neural networks
• DeLaN for energy control
• Symplectic ode-net (Symoden)
• Dissipative symoden
• Lagrangian neural networks
• Simplifying hamiltonian and lagrangian neural networks via explicit constraints
• Extending lagrangian and hamiltonian neural networks with differentiable contact models

The following survey paper nicely summarizes some of the work in this area,

• Benchmarking energy-conserving neural networks for learning dynamics from data.