Submitted by vadhavaniyafaijan t3_112sunq in deeplearning
Comments
humpeldumpel t1_j8m4czr wrote
exactly my thought.. wtf?
crimson1206 t1_j8mb0gu wrote
The normal NN will not learn this function even with more steps. It’s a bit strange that the graphic didn’t show more steps but it doesn’t really change results
danja t1_j8nerv0 wrote
What's a normal NN? How about https://en.wikipedia.org/wiki/Universal_approximation_theorem ?
How efficiently is another matter. Perhaps there's potential for using an activation function somewhere around Chebyshev polynomials that would predispose the net to getting sinusoids.
crimson1206 t1_j8njti4 wrote
By normal NN I'm referring to a standard MLP without anything fancy going on. I.e. input -> hidden layers & activations -> output.
The universal approximation theorem isn't relevant here. Obviously a NN could fit this function given training data. This post is about lacking extrapolation capabilities/how PINNs improve extrapolation though
BrotherAmazing t1_j8q4qdx wrote
Isn’t it more technically correct to state that a “regular NN” could learn to extrapolate this in theory, but is so unlikely to do so that the probability might as well be zero?
PINNs are basically universal function approximators that have additional knowledge about physics-based constraints imposed, so it’s not surprising and shouldn’t be taken as an “dig” on “regular NNs” that they can better decide what solutions may make sense and are admissible vs. something that is basically of an “equivalent” architecture and design but without any knowledge of physics encoded in to regularize it.
danja t1_j8tes8q wrote
I don't quite see how approximation theorems aren't relevant to approximation problems. I'm not criticising the post, I just thought your response was a bit wide of the mark, not much fun.
crimson1206 t1_j8ts496 wrote
Well how is it relevant then? Im happy to be corrected but I dont see how its relevant to this post
It just tells you that there is a well approximating NN for any given function. It doesn't tell you how to find such a NN and it doesnt tell you about extrapolation capabilities of a NN which is well approximating on just a subdomain (which is what this post here is mainly about) either.
The universal approximation theorem in practice just gives a justification for why using NNs as function approximators could be a reasonable thing to do. That's already pretty much the extent of their relevancy to practical issues though
demkom58 t1_j8mg3u1 wrote
Is it just overfitting?
MrZwink t1_j8mjeon wrote
Yes mostlikely.
nibbajenkem t1_j8mpv71 wrote
What is the use case if it is already appropriate to model the phenomenon using regular physics?
wallynext t1_j8npy97 wrote
exactly my thought, but I think this is the "trainning and validation set" where you already know the label and then they can use this neural net to discover new "unseen" mathematical equations
BrotherAmazing t1_j8q7wow wrote
Indeed, this example of a simple harmonic oscillator is not a practical use case, but more of a simple example and pedagogical tool.
There are potential use cases for more complex problems. Many complex physical systems are governed by partial differential equations, however it is often impossible to write explicit formulas for the solutions to these equations, and so the physical states must be experimentally observed as they evolve or else computationally demanding high-fidelity simulations must be run, sometimes on supercomputers for many days, in order to numerically estimate how the physical system evolves.
Just think about recent work in protein folding. Would a deep NN that tries to make protein folding predictions benefit from knowing physics-based constraints?
canbooo t1_j8ohxf9 wrote
Esp. in engineering applications, i.e. with complex systems/philysics, fundamental physical equations are known but not how they influence each other and the observed data. Alternatively, these are too expensive to compute for all possible states. In those cases, we already build ML models using the data to, e.g. optimize design or do stuff like predictive maintenance. However, these models often do not generalize well to out of domain samples and producing samples is often very costly since we either need laboratory experiments or actually create some design that are bound to fail (stupid but for clarity: think planes with rectangular wings, cranes so thin they could not even pick up a feather. Real world use cases are more complicated to fit in these brackets). In some cases, the only available data may be coming from products in use and you may want to model failure modes without observing them. In all these cases PINNs could help. However, none of the models I have tested so far are actually robust to real world data and require much more tuning compared to MLPs, RNNS etc, which are already more difficult to tune compared to more conventional approaches. So I am yet to find an actual use case that is not academic.
TLDR; physics (and simulations) may be inefficient/inapplicable in some cases. PINNs allow us to embed our knowledge about the first principles in form of inductive bias to improve generalization to unseen/unobservable ststes.
nibbajenkem t1_j8ojasp wrote
Of course, more inductive biases trivially lead to better generalization. Its just not clear to me why you cannot forego the neural network and all its weaknesses and instead simply optimize the coefficients of the physical model itself. I.e in the example in OP, why have a physics-based loss with a prior that it's a damped oscillator instead of just doing regular interpolation on whatever functional class(es) describe the damped oscillators?
I don't have much physics expertise beyond the basics so I might be misunderstanding the true depth of the problem though
canbooo t1_j8onqb4 wrote
No, valid question, I just find it difficult to give examples that are easy to understand but let me try. Yes OPs example is not a good one to demonstrate the use case. Let us think about a swarm of drones and their physics, specifically the airflow around them. Hypothetically, you maybe able to describe the physics for a single drone accurately, although this would probably take quite some time in reality. Think days on a simple laptop for a specific configuration if you rally want high accuracy. Nevertheless, if you want to model say 50 drones, things get more complicated. Airflow of one effects the behavior/airflow of others, new turbulence sources and other effects emerge. Actually simulating such a complex system may be infeasible even with supercomputers. Moreover, you are probably interested in many configurations like flight patterns, drone design etc. so that you can choose the best one. In this case, doing a functional interpolation is not very helpful due to the interactions and new emerging effects as we only know the form of the function for a single drone. Sure, you know the underlying equations but you still can't really predict the behavior of the whole without solving them, which is as mentioned costly. The premise of PINNs in this case is to learn to predict the behaviour of this system and the inductive bias is expected to decrease the number of samples required for generalization.
jgonagle t1_j8pdyti wrote
In other words, good priors result in good posteriors. Mind blown.
Late_Scientist_9344 t1_j8n28nw wrote
Does PINN can be used for parameter estimation somehow?
Skoogy_dan t1_j8o0l3j wrote
Yes, you can discover the underlying parameters of the DiffE from a dataset.
smierdek t1_j8oggk9 wrote
what is this nonsense?
smierdek t1_j8ogszh wrote
i mean pick a value somewhere in the middle between min and max and go home bahaha
pgmali0n t1_j8qt90x wrote
how do you inform nn with physics?
danja t1_j8tgfpo wrote
I like it. On a meta level, giving the machine a bit of a priori knowledge about the shape of things to come, that makes a lot of sense.
When the self-driving car hits an obstacle, they will both obey mostly Newtonian sums.
Effectively embedding that knowledge (the differential equations) might make the system less useful for other applications, but should very cheaply improve it's chances on a lot of real-world problems.
Robotics is largely done with PID feedback things. Some more understanding of the behaviour of springs etc etc should help a lot. Quite possibly in other domains, hard to know where such things apply.
MugiwarraD t1_j8ue1xh wrote
Makes no sense.
Oceanboi t1_j8uygkt wrote
why was the neural network stopped at like 1000 steps? why are we comparing a physics informed neural network to a neural network at a different number of steps lol
Also correct me if I'm wrong but don't we care about how the model generalizes? I think we can show that some NN will fit to any training set perfectly given enough steps, but this is already common knowledge no?
crimson1206 t1_j8w4qnt wrote
The steps really don’t matter. The normal NN will not learn to extrapolate better with more steps.
This post is precisely showing how the PINN has better generalization than the normal NN
Oceanboi t1_j8zdely wrote
Oh I see, I missed the major point that the training data is basically incomplete to model the entire relationship.
Why embed priors into neural networks, doesn’t Bayesian Modeling using MCMC do pretty much what this is attempting to do? We did something similar to this in one of my courses although we didn’t get to spend enough time on it so forgive me if my questions are stupid. I also would need someone to walk me through a motivating example for a PINN because I’d just get lost in generalities otherwise. I get the example, but am failing to see the larger use case.
MrMoussab t1_j8o8b41 wrote
I don't think such post deserves upvoting given redditors comments. Number of steps is weird. What nn is used? Why use nns if physical modeling is possible ...
cassova t1_j8m3bo0 wrote
Those training steps thou..