Viewing a single comment thread. View all comments

danja t1_j8nerv0 wrote

Reply to comment by crimson1206 in Physics-Informed Neural Networks by vadhavaniyafaijan

What's a normal NN? How about https://en.wikipedia.org/wiki/Universal_approximation_theorem ?

How efficiently is another matter. Perhaps there's potential for using an activation function somewhere around Chebyshev polynomials that would predispose the net to getting sinusoids.

3

crimson1206 t1_j8njti4 wrote

By normal NN I'm referring to a standard MLP without anything fancy going on. I.e. input -> hidden layers & activations -> output.

The universal approximation theorem isn't relevant here. Obviously a NN could fit this function given training data. This post is about lacking extrapolation capabilities/how PINNs improve extrapolation though

8

BrotherAmazing t1_j8q4qdx wrote

Isn’t it more technically correct to state that a “regular NN” could learn to extrapolate this in theory, but is so unlikely to do so that the probability might as well be zero?

PINNs are basically universal function approximators that have additional knowledge about physics-based constraints imposed, so it’s not surprising and shouldn’t be taken as an “dig” on “regular NNs” that they can better decide what solutions may make sense and are admissible vs. something that is basically of an “equivalent” architecture and design but without any knowledge of physics encoded in to regularize it.

1

danja t1_j8tes8q wrote

I don't quite see how approximation theorems aren't relevant to approximation problems. I'm not criticising the post, I just thought your response was a bit wide of the mark, not much fun.

1

crimson1206 t1_j8ts496 wrote

Well how is it relevant then? Im happy to be corrected but I dont see how its relevant to this post

It just tells you that there is a well approximating NN for any given function. It doesn't tell you how to find such a NN and it doesnt tell you about extrapolation capabilities of a NN which is well approximating on just a subdomain (which is what this post here is mainly about) either.

The universal approximation theorem in practice just gives a justification for why using NNs as function approximators could be a reasonable thing to do. That's already pretty much the extent of their relevancy to practical issues though

1