underPanther t1_jddpryu wrote on March 23, 2023 at 5:35 PM

Reply to comment by andrew21w in [D] Simple Questions Thread by AutoModerator

Another reason: wide single-layer MLPs with polynomials cannot be universal. But lots of other activations do give universality with a single hidden layer.

The technical reason behind this is that ~~non-discriminatory~~ discriminatory activations can give universality with a single hidden layer (Cybenko 1989 is the reference).

But polynomials are not discriminatory (https://math.stackexchange.com/questions/3216437/non-trivial-examples-of-non-discriminatory-functions), so they fail to reach this criterion.

Also, if you craft a multilayer percepteron with polynomials, does this offer any benefit over fitting a Taylor series directly?

andrew21w t1_jde4ayx wrote on March 23, 2023 at 7:07 PM

The thread you sent me says that polynomials are non discriminatory.

Are there other kinds of functions that are non discriminatory?

underPanther t1_jdeofve wrote on March 23, 2023 at 9:15 PM

Sorry for the confusion! It's discriminatory activations that lead to universality in wide single layer networks. I've editted post to reflect this.

As an aside, you might also find the following interesting which is also extremely well-cited: https://www.sciencedirect.com/science/article/abs/pii/S0893608005801315