mediocregradstudent t1_itcl0rj wrote on October 22, 2022 at 4:42 PM

MLPs are universal function approximators but it turns out models with more inductive bias like CNNs are more effective for tasks like image classification.

hellrail t1_itcmak6 wrote on October 22, 2022 at 4:51 PM

Does that mean that MLP are not universal function approximators? No.

Its a fact that MLP is capable of fitting arbritrary functions.

Does anything here deviate from the theory? No.

rehrev t1_itfvyxx wrote on October 23, 2022 at 10:15 AM

That's what the theory says

comradeswitch t1_itmhmqh wrote on October 24, 2022 at 6:59 PM

> MLPs are universal function approximators

MLPs with non-polynomial activation functions with either arbitrary width or arbitrary depth have the ability to approximate a function f: S -> R with an arbitrary specified level of error where S is a compact subset of R^n.

Violate any of these assumptions and you lose those guarantees. Any finite MLP will only be able to approximate a subset of functions with the given support for an arbitrary error level. Nothing about their ability in practice contradicts this.

Much like how there exist matrix multiplication algorithms with better than O(n^2.4) running time but the naive O(n^3) algorithm outperforms them for all physically realizable inputs, the effects of finite sizes are very important to consider.

Professional-Ebb4970 t1_itcyf1z wrote on October 22, 2022 at 6:13 PM

MLP Mixer would like to speak to you