GraciousReformer OP t1_j9ljjqu wrote on February 22, 2023 at 9:11 PM

Reply to comment by activatedgeek in [D] "Deep learning is the only thing that currently works at scale" by GraciousReformer

>inductive biases

Then why does DL have inductive biases and others do not?

activatedgeek t1_j9lu7q7 wrote on February 22, 2023 at 10:18 PM

All model classes have inductive biases. e.g. random forests have the inductive bias of producing axis-aligned region splits. But clearly, that inductive bias is not good enough for image classification because a lot of information in the pixels is spatially correlated that axis-aligned regions cannot capture as specialized neural networks, under the same budget. By budget, I mean things like training time, model capacity, etc.

If we have infinite training time and infinite number of image samples, then probably random forests might be just as good as neural networks.

currentscurrents t1_j9n3o7u wrote on February 23, 2023 at 3:45 AM

Sounds like ideally we'd want a model with good inductive biases for meta-learning new inductive biases, since every kind of data requires different biases.

GraciousReformer OP t1_j9lwe7i wrote on February 22, 2023 at 10:32 PM

Still, why is it that DL has better inductive biases than others?

activatedgeek t1_j9lz6ib wrote on February 22, 2023 at 10:50 PM

I literally gave an example of how (C)NNs have better inductive bias than random forests for images.

You need to ask more precise questions than just a "why".

GraciousReformer OP t1_j9m1o15 wrote on February 22, 2023 at 11:06 PM

So it is like an ability to capture "correlations" that cannot be done with random forests.

currentscurrents t1_j9n8in9 wrote on February 23, 2023 at 4:26 AM

In theory, either structure can express any solution. But in practice, every structure is better suited to some kinds of data than others.

A decision tree is a bunch of nested if statements. Imagine the complexity required to write an if statement to decide if an array of pixels is a horse or a dog. You can technically do it by building a tree with an optimizer; but it doesn't work very well.

On the other hand, a CNN runs a bunch of learned convolutional filters over the image. This means it doesn't have to learn the 2D structure of images and that pixels tend to be related to nearby pixels; it's already working on a 2D plane. A tree doesn't know that adjacent pixels are likely related, and would have to learn it.

It also has a bias towards hierarchy. As the layers stack upwards, each layer builds higher-level representations to go from pixels > edges > features > objects. Objects tend to be made of smaller features, so this is a good bias for working with images.

GraciousReformer OP t1_j9oeeo3 wrote on February 23, 2023 at 12:46 PM

What are the situations that the bias for the hierarchy is not helpful?