You will get a lot of value out of looking at PCA.
Additionally Auto Encoders. Just a quick overview. If you have a set of images of faces as your input data, you should be able to see that any random collection of pixels is not a valid face. We can use deep learning models to "encode" this information of valid faces into lower dimensions. Where originally you have a 28x28 grey scale image, taking with values 0-255, can be compressed to a maybe 100 dimensions. There are quite a few subtleties, Kingma has a great review paper that should help.
For a 10 dimensional vector it really depends on the data, the more structure it has the more easily it can be compressed.
Information theory will be a great Google. I would recommend David MacKay's book if you are really interested.
MRsockman314 t1_izjyueu wrote
Reply to Representation ability of a MLP network [D] by OutOfCharm
You will get a lot of value out of looking at PCA.
Additionally Auto Encoders. Just a quick overview. If you have a set of images of faces as your input data, you should be able to see that any random collection of pixels is not a valid face. We can use deep learning models to "encode" this information of valid faces into lower dimensions. Where originally you have a 28x28 grey scale image, taking with values 0-255, can be compressed to a maybe 100 dimensions. There are quite a few subtleties, Kingma has a great review paper that should help.
For a 10 dimensional vector it really depends on the data, the more structure it has the more easily it can be compressed.
Information theory will be a great Google. I would recommend David MacKay's book if you are really interested.