I really like your last point there. That is a good analogy.
I guess my question boils down to "how to think about information in a trained model". What I am wondering is whether a model can carry more information than it's raw size which I think it may be able to conceptually as the relationship between neurons carries information but isnt reflected in the file size of a model.
So like a regression represents a point cloud, could we now vectorise a book or a movie (if that was what we wanted)?
The sky is a different colour on either side of this rainbow
samyall OP t1_jdqacnn wrote
Reply to comment by adfoucart in Do large language models effectively compress their training dataset? by samyall
I really like your last point there. That is a good analogy.
I guess my question boils down to "how to think about information in a trained model". What I am wondering is whether a model can carry more information than it's raw size which I think it may be able to conceptually as the relationship between neurons carries information but isnt reflected in the file size of a model.
So like a regression represents a point cloud, could we now vectorise a book or a movie (if that was what we wanted)?