Submitted by IamTimNguyen t3_105v7el in MachineLearning
IamTimNguyen OP t1_j3himnu wrote
Reply to comment by AlmightySnoo in [R] Greg Yang's work on a rigorous mathematical theory for neural networks by IamTimNguyen
Having spoken to Greg (who may or may not be chiming in), it appears that the authors of PDLT were only considering one kind of infinite width limit (as evidenced by your use of the word "the"). But Greg considers a general family of them. The NTK limit indeed has no feature learning, whereas Greg analyzes entire families, some that do have feature learning, in particular, one that has maximal feature learning. So there is no contradiction with respect to past works.
eyeofthephysics t1_j4f2w85 wrote
>u/IamTimNguyen
Hi Tim, just to add on to your comment, Sho Yaida (one of the co-authors of PDLT) also wrote a paper on the various infinite width limits of neural nets, https://arxiv.org/abs/2210.04909. He was able to construct a family of infinite width limits and show that in some of them there is representation learning (and he also found agreement with Greg's existing work).
Viewing a single comment thread. View all comments