Submitted by Ash3nBlue t3_xvw467 in MachineLearning
Optional_Joystick t1_ir72t5g wrote
Reply to comment by yldedly in [R] Self-Programming Artificial Intelligence Using Code-Generating Language Models by Ash3nBlue
Really appreciate this. I was excited enough about learning knowledge distillation was a thing. I felt we had the method of extracting the useful single rule from the larger model.
On the interpolation/extrapolation piece: For certain functions like x^2, wouldn't running the result of a function through the function again let you achieve a result that "extrapolates" a new result outside the existing data set? This is kind of my position on why I feel feeding LLM data generated from an LLM can result in something new.
It's still not clear to me how we can verify a model's performance if we don't have data to test it on. I'll have to read more about DreamCoder. As much as I wish I could work in the field, it looks like I've still got a lot to learn.
Viewing a single comment thread. View all comments