Submitted by Syntro42 t3_10aka07 in MachineLearning
Nineshadow t1_j45i8rf wrote
No, 60 problems are not enough, probably not even for fine tuning. I would also strongly advise against starting from scratch.
The best approach in this case would be to fine-tune a pre-trained LLM which was trained on both natural language and code, something like GPT-Neo with 125M parameters. I'm mentioning the small version because the you'll have trouble fitting in memory larger models with billions of parameters!
Personally this is what I used for my Bachelor's where I made a tool to automatically generate input code from competitive programming statements.
AnnualDegree99 t1_j45rbg5 wrote
125m parameters already sounds like it wouldn't be fun on most GPUs that people actually own, I'm imagining a student with like a 1650 laptop waiting days with their laptop sounding like an F-15 with full afterburner
Nineshadow t1_j45ro3w wrote
It's appropriate for Google Colab.
Viewing a single comment thread. View all comments