Submitted by dogonix t3_z6ylfq in MachineLearning
Are there existing solutions/experiments to train LLMs collaboratively on distributed machines over the Internet? Something similar to the SETI@Home project.
Comments
bloc97 t1_iy57elh wrote
With traditional gradient descent, probably not as most operators in modern NN architectures are bottlenecked by bandwidth and not much by compute. There's active research on alternative training methods but they mostly have difficulties in detecting malicious agents in the training pool. If you own all of the machines those algorithms work but when there is a non insignificant amount of malicious agents the training might fail or even produce models that have backdoors or have private training data leaked.
dogonix OP t1_iy78bqr wrote
Interesting. Do you have links to resources with more info about the issue you mention? Thanks.
Away-Gas6172 t1_iy47pqq wrote
Maybe federated learning could be an option. See https://ai.googleblog.com/2017/04/federated-learning-collaborative.html?m=1
bgyoon t1_iy746vr wrote
CKtalon t1_iy432bm wrote
Check out PETAL. https://arxiv.org/pdf/2209.01188.pdf
dogonix OP t1_iy9n1lb wrote
Thanks all for the input.
WokeAssBaller t1_iy42daj wrote
Hugging face put out a paper on this https://huggingface.co/training-transformers-together