Beat GPT-3 which has unlimited money using Open Source community

Submitted by koyo4ever t3_10ugxmc in deeplearning

Is it technically possible to train some model using a lot of personal computers like a cluster.

Eg: an Algorithm to train tiny parts of some model using personal computer of volunteers. Like a community that makes your gpu capacity available, even if it's little.

The idea is train tiny parts of a model, with a lot of volunteers, then bring it together to make some powerful deepmind.

Can this model beat a lot of money spent in models like GPT-3?

23

Comments

You must log in or register to comment.

ze_baco t1_j7c6yhd wrote

This is federated learning. We sure can do this, but it would require a lot of cooperation...

10

junetwentyfirst2020 t1_j7c7gpj wrote

You should consider diving into the topic a little deeper. What you’re talking about is distributing the computation, which is something that is already being done at some scale or another when there is more than one gpu or multiple machines. An outside example of this that you can donate your computers compute to SETI.

Your question about wether it can beat an existing implementation of gpt is the most complicated question ever posed in the history of humanity. It sounds like you’re assuming that this will have more compute than a dedicated system, but there’s a little more to getting something that performs better than just compute. Compute is a bottle neck, but only one of many.

9

lawless_c t1_j7cbt36 wrote

Communication between nodes would become a big bottle neck.

17

earthsworld t1_j7cc7l5 wrote

> train some model

which model? who's creating it? who's testing it? and who the fuck is paying for it?

−6

Appropriate_Ant_4629 t1_j7clc8s wrote

The LAION project ( https://laion.ai/ ) is probably the closest thing to this.

They're looking for volunteers to help work on their fully F/OSS ChatGPT successor now. A video describing the help they need can be found here.

They have a great track record on similar scale projects. They've partnered with /r/datahoarders and volunteers on creation of training sets including their 5.8 billion image/text-pair dataset that they used to train a better version of CLIP.

Their actual training of models tends to be done on some of the larger European supercomputers, though. If I recall correctly, their CLIP-derivative was trained with time donated on JUWELS. Too hard to split up such jobs into average-laptop-sized tasks.

29

sEi_ t1_j7e113t wrote

Afaik you can not split the process into 'small parts' as the whole model needs to be in the VRAM when processing it. And a consumer computer have a hard time to utilize 250+ GB VRAM.

But with the development speed in this area maybe the hurdle will be overcome soon™.

1