Viewing a single comment thread. View all comments

kaibee t1_j05omff wrote

Reply to comment by ReginaldIII in [Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero

> I earnestly believe it solves problems that contain similar words. But it just does not present a practical solution to this problem. > > > > We can't put the returned values on the blockchain. It just isn't possible to store them they are too big and too many, and there is no reason to store them, we only want to pass them onto the next worker or workers that immediately need them. We do care about fault tolerance to make sure they get to their destination. > > > > So there's no way for this pool of blockchain nodes to form a consensus over the returned values being "correct" like this. We can't put the relevant information on the blockchain to allow it be compared. > > > > What you end up with is just a classic non-blockchain vote by agreement system between workers of unknown trustworthiness. No blockchain needed. > > > > You are correct that voting by consensus is needed, you just don't need all the rest of the things that turn that into a blockchain.

This is basically solving a very similar problem. https://rendertoken.com/#intro

1

ReginaldIII t1_j06mqeo wrote

In rendertoken's scenario we don't have a requirement on high throughput of one job feeding into another.

The individual units of work are expensive and long lived. Rendering a frame of a film takes roughly the same amount of time it did a few years ago, we just get higher fidelity output for that same render budget. All the frames can be processed lazily by the compute farm, and the results just go into a pool for later collection.

Because the collation of the results happens in a more offline fashion from the actual computation, you have time and resources to encode the results on a blockchain. Auditing that your requested work was processed is a desirable quality, and so a blockchain does provide a benefit.

In the case of distributed model training the scenario is different. We have high throughput of comparatively small chunks of work. Other than passing the results to the next immediate worker to do the next part of the computation, we have no desire (or storage capacity) to keep any of the intermediate results. Because we have high throughput of many small chunks a blockchain encoding these chunks would need a small proof of work and so would not be a reliable source of truth anyway.

Then consider that we don't even care about having an audit trail to prove historical chunks really were processed when we think they were. We only care about checking results are valid on the fly as we are doing the compute.

We just need a vote by agreement on the immediate results so they can be handed off to the next workers. Yes blockchains often have a vote by agreement part to how they decide what the actual state of the blockchain is, but we just need that part. We don't actually need the blockchain itself.

2