Viewing a single comment thread. View all comments

gBoostedMachinations t1_j155zas wrote

Reply to comment by caedin8 in [D] Running large language models on a home PC? by Zondartul

Training is what takes so much computation in almost all cases. Once the model itself is trained only a tiny fraction of the compute is needed. Most trained ML models that ship today can generate predictions on a raspberry pi or a cell phone. LLMs still require more hardware for inference, but you’d be surprised how little they need compared to what’s needed for training.

8

calv420 t1_j15ytb1 wrote

Don't see why you're getting down voted, inference requires significantly less compute vs training.

5

gBoostedMachinations t1_j16pzea wrote

If there’s on thing I’ve learned about Reddit, it’s that you can make the most uncontroversial comment of the year and still get downvoted. I mean, I got banned from r/coronavirus for pointing out that people who recover from covid probably have at least a little tiny bit of immunity to re-infection.

After covid, I’ve learned to completely ignore my comment scores when it comes to feedback on Reddit. The only way to know if one of my comments is valued is to read the replies.

7