Viewing a single comment thread. View all comments

Lajamerr_Mittesdine t1_jc5b99n wrote

Reply to comment by light24bulbs in [P] Discord Chatbot for LLaMA 4-bit quantized that runs 13b in <9 GiB VRAM by Amazing_Painter_7692

I imagine 1 token per 0.2 seconds would be fast enough. That'd be equivalent to a 60 WPM typer.

Someone should benchmark it on an AMD 7950X3D or Intel 13900-KS

1

light24bulbs t1_jc5e0zk wrote

yeah theres definitely a threshold in there where its fast enough for human interaction. It's only an order of magnitude off, that's not too bad.

3