Viewing a single comment thread. View all comments

besabestin OP t1_j5ya2af wrote

Reply to comment by vivehelpme in Few questions about scalability of chatGPT [D] by besabestin

I see. Interesting. I thought it was generating one by one like that. I wonder why it sometimes encounters error after generating a long text and just stops half way through the task - which happened to me frequently.

7

crt09 t1_j5ytazq wrote

the guy above was kind of unclear, its an autoregressive langauge model so it does generate one at a time, puts it back into the input and generates the next one. It could be printed out in one go once they waitied for it to stop and then be sent to the client and pritned all at once but they went with the fancy GUI type, possibly yeah as a way to slow down spamming

21

visarga t1_j6c0e8m wrote

They might use a second model to flag abuse, not once every token, but once every line or phrase. Their models are already trained to avoid being abused, but this second model is like insurance in case the main one doesn't work.

1

suntehnik t1_j5ykwei wrote

Just speculation here: maybe they store generated text in a buffer and when they run out of memory buffer can be flushed to get allocation back for other tasks.

−1