Viewing a single comment thread. View all comments

DB6135 t1_jd68vdi wrote

Reply to comment by not_particulary in [D] Running an LLM on "low" compute power machines? by Qwillbehr

What are the recommended parameters? I tried the 7B model with default settings but it kept generating repeated garbage output.

9

Qwillbehr OP t1_jd6baxv wrote

I played with it for a few minutes and noticed that the 16B alpaca model gave significantly better responses. From what I can tell though the issue seems to be in how dalai prompts alpaca.CPP (just tells it to complete the sentence with all possible outputs rather then just one of the possible answers). The 16B model fixed most of it for me

4