I played with it for a few minutes and noticed that the 16B alpaca model gave significantly better responses. From what I can tell though the issue seems to be in how dalai prompts alpaca.CPP (just tells it to complete the sentence with all possible outputs rather then just one of the possible answers). The 16B model fixed most of it for me
Qwillbehr OP t1_jd6baxv wrote
Reply to comment by DB6135 in [D] Running an LLM on "low" compute power machines? by Qwillbehr
I played with it for a few minutes and noticed that the 16B alpaca model gave significantly better responses. From what I can tell though the issue seems to be in how dalai prompts alpaca.CPP (just tells it to complete the sentence with all possible outputs rather then just one of the possible answers). The 16B model fixed most of it for me