Alpaca-7B and Dalai, how can I get coherent results?

Submitted by Haghiri75 t3_11wdi8m in deeplearning

Recently, I installed dalai on my Macbook Pro (late 2019, i7 processor and 16GB of RAM) and I also installed Alpaca-7B model. Now when I ask it to write a tweet, it writes a wikipedia article and it does the same pretty much every time 😂

First, should I fine-tune it?

Second, is there any "prompt magic" going on here?

P.S: using this one, I got much better results. What's the difference between the two?

15

Comments

You must log in or register to comment.

Haghiri75 OP t1_jcxt80d wrote

I guess I found the reason. Dalai system does quantization on the models and it makes them incredibly fast, but the cost of this quantization is less coherency.

8

Board_Stock t1_jczmxht wrote

Hello, I've been running the alpaca.cpp on my laptop, and have you figured out how to make it remember conversations yet? sorry if this is a beginner question

1

Jaffa6 t1_jd03par wrote

Replying to Haghiri75 (#2,278,418)

That's odd.

Quantisation should make it go from (e.g.) 32 bit floats to 16bit floats, but I wouldn't expect it to lose that much coherency at all. Did they say somewhere that that's why?

3

Haghiri75 OP t1_jd32s29 wrote

Replying to Jaffa6 (#2,284,150)

Apparently I was wrong, the problem is not only quantization. It is because it's not Stanford's Alpaca and another alpaca-like model. This was what I can surely say about that.

1