darkbluetwilight OP t1_jeesol1 wrote on March 31, 2023 at 2:21 PM

Reply to comment by machineko in [D]Suggestions on keeping Llama index cost down by darkbluetwilight

Thanks, I'll give Cerebras another look. Yes, I'm using the APIs

darkbluetwilight OP t1_je99g95 wrote on March 30, 2023 at 10:32 AM

Reply to comment by machineko in [D]Suggestions on keeping Llama index cost down by darkbluetwilight

Correct, it's for personal use only. I did look into a few different options - Huggingface, Alpaca, BERT, Chinchilla, Cerebras but they all appear to have charges too (with the exception of Alpaca which was taken down). I already had openai nicely implemented in my GUI so wasn't really drawn by any of them.
Can you suggest a model that is free or cheaper than openai that I could integrate into my python gui?
On the database side I tried Mongo DB and Atlas but found these very difficult to use. Since I only need to generate the database once, Llama index was fine to use

darkbluetwilight OP t1_jdy6eu6 wrote on March 28, 2023 at 1:23 AM

Reply to comment by rshah4 in [D]Suggestions on keeping Llama index cost down by darkbluetwilight

Nice suggestion thanks! llama-index currently uses an embedding version of Ada which has negligible pricing (0.0002/1000tokens I think) The once-off index creation (1.3million tokens) cost about 40c.
It was the AI text generation costs that was killing me.

darkbluetwilight OP t1_jdy5v9e wrote on March 28, 2023 at 1:19 AM

Reply to comment by supreethrao in [D]Suggestions on keeping Llama index cost down by darkbluetwilight

I think you are correct. I started being more specific in my prompts and even telling it what level of detail I wanted back and it is showing a lot more promise now, the responses are much more useful. Makes me a little concerned for when asking it about things I'm less familiear with, might need to fall back to davinci.
I wonder why davinci comes across as being more intelligent than gpt3.5? Maybe the reduced cost has something to do with this, less compute power behind it maybe?
With regard to the token count my program is a lot more complex than the code I provided in the OP, with a lot of context and token management features in there already so was able to rule out potential token availability issues.

darkbluetwilight OP t1_jdv9560 wrote on March 27, 2023 at 1:31 PM

Reply to comment by supreethrao in [D]Suggestions on keeping Llama index cost down by darkbluetwilight

You are a gentleman! There doesn't appear to be any documentation in the llama-index docs yet but there is support added via the langchain module. It looks like I can "from langchain.llms import OpenAIChat" and then use this function to build a new index using "gpt-3.5-turbo" model. I will give this a go and see if it works. I will look into Treeindex too, reading the docs around these different indexing tools was getting a bit too complex for me