supreethrao t1_jdy0xtd wrote on March 28, 2023 at 12:43 AM

Hi , to address Update2 , I think you’ll have to change your prompt to GPT3.5-turbo significantly. LLama index also has a cost estimator function that assumes a dummy LLM backend and calculates the expected cost , you can also use OpenAI’s tokenizer called “tiktoken” which is available on GitHub to calculate the exact number of tokens your text produces

darkbluetwilight OP t1_jdy5v9e wrote on March 28, 2023 at 1:19 AM

I think you are correct. I started being more specific in my prompts and even telling it what level of detail I wanted back and it is showing a lot more promise now, the responses are much more useful. Makes me a little concerned for when asking it about things I'm less familiear with, might need to fall back to davinci.
I wonder why davinci comes across as being more intelligent than gpt3.5? Maybe the reduced cost has something to do with this, less compute power behind it maybe?
With regard to the token count my program is a lot more complex than the code I provided in the OP, with a lot of context and token management features in there already so was able to rule out potential token availability issues.