More recently GPT-3 can load 4000 tokens in the context. If you have a dataset of texts you can make a search engine that will put the top results in the context. Then GPT-3 can reference that and answer as if it was up to date.

Using this trick a 25x smaller model could have similar results with a big model, they had 1 trillion tokens of text in the reference.