There's a Github project that an individual put together based on the RETRO paper. If you checkout the issues list there is some info on work on a pretrained model.

There is also the Huggingface RAG Model and Facebook has a couple of models on the HF hub.

Note that the RAG model is an an older approach to retrieval so you probably want to be looking at the RETRO project above.