You should check out https://github.com/jina-ai/jina and https://github.com/jina-ai/finetuner

But what you need is a feature extractor ( a pretrained neural network) and perform nearest neighbors do get the closest vectors