In the last month I came across a blog post about vector databases. The post argued that there are a few basic types of distances (L1, L2, cosine) and that you are going to have better fortune using a vector database that supports those than searching using your own heuristic and hybrid solutions. So my suggestion would be to represent faces in some space that you can search over with a vector database or with some nearest neighbors index