There are two lines of work that come to mind you might be interested in.

1. Geometric deep learning primarily studies various types of invariances (translation, permutation, etc) that can be encoded in DL architectures.
2. Algorithmic alignment studies the relationship between information flow in classical algorithms and DL architectures and how "aligning" the latter to the former can improve performance.

Edit: Spelling

Have you posted actual technical details to share and get feedback? As a long time member of this sub I would be interested, and I don’t think I’m alone here.

There is some work on Frustratingly Short Attention Spans in Neural Language Modeling

Use a pairwise loss.

Glad it was helpful!

You could use a BK-tree.

>average fps across multiple runs gives a more realistic performance and eliminates any outliers

Thanks for the laugh. I'll just leave this here so you can read about why the mean (average) is not a robust measure of central tendency because it is easily skewed by outliers.