huehue9812 t1_iw6aao8 wrote on November 13, 2022 at 7:33 AM

Can someone please enlighten me why this is huge?

The concept of a "true layout" (given the same data and modeling choice), imo, seemed to be implicitly known or acknowledged.

machinelearner77 t1_iw6l27e wrote on November 13, 2022 at 10:04 AM

I don't get the huge thing either. Seems to me like a thorough (and valuable) analysis of something that's probably already been known and tried out in one form or another a couple of times, since the idea is so simple. But is it a big or even huge finding? I don't know..

vwings t1_iw768hl wrote on November 13, 2022 at 2:11 PM

I think it's valuable, but not huge. There have been several recent works that use this concept that a sample is described by similar samples to enrich representations:

the cross-attention mechanism in Transformers does this to some extent
AlphaFold: a protein is enriched with similar (by multiple sequence alignment) proteins
CLOOB: a sample is enriched with similar samples from the current batch
MHNfs: a sample is enriched with similar samples from a large context.

This paper uses this concept, but does it differently: it uses the vector of cosine similarities, which in other works is softmaxed and then used a weights for averaging, directly as representation. That this works and that you can backprop over this is remarkable, but not huge... Just my two cents... [Edits: typos, grammar]

machinelearner77 t1_iw7omuy wrote on November 13, 2022 at 4:29 PM

That it works seems interesting, especially since I would have thought that it might depend too much on the hyper-parameter (anchors), which apparently it doesn't. But why shouldn't you be able to "backprop over this"? It's just cosine, everything is naturally differentiable

vwings t1_iw857q2 wrote on November 13, 2022 at 6:20 PM

Yes, sure you can backprop, but what I meant is that you are able to train a network reasonably with this -- although in the backward pass the gradient gets diluted to all anchor samples. I thought you would at least need softmax attention (forward pass) to be able to route the gradients back reasonably.