DaLameLama t1_iu2jo4l wrote on October 28, 2022 at 2:00 AM

You need a way to prevent the training from collapsing to trivial solutions (like, both NNs output the same constant vector for all inputs).

Methods similar to your idea are Barlow Twins or VICReg.

AmalgamDragon OP t1_iu2qt5u wrote on October 28, 2022 at 2:55 AM

Thanks! Your reference to the issue with constant vectors and the same reference in the relevant papers for those methods you mentioned completes my investigation on this (i.e. this isn't an approach worth pursuing).

DaLameLama t1_iu2rebs wrote on October 28, 2022 at 3:00 AM

Why not worth pursuing? LeCun still believes VICReg is amazing. Feel free to come up with your own twist :)

AmalgamDragon OP t1_iu2souc wrote on October 28, 2022 at 3:10 AM

I meant the general approach I laid out in my original post. That said, I'm also not working with image data (or audio or NLP) and generalizing VICReg seems like its more in theory then in practice at the moment.