I'm not sure how it would really learn something from the input if you don't define a more useful task. How would this model penalize a "collapse" situation where both models always predict 0 for example or any random value?

Contrastive learning algorithms try to build two different embeddings for different parts of the same input, penalize collapse, and train a model to make the two embeddings as close as possible, knowing they come from the same input, even if they are from different parts of that input. It looks a bit like what you said but I don't know an implementation that is like the one you said.