> I'm not sure how it would really learn something from the input if you don't define a more useful task. How would this model penalize a "collapse" situation where both models always predict 0 for example or any random value?

Yeah, it may not work well. I haven't been able to track down if this is something that has been tried and been found wanting or not.