https://arxiv.org/abs/2212.14034 might be a good starting point.

> This means, with an average of 60 watt idle and 350 watt max for a RTX 4090

RTX 4090 "idles" (stream at background) at 10-15 watt. 4k144hz monitor might change it, but 60 watt is way too much for GPU only.

Is there any reason for them to be worse than random weights? Because if not, then you have no reason not to use pretrained models.

> Am I missing something about transfer learning here?

Theoretical answer: Both images and spectrograms have continuous curved lines as a signal, and therefore some transfer should happen.

Practical answer: If it works, it works.