>main problem with that approach is that a catch-all class like that has infinite variance

Sometimes it doesn't, and I've seen an 'other' class work well in these cases. In cases where the data being fed to the model already constrains the variance, then an other class won't have infinite variance. Eg. you know that all of the data will be pictures of fruit, but you only want to label apples, bananas and oranges. In this case, there is a finite number of fruits to take pictures of.

If you are going to use an 'other' label, I think it should be ok in cases where you could label the data, but the labels that the other class comprises are unimportant to your application.