JH4mmer t1_ixhfmew wrote on November 23, 2022 at 1:55 PM

My recommendation would be to drop the "other" class entirely. Thats a classic mistake that I've seen juniors do many times, and it doesnt really work out like you expect it to in the real world. The main problem with that approach is that a catch-all class like that has infinite variance (theoretically requiring infinite training data). Plus your labels often become massively unbalanced relative to the positive classes.

Instead, think of your model as having multiple tails, one for each class you actually care about (e.g. what is the probability that a dog is in this image?, what is the probability that a cat is in this image? Etc.) Each output has its own logistical activation that's independent of the other classes. Where before you might have had a softmax layer that returned [0.2, 0.3, 0.5] for (dog, cat, other), you might now have [0.8, 0.7] for (dog, cat). The labels will not sum to 1 because they are independent of one another.

Note that this is the approach you would take for multi-class classification as well, so you might want to read up on that pattern for more information.

Lastly, if you have a trained model in this format, adding a new class is very easy. The first N layers of the network are shared for all classes and so are already pretrained for you. You would add a new tail to the model using whichever weight initialization strategy you care about, add some samples of the new class, and then do some fine tuning on the new tail layer(s) to make sure that your network can effectively detect the new class.

Of course there are many variations to this training approach. You may choose to also do some fine tuning of the entire network with a dataset that includes samples of the new class, but hopefully you get the idea.

I hope this points you in the right direction! Cheers.

First_Bullfrog_4861 t1_ixhi8ki wrote on November 23, 2022 at 2:16 PM

but finetuning requires the relabeling of the original dataset to include both old and the new label, which op specifically does not want to do.

i don’t think what op wants is doable. or is there some approach i’m missing? i think what op basically wants to do is retrain, but with only data from the new class, and still avoid catastrophic forgetting of the other labels.

is there a way to do this?

JH4mmer t1_ixhvl5k wrote on November 23, 2022 at 3:49 PM

You may be making a different set of assumptions about the training data than I am, so let me clarify a bit. :-)

If you start with images that truly do contain just one class, the addition of a new class label wouldn't change anything. Your level vector for the exisiting images would migrate from [1, 0] to [1, 0, 0], something that can be done automatically without additional human intervention. Your new images (used for training the new class) would have a label of [0, 0, 1].

If, however, your images do already contain more than one possible class (which is far and away much more common in real-world data), the original labels would be already invalid, since the original labeling assumed that there was only one correct answer. Those images that do contain multiple classes would have to be relabeled, yes.

The process I'm describing is a mechanical one that doesn't involve a separate knowledge distillation step. It's a technique my team has used successfully in industrial retail applications, where the number of classes is truly an unknown, and we have to add or remove classes from our trained models frequently.

First_Bullfrog_4861 t1_ixjazoo wrote on November 23, 2022 at 9:27 PM

ok, got it. however, in my experience the number of labels is far less obvious in real world datasets than one might expect. consider an example with images of bottles, cups and glasses, so three labels.

a model trained on these three labels will need revision if further down after the deployment process ‚bottles‘ need to be split in ‚plastic bottles‘ and ‚glass bottles‘. both label sets are perfectly valid, due to the hierarchical nature of things.

anyway, my point is actually another one: afaik this will require dataset relabeling and fully iterate the training process on the newly labeled dataset.

or is there a faster way to make the model aware of the more finegrained bottle labels?

i mean, without access to data of cups and glasses, basically inform it of more finegrained bottle types but let it still keep its knowledge of cups and glasses.

LinuxSpinach t1_ixiy1zy wrote on November 23, 2022 at 8:01 PM

>main problem with that approach is that a catch-all class like that has infinite variance

Sometimes it doesn't, and I've seen an 'other' class work well in these cases. In cases where the data being fed to the model already constrains the variance, then an other class won't have infinite variance. Eg. you know that all of the data will be pictures of fruit, but you only want to label apples, bananas and oranges. In this case, there is a finite number of fruits to take pictures of.

If you are going to use an 'other' label, I think it should be ok in cases where you could label the data, but the labels that the other class comprises are unimportant to your application.