To answer your title question, no. You may wanna look up Bayesian deep learning!

Depends on what you mean by confidence. With softmax, you model probability. You can train your network to give out near 100% probabilities per class, but this tells you nothing about how confident it is.

Instead, what you could do is the following:

• get a prediction
• define the target of your prediction as the resolved label
• calculate loss on this
• now define your target as the inverse of the initial one
• calculate the loss again
• divide the 2nd one by the first one

Voila, you've got a confidence score for your sample. However, this will only give you a number that will be comparable to other samples. This will not give you a % of confidence. You do know, however, that the higher the confidence score, the closer the confidence is to 100%. And you know the smaller your score is, the closer the confidence to 0%. Based on the ranges of your loss, you can probably figure out how to map it to whatever range you want.

For a multi-class problem, you could just sum the loss ratio between all classes other than your predicted class over the loss of your prediction. So, if you had a classifier that classifies an imagine into dog, cat, and fish, and your softmax layer spits out 0.9, 0.09, 0.01, your confidence score would be loss(cat)/loss(dog) + loss(fish)/loss(dog).

What you are describing is exactly the problem of OOD detection in image classification. The canonical reference for that would be Dan Hendrycks et. al. 2019. In short, no. Softmax probability might not be the best indicator of confidence as deep networks are often over confident on the wrong outputs.

The only approach that gives valid uncertainty quantification is conformal prediction, a quick google search should result in a good number of tutorials.

The root of your problem is that you are stuck assigning 100% confidence to the prediction that your horse is a cat or a dog. It’s perfectly rational that you might have p(cat)=1e-5 and p(dog)=1e-7 for a horse picture, right?

So when you normalize those, you get basically p(cat)=1.

Try binary classification of cat vs not cat and dog vs not dog. Don’t make them sum to one.

Could you please share the title of the paper you’re referring to since Hendrycks published multiple highly cited papers in 2019?

There are several papers on this idea - the best one is probably On Calibration of Modern Neural Networks by Guo et al. The gist is that you want your softmax output to be the same as the probability of your prediction being correct. For your architecture they do this through something that is called temperature scaling. Why this works is a more involved topic, but you can get a better handle of what the consequences are of cross entropy using hard (1s and 0s) vs soft labels (not so much a 1 or 0).

I think then going into OOD as the others suggest would be more fruitful. The whole deal with distribution shift and then the extremes of OOD, gets very murky and detached from what happens in practice, but ultimately the goal is to have the ability to know that a mismatch of input to model is happening, vice just having low confidence.

My bad! Here you go! (Also looks like I messed up the years, but all of these are good references)

1. MSP: https://arxiv.org/abs/1610.02136
2. OE: https://arxiv.org/pdf/1812.04606.pdf
3. One vs all: https://arxiv.org/abs/2007.05134

[deleted]

It's a hard problem, nobody has a definitive solution. From my lectures and experience:

• interval calibration (easy)

• temperature scaling (easy)

• ensembling (expensive)

• Monte Carlo dropout (not great)

• using prior networks or auxiliary networks (for OOD detection)

• error correcting output codes (ECOC)

• conformal prediction (slightly different task than confidence estimation)

Here's my Zeta-Alpha confidence estimation paper feed.

Quick question. Wouldn't a simple solution be to include a 'neither'/'other' output class?

Given that a network should classify an image as a dog or a cat, in reality a lot of use cases actually want a multi-class prediction rather than binary, because a picture of a monkey should not be a dog or a cat. Just on a hunch I would guess the performance goes down significantly and obviously requires more training data.

The hacky solutions don't really work... the fact is that if you're going to show the model ones and zeros, the model will learn to predict ones and zeros. If the label however would be a probability, then yes it would be a good measure of confidence.

Yes, that's an option but you have absolutely no guarantees about it's ability to produce anything meaningful. What images to you introduce in the "other" class? There are infinitely more images falling in the other category than there are cat-or-dog images. For any training set you come up with for the "other" class, the model can still be tested with an image totally different from your training set, and the model output will have no reason what-so-ever to favour "other" for the new image.

True, the thought did occur to me, but I thought you could train the other category with a diverse set of animals and also people, nature, cars, landscapes etc. While there are a larger infinite set of "non-dog" or "non-cat" images, it must be possible to classify features that absolutely don't indicate a dog or cat...I don't think it's the most effective method perhaps...though it would be interesting to give it a go, maybe after my exams I'll try...

I can't shake the feeling that it might be somehow informative on the classification layer, either for reducing the confidence of the other categories or weighting it somehow

Softmax class probabilities always have to sum to 100%, so unless you create an “other” class, this will continue to be an issue.

Just replace your the activation on your output layer with sigmoid activations, which will convert your model to a multilabel output.

Yes, indeed the Hendrycks paper is the first to read in this context: https://arxiv.org/abs/1610.02136

Great question and comment! In think the first statement here is that usually the CNNs are overconfident.

One thing that the original post is looking for is calibration of the classifier on a calibration set. On the calibration set, the softmax values can be re-adjusted to be close to probabilities. This is essentially what Conformal Prediction and Platt Scaling do.

I strongly recommend this year's talk on Conformal Prediction which provides insights into these problems. Will try to find the link...

This is not true... There are many other works,e.g. Platt Scaling that also provide calibrated classifiers (i suppose this is what you call "valid"). But conformal prediction indeed tackles this problem...

A hard problem indeed. The methods in your list have use different settings. Deep Ensembles, MC Dropout don't require a calibration set. The prior networks (i love this paper) assume that during training OOD samples are available. Conformal prediction assumes the availability of a calibration set that follows the distribution of future data... For the other methods,I would have to check ...

You will get some results, for sure. Depending on your application may even be good enough. But as a general probability that an image is something other than cat and dog, not really.
As other commenters have mentioned the general problem is known as OOD (out of distribution) sample detection. There are Deep Learning models which model probabilities explicitly and can in principle used for OOD sample detection - Variational Autoencoders. The original formulation of this model performs poorly in practice at OOD sample detection, but there is work addressing some shortcomings, for instance Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation. But with VAEs things get very mathematical, very fast.
Coming back to you initial question, no, softmax is not appropriate for "confidence", but this is an open problem in Deep Learning.

I have a classification model that uses conformal prediction. This has been helpful in working towards building out a high confidence dataset.

IIRC, CP extends Platt.

Don't you need an bayesian approach for that?

From my experience the loss function also plays an important part. Cross entropy forces the model to be very certain with a decision. Focal loss can produce a smoother output distribution.

[removed]

That's what the CP guys say. :)

I would even say that Platt generalizes CP. Whereas CP focuses on the empirical distribution of prediction score only around a particular location at the tails, e.g. at confidence level 5%, Platt scaling tries to mold the whole empirical distribution into calibrated probabilities -- thus Platt considers the whole range of the distr of scores.

Great point with conformal prediction! Would love a link too!

I can confirm this. I did NER and most tokens are not names entities, so they are "other". It's really hard to define what "other" means, even with lots of text the model is unsure. No matter how much "other" I provide, I couldn't train a negative class properly.

How about a bronze statue of a dog, a caricature of a cat, a fantasy image that is hard to classify, etc? Are they "other"?

You should use a Bayesian network. Monte Carlo dropout is an easy place to start. Then you can get a measure of the uncertainty of your predictions.

This is the link to the NeurIPS2022 invited talk: https://neurips.cc/virtual/2022/invited-talk/55872 (don't know if it's accessible without registration)

Here is almost the same talk: https://www.youtube.com/watch?v=dmZBxW7oY1o

Platt scaling does not have any guarantee and in fact it is easy to construct examples where it fails. On the other hand, conformal prediction methods, under very weak assumptions, on the multiclass problem of the question would give you a set of labels that is guaranteed to contain the true label with a specified probability.

What do you mean by confidence? Is that a new metric or smth

Now you are completely deviating from the original scope of the discussion. We discussed what is more general, but - since you changed scope - you agree with me on that.

About "guarantees": also for CP, it is easy to construct examples where it fails. If the distribution of the new data is different from the calibration set, it's not exchangeable anymore, and the guarantee is gone.

Not sure why I got downvoted. This is a very common problem in industry, and often how I have my team handle it.

But what is confidence really? It's a measure based on how likely an outcome is given a specific model. The idea of confidence is completely broken if you are not certain about your model. E.g., if you think that our error is normal distributed with a certain variance you can make statements if a divination from the expected value is noise or not. But this assumes that your normal distribution assumption is correct! If you cannot be certain about the model, which you never really are if you use neural networks, then the confidence is measured against your own implicit model. And since NNs are very different from your own brain and the models used in both cases are likely computing a different function, AND the NNs is not trained to predict confidence (from a human perspective) there is no meaningful way of talking about confidence.

Softmax is not good. I would look into conformal prediction. It's a beautiful solution to this problem but it requires extra data. Worth it imo.