bremen79 t1_j0n8wsn wrote on December 17, 2022 at 11:22 PM

Reply to comment by vwings in [D] Is softmax a good choice for confidence? by thanderrine

Platt scaling does not have any guarantee and in fact it is easy to construct examples where it fails. On the other hand, conformal prediction methods, under very weak assumptions, on the multiclass problem of the question would give you a set of labels that is guaranteed to contain the true label with a specified probability.

vwings t1_j0osaa7 wrote on December 18, 2022 at 7:44 AM

Now you are completely deviating from the original scope of the discussion. We discussed what is more general, but - since you changed scope - you agree with me on that.

About "guarantees": also for CP, it is easy to construct examples where it fails. If the distribution of the new data is different from the calibration set, it's not exchangeable anymore, and the guarantee is gone.