Viewing a single comment thread. View all comments

florinandrei OP t1_jb7pefb wrote

Reply to comment by Mediocre-Bullfrog686 in [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics. by florinandrei

> Isn't this what the ignore_index is doing?

No, it is not.

Let me repeat: ignore_index cuts holes in both the ground truth label frames, and in the prediction frames coming out of the model. Any pixels in those holes are ignored.

This includes pixels in the predictions from the model. You are ignoring chunks of the model's output.

> How else should we exclude them from the average metric?

By not computing metrics for that pixel value.

average_metric = sum(metric_index1 + metric_index2 + ... + metric_indexN) / N

Simply do not include it in the sum, and then just divide by N-1 instead.

What you are doing is not equivalent to that. What you are doing is: you discard pixels from both label frame and prediction frame based on the shape of some regions in the label frame alone. That makes no sense. Whatever the model's predictions happen to be in those holes, they are ignored even if they have pixel values different from ignore_index.

You are ignoring all the model's predictions in those holes, regardless of their pixel values.

You are discarding pixels from the model's output even if they have values different from ignore_index.

2

lynnharry t1_jb8n8p0 wrote

Pixels with ignore_index does not mean the model's output should also be ignore_index. It means the groundtruth label is not determined on those pixels and whatever your model's output is, its correctness is undetermined.

For those undetermined pixels, we simply ignore those outputs completely.

ignore_index is not used to ignore a specific category during the metric calculation, which is what you're proposing. ignore_index is simply notifying intersect_and_union some areas of the image have undetermined labels and should be ignored, and those areas are marked by the value of ignore_index.

2