netcal.metrics.MMCE

class netcal.metrics.MMCE(detection: bool = False)

Maximum Mean Calibration Error (MMCE). The MMCE [1] is a differentiable approximation to the Expected Calibration Error (ECE) using a reproducing _kernel Hilbert space (RKHS). Using a dataset \(\mathcal{D}\) of size \(N\) consisting of the ground truth labels \(\hat{y} \in \{1, ..., K \}\) with input \(x \in \mathcal{X}\), the MMCE is calculated by using a scoring classifier \(\hat{p}=h(x)\) that returns the highest probability for a certain class in conjunction with the predicted label information \(y \in \{1, ..., K \}\) and is defined by

\[MMCE = \sqrt{\sum_{i, j \in \mathcal{D}} \frac{1}{N^2}(\mathbb{1}(\hat{y}_i = y_i) - \hat{p}_i) (\mathbb{1}(\hat{y}_j = y_j) - \hat{p}_j)k(\hat{p}_i, \hat{p}_j)} ,\]

with \(\mathbb{1}(*)\) as the indicator function and a Laplacian _kernel \(k\) defined by

\[k(\hat{p}_i, \hat{p}_j) = \exp(-2.5 |\hat{p}_i - \hat{p}_j|) .\]

Parameters:

detection (bool, default: False) – Detection mode is currently not supported for MMCE! If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

References

[1]

Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain: “Trainable calibration measures for neural networks from _kernel mean embeddings.” International Conference on Machine Learning. 2018. Get source online.

Methods

__init__([detection])	Constructor.
measure(X, y[, batched])	Measure calibration by given predictions with confidence and the according ground truth.