netcal.presentation.ReliabilityDiagram

class netcal.presentation.ReliabilityDiagram(bins: int | Iterable[int] = 10, *, equal_intervals: bool = True, detection: bool = False, sample_threshold: int = 1, fmin: float | None = None, fmax: float | None = None, metric: str = 'ECE', **kwargs)

Plot Confidence Histogram and Reliability Diagram to visualize miscalibration in the context of confidence calibration. On classification, plot the gaps between average confidence and observed accuracy bin-wise over the confidence space [1], [2]. On detection, plot the miscalibration w.r.t. the additional regression information provided (1-D or 2-D) [3].

Parameters:

  • bins (int or iterable, default: 10) – Number of bins used by the ACE/ECE/MCE. On detection mode: if int, use same amount of bins for each dimension (nx1 = nx2 = … = bins). If iterable, use different amount of bins for each dimension (nx1, nx2, … = bins).

  • equal_intervals (bool, optional, default: True) – If True, the bins have the same width. If False, the bins are splitted to equalize the number of samples in each bin.

  • detection (bool, default: False) – If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

  • fmin (float, optional, default: None) – Minimum value for scale color.

  • fmax (float, optional, default: None) – Maximum value for scale color.

  • metric (str, default: 'ECE') – Metric to measure miscalibration. Might be either ‘ECE’, ‘ACE’ or ‘MCE’.

References

Methods

__init__([bins, equal_intervals, detection, ...])

Constructor.

plot(X, y, *[, batched, uncertainty, ...])

Reliability diagram to visualize miscalibration.