netcal.metrics.ENCE

class netcal.metrics.ENCE(bins: int | Iterable[int] = 10, sample_threshold: int = 1)

Expected Normalized Calibration Error (ENCE) for a regression calibration evaluation to test for variance calibration. A probabilistic regression model takes $X$ as input and outputs a mean ${\mu }_Y(X)$ and a standard deviation ${\sigma }_Y(X)$ targeting the ground-truth $y$. Similar to the netcal.metrics.confidence.ECE, the ENCE applies a binning scheme with $B$ bins over the predicted standard deviation ${\sigma }_Y(X)$ and measures the absolute (normalized) difference between root mean squared error (RMSE) and root mean variance (RMV) [1]. Thus, the ENCE [1] is defined by

$$\text{ENCE}:=\frac{1}{B}\sum _{b=1}^B\frac{|RMSE(b)-RMV(b)|}{RMV(b)},$$

where $RMSE(b)$ and $RMV(b)$ are the root mean squared error and the root mean variance within bin $b$, respectively.

If multiple dimensions are given, the ENCE is measured for each dimension separately.

Parameters:

• bins (int or iterable, default: 10) – Number of bins used by the ENCE binning. If iterable, use different amount of bins for each dimension (nx1, nx2, … = bins).

• sample_threshold (int, optional, default: 1) – Bins with an amount of samples below this threshold are not included into the miscalibration metrics.

References

[1] (1,2)

Levi, Dan, et al.: “Evaluating and calibrating uncertainty prediction in regression tasks.” arXiv preprint arXiv:1905.11659 (2019). Get source online

Methods

__init__([bins, sample_threshold])	Constructor.
binning(bin_bounds, samples, *values[, nan])	Perform binning on value (and all additional values passed) based on samples.
frequency(X, y[, batched, uncertainty])	Measure the frequency of each point by binning.
measure(X, y, *[, kind, range_])	Measure the ENCE for given input data either as tuple consisting of mean and stddev estimates or as NumPy array consisting of a sample distribution.
prepare(X, y[, batched, uncertainty])	Check input data.
process(metric, acc_hist, conf_hist, ...)	Determine miscalibration based on passed histograms.
reduce(histogram, distribution, axis[, ...])	Calculate the weighted mean on a given histogram based on a dedicated data distribution.