netcal.binning.ENIR

class netcal.binning.ENIR(score_function: str = 'BIC', quick_init: bool = True, detection: bool = False, independent_probabilities: bool = False)

Ensemble of Near Isotonic Regression (ENIR) models. This method has initially been proposed by [1]. These models allow - in contrast to standard IsotonicRegression method - a violation of the monotony restrictions. Using the modified Pool-Adjacent-Violators Algorithm (mPAVA), this method build multiple Near Isotonic Regression models and weights them by a certain score function.

Let \(\mathcal{D} = \{(x_0, y_0), (x_1, y_1), ... \}\) denote a data set with input data \(x\) and ground truth labels \(y \in \{0, 1\}\) of length \(N\). Let \(M\) denote a model with estimated parameters \(\hat{\theta}\) of length \(K\) and let \(\hat{p}\) denote the confidence estimates on \(\mathcal{D}\) of model \(M\) by parameters \(\hat{\theta}\). The score function might either be the Aikaike Information Criterion (AIC) given by

\[AIC = -2 L ( \hat{\theta} ) + 2K\]

or the Bayesian Information Criterion given by

\[BIC = -2 L ( \hat{\theta} ) + \log(N)K\]

with \(L (\hat{ \theta })\) as the log-likelihood given by

\[L (\hat{ \theta }) = \sum_{i=1}^N y^{(i)} \log(\hat{p}^{(i)}_\hat{\theta}) + (1-y^{(i)}) \log(1 - \hat{p}^{(i)}_\hat{\theta}) .\]

These scores can be used to calculate a model posterior given by

\[p(M | \mathcal{D}) \propto p( \mathcal{D} | M )p(M) \approx \exp( -BIC/2 )p(M) .\]

Using the elbow method to sort out models with a low relative score, the weights for each model can be obtained by normalizing over all model posterior scores.

Parameters:

• score_function (str, default='BIC') – define score functions: - ‘BIC’: Bayesian-Information-Criterion - ‘AIC’: Akaike-Information-Criterion

• quick_init (bool, default=True) – Allow quick initialization of NIR (equal consecutive values are grouped directly).

• detection (bool, default: False) – If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

• independent_probabilities (bool, optional, default: False) – Boolean for multi class probabilities. If set to True, the probability estimates for each class are treated as independent of each other (sigmoid).

References

[1]

Naeini, Mahdi Pakdaman, and Gregory F. Cooper: “Binary classifier calibration using an ensemble of near isotonic regression models.” 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016. Get source online

Methods

__init__([score_function, quick_init, ...])	Constructor.
clear()	Clear model parameters.
epsilon(dtype)	Get the smallest digit that is representable depending on the passed dtype (NumPy or PyTorch).
fit(X, y)	Build ENIR calibration model.
fit_transform(X[, y])	Fit to data, then transform it.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
load_model(filename)	Load model from saved torch dump.
save_model(filename)	Save model instance as with torch's save function as this is safer for torch tensors.
set_output(*[, transform])	Set output container.
set_params(**params)	Set the parameters of this estimator.
transform(X)	After model calibration, this function is used to get calibrated outputs of uncalibrated confidence estimates.