netcal.binning.BBQ

class netcal.binning.BBQ(score_function: str = 'BIC', equal_intervals: bool = True, detection: bool = False, independent_probabilities: bool = False)

Bayesian Binning into Quantiles (BBQ). This method has been proposed by [1]. This method utilizes multiple HistogramBinning instances with different amounts of bins and computes a weighted sum of all methods to obtain a well-calibrated confidence estimate. The scoring function “BDeu”, which is proposed in the original paper, is currently not supported.

Let \(\mathcal{D} = \{(x_0, y_0), (x_1, y_1), ... \}\) denote a data set with input data \(x\) and ground truth labels \(y \in \{0, 1\}\) of length \(N\). Let \(M\) denote a model with estimated parameters \(\hat{\theta}\) of length \(K\) and let \(\hat{p}\) denote the confidence estimates on \(\mathcal{D}\) of model \(M\) by parameters \(\hat{\theta}\). The score function might either be the Aikaike Information Criterion (AIC) given by

\[AIC = -2 L ( \hat{\theta} ) + 2K\]

or the Bayesian Information Criterion given by

\[BIC = -2 L ( \hat{\theta} ) + \log(N)K\]

with \(L (\hat{ \theta })\) as the log-likelihood given by

\[L (\hat{ \theta }) = \sum_{i=1}^N y^{(i)} \log(\hat{p}^{(i)}_\hat{\theta}) + (1-y^{(i)}) \log(1 - \hat{p}^{(i)}_\hat{\theta}) .\]

These scores can be used to calculate a model posterior given by

\[p(M | \mathcal{D}) \propto p( \mathcal{D} | M )p(M) \approx \exp( -BIC/2 )p(M) .\]

Using the elbow method to sort out models with a low relative score, the weights for each model can be obtained by normalizing over all model posterior scores.

Parameters:

• score_function (str, default='BIC') – define score functions: - ‘BIC’: Bayesian-Information-Criterion - ‘AIC’: Akaike-Information-Criterion

• equal_intervals (bool, optional, default: True) – If True, the bins have the same width. If False, the bins are splitted to equalize the number of samples in each bin.

• detection (bool, default: False) – If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

• independent_probabilities (bool, optional, default: False) – Boolean for multi class probabilities. If set to True, the probability estimates for each class are treated as independent of each other (sigmoid).

References

[1]

Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht: “Obtaining well calibrated probabilities using bayesian binning.” Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015. Get source online

Methods

__init__([score_function, equal_intervals, ...])	Constructor.
clear()	Clear model parameters.
epsilon(dtype)	Get the smallest digit that is representable depending on the passed dtype (NumPy or PyTorch).
fit(X, y)	Build BBQ calibration model.
fit_transform(X[, y])	Fit to data, then transform it.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
load_model(filename)	Load model from saved torch dump.
save_model(filename)	Save model instance as with torch's save function as this is safer for torch tensors.
set_output(*[, transform])	Set output container.
set_params(**params)	Set the parameters of this estimator.
transform(X)	After model calibration, this function is used to get calibrated outputs of uncalibrated confidence estimates.