netcal.scaling.LogisticCalibration

class netcal.scaling.LogisticCalibration(*args, temperature_only: bool = False, **kwargs)

On classification, apply the logistic calibration method aka Platt scaling to obtain a calibration mapping. This method is originally proposed by [1]. For the multiclass case, we use the Vector scaling proposed in [2]. On detection mode, this calibration method uses multiple independent normal distributions to obtain a calibration mapping by means of the confidence as well as additional features [3]. This calibration scheme assumes independence between all variables.

On detection, it is necessary to provide all data in input parameter X as an NumPy array of shape (n_samples, n_features), whereas the confidence must be the first feature given in the input array. The ground-truth samples y must be an array of shape (n_samples,) consisting of binary labels \(y \in \{0, 1\}\). Those labels indicate if the according sample has matched a ground truth box \(\text{m}=1\) or is a false prediction \(\text{m}=0\).

Mathematical background: For confidence calibration in classification tasks, a confidence mapping \(g\) is applied on top of a miscalibrated scoring classifier \(\hat{p} = h(x)\) to deliver a calibrated confidence score \(\hat{q} = g(h(x))\).

For detection calibration, we can also use the additional box regression output which we denote as \(\hat{r} \in [0, 1]^J\) with \(J\) as the number of dimensions used for the box encoding (e.g. \(J=4\) for x position, y position, width and height). Therefore, the calibration map is not only a function of the confidence score, but also of \(\hat{r}\). To define a general calibration map for binary problems, we use the logistic function and the combined input \(s = (\hat{p}, \hat{r})\) of size K by

\[g(s) = \frac{1}{1 + \exp(-z(s))} ,\]

According to [1], we can interpret the logit \(z\) as the logarithm of the posterior odds

\[z(s) = \log \frac{f(\text{m}=1 | s)}{f(\text{m}=0 | s)} \approx \log \frac{f(s | \text{m}=1)}{f(s | \text{m}=1)} = \ell r(s)\]

If we assume independence of all variables given in \(s\), we can use multiple univariate probability density distributions with the same variance to obtain a calibration mapping. Using this formulation, we can simply extend the scaling factor (from classification logistic calibration) to a scaling vector \(w \in \mathbb{R}^K\). However, instead of using the uncalibrated confidence estimate \(\hat{p}\), we use the logit of the network as part of \(s\) to be conform with the original formulation in [1] and [2]. Thus, the log-likelihood ratio can be expressed as

\[\ell r(s) = s^T w + c,\]

with bias \(c \in \mathbb{R}\). We utilize standard optimization methods to determine the calibration mapping \(g(s)\).

Capturing epistemic uncertainty of the calibration method is also able with this implementation [4].

Parameters:

• temperature_only (bool, default: False) – If True, use Temperature Scaling instead of Platt/Vector Scaling.

• method (str, default: "mle") – Method that is used to obtain a calibration mapping: - ‘mle’: Maximum likelihood estimate without uncertainty using a convex optimizer. - ‘momentum’: MLE estimate using Momentum optimizer for non-convex optimization. - ‘variational’: Variational Inference with uncertainty. - ‘mcmc’: Markov-Chain Monte-Carlo sampling with uncertainty.

• momentum_epochs (int, optional, default: 1000) – Number of epochs used by momentum optimizer.

• mcmc_steps (int, optional, default: 20) – Number of weight samples obtained by MCMC sampling.

• mcmc_chains (int, optional, default: 1) – Number of Markov-chains used in parallel for MCMC sampling (this will result in mcmc_steps * mcmc_chains samples).

• mcmc_warmup_steps (int, optional, default: 100) – Warmup steps used for MCMC sampling.

• vi_epochs (int, optional, default: 1000) – Number of epochs used for ELBO optimization.

• detection (bool, default: False) – If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

• independent_probabilities (bool, optional, default: False) – Boolean for multi class probabilities. If set to True, the probability estimates for each class are treated as independent of each other (sigmoid).

• use_cuda (str or bool, optional, default: False) – Specify if CUDA should be used. If str, you can also specify the device number like ‘cuda:0’, etc.

References

[1] (1,2,3)

Platt, John: “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.” Advances in large margin classifiers 10.3: 61-74, 1999 Get source online

[2] (1,2)

Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger: “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017. Get source online

[3]

Fabian Küppers, Jan Kronenberger, Amirhossein Shantia and Anselm Haselhoff: “Multivariate Confidence Calibration for Object Detection.” The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.

[4]

Fabian Küppers, Jan Kronenberger, Jonas Schneider and Anselm Haselhoff: “Bayesian Confidence Calibration for Epistemic Uncertainty Modelling.” 2021 IEEE Intelligent Vehicles Symposium (IV), 2021

Methods

__init__(*args[, temperature_only])	Create an instance of LogisticCalibration.
clear()	Clear model parameters.
convex(data, y, tensorboard, log_dir)	Convex optimization to find the global optimum of current parameter search.
epsilon(dtype)	Get the smallest digit that is representable depending on the passed dtype (NumPy or PyTorch).
fit(X, y[, random_state, tensorboard, log_dir])	Build logitic calibration model either conventional with single MLE estimate or with Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) algorithm to also obtain uncertainty estimates.
fit_transform(X[, y])	Fit to data, then transform it.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
guide([X, y])	Variational substitution definition for each parameter.
load_model(filename)	Load model from saved torch dump.
mask()	Seek for all relevant weights whose values are negative.
mcmc(data, y, tensorboard, log_dir)	Perform Markov-Chain Monte-Carlo sampling on the (unknown) posterior.
model([X, y])	Definition of the log regression model.
momentum(data, y, tensorboard, log_dir)	Momentum optimization to find the global optimum of current parameter search.
prepare(X)	Preprocessing of input data before called at the beginning of the fit-function.
prior(dtype)	Prior definition of the weights used for log regression.
save_model(filename)	Save model instance as with torch's save function as this is safer for torch tensors.
set_fit_request(*[, log_dir, random_state, ...])	Configure whether metadata should be requested to be passed to the fit method.
set_output(*[, transform])	Set output container.
set_params(**params)	Set the parameters of this estimator.
set_transform_request(*[, mean_estimate, ...])	Configure whether metadata should be requested to be passed to the transform method.
to(device)	Set distribution parameters to the desired device in order to compute either on CPU or GPU.
transform(X[, num_samples, random_state, ...])	After model calibration, this function is used to get calibrated outputs of uncalibrated confidence estimates.
variational(data, y, tensorboard, log_dir)	Perform variational inference using the guide.