netcal.scaling.TemperatureScaling

class netcal.scaling.TemperatureScaling(*args, **kwargs)

On classification or detection, apply the temperature scaling method to obtain a calibration mapping. This method has been proposed by [1]. For confidence calibration in classification tasks, a confidence mapping \(g\) is applied on top of a miscalibrated scoring classifier \(\hat{p} = h(x)\) to deliver a calibrated confidence score \(\hat{q} = g(h(x))\).

For detection calibration, we can also use the additional box regression output which we denote as \(\hat{r} \in [0, 1]^J\) with \(J\) as the number of dimensions used for the box encoding (e.g. \(J=4\) for x position, y position, width and height). Therefore, the calibration map is not only a function of the confidence score, but also of \(\hat{r}\). To define a general calibration map, we use the the combined input \(s = (\hat{p}, \hat{r})\) of size K and perform a temperature scaling defined by

\[\hat{q} = \sigma(s / T)\]

with the temperature \(T \in \mathbb{R}\) as a single scalar value. The function \(\sigma(*)\) is either the sigmoid (on detection or binary classification) or the softmax operator (multiclass classification).

We utilize standard optimization methods to determine the calibration mapping \(g(s)\). Capturing epistemic uncertainty of the calibration method is also able with this implementation [2].

Parameters:

• method (str, default: "mle") – Method that is used to obtain a calibration mapping: - ‘mle’: Maximum likelihood estimate without uncertainty using a convex optimizer. - ‘momentum’: MLE estimate using Momentum optimizer for non-convex optimization. - ‘variational’: Variational Inference with uncertainty. - ‘mcmc’: Markov-Chain Monte-Carlo sampling with uncertainty.

• momentum_epochs (int, optional, default: 1000) – Number of epochs used by momentum optimizer.

• mcmc_steps (int, optional, default: 20) – Number of weight samples obtained by MCMC sampling.

• mcmc_chains (int, optional, default: 1) – Number of Markov-chains used in parallel for MCMC sampling (this will result in mcmc_steps * mcmc_chains samples).

• mcmc_warmup_steps (int, optional, default: 100) – Warmup steps used for MCMC sampling.

• vi_epochs (int, optional, default: 1000) – Number of epochs used for ELBO optimization.

• detection (bool, default: False) – If False, the input array ‘X’ is treated as multi-class confidence input (softmax) with shape (n_samples, [n_classes]). If True, the input array ‘X’ is treated as a box predictions with several box features (at least box confidence must be present) with shape (n_samples, [n_box_features]).

• independent_probabilities (bool, optional, default: False) – Boolean for multi class probabilities. If set to True, the probability estimates for each class are treated as independent of each other (sigmoid).

• use_cuda (str or bool, optional, default: False) – Specify if CUDA should be used. If str, you can also specify the device number like ‘cuda:0’, etc.

References

[1]

Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger: “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017. Get source online

[2]

Fabian Küppers, Jan Kronenberger, Jonas Schneider and Anselm Haselhoff: “Bayesian Confidence Calibration for Epistemic Uncertainty Modelling.” 2021 IEEE Intelligent Vehicles Symposium (IV), 2021

Methods

__init__(*args, **kwargs)	Create an instance of TemperatureScaling.
clear()	Clear model parameters.
convex(data, y, tensorboard, log_dir)	Convex optimization to find the global optimum of current parameter search.
epsilon(dtype)	Get the smallest digit that is representable depending on the passed dtype (NumPy or PyTorch).
fit(X, y[, random_state, tensorboard, log_dir])	Build logitic calibration model either conventional with single MLE estimate or with Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) algorithm to also obtain uncertainty estimates.
fit_transform(X[, y])	Fit to data, then transform it.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
guide([X, y])	Variational substitution definition for each parameter.
load_model(filename)	Load model from saved torch dump.
mask()	Seek for all relevant weights whose values are negative.
mcmc(data, y, tensorboard, log_dir)	Perform Markov-Chain Monte-Carlo sampling on the (unknown) posterior.
model([X, y])	Definition of the log regression model.
momentum(data, y, tensorboard, log_dir)	Momentum optimization to find the global optimum of current parameter search.
prepare(X)	Preprocessing of input data before called at the beginning of the fit-function.
prior(dtype)	Prior definition of the weights used for log regression.
save_model(filename)	Save model instance as with torch's save function as this is safer for torch tensors.
set_fit_request(*[, log_dir, random_state, ...])	Request metadata passed to the fit method.
set_output(*[, transform])	Set output container.
set_params(**params)	Set the parameters of this estimator.
set_transform_request(*[, mean_estimate, ...])	Request metadata passed to the transform method.
to(device)	Set distribution parameters to the desired device in order to compute either on CPU or GPU.
transform(X[, num_samples, random_state, ...])	After model calibration, this function is used to get calibrated outputs of uncalibrated confidence estimates.
variational(data, y, tensorboard, log_dir)	Perform variational inference using the guide.