Calibration Metrics

Uncertainty-calibration metrics for surrogate model predictions. expected_calibration_error and rank_expected_calibration_error are registered in regression_metric_registry via @register_requires_variance.

Uncertainty calibration metrics for probabilistic regression predictions.

Contains the registered calibration metrics (expected_calibration_error, rank_expected_calibration_error).

alf_core.utils.metrics.calibration.expected_calibration_error(means, variances, targets, n_grid_points=100)

Compute Expected Calibration Error (ECE).

For each confidence level alpha in a grid from 0 to 1: - Find alpha% confidence intervals for all predictions - Count percentage of targets that fall within the confidence intervals - ECE = area between x=y line and the observed coverage curve

Lower ECE values indicate better calibration.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• n_grid_points (int) – Number of grid points for confidence level discretization. Defaults to 100.

Return type:

dict[str, float]

Returns:

Dictionary with key “ece” mapping to the ECE value.

alf_core.utils.metrics.calibration.rank_expected_calibration_error(means, variances, targets)

Compute Expected Calibration Error (ECE) in rank space.

Computes ECE using Monte Carlo ranking to estimate rank distributions, then applies the standard ECE computation in rank space.

Lower ECE values indicate better calibration.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

Return type:

dict[str, float]

Returns:

Dictionary with key “rank_ece” mapping to the ECE value.