Regression Metrics

Regression and uncertainty-calibration metrics for evaluating surrogate model predictions. All functions are registered in regression_metric_registry at import time via @register_requires_variance (for metrics that need uncertainty estimates) or @register_no_variance_required (for point-prediction metrics).

Accuracy metrics (no variance required): MSE, Spearman, Pearson, pairwise cross-entropy, top-k mean, top-k max, and hit rate.

UQ metrics (variance required): coverage, rank coverage, interval width, rank width, residual Spearman, residual Pearson, NLL (Gaussian), UCB regret, and UCB regret sweep. Calibration error metrics live in the Calibration module.

Regression metrics and supporting infrastructure.

All metrics are automatically registered in regression_metric_registry at import time via the @register_requires_variance and @register_no_variance_required decorators.

alf_core.utils.metrics.regression.coverage(means, variances, targets, alpha=0.95)

Compute coverage at alpha% confidence level.

Computes alpha% confidence intervals for all predictions and measures what percentage of targets fall within these intervals. Ideal coverage should be close to alpha.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float) – Confidence level (e.g., 0.95 for 95% CI). Defaults to 0.95.

Returns:

.2f}” mapping to the coverage percentage.

Return type:

dict[str, float]

Raises:

AssertionError – If alpha is not in [0, 1].

alf_core.utils.metrics.regression.hit_rate(_means, _variances, targets, threshold=0.5)

Compute the fraction of acquired candidates whose label exceeds a threshold.

Counts how many oracle-labelled candidates are considered “hits” (i.e. their label is at or above threshold) and returns this as a fraction of the total. Suitable for discovery-framing tasks such as drug screening or protein fitness optimisation, where “active” or “fit” is a binary concept derived from a continuous score.

Parameters:

• _means (Float[ndarray, 'b']) – Array of shape (b,). Unused (for API consistency with registry).

• _variances (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Unused.

• targets (Float[ndarray, 'b']) – Array of shape (b,). Oracle labels for acquired candidates.

• threshold (float) – Minimum label value to count as a hit. Defaults to 0.5.

Returns:

.3f}` mapping to the hit rate in [0, 1].

Return type:

dict[str, float]

alf_core.utils.metrics.regression.monte_carlo_ranking(means, variances, num_samples=10000, seed=42)

Compute ranks, their means and variances using Monte Carlo simulation.

For predicted means and variances, estimates normally distributed means using Monte Carlo simulation and computes ranking statistics.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• num_samples (int) – Number of random samples to draw for the simulation. Defaults to 10000.

• seed (int) – Random seed for reproducibility. Defaults to 42.

Returns:

• mean_rank: Mean rank for each candidate

• rank_variances: Variance of ranks for each candidate

Return type:

tuple[Float[ndarray, 'b'], Float[ndarray, 'b']]

alf_core.utils.metrics.regression.mse(means, _, targets)

Compute the mean squared error.

For each sample compute the squared euclidean distance between the true label and the mean prediction. Compute the mean over these distances.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions

• _ (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Predicted variances

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels

Returns:

MSE float}

Return type:

dict[str, float]

alf_core.utils.metrics.regression.nll_gaussian(means, variances, targets)

Compute the mean negative log-likelihood under a Gaussian predictive distribution.

Evaluates the per-sample NLL assuming the model produces independent Gaussian predictions N(μ_i, σ²_i) for each candidate i:

NLL = 0.5 * mean(log(2π) + log(σ²_i) + (y_i - μ_i)² / σ²_i)

Lower values indicate that the model places high probability mass on the true targets. Use together with expected_calibration_error to characterise both sharpness and calibration of the surrogate’s uncertainty estimates. Variances are clipped to a small positive value before computing the logarithm to guard against numerical instability.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances. Must be non-negative (enforced by the registry decorator).

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

Return type:

dict[str, float]

Returns:

Dictionary with key nll mapping to the mean NLL value.

alf_core.utils.metrics.regression.pairwise_xent(means, _, targets)

Compute the ranking loss for a pairwise classification problem.

For each pair of items in the batch, predict which item has the higher target value. Derive logits from pairs of predictions and treat them as logits of a binary classifier.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions

• _ (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Predicted variances

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels

Returns:

Ranking Loss float}

Return type:

dict[str, float]

alf_core.utils.metrics.regression.pearson(means, _, targets)

Compute the pearson correlation.

This is a statistical test which measures the strength and direction of the linear relationship between variables. We compute the correlation coefficient between the mean predictions and the target. This takes a value between [-1,1]

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions

• _ (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Predicted variances

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels

Returns:

Pearson correlation float}

Return type:

dict[str, float]

alf_core.utils.metrics.regression.rank_coverage(means, variances, targets, alpha=0.95)

Compute coverage at alpha% confidence level in rank space.

Computes coverage metric using Monte Carlo ranking to estimate rank distributions, then applies the standard coverage computation in rank space.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float) – Confidence level (e.g., 0.95 for 95% CI). Defaults to 0.95.

Returns:

.2f}” mapping to the coverage percentage in rank space.

Return type:

dict[str, float]

Raises:

AssertionError – If alpha is not in [0, 1].

alf_core.utils.metrics.regression.rank_width(means, variances, targets, alpha=0.95)

Compute average confidence interval width in rank space.

Computes width metric using Monte Carlo ranking to estimate rank distributions, then applies the standard width computation in rank space.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float) – Confidence level (e.g., 0.95 for 95% CI). Defaults to 0.95.

Returns:

.2f}” mapping to the normalised average width value in rank space.

Return type:

dict[str, float]

Raises:

AssertionError – If alpha is not in [0, 1].

alf_core.utils.metrics.regression.register_no_variance_required(metric_fn)

Decorator to mark a metric as not requiring variance.

Automatically registers the metric in the global registry and applies input validation. The decorated function will receive validated inputs without variance checks.

Parameters:

metric_fn (Callable) – The metric function to decorate.

Return type:

Callable

Returns:

Wrapped metric function with validation and registration.

alf_core.utils.metrics.regression.register_requires_variance(metric_fn)

Decorator to mark a metric as requiring variance.

Automatically registers the metric in the global registry and applies input validation. The decorated function will receive validated inputs with variance checks.

Parameters:

metric_fn (Callable) – The metric function to decorate.

Return type:

Callable

Returns:

Wrapped metric function with validation and registration.

alf_core.utils.metrics.regression.regret_ucb_alpha(means, variances, targets, alpha=0.1, num_acquisitions=100)

Compute UCB (Upper Confidence Bound) acquisition regret.

Computes UCB values (means + alpha * sqrt(variances)) and compares the sum of labels from the top num_acquisitions candidates selected by UCB versus the best possible sum. Lower regret indicates better acquisition performance.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float) – UCB exploration parameter. Defaults to 0.1.

• num_acquisitions (int) – Number of candidates to acquire. Defaults to 100.

Returns:

.2f}” mapping to the cumulative regret value.

Return type:

dict[str, float]

Raises:

AssertionError – If num_acquisitions is not a positive integer.

alf_core.utils.metrics.regression.regret_ucb_alpha_sweep(means, variances, targets, alpha=None, num_acquisitions=100)

Compute UCB regret for multiple alpha values.

Computes UCB regret for each alpha value in the provided list (or default range). Useful for evaluating acquisition function performance across different exploration-exploitation trade-offs.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float | list[float] | None) – Single float, list of floats, or None. UCB exploration parameter(s). If None, uses default list [0.1, 0.3, 0.5, 1.0]. Defaults to None.

• num_acquisitions (int) – Number of candidates to acquire. Defaults to 100.

Returns:

.2f}” to the regret value for each alpha value. The “sweep” namespace avoids colliding with the standalone regret_ucb_alpha metric, which emits “regret_ucb_{alpha:.2f}”.

Return type:

dict[str, float]

Raises:

• ValueError – If alpha is an empty list.

• TypeError – If alpha is not a float or list of floats.

alf_core.utils.metrics.regression.residual_pearson(means, variances, targets)

Compute Pearson correlation between residuals and standard deviations.

Computes absolute residuals (|targets - means|) and measures their Pearson correlation with predicted standard deviations (sqrt(variances)). Higher correlation indicates better uncertainty estimation.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

Return type:

dict[str, float]

Returns:

Dictionary with key “residual_pearson” mapping to the correlation coefficient.

alf_core.utils.metrics.regression.residual_spearman(means, variances, targets)

Compute Spearman correlation between residuals and variances.

Computes absolute residuals (|targets - means|) and measures their Spearman correlation with predicted variances. Higher correlation indicates better uncertainty estimation.

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions.

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

Return type:

dict[str, float]

Returns:

Dictionary with key “residual_spearman” mapping to the correlation coefficient.

alf_core.utils.metrics.regression.spearman(means, _, targets)

Compute the spearman correlation.

This is a non-parametric statistical test which measures the strength of the monotonic relationship between two variables. We rank the mean predictions and compute the correlation coefficient between these ranks and the true ranks. This takes a value between [-1,1]

Parameters:

• means (Float[ndarray, 'b']) – Array of shape (b,). Mean predictions

• _ (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Predicted variances

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels

Returns:

Spearman correlation float}

Return type:

dict[str, float]

alf_core.utils.metrics.regression.top_k_max(_means, _variances, targets, k=10)

Compute the maximum label of the top-k oracle-labelled candidates.

Selects the k highest target values and returns the maximum. When k exceeds the number of available targets the function falls back to all available targets. Use alongside top_k_mean to distinguish between experiments that find one very good candidate versus many good ones.

Parameters:

• _means (Float[ndarray, 'b']) – Array of shape (b,). Unused (for API consistency with registry).

• _variances (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Unused.

• targets (Float[ndarray, 'b']) – Array of shape (b,). Oracle labels for acquired candidates.

• k (int) – Number of top candidates to consider. Defaults to 10.

Return type:

dict[str, float]

Returns:

Dictionary with key top_{k_eff}_max mapping to the maximum value of the top-k targets.

alf_core.utils.metrics.regression.top_k_mean(_means, _variances, targets, k=10)

Compute the mean label of the top-k oracle-labelled candidates.

Selects the k highest target values and returns their mean. When k exceeds the number of available targets the function falls back to all available targets. Use this as the primary optimisation metric to track how quickly an active-learning experiment surfaces high-performing candidates across rounds.

Parameters:

• _means (Float[ndarray, 'b']) – Array of shape (b,). Unused (for API consistency with registry).

• _variances (Float[ndarray, 'b'] | None) – Array of shape (b,) or None. Unused.

• targets (Float[ndarray, 'b']) – Array of shape (b,). Oracle labels for acquired candidates.

• k (int) – Number of top candidates to consider. Defaults to 10.

Return type:

dict[str, float]

Returns:

Dictionary with key top_{k_eff}_mean mapping to the mean value of the top-k targets.

alf_core.utils.metrics.regression.width(_, variances, targets, alpha=0.95)

Compute average confidence interval width normalised by dataset range.

Computes alpha% confidence intervals for all predictions, then calculates the average width normalised by the maximum distance between any two targets. Lower values are better while maintaining good calibration.

Parameters:

• _ (Float[ndarray, 'b']) – Array of shape (b,). Unused parameter (for API consistency).

• variances (Float[ndarray, 'b']) – Array of shape (b,). Predicted variances.

• targets (Float[ndarray, 'b']) – Array of shape (b,). True labels.

• alpha (float) – Confidence level (e.g., 0.95 for 95% CI). Defaults to 0.95.

Returns:

.2f}” mapping to the normalised average width value.

Return type:

dict[str, float]

Raises:

AssertionError – If alpha is not in [0, 1].