Design Task Metrics

Summary metrics for design experiments, not registered in regression_metric_registry.

auc_top_k: normalised area under the top-k mean curve — primary sample-efficiency ranking metric that accepts per-round aggregated values rather than per-candidate prediction arrays.

Standalone metrics not registered in regression_metric_registry.

Contains multi-round summary metrics (auc_top_k) that operate on per-round aggregates rather than per-candidate prediction arrays, and so are not registered in the registry.

alf_core.utils.metrics.aggregate.auc_top_k(round_values, best_value)

Compute the normalised area under the top-k mean curve.

Integrates the per-round curve using the trapezoidal rule, then divides by best_value so the AUC lands in [0, 1]. The value 1.0 is only reachable when the curve itself is bounded by best_value and saturates at it from the first round — true for a max-based curve (e.g. classification accuracy with best_value=1.0), but not for a top-k mean curve normalised by the global max, where the mean of the top k can equal the max only if every top-k label equals the global optimum. For that reason the AUC is best read as a relative sample-efficiency ranking within a fixed dataset and round budget, not as an absolute “fraction of optimal”. Lower values indicate that high-performing candidates were found later in the experiment.

Parameters:

• round_values (Float[ndarray, 'n_rounds']) – Array of shape (n_rounds,). Per-round top-k mean, where entry i is the top-k mean of all candidates acquired by round i (inclusive).

• best_value (float) – The global best oracle label in the dataset. Must be strictly positive. Used to normalise the AUC to [0, 1]. If any entry in round_values exceeds best_value, a warning is issued and the result is clamped to 1.0.

Return type:

dict[str, float]

Returns:

Dictionary with key auc_top_k mapping to the normalised AUC in [0, 1].

Raises:

ValueError – If round_values has fewer than 2 entries or best_value is not strictly positive (non-zero).

alf_core.utils.metrics.aggregate.compute_aggregate_metrics(round_values, best_value)

Run every aggregate metric over a per-round curve and merge the results.

Single entry point for end-of-experiment summary metrics: new aggregate metrics only need to be added to _AGGREGATE_METRICS here, without touching the calling task. Metrics whose requirements are not met (e.g. fewer than two rounds) raise ValueError internally and are skipped, so the caller does not need to guard against partial failures.

Parameters:

• round_values (Float[ndarray, 'n_rounds']) – Array of shape (n_rounds,) with one aggregated value per round.

• best_value (float) – The global best oracle label in the dataset, used for normalisation.

Return type:

dict[str, float]

Returns:

Merged dictionary of all aggregate metrics that could be computed. Empty when none could be computed.

alf_core.utils.metrics.aggregate.compute_experiment_summary(state)

Run all aggregate metrics over a task’s per-round sample-efficiency curve.

State-based entry point for alf_core.tasks: builds the per-round curve and normaliser from the final task state and delegates to compute_aggregate_metrics, so the calling task stays free of metric logic and new metrics only need adding to _AGGREGATE_METRICS.

Parameters:

state (State) – Final task state after all acquisition rounds.

Return type:

dict[str, float]

Returns:

Merged dictionary of all aggregate metrics that could be computed. Empty when none could be computed.