Acquisition Batch Metrics

Metrics that evaluate the quality and diversity of the candidate batch acquired in each round. Unlike the regression and classification registries, these metrics operate on Candidate objects or LabelledCandidates rather than prediction arrays.

intra_batch_diversity measures how spread out the acquired batch is in design space, using normalised Levenshtein distance for SEQUENCE candidates and cosine distance for TABULAR candidates (MOLECULE is not yet supported). compute_recall and compute_regret measure the quality of the acquired set relative to the full candidate pool.

Metrics for the acquisition function on the acquired batch.

Covers both within-batch diversity (how spread out the batch is) and pool-relative quality metrics (recall and regret against the full candidate pool).

alf_core.utils.metrics.acquisition_batch.compute_recall(init_candidate_pool, acquired_candidates, top_percentile=0.1, top_n=100)[source]

Compute recall metrics for acquired candidates.

Measures how many of the acquired candidates are in the top performers of the initial candidate pool, using both percentile-based and top-N thresholds.

Parameters:

  • init_candidate_pool (LabelledCandidates) – Initial candidate pool before acquisition.

  • acquired_candidates (LabelledCandidates) – Candidates acquired during optimization, excluding the initial labelled seed.

  • top_percentile (float) – Percentile threshold (e.g., 0.1 for top 10%). Defaults to 0.1.

  • top_n (int) – Number of top candidates to consider. Defaults to 100.

Returns:

  • “optimizer/top_{top_percentile * 100:.0f}pc_recall”: Recall at the top_percentile threshold

  • ”optimizer/top_{top_n}_recall”: Recall at the top_n threshold

Rank counts are clamped to [1, pool size]; the key names always reflect the nominal top_percentile/top_n arguments, so for pools smaller than top_n the “top_{top_n}_recall” value is computed over the whole pool.

Return type:

dict[str, float]

alf_core.utils.metrics.acquisition_batch.compute_regret(init_candidate_pool, acquired_candidates)[source]

Compute regret of acquired candidates relative to the best possible candidate.

Regret is the difference between the best possible label in the initial pool and the best label found in the acquired candidates.

Parameters:

  • init_candidate_pool (LabelledCandidates) – Initial candidate pool before acquisition.

  • acquired_candidates (LabelledCandidates) – Candidates acquired during optimization, excluding the initial labelled seed.

Returns:

  • “optimizer/regret”: The regret value (lower is better).

Return type:

dict[str, float]

alf_core.utils.metrics.acquisition_batch.intra_batch_diversity(candidates)[source]

Compute average pairwise dissimilarity within a batch of candidates.

Measures how spread out the acquired batch is in the design space. Low diversity indicates the acquisition function is proposing near-duplicate candidates, which wastes the oracle budget.

Dissimilarity is computed as follows depending on candidate modality:

  • SEQUENCE: normalised Levenshtein distance over the string representation. dissimilarity = levenshtein(a, b) / max(len(a), len(b)), yielding 0 for identical strings and 1 when every character must be substituted or the strings differ by their full length.

  • MOLECULE: not yet supported (raises NotImplementedError). Edit distance over raw SMILES is not chemically meaningful — two strings denoting the same molecule (e.g. "CCO" and "OCC") would score as dissimilar. Fingerprint-based Tanimoto distance is a planned follow-up.

  • TABULAR: cosine distance computed via scipy.spatial.distance.pdist. Candidates are flattened to 1-D feature vectors before comparison.

The average of all pairwise dissimilarities is returned.

Parameters:

candidates (list[Candidate]) – List of Candidate objects representing the candidates acquired in one round. All candidates must share the same modality.

Return type:

dict[str, float]

Returns:

Dictionary with key intra_batch_diversity mapping to the average pairwise dissimilarity in [0, 1]. Returns an empty dict when fewer than 2 candidates are provided.

Raises:

  • ValueError – If candidates span multiple modalities, if the modality is not one of SEQUENCE, MOLECULE, or TABULAR, or if any candidate has an all-zero feature vector (cosine distance undefined).

  • NotImplementedError – If the modality is MOLECULE (not yet supported).