Scoring

mlipaudit.scoring.compute_metric_score(values: ndarray, threshold: float, alpha: float) → ndarray

Compute the normalized score for an array of values using a soft thresholding function given a max. desired deviation threshold.

Parameters:

  • values – A NumPy array of metric values.

  • threshold – The maximum threshold accepted for each value.

  • alpha – The alpha parameter. Must be a positive float.

Returns:

A NumPy array of normalized scores.

Raises:

ValueError – If alpha is not positive.

mlipaudit.scoring.compute_benchmark_score(errors: list[list[float | None]], thresholds: list[float]) → float

Given a list of metric values and its associated list of acceptable thresholds, compute the benchmark score by taking the average of the normalized scores. This function handles None’s in the errors by assigning a score of 0.

Parameters:

  • errors – The list of metric values. Nones will count for a score of 0.

  • thresholds – The list of acceptable max. thresholds.

Returns:

The benchmark score.