Metrics & Scoring Model

A metric is a scoring operation registered for compatible dataset schemas. Reference-based metrics compare predictions with supplied answers; judge-based metrics evaluate responses using their configured judge model.

The generated metric catalogue lists the built-in metrics. Run aip.ops.list_metrics() to see which metrics are available, and aip.ops.get_op(<name>) for a metric's full configuration options.

Scores are normalised to the range [0.0, 1.0]. Many metrics read higher-is-better (including ones like toxicity or bias where the underlying quantity is inverted); where a metric is better when smaller, its generated reference page declares that direction (see How the scorer contract shapes output).

Some metrics are scored by an LLM judge. Their scores depend on the judge model you configure and may vary slightly from run to run; you can set the judge model per metric through the run's metric_configs. Metrics scored by deterministic methods (n-gram overlap, string matching, box overlap) give identical results every run. Where a metric builds on an established open-source method — Hugging Face evaluate, Ragas, or openevals — its entry links to that method.

Choose metrics for the behavior you need to measure. Keep their configuration and dataset version fixed when comparing runs. Read the generated metric catalogue for each metric’s declared inputs, configuration, execution strategy, and direction.