Read scorer output

A metric's scorer_contract decides how it iterates, not where its output lands. Scalar metrics write a score column. Structured dataset metrics write a JSON artifact column (currently matrix and per_class_scores), which run.analysis() exposes under metric_artifacts. The metric runner returns summary = None in both cases.

Contract	Runs…	Consumer-visible output
per_row	once per (prompt, sut_response) pair	one score column (one value per row)
full_dataset	once over the whole accumulated dataset	one scalar or artifact column, with the dataset-level value broadcast to every row

All built-in text metrics (LLM and RAG, gdi_text_v1) are per_row. full_dataset is used by image metrics (gdi_image_v1, e.g. object-detection confusion_matrix). The two never mix within one schema — gdi_text_v1 requires per_row metrics and gdi_image_v1 requires full_dataset metrics.

The output column is named after the bare metric — the family prefix is stripped, so llm.bleu → bleu, rag.faithfulness → faithfulness, and image_classification.confusion_matrix → confusion_matrix. Two metrics that collapse to the same bare name (e.g. llm.toxicity + rag.toxicity) cannot be scored together — scoring raises ValueError rather than let one overwrite the other. Numeric scores are normalised to [0.0, 1.0]; direction controls threshold and colour semantics for lower-is-better metrics such as image_classification.fnr. Artifact columns are excluded from scalar summaries and are returned through run.analysis()["metric_artifacts"].

Per-row — ScoreResult

run.results() melts each row into one ScoreResult per scorer. It returns a RunResultsPage, and each item in page.results is a ScoreResult:

run = aip.get_run("run_abc123")
page = run.results(page=1, page_size=100)

for r in page.results:
    print(r.input_id, r.scorer, r.score, r.explanation)

The loop prints:

row_0001 bleu  0.62 None
row_0001 rouge 0.71 None

ScoreResult fields: input_id: str, scorer: str, score: float, explanation: str | None, config_identity: list[MetricConfigIdentity] (the stored config(s) this scorer resolved; empty if none was pinned). explanation is currently always None — it is a forward-compatible hook for per-row rationale, not yet a populated contract.

For a scalar full_dataset metric such as image_classification.recall, every row carries the same broadcast value:

scores = [r.score for r in page.results if r.scorer == "recall"]
assert len(set(scores)) == 1

The assertion holds because all the values are identical — a broadcast, not a per-row score.

For an artifact metric, use the analysis payload instead of trying to coerce the column into a ScoreResult:

analysis = run.analysis()
matrix = analysis["metric_artifacts"]["confusion_matrix"]
assert matrix["type"] == "matrix"

Aggregated — Metric

RunResultsPage.metrics carries one Metric per scorer — the run's aggregate. Each item in page.metrics is a Metric:

for m in page.metrics:
    print(m.scorer, m.mean, m.std, m.count, m.pass_rate)

The loop prints:

bleu        0.58 0.21 100 0.74
correctness 0.86 0.14 100 0.91

Field	Type	Meaning
scorer	str	Bare metric name (e.g. bleu)
mean	float	Mean score across scored rows
std	float	Standard deviation
count	int	Rows scored (excludes nulls)
pass_rate	float	Fraction at or above the metric threshold (1 − below_threshold / count)

For richer aggregates use run.analysis() (optionally group_by="<dimension>"), which returns per-scorer scorer_stats (count/mean/std/min/p25/p50/p75/p95/below_threshold/direction), a score_distribution histogram, and failure clusters.

Metric ops never return a summary dict — the runner returns summary=None and writes scores to the output Parquet. The summary envelope is populated only by non-metric full-dataset ops (quality checks, analysis). For any metric, read from the score column, never from summary.

Worked example — text generation

A gdi_text_v1 dataset with prompt and expected_output, scored with two per_row metrics:

with aip.run(project="support", dataset="support_qa@v1", metrics=["llm.bleu", "llm.rouge"]) as run:
    df = run.dataset.pull()
    df["sut_response"] = my_model(df["prompt"])
    run.upload(aip_metrics.score(df, scorers=run.required_scorers))
    run.poll_status()
    page = run.results()

The per-row results (page.results) hold one ScoreResult per (row, metric):

ScoreResult(input_id="row_0001", scorer="bleu",  score=0.62, explanation=None)
ScoreResult(input_id="row_0001", scorer="rouge", score=0.71, explanation=None)

The aggregated results (page.metrics) hold one Metric per scorer:

Metric(scorer="bleu",  mean=0.58, std=0.21, count=100, pass_rate=0.74)
Metric(scorer="rouge", mean=0.69, std=0.18, count=100, pass_rate=0.82)