Metrics & Scorers

Metrics are the scoring ops AIP runs against your dataset. They live in the Op Registry alongside schemas, transforms, checks, and analysis ops; a metric is an op of kind == "metric". This section covers how to discover them, how to tell which ones fit your dataset's schema, the built-in families, the output shape each metric produces, and the validation errors you get when a metric doesn't fit.

Discovering metrics

The catalogue is the source of truth — don't hard-code metric lists. Query it from the SDK.

SDK — list_metrics / list_ops / get_op

import aip_sdk as aip

metrics = aip.ops.list_metrics()
text_metrics = aip.ops.list_metrics(schema="gdi_text_v1")

catalogue = aip.ops.list_ops()
for op in catalogue["scorers"]:
    print(f"{op['name']}: {op.get('description', '')}")

op = aip.ops.get_op("llm.bleu")
print(op["category"])
print(op["config_schema"])

list_metrics() is a convenience over list_ops() that returns all metric ops and takes an optional schema filter. list_ops() returns the full op catalogue grouped by kind as an OpCatalogue (a dict). get_op(name) returns the config schema and metadata for one op — for llm.bleu, op["category"] is "scorers" and op["config_schema"] is the JSON Schema dict for the op config ({} means no config is needed).

list_ops() returns an OpCatalogue keyed by kind:

Key	Contents
"schemas"	GDI input schema descriptors
"transforms"	DataFrame preprocessing ops
"checks"	Quality-check ops
"analysis"	Dataset diagnostics / profiles / histograms
"scorers"	Metric ops — pass these names to metrics= on a run

Each list item is an OpEntry:

Field	Type	Description
name	str	Op name, e.g. "llm.bleu"
description	str	Human-readable summary
config_schema	dict	JSON Schema for op-level config; {} = no config needed
execution_strategy	str | None	"per_row" or "full_dataset" for metrics; None for non-metric ops
direction	str | None	"higher_is_better" or "lower_is_better" — the metric's intrinsic semantic (None for non-metric ops)

Registry entries carry source (builtin | upload | custom) and workspace_id (None for globals, set for a workspace's custom ops), so you can scope discovery to your workspace. They also carry deployment_name — None for every op in normal use; only set when an op's dispatch has been redirected to a different deployed function (e.g. a PR preview environment overriding one changed op to its own instance while every other op still resolves to the shared fleet).

custom_metrics = aip.ops.list_metrics(source="custom", workspace_id=ws.id)

Filtering to metrics that accept your schema — the accepts field

Every metric declares the GDI schemas it can process in its accepts list. To find which metrics are valid for your dataset, filter on it. list_ops() groups by kind but does not include accepts — use GET /ops/registry when you need schema-level filtering:

registry = client.get("/ops/registry")["ops"]

text_scorers = [
    op["name"]
    for op in registry
    if op.get("kind") == "metric" and any(ref["name"] == "gdi_text_v1" for ref in (op.get("accepts") or []))
]

GET /ops/registry returns {"ops": [...OpRegistryEntry...], "unparseable": [...]}. The text_scorers list built above yields, for example, ["llm.bleu", "llm.correctness", "rag.hit_at_k", ...].

One metric's config schema

aip.ops.get_op(<name>) returns the metric's full contract, including its config_schema (a JSON Schema dict; {} means no config is required). Config values are passed at run time under a metric's test_params — for example {"object_detection.recall": {"iou_threshold": 0.5}} or {"rag.hit_at_k": {"test_params": {"metric_params": {"k": 5}}}}.

Built-in metric families

The generated metric catalogue lists the built-in metrics from their op manifests, including inputs, configuration, execution strategy, and score direction. Query aip.ops.list_metrics() to see what is available in your workspace.

Read scorer output for row scores, aggregate metrics, and dataset artifacts. For metrics that generate new prompts before scoring, follow generation-dependent metrics.

Run-launch validation errors

POST /runs validates every requested metric against the dataset before the run is created. All errors across all metrics are collected and returned together as HTTP 422; the Python SDK raises aip_sdk.exceptions.UnprocessableEntityError.

{
  "detail": "Run configuration has 2 metric incompatibility error(s)",
  "validation_errors": [
    {
      "error_code": "OpSchemaIncompatible",
      "op_name": "object_detection.specificity",
      "detail": "metric 'object_detection.specificity' accepts ['gdi_image_v1'] but dataset schema is 'gdi_text_v1'"
    }
  ]
}

The SDK folds every reported failure into the exception message, so printing the exception is enough to see which metric failed and why:

Run configuration has 2 metric incompatibility error(s):
  - object_detection.specificity: metric 'object_detection.specificity' accepts ['gdi_image_v1'] but dataset schema is 'gdi_text_v1'

Read exc.validation_errors for the same failures as structured entries — the array shown above — when you want to branch on error_code rather than display the message.

error_code	Cause	Fix
OpNotFound	Metric name not in the Op Registry	Run aip.ops.list_metrics() — check spelling and package prefix (llm.bleu, not bleu)
OpSchemaIncompatible	Dataset GDI schema not in the op's accepts list	Text metrics accept gdi_text_v1; CV metrics accept gdi_image_v1. Check GET /ops/registry/{name}
OpMissingColumns	One or more required_columns absent from the dataset version	Re-map the dataset version with the missing columns; check required_columns via GET /ops/registry/{name}
ScorerContractMismatch	Op scorer_contract doesn't match the contract implied by the dataset schema	gdi_text_v1 needs per_row metrics; gdi_image_v1 needs full_dataset metrics

Commonly missing columns by metric type:

Metric type	Typically missing
LLM answered (llm.correctness, llm.bleu, …)	expected_output
RAG contextualized (rag.hit_at_k, …)	retrieved_context, reference_contexts
Object Detection	predictions

Storing, pinning, and comparing metric configs

The sections above cover discovering a metric and its config_schema. This section covers what comes after: storing a tuned config, pinning it into a run, and telling whether two runs are comparable. It applies to agent-trace metrics, whose config_schema is the only authoritative source for what a given metric's MetricConfig accepts — don't guess param names from a docstring or another metric's config, since not every metric supports every setting and a typo'd param can silently do nothing.

Per-run discovery workflow

1. List the deployed trace metrics — aip.ops.list_metrics(kind="trace_metric") returns one OpEntry per metric. Don't add a schema= filter here: every trace metric's accepts is canonical_partition_v1 (the partition contract), not your project's ingest schema.

2. Read config_schema on the entry that interests you.

3. List existing configs to reuse or create a new one — aip.list_metric_configs(metric_name="agent.hallucination").

4. Run with a metric_config_refs pin (see below).

5. Post-run: inspect the run's effective config, per row. Run.results()'s ScoreResult/TraceScoreResult rows carry config_identity, the stored config(s) that scorer actually resolved:

   page = run.results()
   for result in page.results:
       for identity in result.config_identity:
           print(result.scorer, "->", identity.config_name, identity.config_version)

   config_identity is empty for a scorer that pinned no stored config, or a platform predating this field — never an error to read, just possibly empty.

aip.discover_metric_config_status(workspace_id=...) reports, per metric, the workspace default (with usage), every named override (with usage), and whether the metric can run at all without an explicit config. is_required is True only when a metric has neither a published default nor any named override — rare, since every agentic metric's default is auto-seeded at workspace creation.

Per-metric config examples

Parameter names below are taken directly from each metric's config_schema — re-check aip.ops.list_metrics() before relying on this, since a metric's schema can gain fields over time.

agent.hallucination (judge-backed) has no bare threshold field. The tunable knobs include judge_temperature (default 0.0, deterministic; raising it lets judge sampling variance affect the verdict), model (judge model override), and max_response_length / max_tool_calls_length / max_total_context — score-defining truncation limits on evidence shown to the judge, so runs using different values are not comparable:

strict = aip.create_metric_config(
    "hallucination_strict", metric_ref, params={"judge_temperature": 0.0}
)

agent.subagent_validity (rule-based) has one knob: min_response_length (default 10) — responses shorter than this (chars) are flagged short. Raise it where valid answers are short (a coordinate, an ID). It's deterministic, so it's the easiest metric to A/B test different thresholds on the same data:

lenient = aip.create_metric_config(
    "subagent_lenient", metric_ref, params={"min_response_length": 5}
)

Config storage best practices

• Name semantically: hallucination_strict, subagent_lenient, tool_selection_baseline — not config_1.

• Version deliberately: bump config_version (e.g. "2.0.0") for a tuning iteration rather than editing in place once a run has scored with it — a config is frozen the moment a run pins it (MetricConfigReferencedByRunsError on .update()), by design: a (config_name, config_version) pair must never change meaning after creation, or "which config scored this run" stops meaning anything.

• .delete() is permanent: there is no soft-delete tier or restore, and it frees the (name, version) key for reuse. It is refused (MetricConfigReferencedByRunsError) while any run still references the config — check usage_count on the summary/detail object beforehand if you want to know without attempting the delete.

• Track config→run provenance via aip.iter_runs(metric=..., metric_config_name=...) — auto-paginates every run pinning a config version, useful before retiring one.

• Compare configs across a diff, not just identities, via config_identity_match(include_params=True) — fetches each mismatched identity's params and reports param_diff:

  for identity, entry in run_diff.config_identity_match(include_params=True).items():
      if entry.param_diff:
          print(f"{entry.metric}: {entry.param_diff}")

Config vs inline trade-offs

	Stored MetricConfig	Inline metric_configs
Reusable across runs	Yes	No — one run only
Versioned / auditable	Yes (config_name, config_version)	No
Comparable via aip.diff().config_identity_match()	Yes	N/A — not a named identity
Best for	Tuning you want to repeat or compare	One-off experiments, quick validation

Precedence when more than one is present, highest first: an explicit run-level op_config value; then the target's own metric_configs[metric] entry; then a stored config pinned via metric_config_refs[metric]. A pinned MetricConfigRef is opt-in per metric — a run that never sets metric_config_refs never touches the MetricConfig store at all.

metric_config_refs is also a field on EvalConfig (the object aip.load_eval_config() returns), so a pin can live in a version-controlled YAML file rather than only as a direct aip.run() keyword argument:

project: proj-abc123
dataset: agent-traces
version: "2"
targets:
  - partition_type: trace
    metrics: [agent.hallucination]
metric_config_refs:
  agent.hallucination: {config_name: hallucination_strict, config_version: "1.0.0"}

It stays top-level for both the v1 (flat) and v2/v3 (targeted) shapes, unlike metrics/metric_configs which move under targets[] for a targeted config — which stored config a metric pins is a property of the metric, not of any one target it runs under. EvalConfig.to_run_kwargs() forwards it unchanged into aip.run(metric_config_refs=...).