Eval Configs

An eval config is the declarative definition of what an evaluation measures — separate from any single run of it. One config names the dataset under test, the SUT and its connection, the scorers/metrics to compute, the quality gates that decide pass or fail, and a set of queryable tags. Because it is plain YAML, you version-control it alongside your code and replay the exact same evaluation across branches, CI pipelines, and release candidates. Publishing a config to the platform stores it under a stable name and returns a server-assigned UUID that runs can reference later.

The workflow has four parts: discover the metrics your dataset's GDI schema supports, select and confirm the ones you want, assemble a typed EvalConfig (from code or loaded from YAML), and publish it. The steps below follow that order.

Discovering available metrics

Metrics are registered ops of kind metric. List them and filter to those that accept your dataset's GDI schema — for a text dataset that is gdi_text_v1, for images gdi_image_v1. Each metric name is namespaced by family (for example llm. for LLM scorers), and info prints one metric's full contract. For the complete catalogue of metrics and their input/output contracts, see the metrics catalog in Reference.

You can also ask the platform which metrics suit a specific dataset. check_suggestions returns a ranked list keyed off the dataset's latest-version schema, along with warnings (for example a mixed-schema or small-dataset caveat) and the source_family that produced each suggestion. Treat this as advice — it does not select anything; real selection happens next.

SDK

list_ops returns the registered metric ops, which you then filter to the family you need. check_suggestions gives the ranked, dataset-aware suggestions and is advisory only. The source_family on the response is the detected schema (for example "gdi_text_v1"), or None if it could not be detected.

import aip_sdk as aip

catalogue = aip.ops.list_ops(client=aip._context.get_default_client())
metrics = catalogue["metrics"] if isinstance(catalogue, dict) else catalogue
llm_metrics = [m for m in metrics if m["name"].startswith("llm.")]
for m in llm_metrics:
    print(m["name"], m.get("description", ""))

response = aip.check_suggestions(dataset.id)
for warning in response.warnings:
    print(warning)
for s in response.suggestions:
    print(s.rank, s.metric, s.reason)
print(response.source_family)

API

The same ranked suggestions are available over HTTP:

curl -sS "$AIP_API_URL/check-suggestions?dataset_id=<dataset-id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

Selecting and confirming metrics

Selection is just cross-referencing the ranked suggestions against the metrics that are actually registered, and keeping the ones you want. A useful default is to take the registered suggestions plus a few reference-based scorers (llm.bleu, llm.rouge, llm.exact_match). Judge-backed metrics — the LLM-as-judge scorers such as llm.toxicity, llm.correctness, or llm.answer_relevance — are computed server-side on the AIP API host, which already holds the judge credential; the per-metric judge model itself is set at run time in the run config, not here (see Runs & Results).

SDK

The snippet below keeps the suggested metrics that are actually registered, then appends the reference scorers, deduplicating as it goes:

metric_by_name = {m["name"]: m for m in llm_metrics}

suggested = [
    f"llm.{s.metric}" for s in sorted(response.suggestions, key=lambda s: s.rank) if f"llm.{s.metric}" in metric_by_name
]
candidates = suggested + ["llm.bleu", "llm.rouge", "llm.exact_match"]

selected_metrics = []
for metric in candidates:
    if metric in metric_by_name and metric not in selected_metrics:
        selected_metrics.append(metric)

if not selected_metrics:
    raise RuntimeError("No registered metrics available to run.")

Loading a config from YAML

SDK

load_eval_config reads a YAML file and returns a typed EvalConfig object with validated fields, so a malformed file fails fast rather than at run time. Pass the object straight to run() via to_run_kwargs(), or edit a field and write a new version with save_eval_config. Note that YAML comments are not preserved through a load → save round-trip.

import aip_sdk as aip

config = aip.load_eval_config("eval_config_ci.yaml")

with aip.run(**config.to_run_kwargs()) as run:
    ...

updated = config.model_copy(update={"tags": {**config.tags, "branch": "feature/x"}})
aip.save_eval_config(updated, "eval_config_branch.yaml")

SDK

Loading through the SDK surfaces distinct exceptions so you can tell a missing file from a bad schema version from an invalid field:

from aip_sdk import ConfigParseError, ConfigValidationError, ConfigVersionError

try:
    config = aip.load_eval_config("eval.yaml")
except FileNotFoundError:
    print("File not found")
except ConfigParseError as e:
    print(f"Malformed YAML: {e}")
except ConfigVersionError as e:
    print(f"Unsupported version: {e}")
except ConfigValidationError as e:
    print(f"Invalid fields: {e}")

SDK

You can also build an EvalConfig directly in code — for example from the metrics you just selected — without a file on disk:

from aip_core.schemas.eval_config import EvalConfig

cfg = EvalConfig(
    project=str(project.id),
    dataset=f"{dataset.id}@v{version.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
)

Publishing and pulling

SDK

publish_config uploads a config under a name and returns the server-assigned UUID; store that UUID (or the name) to reference the config from runs and the AIP UI. pull_config retrieves a config by either name or UUID and returns a typed EvalConfig; scope a name lookup to a workspace with workspace_id when the same name exists across workspaces.

config = aip.load_eval_config("eval_config_ci.yaml")
config_id = aip.publish_config(
    config,
    name="ci-baseline",
    description="Baseline CI run",
    workspace_id=ws.id,
)

config = aip.pull_config("ci-baseline")
config = aip.pull_config("<config-uuid>")
config = aip.pull_config("ci-baseline", workspace_id="<ws-id>")

API

Publish with a POST whose content must be a valid EvalConfig object; fetch a single config by ID with a GET, which returns both its YAML and parsed content:

curl -sS -X POST "$AIP_API_URL/eval-configs" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ci-baseline",
    "description": "Baseline CI run config",
    "content": {
      "version": "1",
      "project": "my-project",
      "dataset": "golden-v1@v3",
      "sut_id": "sut-abc-123",
      "connection_id": "conn-xyz-456"
    }
  }'

curl -sS "$AIP_API_URL/eval-configs/<config-id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

Listing configs (reference)

To enumerate the configs in a workspace, use the list call — handy for confirming a publish or building a picker. Each record exposes .id, .name, .workspace_id, .description, .content (dict), .content_yaml (str), .created_by, .created_at, and .updated_at. get_eval_config fetches one config's full YAML by ID.

SDK

for record in aip.list_eval_configs(workspace_id=ws.id, per_page=100):
    print(record.id, record.name, record.created_at)

record = aip.get_eval_config("<config-uuid>")
print(record.content_yaml)

API

The same listing is available over HTTP with a GET:

curl -sS "$AIP_API_URL/eval-configs" -H "Authorization: Bearer $AIP_TOKEN"

YAML schema

A minimal config needs only project and dataset; everything else falls back to project defaults.

version: "1"
project: my-project
dataset: golden-v1

A full production config names the SUT, the metrics to compute, the quality gates that decide pass/fail, and queryable tags. In the example below, the dataset is pinned to a version (golden-v1@v3) for reproducible CI runs; the run is flagged failed if any thresholds entry is breached; and tags are queryable key/value labels shown in the AIP UI. Leave metrics out to fall back to the project defaults.

version: "1"

project: my-project
dataset: golden-v1@v3

sut_id: sut-abc-123
connection_id: conn-xyz-456

metrics:
  - accuracy
  - f1_score

thresholds:
  accuracy: 0.85
  f1_score: 0.80

tags:
  env: ci
  branch: main
  model_version: "2.1.0"

Field reference

Field	Type	Required	Description
version	string	yes	Schema version — "1" (flat) or "2" (target-scoped).
project	string	yes	AIP project id (a project name is still accepted).
dataset	string	yes	Dataset name; optionally pin with "name@v3".
sut_id	string	no	UUID of the registered SUT.
connection_id	string	no	UUID of the SUT connection endpoint.
judge_connection_id	string	no	Run-level default judge connection.
metrics	list[string]	no	v1 only. Metric names to require. Defaults to project metrics.
metric_configs	dict[string, dict]	no	v1 only. Per-metric config, keyed by metric name.
targets	list[object]	v2 only	v2 only. Metric selection per evaluation target.
thresholds	dict[string, float]	no	Minimum passing score per metric (0.0–1.0). Keyed by metric name only; see the v2 note below.
tags	dict[string, string]	no	Free-form key/value labels.
parameters	dict[string, any]	no	Reserved; stored but not forwarded to run() yet.

The legacy scorers key is still accepted in place of metrics on a v1 config; send only one of the pair.

Target-scoped configs (schema v2)

A published v2 config runs the same way a v1 one does. aip.run(**config.to_run_kwargs()) works for either version: for a v2 config the kwargs carry pipeline="trace_metric_invoke" and the config's targets, and all of them are scored in ONE run so their rows share a result artifact and stay groupable by partition_type / partition_id. Each target keeps its own metrics / metric_configs rather than being flattened into a run-level union.

Two fields and two shapes do not reach the run. A target's span_kind and the config's connection_id are the fields: span_kind is refused, since scoping to spans is done by a SPAN partition's selector and forwarding it would widen the target to every span; name a SPAN partition_id instead. connection_id is dropped with a warning — a trace run scores already-recorded traces and never invokes a SUT. The two shapes are refused for the same reason a run would reject them, but named at the config level: a target that selects no metrics (a trace run has no project-default selection to fall back on), and a target with no partition_type alongside others (their results would be indistinguishable). All four raise InvalidArgumentError, the same type aip.run(targets=[...]) raises for them.

A flat metrics list says what to run but not where. Trace evaluation is inherently per-(target, metric) — trajectory adherence is judged once per session, sub-agent validity per individual sub-agent span — so schema "2" moves metric selection under a targets list. Each entry names a target by partition_type (the granularity), partition_id (a saved Partition selector), or both, and carries its own metrics / metric_configs in the same shapes v1 uses.

Both versions are supported permanently, and each admits exactly one shape: v1 uses top-level metrics, v2 uses targets. Mixing them is rejected rather than merged.

version: "2"
project: my-project
dataset: agent-traces@v3

targets:
  - partition_type: session
    metrics:
      - agent.step_accuracy
    metric_configs:
      agent.step_accuracy:
        expected_steps: ["search", "summarize", "respond"]

  - partition_type: span
    partition_id: part_abc123   # the selector scopes WHICH spans are scored
    metrics:
      - agent.subagent_validity
    metric_configs:
      agent.subagent_validity:
        min_response_length: 15

thresholds stays top-level and metric-keyed, so it cannot name a target. A v2 config may score one metric at several granularities, but it may not put a threshold on a metric that more than one target could claim — that config is rejected instead of gating an unspecified target. Name the metric in exactly one target, or split the targets across configs. (A target that omits metrics inherits the project defaults, so it counts as a possible claimant for any threshold key.)

resolved_targets() reads either version through one code path — a v1 config yields a single untargeted entry whose partition_type is None.

For the full target-field reference and validation rules, see the eval-config section of docs/platform-reference.md.