Runs

An evaluation run takes a dataset version, produces system-under-test (SUT) outputs, scores them against the metrics you select, and stores per-row results plus aggregated metrics you can retrieve, export, and diff. AIP offers two ways to produce the SUT outputs.

Two run modes

AIP produces the SUT outputs in one of two runner modes. The mode is a property of how you create the run, not a separate object. When you leave runner unset, aip.run() infers the mode: passing connection_id= (or a sut= HostedSUT) selects the hosted runner, otherwise it defaults to external. Setting runner="hosted" or runner="external" explicitly always wins — and pairing runner="external" with connection_id=/sut= raises ValueError rather than silently switching to hosted, so an external run never triggers platform-side SUT invocation you didn't ask for. sut_id= is attribution-only: it records which registered SUT a run is associated with and never changes the runner mode.

Mode	Who runs inference	Use when
External (default)	You do, on your own infrastructure	Your SUT can't be reached as a registered connection, or you'd rather not expose it.
Hosted	AIP calls your registered SUT connection	Your SUT is reachable as a registered connection (OpenAI-compatible chat endpoint, RAG API, CV detector, …).

External runner

You own inference, not scoring — scoring itself always happens server-side. AIP hands you the golden data; you generate predictions locally, then hand the SUT-filled rows back to AIP as a new dataset. project.upload_dataset() uploads them, dataset.promote() promotes the version once it clears quality checks, and aip.run(pipeline="metric_invoke", dataset=..., metrics=[...]) creates a run that scores that golden version — the platform dispatches your chosen metrics to its own scoring infrastructure, so no local aip_metrics.score() call is involved. If you've already produced an external run's raw transcript instead of a fresh dataset, metric_invoke also accepts source_run_id in place of dataset — see Score an existing run server-side.

In the example below, dataset.latest_version() plus aip.DatasetHandle(...).pull() download the existing golden data, and my_sut() fills in sut_response locally. project.upload_dataset() uploads the filled rows as a new dataset; map_version(), run_checks(), and promote() clear that version for use (a WARN verdict can be force-promoted with a reason, FAIL/STALE/NOT_RUN cannot — see the promote_and_score walkthrough under Generation-dependent metrics for the full decision tree). The with aip.run(pipeline="metric_invoke", ...) block then scores the promoted version: run.poll_status() waits for completion and run.results() returns a RunResultsPage. run.url is the web UI link for the run, and run.id is the identifier to keep if you want to re-open the run later.

SDK

import aip_sdk as aip

version = dataset.latest_version()
golden_df = aip.DatasetHandle(dataset.id, version.version, version.id, aip._context.get_default_client()).pull()
golden_df["sut_response"] = my_sut(golden_df["prompt"])

ds = project.upload_dataset(golden_df, name="my-sut-results")
version = ds.latest_version()
version = ds.map_version(version.id)  # columns already match gdi_text_v1, no renaming needed
report = ds.run_checks(version.id, label="external-mode-scored")
version = ds.promote(version.id)  # or force=True + reason after reviewing a WARN

with aip.run(
    project=project.id,
    pipeline="metric_invoke",
    dataset=f"{ds.id}@v{version.version}",
    metrics=["llm.bleu", "llm.rouge"],
) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results(page=1, page_size=100)
    print(run.url)
    run_id = run.id

A note on generation-dependent metrics. Some metrics generate the prompts themselves — adversarial, reworded, or counterfactual variants of a seed question — before scoring, rather than just scoring the prompts you hand them:

• llm.data_leakage
• llm.decision_flip
• llm.factual_consistency
• llm.group_interaction_bias
• llm.instruction_following
• llm.ood_detection
• llm.perturbation_robustness
• llm.safety_consistency
• llm.semantic_consistency
• llm.toxicity_jailbreak
• llm.toxicity_robustness
• rag.content_bias.

All of them already work in hosted mode today, for every client tier including Trials — it's only here, in external mode, that support is partial: llm.toxicity_jailbreak and llm.perturbation_robustness are ready, and we're actively working on the rest for Trials customers. See Generation-dependent metrics in the metrics reference for how to use the two that are ready.

Hosted runner

AIP calls your registered SUT connection for you: it pulls the golden data, invokes your endpoint for every row, scores the responses, and moves the run to a terminal state — you only wait and read results. Pass sut_id=/connection_id=, and the block only blocks on run.poll_status() until a terminal state, then reads the RunResultsPage with run.results().

SDK

with aip.run(project=project.id, dataset=f"{dataset.id}@v1", sut_id=sut.id, connection_id=conn.id) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results()

Creating and enqueuing a run

aip.run() (and its async twin aip.arun()) both create and enqueue the run when the with block opens. Key parameters: project, dataset (accepts a bare id or a pinned id@v1), metrics (the scorers to compute — omit for a generate-only run that produces SUT outputs with no scoring), and, to select the hosted runner, connection_id (with sut_id for attribution). Pass run_config=aip.RunConfig(...) to tune external-runner connection backoff/retry (the hosted runner ignores it). To score an already-produced transcript with hosted judges instead of running end-to-end, pass pipeline="metric_invoke" with source_run_id — see Score an existing run server-side.

SDK

You can also keep the run definition in a version-controlled eval-config YAML and hand it straight to run(). aip.load_eval_config() returns a typed EvalConfig, and config.to_run_kwargs() expands into the same parameters shown above.

config = aip.load_eval_config("eval_config_ci.yaml")
with aip.run(**config.to_run_kwargs()) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results()

In the eval config (eval_config_ci.yaml), pinning the dataset to a version (golden-v1@v3) keeps CI runs reproducible, and the thresholds: block flags the run as failed if a metric breaches its threshold.

version: "1"
project: my-project
dataset: golden-v1@v3
sut_id: sut-abc-123
connection_id: conn-xyz-456
metrics:
  - accuracy
  - f1_score
thresholds:
  accuracy: 0.85
tags:
  env: ci
  branch: main

API

POST /runs creates a run directly.

curl -sS -X POST "$AIP_API_URL/runs" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "metric_invoke",
    "dataset_version_id": "<version_id>",
    "run_config": {"metrics": ["llm.correctness", "llm.toxicity"]}
  }'

Score an existing run server-side (pipeline="metric_invoke")

By default aip.run() runs the evaluate_sut_hosted pipeline end-to-end. When you have already produced SUT responses off-platform — you generated them yourself and uploaded them to an external run — you can have the platform run judge scoring server-side against that existing transcript, without re-invoking the SUT. Pass pipeline="metric_invoke" with source_run_id (the completed run to score) and the metrics to compute. The run is always hosted; the platform reads the source run's stored responses and dispatches your metrics to its own judge infrastructure, so no local aip_metrics.score() step is needed.

metric_invoke takes exactly one input source: source_run_id to score an existing run's transcript, or dataset to score a golden dataset version. Supplying both, neither, or an empty metrics list raises InterfaceValidationError before any run is created, and runner="external" is rejected because the scoring runs server-side.

SDK

The example is end-to-end: it first produces the source run off-platform — a generate-only external run (metrics omitted, so no local scoring) that uploads just the SUT responses — then scores that stored transcript server-side by passing its id as source_run_id.

import aip_sdk as aip

# Step 1 — produce the transcript off-platform. This is a generate-only external
# run: you own inference and upload only the SUT responses, no local scoring.
with aip.run(project=project.id, dataset=f"{dataset.id}@v1") as prior:
    golden_df = prior.dataset.pull()
    golden_df["sut_response"] = my_sut(golden_df["prompt"])
    prior.upload(golden_df)  # stamps aip_version itself when no local scoring produced one
    prior.poll_status(interval=5, timeout=300)
    prior_run_id = prior.id  # keep this to score later

# Step 2 — score the stored transcript server-side; the SUT is never re-invoked.
with aip.run(
    project=project.id,
    pipeline="metric_invoke",
    source_run_id=prior_run_id,
    metrics=["llm.correctness", "llm.toxicity"],
) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results()

aip.arun() accepts the same pipeline and source_run_id parameters for async callers.

A runnable, turnkey version of this flow — with typed InterfaceValidationError handling for each malformed request — lives at metric-invoke recipe.

API

POST /runs with source_run_id (top-level or under run_config) targets an existing run:

curl -sS -X POST "$AIP_API_URL/runs" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "metric_invoke",
    "runner_mode": "hosted",
    "source_run_id": "<run_id>",
    "run_config": {"metrics": ["llm.correctness", "llm.toxicity"]}
  }'

Per-metric judge model (run_config.metric_configs)

Networked LLM-judge metrics (llm.toxicity, rag.faithfulness, …) default to the worker's global judge credential (AIP_JUDGE_API_KEY / AIP_JUDGE_MODEL). To give individual metrics their own judge model — and optionally their own key — pass a metric_configs mapping (dict[metric_name → config]). judge_model and judge_api_key resolve per-field: an entry that sets only judge_model keeps the global key, and vice versa. A metric with no entry is untouched, and any key other than those two is forwarded to the scorer as test_params. Overrides apply to networked judge metrics only.

An entry may instead name a registered judge connection, which is preferred over pasting a key: only the reference is stored in run_config, and the credential stays encrypted until scoring. Each metric resolves its judge from the first source that is set — metric_configs[metric].judge_api_key (inline, deprecated) → metric_configs[metric].judge_connection_id → run_config.judge_connection_id (the run-level default) → AIP_JUDGE_API_KEY / AIP_JUDGE_MODEL. A metric's own judge_connection_id replaces the run-level one rather than layering over it, so each metric resolves exactly one connection and takes that connection's key, endpoint, and model together — a key is never paired with another judge's endpoint or model. An inline judge_model can still retarget the model on its own; it falls back to the selected connection's model, then to AIP_JUDGE_MODEL. An inline judge_api_key still outranks a connection on the same metric, so older configs behave identically, but it is deprecated because the key is stored in plaintext with the run. Unknown, inaccessible, or cross-workspace references are rejected when the run is created and the error names the metric. An endpoint cannot be set inline — a base_url in an entry is treated as a scorer param, not as judge wiring.

{
  "metrics": ["llm.correctness", "llm.toxicity"],
  "judge_connection_id": "judge_default123",
  "metric_configs": {"llm.correctness": {"judge_connection_id": "judge_strong456"}}
}

Multi-client metrics (llm.data_leakage, rag.content_bias) use three clients: the selected judge drives both the evaluator and the generation client, while the system-under-test client comes from the run's SUT connection and is unaffected by the judge selection.

metric_configs is a first-class parameter on every surface, so you never need to hand-roll a raw request body.

SDK

Pass it directly to aip.run().

with aip.run(
    project="my-project",
    dataset="gds@v1",
    metrics=["llm.toxicity", "rag.faithfulness"],
    metric_configs={
        "llm.toxicity": {"judge_model": "gpt-4o-mini"},
        "rag.faithfulness": {"judge_model": "claude-opus-4-8", "judge_api_key": "sk-…"},
    },
) as run:
    ...

API

metric_configs lives inside run_config.

curl -sS -X POST "$AIP_API_URL/runs" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": "metric_invoke",
    "dataset_version_id": "<version_id>",
    "run_config": {
      "metrics": ["llm.toxicity", "rag.faithfulness"],
      "metric_configs": {
        "llm.toxicity": {"judge_model": "gpt-4o-mini"},
        "rag.faithfulness": {"judge_model": "claude-opus-4-8", "judge_api_key": "sk-…"}
      }
    }
  }'

The legacy aliases scorers / scorer_configs are still accepted in place of metrics / metric_configs; send only one of each pair. In eval-config YAML the same mapping lives under a top-level metric_configs: key.

Async runs (aip.arun())

aip.arun() is the async context-manager twin of aip.run() — identical parameters and the same Run object — for use inside asyncio event loops or notebook await cells. The Run methods make blocking HTTP calls, so wrap long-polling in asyncio.to_thread.

SDK

async with aip.arun(project=project.id, dataset=f"{dataset.id}@v1", sut_id=sut.id, connection_id=conn.id) as run:
    await asyncio.to_thread(run.poll_status, interval=5, timeout=300)
    page = await asyncio.to_thread(run.results, page=1, page_size=100)

Checking run status

run.status returns one of "pending", "running", "completed", "failed", or "cancelled" (the backend may report "complete", which the SDK treats as a synonym of "completed"). run.poll_status(interval, timeout) blocks until a terminal state and raises RunFailedError or RunCancelledError on failure. run.cancel() requests graceful cancellation of a run that hasn't finished. Rather than poll a long run yourself, you can subscribe to run-completion webhooks — see the Notifications feature (aip.NotificationConfig.create(...)), which delivers a signed event envelope when a run reaches a terminal state.

API

A GET /runs/<run_id> syncs and returns the run status.

curl -sS "$AIP_API_URL/runs/<run_id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

Retrieving results and aggregated metrics

run.results(page=1, page_size=100) returns a RunResultsPage — the per-row scores and the aggregated metric summaries for the run. page_size must be between 1 and 1000; page server-side by incrementing page. Calling it before the run finishes raises RunNotCompleteError. The three result types are frozen dataclasses exported at the top level (from aip_sdk import ScoreResult, Metric, RunResultsPage):

• RunResultsPage — .results (list[ScoreResult]), .metrics (list[Metric], repeated on every page), .total, .page, .page_size.
• ScoreResult — one entry per (input_id, scorer) pair: .input_id, .scorer, .score (float), .explanation (str | None), .config_identity (list[MetricConfigIdentity] — the stored config(s) that scorer resolved; empty if none was pinned or the platform predates this field).
• Metric — one aggregated entry per scorer: .scorer, .mean, .std, .count, and .pass_rate (fraction of rows at or above the scorer threshold, 0.0–1.0).

For descriptive statistics and per-dimension breakdowns, run.analysis(group_by=None) returns a dict with run_id, row_count, dimension_columns, scorer_stats, and score_distribution — plus per-value group_stats when group_by names a dimension column.

SDK

page = run.results(page=1, page_size=100)
for r in page.results:
    print(r.input_id, r.scorer, r.score)
for m in page.metrics:
    print(f"{m.scorer}: mean={m.mean:.3f} pass_rate={m.pass_rate:.0%} (n={m.count})")

analysis = run.analysis(group_by="intent")

API

The /results endpoint returns paginated row-level output and the /analysis endpoint returns aggregated scorer stats plus histograms.

curl -sS "$AIP_API_URL/runs/<run_id>/results?page=1&page_size=100" \
  -H "Authorization: Bearer $AIP_TOKEN"
curl -sS "$AIP_API_URL/runs/<run_id>/analysis" \
  -H "Authorization: Bearer $AIP_TOKEN"

Listing runs (aip.list_runs, aip.iter_runs)

To enumerate past runs — for a project dashboard, or to find a run id to re-open — use aip.list_runs(). It returns RunSummary records (newest first), optionally filtered by project and status, and paginated. Each RunSummary is a read-only snapshot carrying .id, .pipeline, .status, .runner_mode, the project/dataset/SUT ids, .error, and the run timestamps; pass a summary's .id to aip.get_run() to attach to the run and read its results. status accepts any of pending | running | completed | failed | cancelled; page/per_page are validated at the boundary (InvalidArgumentError), and per_page is capped at 100.

Pass metric with metric_config_name/metric_config_version to filter to runs whose scoring snapshot for that metric resolved a specific stored config — metric and at least one config filter are each required by the other. aip.iter_runs() takes the same filters and auto-paginates, for sweeping every run pinning a config version (e.g. before retiring it) instead of paging by hand.

SDK

for summary in aip.list_runs(status="completed", per_page=50):
    print(summary.id, summary.pipeline, summary.status, summary.created_at)

# Attach to a listed run to read its results.
latest = aip.list_runs(project_id=project.id)[0]
page = aip.get_run(latest.id).results()

# Auto-paginate every run pinning a specific stored config.
for summary in aip.iter_runs(metric="agent.custom_judge_rubric", metric_config_name="rubric_prompt_a"):
    print(summary.id, summary.status)

Inspecting a past run (aip.get_run)

To re-open a run created earlier — for example yesterday's run — fetch it by id. The returned Run is not a new run and does not own the HTTP client, so you can call status, results(), and analysis() directly, outside any with block. results() still raises RunNotCompleteError until the run finishes.

In the example below, aip.get_run(run_id) attaches to the existing run, run.status reads its state (e.g. "completed"), run.results() returns a RunResultsPage, and run.analysis() returns the aggregated stats.

SDK

run = aip.get_run(run_id)
print(run.status)
page = run.results(page=1, page_size=100)
analysis = run.analysis(group_by="intent")

Diffing two runs (aip.diff)

Compare two completed runs row-by-row and export the full diff as CSV. aip.diff() accepts run ids or Run objects and returns a RunDiff handle; .export() streams the diff to a path or returns the CSV as bytes. Only format="csv" is supported; a 5xx or transport failure raises ResultsUnavailableError.

In the example, aip.diff(...) returns a RunDiff, run_diff.export(path="diff.csv") streams the CSV to disk and returns None, and calling export() with no path returns the CSV payload as bytes.

SDK

run_diff = aip.diff(run_a_id, run_b_id)
run_diff.export(path="diff.csv")
csv_bytes = run_diff.export()

# Narrow + sort server-side — the CSV contains only the matching, ordered rows
run_diff.export(
    path="diff.csv",
    row_status="new",  # matched | new | missing
    scorer_name=["accuracy", "relevance"],  # str or list of scorer names
    delta_min=0.1,  # bound the max abs per-scorer delta
    sort="accuracy__delta_desc",  # <dim>_asc / <dim>_desc
)

export() accepts optional narrowing params, applied server-side so the CSV holds only the matching, sorted rows: row_status (matched/new/missing), scorer_name (a scorer name or list — keep rows where any named scorer has a score), delta_min/delta_max (bound the maximum absolute per-scorer delta), and sort (<dimension>_asc/_desc, where <dimension> is input_id, row_status, or a per-scorer column like accuracy__delta). An invalid filter or sort value raises APIError (HTTP 400).

API

The raw API streams CSV from the /runs/diff/export endpoint; format=csv is the only supported format.

curl -sS "$AIP_API_URL/runs/diff/export?run_a=<run_a>&run_b=<run_b>&format=csv" \
  -H "Authorization: Bearer $AIP_TOKEN" -o diff.csv