Skip to main content

aip_sdk.Run

aip_sdk.Run(run_id: str, dataset_version_id: str, runner_mode: str, upload_token: str | None, client: APIClient, owns_client: bool = False, backend: RunnerBackend | None = None)

Active evaluation run context.

aip_sdk.Run.analysis​

aip_sdk.Run.analysis(group_by: str | None = None) -> dict[str, Any]

Return pre-computed statistics, distributions, and failure clusters for this run.

Wraps GET /runs/{id}/analysis. When group_by is set, also returns per-dimension-value breakdowns under group_stats.

Parameters

  • group_by str | None: Optional dimension to stratify results by. Takes either a qualified dataset dimension id — declared:<name>, derived:<key>.<property> or assigned:<name>, including one saved by save_dimension() — or a string column of the run output. Qualified ids are the primary use: they are joined to the run's rows and are not listed in the response's dimension_columns, which names the output columns only. Discover the ids available for a dataset version with GET /datasets/{dataset_id}/versions/{version_id}/dimensions. For a wide (non-trace) run the output columns are dataset columns (e.g. "scenario", "intent"); for a trace_metric_invoke run they are "partition_type", "target_ref", "parent_id", "partition_id", "metric_config_version" and "metric_config_name". Group by metric_config_name to separate two variants of one metric that share a version string — grouping by the version alone would merge them, since uniqueness is per (name, version).

Returns

  • dict[str, Any]: Dict with keys run_id, row_count, dimension_columns,
  • dict[str, Any]: scorer_stats (per-scorer descriptive stats), score_distribution
  • dict[str, Any]: (histogram buckets), clusters (pre-computed failure groupings), and
  • dict[str, Any]: optionally group_by + group_stats when a dimension is requested.
  • dict[str, Any]: A trace_metric_invoke run additionally carries series — one entry per
  • dict[str, Any]: (metric, partition_type, partition_id) measurement, each with its own
  • dict[str, Any]: identity and stats, plus session_scopes (every session it holds results for)
  • dict[str, Any]: and rollups: the SESSION/SYSTEM-level summaries that results() cannot
  • dict[str, Any]: show, since it returns PARTITION-level rows only. Each summary carries
  • dict[str, Any]: reduction — the name and version of the logic that produced the number — and
  • dict[str, Any]: is present only where the metric declares that reduction admissible over the
  • dict[str, Any]: granularity it was scored at; it may be absent even then. rollups.session and
  • dict[str, Any]: rollups.system are always present and are empty when no summary was produced,
  • dict[str, Any]: so read session_scopes to discover a run's sessions. Top-level
  • dict[str, Any]: rollup_stats mirrors series[].rollups for callers not yet on series
  • dict[str, Any]: and is deprecated.

Raises

aip_sdk.Run.cancel​

aip_sdk.Run.cancel() -> None

Cancel the run gracefully.

Sends a cancellation request to the platform. After cancellation run.status returns "cancelled".

Raises

  • RunCancelledError: If the API rejects the cancellation (wraps the underlying APIError and preserves its status_code).

aip_sdk.Run.close​

aip_sdk.Run.close() -> None

Close the HTTP client if we own it.

aip_sdk.Run.dataset​

aip_sdk.Run.dataset: DatasetHandle

Get handle for accessing the golden dataset.

Resolves the dataset version via GET /dataset-versions/{id} and caches the handle for subsequent calls.

Returns

Raises

aip_sdk.Run.dataset_version_id​

aip_sdk.Run.dataset_version_id = dataset_version_id

No docstring is defined in the source.

aip_sdk.Run.embeddings​

aip_sdk.Run.embeddings: EmbeddingsHandle

Read this run's embedding coordinates.

Returns

Examples:

with aip.run(project=project.id, dataset="gds@v1", embeddings=config) as run:
    run.poll_status()
    coords = run.embeddings.coordinates

aip_sdk.Run.finish​

aip_sdk.Run.finish(df: pd.DataFrame) -> None

Upload results and close the run context.

This is a convenience method that calls upload() - the actual context cleanup happens in the context manager.

Parameters

  • df pd.DataFrame: DataFrame with scored results

aip_sdk.Run.get​

aip_sdk.Run.get(run_id: str, client: APIClient | None = None) -> Run

Attach to an existing run by id for post-hoc inspection.

Unlike run(), this does not create a new run — it builds a Run bound to an id that already exists so you can read status and call results() / analysis() outside any with block. The returned Run never owns the client, so the client is never auto-closed and the run stays usable for the life of the session.

Parameters

  • run_id str: Id of an existing run (e.g. a previous run's run.id).
  • client APIClient | None: Optional API client. Defaults to the module-level client.

Returns

  • Run: A Run bound to run_id.

Raises

  • AuthError: If no credentials are configured.

Examples:

run = aip.get_run("run_abc123")
page = run.results()
print(page.metrics)

aip_sdk.Run.id​

aip_sdk.Run.id = run_id

No docstring is defined in the source.

aip_sdk.Run.invalidate_output_cache​

aip_sdk.Run.invalidate_output_cache() -> None

Discard a cached output() frame so the next read re-downloads it.

aip_sdk.Run.output​

aip_sdk.Run.output(max_retries: int = 3, backoff_factor: float = 1.0) -> pd.DataFrame

Download this run's full row-level output as a DataFrame.

Everything the run produced, one row per evaluated input: the golden columns, the SUT response, every metric's score, and — when the run was created with embeddings= — the nested embed column holding the 2D coordinates. This is the offline-analysis entry point: embeddings reads the coordinates out of the same download.

Available once the run has completed. Unlike results(), which pages through the API, this fetches the whole Parquet in one download.

Parameters

  • max_retries int: Maximum download attempts (default 3).
  • backoff_factor float: Multiplier for exponential wait between retries (default 1.0).

Returns

  • pd.DataFrame: DataFrame with one row per evaluated input.

Raises

Examples:

run = aip.get_run("run_abc123")
df = run.output()
df.to_parquet("run_abc123.parquet")

aip_sdk.Run.poll_status​

aip_sdk.Run.poll_status(interval: float = 5.0, timeout: float = 300.0) -> str

Poll run status until completion or timeout with exponential backoff.

Parameters

  • interval float: Initial seconds between polls
  • timeout float: Maximum seconds to wait

Returns

  • str: "completed" when the run finishes successfully.

Raises

  • RunFailedError: If the run fails during execution.
  • RunCancelledError: If the run is cancelled before completion.
  • RunConnectionError: If the external runner cannot reach the platform after all retry attempts are exhausted (external runner only).
  • TimeoutError: If run does not complete within timeout.

aip_sdk.Run.required_metrics​

aip_sdk.Run.required_metrics: list[str]

Get the list of metrics required for this run.

Fetches run configuration from the API to determine which metrics should be applied.

Returns

  • list[str]: List of metric names

aip_sdk.Run.required_scorers​

aip_sdk.Run.required_scorers: list[str]

Deprecated alias for required_metrics.

aip_sdk.Run.results​

aip_sdk.Run.results(page: int = 1, page_size: int = 100) -> RunResultsPage

Retrieve typed per-row results and aggregated metrics for this run.

Wraps GET /runs/{id}/results and GET /runs/{id}/analysis. Rows are melted into per-(input_id, scorer) ScoreResult objects. Aggregated Metric summaries are fetched from the analysis endpoint and included in every page response.

For a trace_metric_invoke run, results/metrics are actually TraceScoreResult/TraceMetric instances at runtime — subclasses carrying partition_type, partition_id, and (on TraceScoreResult) target_ref/trace_id. This method's declared return type only names the base ScoreResult/Metric fields, though, so a static type checker won't see the extra ones — call trace_results() instead for a return type that names them. A metric measured at more than one target appears more than once in metrics either way, each entry distinguished by that identity rather than by scorer alone.

For a hosted-SUT (non-trace) run, per-call SUT telemetry (HTTP status, retry count, time-to-first-byte) is reported separately from evaluation metrics — it describes how the call to the system under test behaved, not how well it scored, so it is never mixed into metrics. Numeric telemetry (retry count, time-to-first-byte) is a mean/std/count distribution, as Telemetry entries. Categorical telemetry (HTTP status) is a count per value — a mean HTTP status code is meaningless — as TelemetryBreakdown entries. A trace_metric_invoke run has no SUT call telemetry, so both are empty.

row_coverage reports how much of the dataset the system under test answered. Rows it refused carry no response to evaluate, so they are excluded from every metric rather than scored as failures — which makes each metrics entry a number about row_coverage.answered_rows, not the whole dataset. It is None for a run that never called a system under test.

Parameters

  • page int: 1-based page number (default 1)
  • page_size int: Rows per page (default 100)

Each ScoreResult's config_identity names the stored metric config(s) that scorer resolved when scored — empty for a scorer that pinned no stored config, or a run scored before this field existed.

Returns

Raises

  • RunNotCompleteError: If the run has not yet completed.
  • ResultsUnavailableError: If the results backend returns a 5xx error or is unreachable.
  • ResponseParseError: If this run's pipeline is trace_metric_invoke and the analysis response carries no series, or its results carry a partition_type this SDK version does not recognise. For any other pipeline, if scorer_stats, telemetry, telemetry_breakdown, or row_coverage in the analysis response does not match the shape this SDK version expects.

aip_sdk.Run.runner_mode​

aip_sdk.Run.runner_mode = runner_mode

No docstring is defined in the source.

aip_sdk.Run.session​

aip_sdk.Run.session(parent_id: str, *, compare_run: str | Run | None = None) -> RunSession

Retrieve one session's span structure with this run's results joined onto it.

Wraps GET /runs/{id}/sessions/{parent_id}. Answers where inside a session an evaluation failed: components holds every span in the session, including ones no partition covered (an empty results list, not a missing component), while end_to_end holds the session/trace-level verdicts. Call sessions() first to discover which parent_id values exist in this run.

When this read is one side of a session comparison, pass the other run as compare_run so the platform refuses a pair whose sessions cannot be matched before either side is read. Read both sides with each naming the other:

left = run_a.session(parent_id, compare_run=run_b)
right = run_b.session(parent_id, compare_run=run_a)

The check needs a platform at aip-api 0.2.1 or later; an older platform ignores compare_run and serves the read unchecked.

Parameters

  • parent_id str: The session key results were rolled up under — the dataset's real session_id where its traces carry one, else the trace's own trace_id. Must be non-empty and must not contain / — not addressable through this endpoint today.
  • compare_run str | Run | None: The run this read will be compared against, by id or as a Run. Omit it for a single-run read.

Returns

  • RunSession: A RunSession for this run and session.

Raises

  • InvalidArgumentError: If parent_id is empty or contains /, or compare_run is an empty id or not a run id or Run.
  • NotFoundError: If the run or compare_run does not exist or belongs to a workspace the caller is not a member of, this run's output is not yet available, or parent_id names no session on the scored dataset version.
  • ForbiddenError: If compare_run belongs to a different workspace from this run.
  • IncomparableRunsError: If compare_run scored separately ingested trace data, so its sessions cannot be paired with this run's.
  • UnprocessableEntityError: If the run is not a trace-metric run.
  • ResultsUnavailableError: If the results backend returns a 5xx error or is unreachable.
  • ResponseParseError: If the response does not match the shape this SDK version expects.

aip_sdk.Run.sessions​

aip_sdk.Run.sessions() -> list[str]

List the session keys (parent_id values) present in this run's results.

Every session that appears anywhere in this run's results is included, even one whose every occurrence errored. Pass any of the returned keys to session() to read that session's component breakdown.

Returns

  • list[str]: Sorted session keys.

Raises

aip_sdk.Run.status​

aip_sdk.Run.status: str

Return current run status from the API.

Returns

  • str: Status string: "pending", "running", "completed",
  • str: "failed", or "cancelled".

aip_sdk.Run.test_plan_id​

aip_sdk.Run.test_plan_id: str | None

Id of the test plan version that supplied this run's evaluation config.

None when no plan governed the run. A run created in this session reads its own creation response and makes no request; a run attached with aip.get_run() reads the run record once. Either way the value is cached for the life of this handle — attach a new handle with aip.get_run() to read it again.

The id addresses one immutable published version. Read the plan itself to see what that version holds, including its name and version number.

Returns

  • str | None: The plan version's id, or None.

Raises

  • (AuthError, NotFoundError, ForbiddenError, RateLimitError, APIError): If the run record has to be fetched and that request fails.

Examples:

run = aip.get_run("run_abc123")
if run.test_plan_id is not None:
    print(run.test_plan_id)

aip_sdk.Run.trace_results​

aip_sdk.Run.trace_results(page: int = 1, page_size: int = 100) -> TraceRunResultsPage

Retrieve typed per-row results and aggregated metrics for a trace_metric_invoke run.

Identical to results() for a trace run, but statically typed to the row identity a trace run actually carries: results/metrics are declared as TraceScoreResult/TraceMetric (not the base ScoreResult/Metric), so partition_type/partition_id/target_ref/trace_id type-check without a cast. Call results() instead for a run whose pipeline isn't known ahead of time.

Parameters

  • page int: 1-based page number (default 1)
  • page_size int: Rows per page (default 100)

Returns

Raises

aip_sdk.Run.upload​

aip_sdk.Run.upload(df: pd.DataFrame) -> None

Upload scored results to the run.

Validates the required columns and normalises aip_version before uploading: a missing or entirely blank column is filled from the installed aip_metrics and aip_sdk versions, and every dict or string cell is re-serialised to the one string form the platform stores.

Parameters

  • df pd.DataFrame: DataFrame with SUT responses and any locally scored columns; it is not modified.

Raises

  • InvalidStateError: If the run is not an external run or has no upload token.
  • InvalidArgumentError: If the frame is empty, a required column is missing, or aip_version is partially filled, differs across rows, or holds a cell that is neither a dict nor a parseable scorer-version map.
  • APIError: If the platform rejects the upload.

aip_sdk.Run.url​

aip_sdk.Run.url: str

Return Web UI URL for this run.

aip_sdk.Run.validate_environment​

aip_sdk.Run.validate_environment() -> None

Validate that local environment meets run requirements.

Checks:

  • aip_metrics is installed
  • Scorer versions meet minimum requirements (if specified in run config)

Raises

  • RuntimeError: If environment is incompatible