Skip to main content

aip_sdk.run

aip_sdk.run(project: str | None = None, dataset: str | None = None, runner: str | None = None, tags: dict[str, str] | None = None, sut: Any = None, sut_id: str | None = None, connection_id: str | None = None, judge_connection_id: str | None = None, metrics: list[str] | None = None, metric_configs: dict[str, dict[str, Any]] | None = None, metric_config_refs: dict[str, Any] | None = None, run_config: RunConfig | None = None, pipeline: PipelineName = 'evaluate_sut_hosted', source_run_id: str | None = None, targets: list[RunTarget | EvalTarget] | None = None, embeddings: EmbedProjectConfig | RegisteredEmbeddingConfig | dict[str, Any] | _Unset | None = _EMBEDDINGS_UNSET, client: APIClient | None = None) -> Generator[Run, None, None]

Context manager for evaluation runs.

For external runner (developer-owned infrastructure): with aip.run(project=project.id, dataset="gds@v1") as run: run.validate_environment() df = load_my_golden_data() df["sut_response"] = my_sut(df["prompt"]) scored = aip_metrics.score(df, scorers=run.required_metrics) run.upload(scored)

with aip.run(project=project.id, dataset="gds@v1", runner="hosted", sut=aip.HostedSUT("https://...")) as run: pass # AIP executes end-to-end

with aip.run(project=project.id, dataset="gds@v1", sut_id=sut.id, connection_id=conn.id) as run: run.poll_status()

To score an existing run's transcript with hosted judges (no SUT re-invocation), e.g. after generating SUT outputs off-platform and uploading them to a prior run: with aip.run(project=project.id, pipeline="metric_invoke", source_run_id=prior_run.id, metrics=["llm.correctness", "llm.toxicity"]) as run: run.poll_status() page = run.results()

To score the agent traces landed in a dataset version with trace metrics, scoped to a partition you authored with aip.create_partition(): with aip.run(project=project.id, dataset="trace-ds@v1", pipeline="trace_metric_invoke", targets=[aip.RunTarget(partition_id=partition.id, partition_type=aip.PartitionType.SPAN, metrics=["agent.tool_selection_quality"])]) as run: run.poll_status() page = run.results()

To score several granularities in one run — one artifact, so a session's verdict and the span verdicts under it can be joined back together with run.session(): with aip.run(project=project.id, dataset="trace-ds@v1", pipeline="trace_metric_invoke", targets=[ aip.RunTarget(partition_type=aip.PartitionType.SESSION, metrics=["agent.hallucination"]), aip.RunTarget(partition_id=tool_partition.id, partition_type=aip.PartitionType.SPAN, metrics=["agent.tool_selection_accuracy"]), ]) as run: run.poll_status() page = run.results()

Parameters

  • project str | None: Optional project ID (or name, for backward compatibility) the run is scoped to. When supplied, it is forwarded on the request and checked against the project the platform resolves from dataset/source_run_id (via the dataset's or source run's own project); a mismatch raises UnprocessableEntityError rather than silently running under a different project. Also raises when the resolved dataset has no project of its own to check against — there project cannot be honoured at all, so it is rejected rather than silently ignored. Omit project entirely for a workspace-scoped, project-less dataset, which the platform rejects any asserted project against.

  • dataset str | None: Dataset version reference (e.g., "dataset-id@v2" or just "dataset-id"). Required for pipeline="evaluate_sut_hosted" and pipeline="trace_metric_invoke". For pipeline="metric_invoke" supply exactly one of dataset or source_run_id.

  • runner str | None: Runner mode — "external" (developer-owned infra) or "hosted" (AIP-managed infrastructure). When left unset (None, the default) the mode is inferred: sut or connection_id selects hosted, otherwise external. An explicit value always wins; passing runner="external" together with sut/connection_id raises ValueError rather than silently switching to hosted. pipeline="metric_invoke" and pipeline="trace_metric_invoke" always run hosted and reject runner="external".

  • tags dict[str, str] | None: Optional metadata tags for the run

  • sut Any: Optional HostedSUT configuration (selects hosted mode when runner is unset)

  • sut_id str | None: Optional SUT registry ID to associate with this run; attribution only — does not select a runner mode.

  • connection_id str | None: Optional connection ID within the SUT (selects hosted mode when runner is unset)

  • judge_connection_id str | None: Optional ID of a registered judge connection to use as the run-level default judge. Every NETWORK judge metric in the run falls back to this connection's model, endpoint, and key unless a per-metric metric_configs override is set. Reference only — the credential stays encrypted on the platform and is never sent from or returned to the SDK. Omitted from the request when not provided.

  • metrics list[str] | None: Optional list of metric names to apply. When omitted, the key is omitted and the server applies its default metric selection. Required (non-empty) for pipeline="metric_invoke"; also required (directly or via a targets entry) for pipeline="trace_metric_invoke". This flat list itself may not repeat a metric — the same metric may still appear in more than one targets entry's own metrics, scoring it at every named target.

  • metric_configs dict[str, dict[str, Any]] | None: Optional per-metric judge configuration, keyed by metric name (e.g. {"llm.toxicity": {"judge_connection_id": "judge_abc123"}}). Forwarded verbatim into the run's run_config. Set judge_connection_id on an entry to score that metric with a registered judge connection, taking precedence over the run-level judge_connection_id. The inline judge_model / judge_api_key keys still work; judge_api_key is deprecated, is stored in plaintext with the run, and outranks a connection set on the same metric. Omitted from the request when not provided.

  • metric_config_refs dict[str, Any] | None: Optional stored MetricConfig pins, keyed by metric name. Names a config by its OWN (config_name, config_version) identity, which is independent of the metric it configures — unlike metric_configs, which carries inline parameters and no version provenance. Pass one ref, or several to score a metric under each in a single run::

    metric_config_refs = { "agent.custom_judge_rubric": [ {"config_name": "rubric_a", "config_version": "1.0.0"}, {"config_name": "rubric_b", "config_version": "1.0.0"}, ] }

Several refs for one metric require pipeline="trace_metric_invoke", whose rows each belong to one metric and so can carry that metric's own config; the wide pipelines merge every metric into shared rows and accept one ref per metric. Each variant's scores come back in their own {metric}__cfg__{config_name} result column. Omitted from the request when not provided.

  • run_config RunConfig | None: Optional backoff/retry configuration for the external runner. Ignored when using the hosted runner.
  • pipeline PipelineName: Pipeline to run — "evaluate_sut_hosted" (the default, end-to-end evaluation), "metric_invoke" (score an existing transcript with hosted judges, without re-invoking the SUT), or "trace_metric_invoke" (score the agent traces landed in an agent-trace dataset version with trace metrics).
  • source_run_id str | None: Completed run whose transcript to score. Only valid with pipeline="metric_invoke", and mutually exclusive with dataset.
  • targets list[RunTarget | EvalTarget] | None: The partition(s) a trace_metric_invoke run scores — the sole way to scope such a run to one or more partitions, whether that's one target or several scored together in one run/artifact (so results across them can be attributed together, e.g. via Run.session()). Only valid with pipeline="trace_metric_invoke". Each RunTarget may carry its own metrics/metric_configs, overriding the run-level metrics/ metric_configs for that target; a target that omits them scores the run-level selection. partition_type is required on every target once more than one is supplied. Omit targets entirely to score with no partition scoping at all — every occurrence the metrics accept. Omitted from the request when not provided. A stored v2 eval config's targets are accepted here too, so run(**config.to_run_kwargs()) needs no conversion.
  • embeddings EmbedProjectConfig | RegisteredEmbeddingConfig | dict[str, Any] | _Unset | None: Embedding config (an EmbedProjectConfig, or the dict it serialises to) to apply to this run's scored output. The platform projects each row to 2D and clusters it, persisting the coordinates with the run output for run.embeddings.coordinates to read back. Validate a config locally with aip.embeddings.compute(df, config) before spending a run on it. Omit the argument to inherit the project's config, or the workspace's if the project sets none. Pass None to compute no embeddings for this run, even where a project or workspace config would otherwise apply. See Project.set_embedding_config for the full resolution order.
  • client APIClient | None: Optional API client

Yields

  • Run: Run instance

Raises

  • ValueError: If runner is invalid, or runner="external" is combined with sut/connection_id (a contradictory request).
  • InterfaceValidationError: If pipeline is unknown; if a metric_invoke run does not name exactly one of dataset/source_run_id, omits metrics, or is paired with external execution or a SUT trigger; if a trace_metric_invoke run has no dataset, omits metrics, names one metric twice, or is paired with external execution or a SUT trigger; or if targets is passed with a pipeline other than trace_metric_invoke.
  • InvalidArgumentError: If a metric_configs entry is not a dict, or targets is empty, holds an entry that is neither a RunTarget nor an EvalTarget, carries a span_kind, or omits partition_type on any target while naming more than one.
  • AuthError: If no credentials are configured.
  • SutNotFoundError: If sut_id references an unknown SUT.
  • JudgeConnectionNotFoundError: If judge_connection_id references an unknown judge connection, or one outside the caller's workspaces (validated before the run is created).
  • ForbiddenError: If the caller's role is below editor in the workspace of the dataset, source run, SUT connection or judge connection the run uses.
  • PlanLockViolationError: If the project was created from a test plan that locks its evaluation config and the call supplied metrics, metric_configs or metric_config_refs — or, on a metric_invoke re-score, named a metric the plan does not carry. Omit them to score with the plan's values.
  • TestPlanReferenceError: If a reference the project's test plan carries no longer resolves for a metric this run scores; failures names each one.
  • ResponseParseError: If a plan refusal's detail does not match the shape this SDK version reads.
  • UnprocessableEntityError: If judge_connection_id belongs to a different workspace than the dataset; if a trace_metric_invoke run's dataset version is not an agent-trace version or is not promoted to golden; if a target's partition_id does not belong to that version; or if project does not match the project resolved from dataset/source_run_id.
  • RunFailedError: If the run fails during execution (raised from poll_status).
  • RunConnectionError: If the external runner cannot reach the platform after all retry attempts are exhausted.
  • APIError: If run creation fails