Skip to main content

aip_sdk.RunDiff

aip_sdk.RunDiff(run_a: str, run_b: str, client: APIClient)

Handle for comparing two completed runs.

Export the full row-level comparison with export(), or read whether the two runs were scored under the same metric configs with config_identity_match().

aip_sdk.RunDiff.config_identity_match​

aip_sdk.RunDiff.config_identity_match(*, include_params: bool = False) -> dict[str, ConfigIdentityMatch]

Report whether the two diffed runs resolved the same metric configs.

Keyed on the opaque "{config_name}@{config_version}@{config_scope}" identity of each stored metric config seen scoring either run. Each ConfigIdentityMatch entry's match is False only when both runs pinned this metric to different stored-config identities; an unpinned side reports True. A metric neither run pinned contributes no entry. The mapping is empty both when neither run pinned any config, and when the connected API predates this field — check config_identity_match_supported to tell those two apart.

The base result is fetched once and cached on this handle, since the flag is pair-level and identical no matter which diff page it is read from. The underlying request costs the same as a full row-diff query, so this call shares export()'s failure modes (e.g. raising NotFoundError for missing stored results, not just a missing run) even though it returns no rows itself. Transient rate-limit or server errors are retried with backoff before either is raised.

Parameters

  • include_params bool: When True, also fetch each identity's params (one extra request per distinct identity, cached separately) and populate param_diff for a metric with one identity per side of a mismatch. False by default so a caller who only wants match keeps the single-request cost.

Returns

  • dict[str, ConfigIdentityMatch]: A mapping from config identity to that identity's comparability.
  • dict[str, ConfigIdentityMatch]: Each call returns an independent copy — mutating the result does
  • dict[str, ConfigIdentityMatch]: not affect the cached value read by a later call.

Raises

  • AuthError: If no credentials are configured, or they have expired.
  • NotFoundError: If either run, or either run's stored results, does not exist, or either run belongs to a workspace the caller is not a member of.
  • ForbiddenError: If the two runs belong to different workspaces.
  • ConflictError: If either run has not reached a completed status.
  • IncomparableRunsError: If the two runs scored separately ingested trace data.
  • UnprocessableEntityError: If the two run ids are identical, or either run's pipeline does not produce a diffable result.
  • RateLimitError: If the caller has exceeded the API's rate limit and retries were exhausted.
  • ResponseParseError: If the platform returns a value this SDK version cannot parse.
  • ResultsUnavailableError: If the diff backend is unreachable or still returns a 5xx after retries are exhausted.
  • APIError: For any other unexpected error status.

aip_sdk.RunDiff.config_identity_match_supported​

aip_sdk.RunDiff.config_identity_match_supported: bool

Whether the connected platform sends the config_identity_match field at all.

False against a platform predating the field, where config_identity_match() returns {} because the comparison is unknown rather than because both runs agree on every config — check this before reading an empty config_identity_match() as "nothing to report" instead of "not reported."

Triggers the same fetch as config_identity_match() on first access, and shares its cache — reading both incurs one request, not two.

Returns

  • bool: True if the connected platform reports this field at all,
  • bool: regardless of whether either run pinned a config.

Raises

aip_sdk.RunDiff.export​

aip_sdk.RunDiff.export(path: str | PathLike[str] | None = None, format: str = 'csv', *, row_status: str | None = None, scorer_name: str | Sequence[str] | None = None, delta_min: float | None = None, delta_max: float | None = None, sort: str | None = None) -> bytes | None

Export the full run diff as CSV, optionally narrowed server-side.

When path is provided, response chunks are streamed directly to disk. Without path, the CSV payload is returned as bytes. The optional filter and sort params are applied by the server, so the CSV contains only the matching, sorted rows.

Columns are input_id, row_status, then a <scorer>__score_a/__score_b/__delta triplet per scorer. input_id is the dataset input the row was evaluated on, except for two agent-trace runs, where it is the reference of the scored occurrence instead.

Two agent-trace runs add four granularity columns after row_status, so a row can be placed even when one run scored several granularities at once: partition_type is span, trace or session; target_ref is the span, trace or session that was scored; trace_id is the trace it belongs to (a span id is unique only within its trace, so key a span row on both); and parent_id is the enclosing session, empty for an occurrence outside one.

Parameters

  • path str | PathLike[str] | None: Destination CSV path. When omitted, the CSV is returned as bytes.
  • format str: Export format. Only "csv" is supported.
  • row_status str | None: Keep only "matched", "new", or "missing" rows.
  • scorer_name str | Sequence[str] | None: A scorer name, or list of scorer names; keep rows where any named scorer has a score present.
  • delta_min float | None: Keep rows whose maximum absolute per-scorer delta is at least this value.
  • delta_max float | None: Keep rows whose maximum absolute per-scorer delta is at most this value.
  • sort str | None: Order the rows by "<dimension>_asc" / "<dimension>_desc", where <dimension> is "input_id", "row_status", or a per-scorer column ("<scorer>__delta", "<scorer>__score_a", "<scorer>__score_b") — e.g. "accuracy__delta_desc".

Returns

  • bytes | None: The CSV payload as bytes when path is omitted, otherwise None.

Raises

  • ValueError: If format is anything other than "csv".
  • APIError: If a filter or sort value is invalid (HTTP 400).
  • IncomparableRunsError: If the two runs scored separately ingested trace data.
  • UnprocessableEntityError: If the two run ids are identical, or either run's pipeline does not produce a diffable result.
  • ResultsUnavailableError: If the export backend is unreachable or returns a 5xx.

aip_sdk.RunDiff.run_a​

aip_sdk.RunDiff.run_a = run_a

No docstring is defined in the source.

aip_sdk.RunDiff.run_b​

aip_sdk.RunDiff.run_b = run_b

No docstring is defined in the source.