Skip to main content

aip_sdk.EvalConfig

aip_sdk.EvalConfig

Typed evaluation configuration consumed by aip.run() and YAML config files.

version is a schema version guard. Two shapes are supported, and each version accepts exactly one of them:

  • v1 — a flat, untargeted selection: metrics is a list[str] and metric_configs is keyed by bare metric name. This is every config written before the partition axis existed; it keeps loading and validating unchanged, and metrics stays a list[str] rather than being rewritten into the v2 shape.
  • v2 — a targets list, each entry pairing a partition (see EvalTarget) with the metrics to run against it. Trace evaluation is inherently per-(target, metric), which a flat list cannot express.

Mixing the two is rejected rather than merged: a v1 config carrying targets, or a v2 config carrying top-level metrics, is ambiguous about which selection wins. Use resolved_targets() to read either version through one code path.

parameters and thresholds are accepted but not forwarded to run(), for either version — they exist so configs written today stay valid when those features land, and run() has no parameter to carry them to. thresholds is keyed by bare metric name and carries no target axis, so a v2 config may not set a threshold for a metric that more than one target could claim; see _thresholds_must_name_one_target().

metric_config_refs pins a metric to a stored, versioned MetricConfig by its own (config_name, config_version) identity — distinct from metric_configs' inline, unversioned params. It stays top-level for both shapes (unlike metrics/ metric_configs, which move under targets for v2/v3): which stored config a metric uses is a property of the metric, not of any one target it runs under.

aip_sdk.EvalConfig.connection_id​

aip_sdk.EvalConfig.connection_id: str | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.dataset​

aip_sdk.EvalConfig.dataset: str

No docstring is defined in the source.

aip_sdk.EvalConfig.judge_connection_id​

aip_sdk.EvalConfig.judge_connection_id: str | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.metric_config_refs​

aip_sdk.EvalConfig.metric_config_refs: dict[str, MetricConfigRefValue] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.metric_configs​

aip_sdk.EvalConfig.metric_configs: dict[str, MetricConfigValue] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.metrics​

aip_sdk.EvalConfig.metrics: list[str] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.parameters​

aip_sdk.EvalConfig.parameters: dict[str, Any] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.project​

aip_sdk.EvalConfig.project: str

No docstring is defined in the source.

aip_sdk.EvalConfig.resolved_targets​

aip_sdk.EvalConfig.resolved_targets() -> list[EvalTarget]

Return this config's metric selection as a target list, whatever its version.

One read path for consumers so they never branch on version. A v1 config has no partition axis, so it yields exactly one untargeted entry (partition_type and partition_id both None) carrying its flat selection; a v2 config yields its authored targets unchanged.

The untargeted entry a v1 config yields is a READ view, not an authorable target: a target with neither partition_type nor partition_id is rejected in a v2 config, so this output cannot be fed straight back into EvalConfig(version="2", targets=...).

Every entry is a deep copy, so the whole result is a snapshot a caller may do anything to. EvalTarget is frozen, but that only refuses attribute assignment — the metrics list and the metric_configs dict a target holds are ordinary mutable objects, so handing out the stored instances would let resolved_targets()[0].metrics.append(...) edit the config through what reads as a read method. It would also make the two versions behave differently: a v1 config builds its entry from validated copies and could never be reached that way. Callers wanting to change a config still go through the model.

Returns

  • list[EvalTarget]: list[EvalTarget] — one entry per authored target for a v2 config,
  • list[EvalTarget]: or exactly one untargeted entry for a v1 config. Never empty, and never
  • list[EvalTarget]: aliased to this config's own state.

aip_sdk.EvalConfig.sut_id​

aip_sdk.EvalConfig.sut_id: str | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.tags​

aip_sdk.EvalConfig.tags: dict[str, str] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.targets​

aip_sdk.EvalConfig.targets: list[EvalTarget] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.thresholds​

aip_sdk.EvalConfig.thresholds: dict[str, float] | None = None

No docstring is defined in the source.

aip_sdk.EvalConfig.to_run_kwargs​

aip_sdk.EvalConfig.to_run_kwargs() -> dict[str, Any]

Return a dict consumable by the run() context manager.

A v2 config expands onto the run's target axis — one run carrying every target, each keeping its own metrics/metric_configs. thresholds is dropped for both versions: the run contract has no gate to forward it to.

Returns

  • dict[str, Any]: dict[str, Any] of keyword arguments for aip.run(). A v2 config also
  • dict[str, Any]: carries pipeline and targets.

Raises

  • InvalidArgumentError: if a v2 config names a shape a run cannot execute — a target carrying span_kind, a target selecting no metrics, or a target with no partition_type alongside others. Each names the config-level fix.

aip_sdk.EvalConfig.version​

aip_sdk.EvalConfig.version: str = Field(default='1')

No docstring is defined in the source.