Configuration & Schema Reference
Golden Dataset Interface schemas
agent_trace_v1
GDI schema — a landed agent-trace dataset, one row per span.
Mirrors aip_traces_agent.parquet's column/JSON split, the actual at-rest shape a
land_traces() ingest writes: the queryable scalar span fields (ids, kind,
status, timings, token counts) are real columns here; the rich nested fields
(messages, tool calls, documents, the aip.* catch-all, the provenance map) ride in
the single attributes_json string column instead of one column each — this model
does not decode that blob, the same way the Parquet writer does not flatten it.
This is the schema a dataset/project declares when agent traces have been landed
(schema_name == "agent_trace_v1") — not the schema trace metrics score against.
Metrics read canonical_partition_v1 rows instead, derived from this table's spans
by resolve_partition() at scoring time; no metric's accepts names this schema
directly. It exists in this registry so a trace project/dataset can be created and
discovered at all (Project.create(), schemas.list_names(), detect()) the
same way a gdi_text_v1/gdi_image_v1 one can, and so a trace dataset's declared
schema matches what land_traces() actually lands, the invariant every other schema
already upholds (a dataset's schema_name must equal its project's).
The landing path (canonical_spans_to_parquet) always writes every field below as a
column, regardless of how many spans happen to be null in it — the Parquet writer fixes
the schema up front so the at-rest shape does not drift batch to batch. This model is
still split into required vs. nullable (| None) fields the same way every sibling
schema in this registry is, matching what a caller validating an arbitrary, hand-built
DataFrame (not one that went through the real landing path) should be allowed to omit —
required_columns/optional_columns on the registered SchemaCapability mirror
this split exactly, the same convention canonical_partition_v1 follows.
Schema: agent_trace_v1 (family: trace)
Description: A landed agent-trace dataset, one row per span — the schema a trace dataset/project actually declares (schema_name), written by land_traces() regardless of source convention (openinference/genai/langsmith all convert to this same shape first). Not what trace metrics score against: metrics read canonical_partition_v1 rows, derived from these spans by resolve_partition() when a run scores. The real landing path always writes every column below; required_columns/optional_columns mirror the model's own required-vs-nullable field split, the same convention canonical_partition_v1 follows, for a caller validating a hand-built DataFrame.
Preview mode: table
Required columns: attributes_json, kind, name, span_id, status, trace_id
Optional columns:
end_ns — Span end time, nanoseconds since epoch, when the source reports one
error — Error message/detail when status is an error status; null otherwise
input_tokens — Input token count, for an LLM/embedding span that reports one
model — LLM/embedding model id (e.g. gpt-4o-mini), for an LLM/embedding span
output_tokens — Output token count, for an LLM span that reports one
parent_id — This span's parent span id within its trace; null for a root span
session_id — Thread identity grouping traces into a session, when the source export carries one
start_ns — Span start time, nanoseconds since epoch, when the source reports one
step_index — Position of this span in its trace's tree, when materialized
Scorer contract: per_row
Extra columns: preserved as-is (strict=False)
{
"assignable_to_project": true,
"column_role_hints": {},
"description": "A landed agent-trace dataset, one row per span \u2014 the schema a trace dataset/project actually declares (schema_name), written by land_traces() regardless of source convention (openinference/genai/langsmith all convert to this same shape first). Not what trace metrics score against: metrics read canonical_partition_v1 rows, derived from these spans by resolve_partition() when a run scores. The real landing path always writes every column below; required_columns/optional_columns mirror the model's own required-vs-nullable field split, the same convention canonical_partition_v1 follows, for a caller validating a hand-built DataFrame.",
"extra_scorer_contracts": [],
"family": "trace",
"name": "agent_trace_v1",
"optional_column_descriptions": {
"end_ns": "Span end time, nanoseconds since epoch, when the source reports one",
"error": "Error message/detail when status is an error status; null otherwise",
"input_tokens": "Input token count, for an LLM/embedding span that reports one",
"model": "LLM/embedding model id (e.g. gpt-4o-mini), for an LLM/embedding span",
"output_tokens": "Output token count, for an LLM span that reports one",
"parent_id": "This span's parent span id within its trace; null for a root span",
"session_id": "Thread identity grouping traces into a session, when the source export carries one",
"start_ns": "Span start time, nanoseconds since epoch, when the source reports one",
"step_index": "Position of this span in its trace's tree, when materialized"
},
"optional_columns": [
"end_ns",
"error",
"input_tokens",
"model",
"output_tokens",
"parent_id",
"session_id",
"start_ns",
"step_index"
],
"preview_mode": "table",
"required_columns": [
"attributes_json",
"kind",
"name",
"span_id",
"status",
"trace_id"
],
"reserved_task_types": [],
"row_id_column": null,
"scorer_contract": "per_row",
"task_types": []
}
canonical_partition_v1
GDI schema — a materialized, resolved Partition occurrence.
One row is one resolved Partition occurrence (SessionPartition /
TracePartition / SpanPartition), not one span — rows carry no copy of
trace/span content, it is reconstructed at scoring time by filtering the at-rest
span table on target_ref. Column-level optionality (column may be absent from
the DataFrame) is expressed via Series[T] | None; row-level nullability
(individual cells may be None/NaN) is expressed via nullable=True. See each
field's own description= below for what it holds.
Ground-truth columns, in general: unlike gdi_text_v1.expected_output (one
column, the same reference value reused verbatim by many scorers) or
gdi_image_v1.label (one column, polymorphic by task_type), trace metrics'
ground-truth needs are genuinely heterogeneous — tool_selection_accuracy wants
an unordered set of tool names, step_accuracy wants an ORDERED step sequence,
and a dataset may need to supply both at once for the same occurrence. So this
schema does not have one universal ground-truth column: each metric that needs one
contributes its own new optional column, named expected_<something>, never
reusing another metric's. A column sharing expected_tools' plain JSON-encoded
list[str] shape should reuse :func:_is_json_list_of_str for its own @pa.check
rather than duplicating the validator; expected_steps instead uses
:func:_is_json_list_of_str_or_str_list, since step_accuracy's own ground truth can nest an
unordered concurrent-step group one level deep — a real shape difference, not an incidental
one, so the two columns are not forced to share a validator.
Schema: canonical_partition_v1 (family: trace)
Description: Materialized resolved Partition occurrences (session/trace/span) from an agent trace dataset. Lets trace metrics reuse the same golden-dataset schema/type and customer-field validation as other dataset kinds. Rows carry no copy of trace/span content. row_id is a stable, single-column identifier result rows key off (f'{partition_type}:{target_ref}:{partition_id}', the trailing segment omitted when no persisted Partition is named), unique on its own so two partitions covering one occurrence do not share it; partition_type is the discriminator; target_ref is the join key back to THIS occurrence (session_id/trace_id/span_id, depending on partition_type). parent_id is a DIFFERENT thing: the largest grouping key ABOVE this occurrence (the dataset's real session_id when its traces share one, else this occurrence's own trace_id) — it answers 'which session is this in' for rollup, where target_ref answers 'which row is this'; the two coincide only for a session row, or a trace row when the dataset has no real sessions. partition_id identifies the persisted Partition (selector) DEFINITION, when one exists — usually null, unlike parent_id which is always populated. optional_columns is a living set, not fixed to one metric: it currently lists expected_tools (tool_selection_accuracy's customer-supplied field) and expected_steps (step_accuracy's) and is expected to grow as more trace metrics adopt this schema, each contributing its own customer-supplied column(s) — the same way gdi_text_v1's optional columns accumulated across its scorer families. Never assigned to a Project or Dataset directly — see assignable_to_project below; the schema you land agent-trace data as is agent_trace_v1.
Preview mode: table
Required columns: parent_id, partition_type, row_id, target_ref
Optional columns:
expected_steps — Customer-supplied ground truth for step_accuracy: JSON-encoded, ordered list of the expected step sequence. Each element is either a bare step name, or itself a JSON list of step names — an unordered group of concurrent steps, one level of nesting only
expected_tools — Customer-supplied ground truth for tool_selection_accuracy: JSON-encoded list[str] of required tool names
partition_id — Id of the persisted Partition (selector) DEFINITION this occurrence resolves, when a saved selector exists — null for an ad hoc resolve with no authored selector behind it
trace_id — The trace enclosing this occurrence, which trace-qualifies target_ref: a span_id is unique only within its trace, so a span occurrence in a multi-trace session is not addressable without it. Null for a session occurrence, which spans every trace in the session, and on rows written before this column existed
Scorer contract: per_row
Extra columns: preserved as-is (strict=False)
{
"assignable_to_project": false,
"column_role_hints": {
"expected_steps": "unknown",
"expected_tools": "unknown",
"parent_id": "identifier",
"partition_id": "identifier",
"partition_type": "category",
"row_id": "identifier",
"target_ref": "identifier",
"trace_id": "identifier"
},
"description": "Materialized resolved Partition occurrences (session/trace/span) from an agent trace dataset. Lets trace metrics reuse the same golden-dataset schema/type and customer-field validation as other dataset kinds. Rows carry no copy of trace/span content. row_id is a stable, single-column identifier result rows key off (f'{partition_type}:{target_ref}:{partition_id}', the trailing segment omitted when no persisted Partition is named), unique on its own so two partitions covering one occurrence do not share it; partition_type is the discriminator; target_ref is the join key back to THIS occurrence (session_id/trace_id/span_id, depending on partition_type). parent_id is a DIFFERENT thing: the largest grouping key ABOVE this occurrence (the dataset's real session_id when its traces share one, else this occurrence's own trace_id) \u2014 it answers 'which session is this in' for rollup, where target_ref answers 'which row is this'; the two coincide only for a session row, or a trace row when the dataset has no real sessions. partition_id identifies the persisted Partition (selector) DEFINITION, when one exists \u2014 usually null, unlike parent_id which is always populated. optional_columns is a living set, not fixed to one metric: it currently lists expected_tools (tool_selection_accuracy's customer-supplied field) and expected_steps (step_accuracy's) and is expected to grow as more trace metrics adopt this schema, each contributing its own customer-supplied column(s) \u2014 the same way gdi_text_v1's optional columns accumulated across its scorer families. Never assigned to a Project or Dataset directly \u2014 see assignable_to_project below; the schema you land agent-trace data as is agent_trace_v1.",
"extra_scorer_contracts": [],
"family": "trace",
"name": "canonical_partition_v1",
"optional_column_descriptions": {
"expected_steps": "Customer-supplied ground truth for step_accuracy: JSON-encoded, ordered list of the expected step sequence. Each element is either a bare step name, or itself a JSON list of step names \u2014 an unordered group of concurrent steps, one level of nesting only",
"expected_tools": "Customer-supplied ground truth for tool_selection_accuracy: JSON-encoded list[str] of required tool names",
"partition_id": "Id of the persisted Partition (selector) DEFINITION this occurrence resolves, when a saved selector exists \u2014 null for an ad hoc resolve with no authored selector behind it",
"trace_id": "The trace enclosing this occurrence, which trace-qualifies target_ref: a span_id is unique only within its trace, so a span occurrence in a multi-trace session is not addressable without it. Null for a session occurrence, which spans every trace in the session, and on rows written before this column existed"
},
"optional_columns": [
"expected_steps",
"expected_tools",
"partition_id",
"trace_id"
],
"preview_mode": "table",
"required_columns": [
"parent_id",
"partition_type",
"row_id",
"target_ref"
],
"reserved_task_types": [],
"row_id_column": "row_id",
"scorer_contract": "per_row",
"task_types": []
}
gdi_image_v1
Unified image golden dataset schema v1.
Image task differences are carried by task_type rather than by separate
schema names. image_path and image are alternatives; at least one
must be present and non-null for every row.
Schema: gdi_image_v1 (family: image)
Description: Image datasets. task_type selects the task-specific label validation.
Preview mode: image_gallery
Task types: classification, detection, instance_segmentation, semantic_segmentation
Required columns: image_id, label, task_type
Optional columns:
image — Embedded image bytes or dict with 'bytes' key; null when image_path is set
image_path — Relative or absolute path to the image file; null when image bytes are embedded
metadata — Per-row metadata dict (capture conditions, source, etc.)
scenario — Label or scenario category for stratified analysis
Scorer contract: full_dataset (also accepts: per_row)
Extra columns: preserved as-is (strict=False)
{
"assignable_to_project": true,
"column_role_hints": {
"image": "image",
"image_id": "identifier",
"image_path": "image",
"label": "annotation",
"metadata": "unknown",
"scenario": "category",
"task_type": "category"
},
"description": "Image datasets. task_type selects the task-specific label validation.",
"extra_scorer_contracts": [
"per_row"
],
"family": "image",
"name": "gdi_image_v1",
"optional_column_descriptions": {
"image": "Embedded image bytes or dict with 'bytes' key; null when image_path is set",
"image_path": "Relative or absolute path to the image file; null when image bytes are embedded",
"metadata": "Per-row metadata dict (capture conditions, source, etc.)",
"scenario": "Label or scenario category for stratified analysis"
},
"optional_columns": [
"image",
"image_path",
"metadata",
"scenario"
],
"preview_mode": "image_gallery",
"required_columns": [
"image_id",
"label",
"task_type"
],
"reserved_task_types": [],
"row_id_column": "image_id",
"scorer_contract": "full_dataset",
"task_types": [
"classification",
"detection",
"instance_segmentation",
"semantic_segmentation"
]
}
gdi_text_v1
Golden Dataset Interface — text schema v1.
Covers LLM, RAG, and VLM evaluation in a single unified schema.
input_id, prompt and task_type are required; all other columns
are optional at the column level.
task_type is the authoritative task-flavour discriminator (see
TEXT_TASK_TYPES). Optional column presence is a suggestion on upload,
not authoritative routing:
- retrieved_context → suggests RAG (single_turn_rag)
- images → VLM scorers
Column-level optionality (column may be absent from the DataFrame) is expressed via Optional[Series[T]]. Row-level nullability (individual cells may be None/NaN) is expressed via nullable=True.
Schema: gdi_text_v1 (family: text)
Description: LLM, RAG, and VLM evaluation in a single unified schema. task_type is the authoritative task-flavour discriminator; column presence (retrieved_context → RAG, images → VLM) is a suggestion on upload, not authoritative routing.
Preview mode: table
Task types: single_turn_llm, single_turn_rag
Reserved task types (declared, not yet accepted): multi_turn_llm, multi_turn_rag
Required columns: input_id, prompt, task_type
Optional columns:
expected_output — Reference answer used by scorer ops; activates answered scorers (bleu, rouge, correctness, exact_match)
extra_params — Reserved: per-row SUT override parameters — dict per row. Accepted and stored but NOT yet applied at SUT-call time (the call uses run-level model_params only); a populated cell currently raises a warning at run launch.
images — Vision inputs — list[str] of base64 data URIs per row; column presence activates VLM scorers
retrieved_context — RAG context chunks — list[str] per row; column presence activates RAG scorers (faithfulness, context_precision, context_recall)
scenario — Label or scenario category for stratified analysis
system_prompt — Optional system instruction prepended to the prompt
Scorer contract: per_row (also accepts: full_dataset)
Extra columns: preserved as-is (strict=False)
{
"assignable_to_project": true,
"column_role_hints": {
"expected_output": "text",
"extra_params": "unknown",
"images": "image",
"input_id": "identifier",
"prompt": "text",
"retrieved_context": "text",
"scenario": "category",
"system_prompt": "text",
"task_type": "category"
},
"description": "LLM, RAG, and VLM evaluation in a single unified schema. task_type is the authoritative task-flavour discriminator; column presence (retrieved_context \u2192 RAG, images \u2192 VLM) is a suggestion on upload, not authoritative routing.",
"extra_scorer_contracts": [
"full_dataset"
],
"family": "text",
"name": "gdi_text_v1",
"optional_column_descriptions": {
"expected_output": "Reference answer used by scorer ops; activates answered scorers (bleu, rouge, correctness, exact_match)",
"extra_params": "Reserved: per-row SUT override parameters \u2014 dict per row. Accepted and stored but NOT yet applied at SUT-call time (the call uses run-level model_params only); a populated cell currently raises a warning at run launch.",
"images": "Vision inputs \u2014 list[str] of base64 data URIs per row; column presence activates VLM scorers",
"retrieved_context": "RAG context chunks \u2014 list[str] per row; column presence activates RAG scorers (faithfulness, context_precision, context_recall)",
"scenario": "Label or scenario category for stratified analysis",
"system_prompt": "Optional system instruction prepended to the prompt"
},
"optional_columns": [
"expected_output",
"extra_params",
"images",
"retrieved_context",
"scenario",
"system_prompt"
],
"preview_mode": "table",
"required_columns": [
"input_id",
"prompt",
"task_type"
],
"reserved_task_types": [
"multi_turn_llm",
"multi_turn_rag"
],
"row_id_column": "input_id",
"scorer_contract": "per_row",
"task_types": [
"single_turn_llm",
"single_turn_rag"
]
}
Configuration JSON Schemas
EvalConfig
{
"$defs": {
"EvalTarget": {
"additionalProperties": false,
"description": "One evaluation target in a v2 config: where metrics run, plus what runs there.\n\n``partition_type`` is the granularity to score at \u2014 ``\"session\"``, ``\"trace\"`` or\n``\"span\"``. ``partition_id`` names a saved partition selector, when the evaluation\nscores one. At least one of the two must be set on a target you author;\n:meth:`EvalConfig.resolved_targets` also uses this model for the untargeted view of\na v1 config, where both are None. ``span_kind`` narrows a span target to a single\nkind (``\"TOOL\"``, ``\"LLM\"``, ``\"AGENT\"``, \u2026) and is only valid when\n``partition_type`` is ``\"span\"``.\n\n``metrics`` and ``metric_configs`` keep their exact v1 shapes \u2014 a flat name list,\nand a mapping keyed by bare metric name whose values stay free-form. Only the\nnesting under a target is new.\n\nInstances are frozen: a target is a value object, and allowing assignment would\nlet a caller set ``span_kind`` on a non-span target after validation had already\npassed.",
"properties": {
"metric_configs": {
"anyOf": [
{
"additionalProperties": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"items": {
"additionalProperties": true,
"type": "object"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Metric Configs"
},
"metrics": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Metrics"
},
"partition_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Partition Id"
},
"partition_type": {
"anyOf": [
{
"$ref": "#/$defs/PartitionType"
},
{
"type": "null"
}
],
"default": null
},
"span_kind": {
"anyOf": [
{
"$ref": "#/$defs/SpanKind"
},
{
"type": "null"
}
],
"default": null
}
},
"title": "EvalTarget",
"type": "object"
},
"PartitionType": {
"description": "The evaluation-target shape a trace ``PartitionSelector`` resolves to.\n\n``SESSION`` \u2014 one occurrence per session, pooling every trace it contains.\n``TRACE`` \u2014 one occurrence per trace.\n``SPAN`` \u2014 one occurrence per span (optionally narrowed to a given kind).",
"enum": [
"session",
"trace",
"span"
],
"title": "PartitionType",
"type": "string"
},
"SpanKind": {
"description": "OpenInference's 11-value superset; the thinner gen_ai taxonomy maps INTO this.",
"enum": [
"LLM",
"CHAIN",
"TOOL",
"RETRIEVER",
"EMBEDDING",
"AGENT",
"RERANKER",
"GUARDRAIL",
"EVALUATOR",
"PROMPT",
"PLAN",
"RETRY",
"UNKNOWN"
],
"title": "SpanKind",
"type": "string"
}
},
"description": "Typed evaluation configuration consumed by aip.run() and YAML config files.\n\n``version`` is a schema version guard. Two shapes are supported, and each version\naccepts exactly one of them:\n\n* **v1** \u2014 a flat, untargeted selection: ``metrics`` is a ``list[str]`` and\n ``metric_configs`` is keyed by bare metric name. This is every config written\n before the partition axis existed; it keeps loading and validating unchanged,\n and ``metrics`` stays a ``list[str]`` rather than being rewritten into the v2\n shape.\n* **v2** \u2014 a ``targets`` list, each entry pairing a partition (see\n :class:`EvalTarget`) with the metrics to run against it. Trace evaluation is\n inherently per-``(target, metric)``, which a flat list cannot express.\n\nMixing the two is rejected rather than merged: a v1 config carrying ``targets``,\nor a v2 config carrying top-level ``metrics``, is ambiguous about which selection\nwins. Use :meth:`resolved_targets` to read either version through one code path.\n\n``parameters`` and ``thresholds`` are accepted but not forwarded to run(), for\neither version \u2014 they exist so configs written today stay valid when those features\nland, and ``run()`` has no parameter to carry them to. ``thresholds`` is keyed by\nbare metric name and carries no target axis, so a v2 config may not set a threshold\nfor a metric that more than one target could claim; see\n:meth:`_thresholds_must_name_one_target`.\n\n``metric_config_refs`` pins a metric to a stored, versioned ``MetricConfig`` by its own\n``(config_name, config_version)`` identity \u2014 distinct from ``metric_configs``' inline,\nunversioned params. It stays top-level for both shapes (unlike ``metrics``/\n``metric_configs``, which move under ``targets`` for v2/v3): which stored config a\nmetric uses is a property of the metric, not of any one target it runs under.",
"properties": {
"connection_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Connection Id"
},
"dataset": {
"title": "Dataset",
"type": "string"
},
"judge_connection_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Judge Connection Id"
},
"metric_config_refs": {
"anyOf": [
{
"additionalProperties": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"items": {
"additionalProperties": true,
"type": "object"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Metric Config Refs"
},
"metric_configs": {
"anyOf": [
{
"additionalProperties": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"items": {
"additionalProperties": true,
"type": "object"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Metric Configs"
},
"metrics": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Metrics"
},
"parameters": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Parameters"
},
"project": {
"title": "Project",
"type": "string"
},
"sut_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Sut Id"
},
"tags": {
"anyOf": [
{
"additionalProperties": {
"type": "string"
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Tags"
},
"targets": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/EvalTarget"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Targets"
},
"thresholds": {
"anyOf": [
{
"additionalProperties": {
"type": "number"
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Thresholds"
},
"version": {
"default": "1",
"title": "Version",
"type": "string"
}
},
"required": [
"project",
"dataset"
],
"title": "EvalConfig",
"type": "object"
}
OpManifest
{
"$defs": {
"AnyOfKinds": {
"description": "An OR-group inside a trace metric's ``required_kinds``: at least one of these kinds present.",
"properties": {
"anyOf": {
"items": {
"$ref": "#/$defs/SpanKind"
},
"minItems": 1,
"title": "Anyof",
"type": "array"
}
},
"required": [
"anyOf"
],
"title": "AnyOfKinds",
"type": "object"
},
"ClassAttributionContract": {
"additionalProperties": false,
"description": "Machine-readable contract for per-row class attribution details.\n\n``artifact_type`` identifies the companion payload and ``version`` pins its\nschema. Consumers use this declaration to discover support. They must not\ninfer class attribution from a metric name or task type.",
"properties": {
"artifact_type": {
"const": "class_attribution",
"title": "Artifact Type",
"type": "string"
},
"version": {
"const": 1,
"title": "Version",
"type": "integer"
}
},
"required": [
"artifact_type",
"version"
],
"title": "ClassAttributionContract",
"type": "object"
},
"ExecutionHints": {
"description": "Coarse routing and resource hints for Op runtime dispatch.\n\n``processing_kind`` determines which function pool handles the op.\n``timeout_seconds=None`` means no timeout.\n\n``max_concurrency`` has exactly one consumer: ``aip_metrics.score`` reads it to\nsize how many dataset *rows* a single invocation scores at once, and only for the\nmetrics that driver has opted in. Nothing derives a cap on concurrent op\n*invocations* from it \u2014 dispatch parallelism is bounded by the run queue and by a\nfunction's ``numWorkers``, neither of which consults this field. So setting\n``max_concurrency=1`` on a GPU op does not serialise its invocations; it only\nserialises row scoring within one, once that metric is opted in. Set it as a\nrows-per-invocation ceiling and nothing else.\n\n``stream_batch_size`` declares that the op image reads its input in batches of this\nmany rows instead of loading the whole dataset (the ``STREAM_BATCH_SIZE`` its\n``function.yaml`` sets, which is what the op itself reads). ``None`` \u2014 the default \u2014\nis the whole-file path. Declaring it here is what makes an op's streaming visible to\nthe platform without a second list of op names to keep in sync with the images: row\nstaging, for one, only functions on the streaming path, so aip-api reads this off the\nop's registry row to decide which ops can be handed a staging credential.\n\n``function_name`` optionally routes a logical op to a shared runtime function.\n``emits_metric_family`` declares that one invocation of that shared function\nemits score columns for every logical metric routed to it. It defaults to\nfalse because sharing a deployment alone does not guarantee shared output.",
"properties": {
"emits_metric_family": {
"default": false,
"title": "Emits Metric Family",
"type": "boolean"
},
"function_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Function Name"
},
"max_concurrency": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Max Concurrency"
},
"processing_kind": {
"$ref": "#/$defs/ProcessingKind",
"default": "cpu"
},
"stream_batch_size": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Stream Batch Size"
},
"timeout_seconds": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Timeout Seconds"
}
},
"title": "ExecutionHints",
"type": "object"
},
"GdiSchemaRef": {
"additionalProperties": false,
"description": "Reference to a registered GDI schema, optionally narrowed to some of its task types.\n\nExample: {\"name\": \"gdi_image_v1\", \"task_types\": [\"detection\"]}\n\n``task_types`` exists because a schema name stopped being a sufficient compatibility\nkey once ``gdi_image_v1`` consolidated four tasks under one name (ADR 2026-05-22):\nan object-detection metric and a classification metric declare byte-identical\n``accepts`` and ``required_columns``, so ``accepts`` alone cannot tell them apart.\nNested here rather than a sibling field on the op because a task type is only\nmeaningful relative to the schema that defines it.\n\nEmpty means every task type of that schema \u2014 the backward-compatible default, and the\nright answer for ops that genuinely span tasks (``llm.*`` over both text task types).",
"properties": {
"name": {
"title": "Name",
"type": "string"
},
"task_types": {
"default": [],
"items": {
"$ref": "#/$defs/TaskType"
},
"title": "Task Types",
"type": "array"
}
},
"required": [
"name"
],
"title": "GdiSchemaRef",
"type": "object"
},
"MetricDirection": {
"description": "Intrinsic semantic of a metric's raw score \u2014 which way is \"good\".\n\n``HIGHER_IS_BETTER`` \u2014 a larger raw value is the better outcome (accuracy, recall,\nhelpfulness).\n\n``LOWER_IS_BETTER`` \u2014 a smaller raw value is the better outcome (toxicity, bias, an\nerror or miss rate). Such a metric is normalised with ``inverted=True`` so its\n*normalised* score still runs higher-is-better; ``direction`` records the intrinsic\nraw semantic so surfaces can label it without special-casing by name.\n\n``StrEnum`` so it serialises as a plain string in API responses and YAML run configs.",
"enum": [
"higher_is_better",
"lower_is_better"
],
"title": "MetricDirection",
"type": "string"
},
"MetricMetadata": {
"additionalProperties": false,
"description": "Structured, human-facing catalogue metadata for an op.\n\nCarries the content dimensions the picker UI renders *separately* from the\nshort ``description`` \u2014 most importantly ``methodology`` (\"how the score is\ncalculated\", as ordered steps), which a tester can expand beneath the\none-line summary. Kept as a dedicated structured field (not packed into\n``description``) so consumers can render each dimension independently.\n\nThis is the type ``OpManifest.metric_metadata`` and ``OpRegistryEntry.metric_metadata``\nactually carry \u2014 every op's catalogue prose flows through *this* model on its way to the\nDB and the API/SDK, regardless of which package registers the op. A kind-specific\nmetadata model (e.g. one scoped to trace metrics) cannot substitute for it here: those\ntwo fields are declared on the shared, kind-agnostic ``OpManifest``/``OpRegistryEntry``,\nand pydantic validates/copies whatever type is actually declared there.\n\nAttributes\n----------\nsummary : str\n One-line \"what it does\", shown at a glance in the catalogue. May repeat\n or refine ``description``; kept distinct so authored catalogue copy is\n not coupled to the terse registry ``description``.\nscore_semantics : str\n What the score *means* \u2014 range, and whether higher or lower is better,\n in prose (complements the machine-readable ``direction``).\nmethodology : list[str]\n Ordered \"how it is calculated\" steps, rendered as a numbered list.\nworked_example : str\n A concrete worked example (inputs \u2192 score) illustrating the metric.\nclass_attribution : ClassAttributionContract | None\n Machine-readable per-row class attribution contract, when supported.\nscore_range : MetricScoreRange | None\n Bounds of the emitted score after normalisation, when declared.",
"properties": {
"class_attribution": {
"anyOf": [
{
"$ref": "#/$defs/ClassAttributionContract"
},
{
"type": "null"
}
],
"default": null
},
"methodology": {
"items": {
"type": "string"
},
"title": "Methodology",
"type": "array"
},
"score_range": {
"anyOf": [
{
"$ref": "#/$defs/MetricScoreRange"
},
{
"type": "null"
}
],
"default": null
},
"score_semantics": {
"default": "",
"title": "Score Semantics",
"type": "string"
},
"summary": {
"default": "",
"title": "Summary",
"type": "string"
},
"worked_example": {
"default": "",
"title": "Worked Example",
"type": "string"
}
},
"title": "MetricMetadata",
"type": "object"
},
"MetricScoreRange": {
"additionalProperties": false,
"description": "Declared bounds of the emitted score, after normalisation.\n\nThese bounds describe stored scores, not the raw inputs accepted by a normaliser.",
"properties": {
"max": {
"title": "Max",
"type": "number"
},
"min": {
"title": "Min",
"type": "number"
}
},
"required": [
"min",
"max"
],
"title": "MetricScoreRange",
"type": "object"
},
"OpKind": {
"description": "Coarse category discriminator for op registry entries.\n\n``TRANSFORM`` \u2014 DataFrame-in, DataFrame-out preprocessing op. Mutates\ndata without producing scores or quality gates. Examples: ``normalise_v2``,\n``dedupe``, ``remove_pii``.\n\n``QUALITY_CHECK`` \u2014 inspects a DatasetVersion and returns a\n``CheckResult`` (PASS/WARN/FAIL). Gates ``raw \u2192 mapped \u2192 golden``\npromotion.\n\n``ANALYSIS`` \u2014 computes dataset diagnostics, profiles, histograms, and\nrow-aligned explanatory values. Does not update quality verdicts or gate\npromotion.\n\n``METRIC`` \u2014 scores run-output Parquet. ``PER_ROW`` ops append score\ncolumns per row; ``FULL_DATASET`` ops receive the entire dataset and\nreturn an aggregated summary via ``OpInvokeResult.summary``. Requires\n``scorer_contract`` to be set on the ``OpRegistryEntry``.\n\n``INFERENCE`` \u2014 calls the SUT for each row in a golden DatasetVersion\nand writes the run-output Parquet. Exactly one inference op per run.\n\n``SDG`` \u2014 synthetic data generation. Produces a new ``DatasetVersion``\nwith ``lineage_parent_id`` pointing to the seed; output is\n``stage=\"raw\"`` and must re-clear quality checks before promotion.\n\n``TRACE_METRIC`` \u2014 scores agent-trace partition occurrences (a whole trace,\na pooled session, or a single span) rather than golden-dataset run output.\n``PER_ROW`` over ``canonical_partition_v1`` rows: one materialized partition\noccurrence = one row. Carries the trace-specific contract fields\n(``partition_types``, ``required_kinds``, ``target_kind``) that ``METRIC``\ndoes not \u2014 see ``OpManifest``.",
"enum": [
"transform",
"quality_check",
"analysis",
"metric",
"inference",
"sdg",
"trace_metric"
],
"title": "OpKind",
"type": "string"
},
"PartitionType": {
"description": "The evaluation-target shape a trace ``PartitionSelector`` resolves to.\n\n``SESSION`` \u2014 one occurrence per session, pooling every trace it contains.\n``TRACE`` \u2014 one occurrence per trace.\n``SPAN`` \u2014 one occurrence per span (optionally narrowed to a given kind).",
"enum": [
"session",
"trace",
"span"
],
"title": "PartitionType",
"type": "string"
},
"ProcessingKind": {
"description": "Execution routing hint for Op runtime dispatch.\n\n``CPU`` \u2014 standard CPU-only workload; dispatched to the default function\npool. Suitable for token-based metrics (BLEU, ROUGE) and most checks.\n\n``GPU`` \u2014 requires GPU access; dispatched to a GPU-pool function variant.\nSuitable for embedding-based and vision metrics (BERTScore, VLM scorers).\n\n``NETWORK`` \u2014 op makes outbound LLM-judge calls; primary cost is latency,\nnot compute. Suitable for LLM-as-judge metrics (faithfulness, correctness).",
"enum": [
"cpu",
"gpu",
"network"
],
"title": "ProcessingKind",
"type": "string"
},
"ScorerContract": {
"description": "How a scorer iterates over rows for a given schema family \u2014 an execution-strategy\ndispatch key, nothing else. A metric's implementation maturity (stub vs. real) is not\npart of this vocabulary: an unfinished metric simply is not registered, rather than\nbeing registered under a contract value that isn't actually an iteration strategy.\n\n``PER_ROW`` \u2014 one call per item, dispatched sequentially/concurrently over a collection:\none (test_case, response) pair for LLM/RAG/VLM scoring, or one resolved partition\noccurrence (session/trace/span) for trace scoring \u2014 same dispatch shape either way.\n\n``FULL_DATASET`` \u2014 accumulate all rows then compute once (CV metrics: MeanIoU, mAP, \u2026).",
"enum": [
"per_row",
"full_dataset"
],
"title": "ScorerContract",
"type": "string"
},
"SpanKind": {
"description": "OpenInference's 11-value superset; the thinner gen_ai taxonomy maps INTO this.",
"enum": [
"LLM",
"CHAIN",
"TOOL",
"RETRIEVER",
"EMBEDDING",
"AGENT",
"RERANKER",
"GUARDRAIL",
"EVALUATOR",
"PROMPT",
"PLAN",
"RETRY",
"UNKNOWN"
],
"title": "SpanKind",
"type": "string"
},
"TaskType": {
"description": "The task a dataset's rows represent \u2014 the discriminator a Project/Dataset carries.\n\nImage (``gdi_image_v1``) \u2014 ``CLASSIFICATION``, ``DETECTION``, ``SEMANTIC_SEGMENTATION``,\n``INSTANCE_SEGMENTATION``. One schema covers all four (ADR 2026-05-22), so the task type,\nnot the schema name, is what distinguishes an object-detection metric from a\nclassification one.\n\nText (``gdi_text_v1``) \u2014 ``SINGLE_TURN_LLM`` and ``SINGLE_TURN_RAG`` are usable.\n``MULTI_TURN_LLM`` and ``MULTI_TURN_RAG`` are reserved until the canonical multi-turn\n``messages`` column lands.\n\n``StrEnum`` so it serialises as a plain string everywhere it is already stored as one \u2014\nAPI responses, YAML manifests, and the ``str(64)`` ``task_type`` columns.",
"enum": [
"classification",
"detection",
"semantic_segmentation",
"instance_segmentation",
"single_turn_llm",
"single_turn_rag",
"multi_turn_llm",
"multi_turn_rag"
],
"title": "TaskType",
"type": "string"
},
"TraceShape": {
"description": "A structural pattern in a trace's span tree \u2014 the vocabulary a trace metric names in\n``unsupported_trace_shapes`` when its scoring logic cannot yet handle that topology.\n\nDistinct from ``required_kinds``: a kind requirement asks \"is a span of this kind present\nanywhere\", which cannot express a *relationship* between spans (root vs. descendant, how\nmany of a kind). A shape names that relationship instead, so a metric that only breaks on a\nspecific topology \u2014 not on the kind's mere presence \u2014 can declare exactly that, rather than\na kind constraint that would also exclude traces it scores fine.\n\n``ORCHESTRATOR_SUBAGENT`` \u2014 a single-root trace whose root has one or more non-root ``AGENT``\ndescendants (an orchestrator delegating to subagent(s)). Correctly attributing the final\nanswer to the right agent, with each agent's own tool calls grouped separately, needs\nper-agent extraction a metric may not implement \u2014 see the Partition-Metric Reconciliation\ndoc's \"Case 4\" for the motivating example (``agent.hallucination``'s TRACE path).",
"enum": [
"orchestrator_subagent"
],
"title": "TraceShape",
"type": "string"
}
},
"description": "Declarative registration manifest for one op.\n\nAttributes\n----------\nmanifest_version : str\n Schema version of the manifest *format* (the version guard). Distinct from\n ``version`` (the op's own semver). Only ``\"1\"`` is supported today.\nname : str\n Op identifier (e.g. ``\"llm.demo_always_pass\"``). Maps to ``OpRegistryEntry.name``.\nversion : str\n The op's semantic version (e.g. ``\"1.0.0\"``). Maps to ``OpRegistryEntry.version``.\nkind : OpKind\n Op kind discriminator. Selects the kind-conditional validation branch.\nlifecycle : Literal[\"available\", \"planned\", \"deprecated\"]\n Authored availability declaration, checked by the documentation build\n against the deployed ops catalogue. Defaults to ``\"available\"``.\ndescription : str\n Human-readable description (shown in the UI catalogue).\ndisplay_name : str\n Short presentation label (e.g. ``\"Answer Correctness\"``). Metric presentation\n metadata read by the report generator; not an ``OpRegistryEntry`` field, so it\n is dropped by :meth:`to_registry_entry`.\nmetric_type : str\n Metric case-structure tag (e.g. ``\"pointwise\"``). Presentation metadata; like\n ``display_name`` it is not part of ``OpRegistryEntry``.\naccepts : list[GdiSchemaRef]\n GDI schemas this op accepts. Must be non-empty.\nrequired_columns : list[str]\n Columns required within the accepted schemas.\nexecution : ExecutionHints\n Execution routing hints (processing_kind, timeout, max_concurrency).\nconfig_schema : dict\n JSON Schema for op configuration; empty means no configuration.\nmetric_metadata : MetricMetadata\n Structured, human-facing catalogue metadata (summary, score semantics,\n methodology steps, worked example). Authored here in the manifest and\n carried through to ``OpRegistryEntry`` so the picker UI can render the\n dimensions separately from ``description``.\nentrypoint : str\n ``\"module\"`` or ``\"module:attribute\"`` reference to the op's Python\n implementation. Resolved (importlib, folder-relative fallback) at register\n time so a bad reference fails before any deploy/register. Runtime is Python\n for v1; non-Python entrypoints are a future extension.\ndependencies : list[str]\n Manifest-relative paths to dependency files that must exist alongside the\n manifest (e.g. ``[\"pyproject.toml\", \"uv.lock\"]``). Existence is checked at\n register time.\nscorer_contract : ScorerContract | None\n Required when ``kind == metric`` or ``kind == trace_metric``; ``None`` otherwise.\ndirection : MetricDirection | None\n Required when ``kind == metric`` or ``kind == trace_metric``; ``None`` otherwise.\npartition_types : list[PartitionType]\n Trace-metric only: the partition granularities the op supports \u2014 any of\n ``PartitionType.TRACE``/``SESSION``/``SPAN``. Must be non-empty for a ``trace_metric``\n accepting ``canonical_partition_v1``. Empty for other kinds.\nrequired_kinds : list[SpanKind | AnyOfKinds]\n Trace-metric only: the span kinds (Gate A) that must be present in an occurrence\n for the metric to run (e.g. ``[SpanKind.TOOL]``). A bare ``SpanKind`` is an AND\n requirement; an ``AnyOfKinds`` entry (``{\"any_of\": [...]}`` in YAML) is an OR-group,\n which is how a metric whose partition types need different kinds states its Gate A\n requirement without demanding all of them. Empty for other kinds.\ntarget_kind : SpanKind | None\n Trace-metric only: for a ``PartitionType.SPAN`` partition type, the span kind whose\n occurrences are scored (e.g. ``SpanKind.AGENT``). Must be ``None`` unless\n ``partition_types`` includes ``PartitionType.SPAN``.\nunsupported_trace_shapes : list[TraceShape]\n Trace-metric only: trace topologies (e.g. ``TraceShape.ORCHESTRATOR_SUBAGENT``) this\n op's TRACE scoring logic does not support. Must be empty unless ``partition_types``\n includes ``PartitionType.TRACE``. Describes only the TRACE scorer's limitation \u2014 a\n metric that also declares ``PartitionType.SESSION`` is not implying its (separate)\n session scorer shares it.\nmax_prompt_slots : int | None\n Number of named prompt slots a ``MetricConfig`` may pin for this op. ``0`` by\n default; an op must opt in to accept a pinned prompt. ``None`` means unlimited.\n\nNotes\n-----\n``manifest_version``, ``lifecycle``, ``entrypoint``, and ``dependencies`` are manifest-only\nauthoring/validation concerns and are **not** part of ``OpRegistryEntry`` \u2014 the\ndeployed bundle carries the actual code, so they are dropped by\n:meth:`to_registry_entry`.",
"properties": {
"accepts": {
"items": {
"$ref": "#/$defs/GdiSchemaRef"
},
"minItems": 1,
"title": "Accepts",
"type": "array"
},
"config_schema": {
"additionalProperties": true,
"title": "Config Schema",
"type": "object"
},
"dependencies": {
"items": {
"type": "string"
},
"title": "Dependencies",
"type": "array"
},
"description": {
"default": "",
"title": "Description",
"type": "string"
},
"direction": {
"anyOf": [
{
"$ref": "#/$defs/MetricDirection"
},
{
"type": "null"
}
],
"default": null
},
"display_name": {
"default": "",
"title": "Display Name",
"type": "string"
},
"entrypoint": {
"title": "Entrypoint",
"type": "string"
},
"execution": {
"$ref": "#/$defs/ExecutionHints"
},
"kind": {
"$ref": "#/$defs/OpKind"
},
"lifecycle": {
"default": "available",
"enum": [
"available",
"planned",
"deprecated"
],
"title": "Lifecycle",
"type": "string"
},
"manifest_version": {
"default": "1",
"title": "Manifest Version",
"type": "string"
},
"max_prompt_slots": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"default": 0,
"title": "Max Prompt Slots"
},
"metric_metadata": {
"$ref": "#/$defs/MetricMetadata"
},
"metric_type": {
"default": "",
"title": "Metric Type",
"type": "string"
},
"name": {
"title": "Name",
"type": "string"
},
"partition_types": {
"items": {
"$ref": "#/$defs/PartitionType"
},
"title": "Partition Types",
"type": "array"
},
"required_columns": {
"items": {
"type": "string"
},
"title": "Required Columns",
"type": "array"
},
"required_kinds": {
"items": {
"anyOf": [
{
"$ref": "#/$defs/SpanKind"
},
{
"$ref": "#/$defs/AnyOfKinds"
}
]
},
"title": "Required Kinds",
"type": "array"
},
"scorer_contract": {
"anyOf": [
{
"$ref": "#/$defs/ScorerContract"
},
{
"type": "null"
}
],
"default": null
},
"target_kind": {
"anyOf": [
{
"$ref": "#/$defs/SpanKind"
},
{
"type": "null"
}
],
"default": null
},
"unsupported_trace_shapes": {
"items": {
"$ref": "#/$defs/TraceShape"
},
"title": "Unsupported Trace Shapes",
"type": "array"
},
"version": {
"title": "Version",
"type": "string"
}
},
"required": [
"name",
"version",
"kind",
"accepts",
"entrypoint"
],
"title": "OpManifest",
"type": "object"
}
SUTConnection
{
"$defs": {
"AuthType": {
"description": "Authentication method used for outbound SUT requests.\n\nMaps directly to SUT proxy auth behavior:\n- NONE \u2192 no authentication header\n- BEARER \u2192 Authorization bearer token\n- API_KEY \u2192 API key sent in a caller-chosen header\n- BASIC \u2192 declared in contract, not proxied yet",
"enum": [
"none",
"bearer",
"api_key",
"basic"
],
"title": "AuthType",
"type": "string"
},
"HFDetectionRequestTemplate": {
"description": "Request template for Hugging Face object-detection endpoints.",
"properties": {
"image_field": {
"default": "binary",
"enum": [
"binary",
"url"
],
"title": "Image Field",
"type": "string"
},
"kind": {
"const": "hf_detection",
"default": "hf_detection",
"title": "Kind",
"type": "string"
},
"threshold": {
"default": 0.5,
"maximum": 1.0,
"minimum": 0.0,
"title": "Threshold",
"type": "number"
}
},
"title": "HFDetectionRequestTemplate",
"type": "object"
},
"HFDetectionResponseMapper": {
"description": "Response mapper for Hugging Face object-detection endpoints.\n\nBoxes at ``box_path`` are read as canonical ``[x1, y1, x2, y2]`` corners,\nthe one form the platform stores and scores. An endpoint answering in COCO\n``[x, y, w, h]`` needs converting before its predictions are recorded.",
"properties": {
"box_path": {
"default": "box",
"title": "Box Path",
"type": "string"
},
"confidence_path": {
"default": "score",
"title": "Confidence Path",
"type": "string"
},
"detections_path": {
"default": "$",
"title": "Detections Path",
"type": "string"
},
"kind": {
"const": "hf_detection",
"default": "hf_detection",
"title": "Kind",
"type": "string"
},
"label_path": {
"default": "label",
"title": "Label Path",
"type": "string"
}
},
"title": "HFDetectionResponseMapper",
"type": "object"
},
"HFRAGRequestTemplate": {
"description": "Request template for Hugging Face RAG endpoints.",
"properties": {
"context_field": {
"default": "retrieved_context",
"title": "Context Field",
"type": "string"
},
"kind": {
"const": "hf_rag",
"default": "hf_rag",
"title": "Kind",
"type": "string"
},
"parameters": {
"additionalProperties": true,
"title": "Parameters",
"type": "object"
},
"prompt_field": {
"default": "prompt",
"title": "Prompt Field",
"type": "string"
}
},
"title": "HFRAGRequestTemplate",
"type": "object"
},
"HFRAGResponseMapper": {
"description": "Response mapper for Hugging Face RAG endpoints.",
"properties": {
"answer_path": {
"default": "$[0].generated_text",
"title": "Answer Path",
"type": "string"
},
"context_path": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Context Path"
},
"kind": {
"const": "hf_rag",
"default": "hf_rag",
"title": "Kind",
"type": "string"
}
},
"title": "HFRAGResponseMapper",
"type": "object"
},
"HFTextGenerationRequestTemplate": {
"description": "Request template for Hugging Face text-generation endpoints.",
"properties": {
"inputs_field": {
"default": "prompt",
"title": "Inputs Field",
"type": "string"
},
"kind": {
"const": "hf_text_generation",
"default": "hf_text_generation",
"title": "Kind",
"type": "string"
},
"parameters": {
"additionalProperties": true,
"title": "Parameters",
"type": "object"
}
},
"title": "HFTextGenerationRequestTemplate",
"type": "object"
},
"HFTextGenerationResponseMapper": {
"description": "Response mapper for Hugging Face text-generation endpoints.",
"properties": {
"kind": {
"const": "hf_text_generation",
"default": "hf_text_generation",
"title": "Kind",
"type": "string"
},
"response_path": {
"default": "$[0].generated_text",
"title": "Response Path",
"type": "string"
}
},
"title": "HFTextGenerationResponseMapper",
"type": "object"
}
},
"description": "Endpoint configuration for invoking a registered SUT.",
"properties": {
"auth_header_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Auth Header Name"
},
"auth_header_value_enc": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Auth Header Value Enc"
},
"auth_type": {
"$ref": "#/$defs/AuthType",
"default": "none"
},
"base_url": {
"title": "Base Url",
"type": "string"
},
"gdi_schema": {
"title": "Gdi Schema",
"type": "string"
},
"id": {
"title": "Id",
"type": "string"
},
"label": {
"default": "default",
"enum": [
"prod",
"staging",
"dev",
"default"
],
"title": "Label",
"type": "string"
},
"request_template": {
"discriminator": {
"mapping": {
"hf_detection": "#/$defs/HFDetectionRequestTemplate",
"hf_rag": "#/$defs/HFRAGRequestTemplate",
"hf_text_generation": "#/$defs/HFTextGenerationRequestTemplate"
},
"propertyName": "kind"
},
"oneOf": [
{
"$ref": "#/$defs/HFTextGenerationRequestTemplate"
},
{
"$ref": "#/$defs/HFRAGRequestTemplate"
},
{
"$ref": "#/$defs/HFDetectionRequestTemplate"
}
],
"title": "Request Template"
},
"response_mapper": {
"discriminator": {
"mapping": {
"hf_detection": "#/$defs/HFDetectionResponseMapper",
"hf_rag": "#/$defs/HFRAGResponseMapper",
"hf_text_generation": "#/$defs/HFTextGenerationResponseMapper"
},
"propertyName": "kind"
},
"oneOf": [
{
"$ref": "#/$defs/HFTextGenerationResponseMapper"
},
{
"$ref": "#/$defs/HFRAGResponseMapper"
},
{
"$ref": "#/$defs/HFDetectionResponseMapper"
}
],
"title": "Response Mapper"
},
"sut_id": {
"title": "Sut Id",
"type": "string"
}
},
"required": [
"id",
"sut_id",
"base_url",
"gdi_schema",
"request_template",
"response_mapper"
],
"title": "SUTConnection",
"type": "object"
}
TestPlan
{
"$defs": {
"DimensionSpec": {
"additionalProperties": false,
"description": "A slice of the dataset the methodology requires: a column plus the values expected in it.\n\n``vocabulary`` is required but may be empty, so an author declares the values the methodology\nexpects even when that declaration is \"none\".\n\n``column`` cannot be checked here \u2014 GDI schemas are ``strict=False``, so it may name a\ndeclared optional column or one only this methodology's data carries. Its existence is\nverified against a real dataframe at promotion.",
"properties": {
"column": {
"maxLength": 64,
"minLength": 1,
"title": "Column",
"type": "string"
},
"name": {
"maxLength": 64,
"minLength": 1,
"title": "Name",
"type": "string"
},
"required": {
"default": false,
"title": "Required",
"type": "boolean"
},
"vocabulary": {
"items": {
"type": "string"
},
"title": "Vocabulary",
"type": "array"
}
},
"required": [
"name",
"column",
"vocabulary"
],
"title": "DimensionSpec",
"type": "object"
},
"MetricConfigRef": {
"additionalProperties": false,
"description": "A stored, versioned ``MetricConfig`` a metric runs with, named by that config's own identity.\n\n``(config_name, config_version)`` is the storage layer's key for a config row \u2014 independent of\nthe metric it configures, so ``toxicity-strict`` and ``toxicity-lenient`` can both configure\n``llm.toxicity`` and either is pinnable. Resolved within one workspace, which owns every stored\nconfig and is why no scope is named here. Lengths\nmatch ``MetricConfigRecord.config_name``/``config_version``, so what passes here is what the\nlookup could have stored. ``frozen`` to stay hashable, as the dataclass this replaced was.\n\nThe pin names a version; it does not freeze one. A stored config's ``params`` and ``prompts``\ncan be rewritten in place at the same key until some run has *scored* against it, so bump\n``config_version`` for a content change wherever reproducibility is the point of pinning.",
"properties": {
"config_name": {
"maxLength": 255,
"minLength": 1,
"title": "Config Name",
"type": "string"
},
"config_version": {
"maxLength": 64,
"minLength": 1,
"title": "Config Version",
"type": "string"
}
},
"required": [
"config_name",
"config_version"
],
"title": "MetricConfigRef",
"type": "object"
},
"MetricDirection": {
"description": "Intrinsic semantic of a metric's raw score \u2014 which way is \"good\".\n\n``HIGHER_IS_BETTER`` \u2014 a larger raw value is the better outcome (accuracy, recall,\nhelpfulness).\n\n``LOWER_IS_BETTER`` \u2014 a smaller raw value is the better outcome (toxicity, bias, an\nerror or miss rate). Such a metric is normalised with ``inverted=True`` so its\n*normalised* score still runs higher-is-better; ``direction`` records the intrinsic\nraw semantic so surfaces can label it without special-casing by name.\n\n``StrEnum`` so it serialises as a plain string in API responses and YAML run configs.",
"enum": [
"higher_is_better",
"lower_is_better"
],
"title": "MetricDirection",
"type": "string"
},
"MetricSpec": {
"additionalProperties": false,
"description": "What one metric contributes to the methodology: its bar, the params it runs with, and what\nthe platform recorded it resolving to when the plan was published.\n\nGrouped per metric rather than split into sibling ``thresholds``/``metric_configs`` mappings,\nso a bar for a metric the plan does not name is unrepresentable rather than needing a\ncross-mapping key check.\n\nEvery field is optional: no ``threshold`` grades against the metric's registered score card,\nno ``config`` runs on its op's declared defaults, and no ``config_ref`` pins no stored config.\n\n``config`` and ``config_ref`` layer rather than compete, and naming both is allowed because a\nrun can express the same thing: the stored config's params apply first and an inline\n``config`` key wins over them. The trade-off is per key \u2014 an inline key that *overrides* a\nstored one costs the run's config identity stamp (it records ``metric_config_overridden``\ninstead), so keep the two disjoint where the point of pinning was traceability.",
"properties": {
"config": {
"additionalProperties": true,
"title": "Config",
"type": "object"
},
"config_ref": {
"anyOf": [
{
"$ref": "#/$defs/MetricConfigRef"
},
{
"type": "null"
}
],
"default": null,
"description": "A stored, versioned MetricConfig this metric runs with, pinned by its own `(config_name, config_version)`. Unlike the anonymous kwargs in `config`, a pinned config stamps its identity onto every result the run writes, and can carry judge prompts. One config per metric. Layers with `config`, which wins per key \u2014 at the cost of that identity stamp for any key it overrides. The pin names a version rather than freezing one: a stored config's params and prompts stay editable in place until a run has scored against it, so bump `config_version` for a content change if this plan's scoring has to stay reproducible."
},
"published_required_columns": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"description": "Record, not instruction: the columns this metric required at publish, sorted. Platform-assigned like `published_version`. What a dataset needs *now* is the response's live `required_columns`, never this \u2014 the two answer different questions, and a difference between them is drift. `[]` means the metric genuinely required no columns; `null` means this instance has not been through the publish route.",
"title": "Published Required Columns"
},
"published_version": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Record, not instruction: the version of the metric *implementation* this metric resolved to at publish \u2014 its `OpRegistry.version`, not this plan's own `version`. Platform-assigned, and assigned the same way `version` is: a value supplied here is ignored, and recomputed on every publish. Nothing resolves the metric by it \u2014 a run still resolves by name \u2014 so this is what makes a later mismatch reportable as platform drift rather than as an authoring error.",
"title": "Published Version"
},
"threshold": {
"anyOf": [
{
"ge": 0.0,
"le": 1.0,
"type": "number"
},
{
"$ref": "#/$defs/ThresholdBand"
},
{
"type": "null"
}
],
"default": null,
"title": "Threshold"
}
},
"title": "MetricSpec",
"type": "object"
},
"TaskType": {
"description": "The task a dataset's rows represent \u2014 the discriminator a Project/Dataset carries.\n\nImage (``gdi_image_v1``) \u2014 ``CLASSIFICATION``, ``DETECTION``, ``SEMANTIC_SEGMENTATION``,\n``INSTANCE_SEGMENTATION``. One schema covers all four (ADR 2026-05-22), so the task type,\nnot the schema name, is what distinguishes an object-detection metric from a\nclassification one.\n\nText (``gdi_text_v1``) \u2014 ``SINGLE_TURN_LLM`` and ``SINGLE_TURN_RAG`` are usable.\n``MULTI_TURN_LLM`` and ``MULTI_TURN_RAG`` are reserved until the canonical multi-turn\n``messages`` column lands.\n\n``StrEnum`` so it serialises as a plain string everywhere it is already stored as one \u2014\nAPI responses, YAML manifests, and the ``str(64)`` ``task_type`` columns.",
"enum": [
"classification",
"detection",
"semantic_segmentation",
"instance_segmentation",
"single_turn_llm",
"single_turn_rag",
"multi_turn_llm",
"multi_turn_rag"
],
"title": "TaskType",
"type": "string"
},
"ThresholdBand": {
"additionalProperties": false,
"description": "The object form of :attr:`MetricSpec.threshold`: the four keys score-card resolution reads.\n\n``pass``, ``warn``, ``direction`` and ``inverted`` are all optional \u2014 each merges onto the\nmetric's registered card, leaving an unset key's value untouched. A bare float in\n``threshold`` is shorthand for setting ``pass`` alone.\n\n``extra=\"forbid\"`` because a typo'd key (``{\"passs\": 0.9}``) would otherwise resolve to the\ndefault card and let the run score cleanly against a bar the author never set.\n\nAn unset key dumps as ``null`` rather than being pruned \u2014 dump with ``exclude_none=True`` to\ndrop it. A ``model_serializer`` is deliberately not used, as it would collapse this class to\nan untyped object in FastAPI's JSON Schema.",
"properties": {
"direction": {
"anyOf": [
{
"$ref": "#/$defs/MetricDirection"
},
{
"type": "null"
}
],
"default": null
},
"inverted": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"default": null,
"title": "Inverted"
},
"pass": {
"anyOf": [
{
"ge": 0.0,
"le": 1.0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"title": "Pass"
},
"warn": {
"anyOf": [
{
"ge": 0.0,
"le": 1.0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"title": "Warn"
}
},
"title": "ThresholdBand",
"type": "object"
}
},
"additionalProperties": false,
"description": "A reusable, workspace-scoped evaluation methodology \u2014 a project template, not an instrument.\n\n``version`` is assigned by the publish route, not the author, from the highest existing\nversion of the same name; it is carried on the body so a stored plan states its own version\nwithout a join.\n\n``metrics`` keys and ``required_quality_checks`` name registry entries that only a\nworkspace-scoped lookup can resolve, so they are free strings here. ``recommended_dataset``\nnames a dataset, not a dataset version.\n\n``required_columns`` and ``applicable_quality_checks`` are both deliberately absent: each is\nderived on read, and storing either would snapshot registry state that goes stale the moment a\nmetric gains a column or a check is registered.\nEach metric's :attr:`MetricSpec.published_required_columns` *is* stored, and is not that field\nunder another name \u2014 it records what the metric required at publish, so the two differing is\nthe drift a reader needs to see rather than a staleness to be avoided.\n\nNot ``frozen``: immutability of a published version is enforced on the stored row instead, so\nthe publish route can assign the real version and re-validate rather than using\n``model_copy``, which skips validation.\n\n``required_quality_checks`` states a *requirement*, never a selection: it names a subset of\nthe derived ``applicable_quality_checks`` that must have produced a verdict before a dataset\nversion is promotable. It grades nothing \u2014 ``force``, the FAIL floor and the WARN override\nare the platform's, unchanged at either value of the lock.",
"properties": {
"description": {
"anyOf": [
{
"maxLength": 1000,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "What this methodology is for, in the author's own words \u2014 the one field here that is prose rather than machinery, so a reader browsing the catalogue can tell two plans apart without reading their metric lists. Per version, not per name: an edit may change it to say what changed, and an older version keeps the description that was accurate when it was published. Author-supplied and untouched by the platform, unlike each metric's `published_version` receipt. Nothing reads it, so nothing beyond its length is validated.",
"title": "Description"
},
"dimensions": {
"items": {
"$ref": "#/$defs/DimensionSpec"
},
"title": "Dimensions",
"type": "array"
},
"is_dimensions_locked": {
"default": true,
"description": "Governs the dimension contract only: `PATCH /projects` refuses a changed dimension set, and run creation rejects a dataset version whose `dimension_snapshot` diverges from the plan. Unlocked, the project owns its dimensions and may edit them freely; nothing is refused and nothing is recorded. It governs no other tier.",
"title": "Is Dimensions Locked",
"type": "boolean"
},
"is_metrics_locked": {
"default": true,
"description": "Governs the evaluation tier only: the `metrics` map, meaning each metric named plus its `threshold`, `config` and `config_ref`, as filled into `run_config` at run creation. Locked, a run supplying any of extra keys is rejected, naming the key; unlocked, the plan pre-fills them and the tester may override, with nothing recorded about the choice. It governs no other tier.",
"title": "Is Metrics Locked",
"type": "boolean"
},
"is_quality_checks_locked": {
"default": true,
"description": "Governs the data-quality tier only: the `required_quality_checks` this plan seeds onto a project it creates. Locked, the project may not change that set \u2014 locked with an empty set therefore fixes it at no requirement, unable to add one. Unlocked, the project owns the set and may edit or clear it freely; nothing is refused and nothing is recorded. It governs no other tier, and at neither value does it reach the platform's own gate \u2014 a plan cannot make a dataset promotable that would not otherwise be.",
"title": "Is Quality Checks Locked",
"type": "boolean"
},
"metrics": {
"additionalProperties": {
"$ref": "#/$defs/MetricSpec"
},
"title": "Metrics",
"type": "object"
},
"name": {
"maxLength": 255,
"minLength": 1,
"title": "Name",
"type": "string"
},
"recommended_dataset": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Recommended Dataset"
},
"required_quality_checks": {
"description": "Checks a dataset version must have produced a verdict for before it can be promoted to golden, by the bare name results are reported under (`row_count`, not `quality_check.row_count`). Must be a subset of `applicable_quality_checks`; publish refuses a name outside it. A requirement, not a selection: it changes neither which checks dispatch nor how they are graded \u2014 a required check that came back WARN is still promotable with a reasoned override, and FAIL is still fatal. It refuses only the case where a required check produced no result at all. Empty means no requirement.",
"items": {
"type": "string"
},
"title": "Required Quality Checks",
"type": "array"
},
"schema_name": {
"title": "Schema Name",
"type": "string"
},
"task_type": {
"$ref": "#/$defs/TaskType"
},
"version": {
"default": 1,
"minimum": 1,
"title": "Version",
"type": "integer"
}
},
"required": [
"name",
"schema_name",
"task_type"
],
"title": "TestPlan",
"type": "object"
}