Skip to main content

Partition Available Metrics

GET 
/datasets/:dataset_id/versions/:version_id/partitions/:partition_id/available-metrics

Which trace metrics can score this partition on this dataset version, and why not.

Each metric is checked against the span kinds this partition's occurrences actually carry, and against the canonical_partition_v1 occurrence rows it would be scored on — schema, ground-truth columns, scorer contract. Resolving this server-side is the point: the span-kind half cannot be computed from GET /ops plus a column list, and approximating it client-side is what this endpoint exists to prevent.

The occurrences judged are the ones the stored selector actually resolves to — the same resolve_partition path /resolve runs, so a partition narrowed by name or ancestry_path to nothing reports PartitionHasNoOccurrences rather than an optimistic verdict. It costs what /resolve costs to compute, and returns verdicts instead of spans.

The ground-truth columns judged are the ones this version records, so a metric blocked only on expected_tools reports available on the version a ground-truth upload derived and stays unavailable on the landed version it was derived from. Ask about the version you intend to run.

Two metrics — agent.step_accuracy and agent.tool_selection_accuracy — get one further check on a TRACE partition: each falls back to its session's pooled ground truth when an occurrence carries no annotation of its own, which is only correct for a single-trace session. A metric reports AmbiguousTraceGroundTruth when every occurrence this partition selects would fall back that way for a session with more than one trace — the same condition its own scoring rejects outright rather than silently mis-scoring, surfaced here so a caller checking availability first sees it before running rather than after.

Run creation is partition-aware and agrees on schema and ground-truth columns, but this is not a run-admission preview. POST /runs with pipeline=trace_metric_invoke takes a partition_id and validates through validate_trace_run_metrics, against canonical_partition_v1 with this version's own ground-truth columns. It differs on two axes. Granularity is the request's own partition_type, which run creation does not require to match the saved partition's type (and omitting it skips that gate), so PartitionTypeUnsupported here means "not at this partition's granularity", not "the run will be refused". And run creation enforces preconditions this endpoint says nothing about — the version must be golden — so an available metric can still be refused at submit for a reason that is not about the metric.

A metric is also checked against the cluster: one that is registered and contract-compatible but whose Nuclio function is absent or not ready reports OpNotDeployed, because the registry is seeded from the op manifests in the source tree and says nothing about what this environment deploys. When the Nuclio control plane cannot be reached the response says so in deployment_checked=false and every verdict is registry-only — an available metric is then contract-compatible but unproven runnable. The cluster is read only once the caller's role and both path ids have been checked, so an unauthorized or misaddressed request costs the control plane nothing.

The ambiguous-trace-ground-truth check (agent.step_accuracy / agent.tool_selection_accuracy, see MetricUnavailableReason.AMBIGUOUS_TRACE_GROUND_TRUTH) has the same shape on a TRACE partition: it reads the version's ground-truth annotation values, and when that read fails the response says so in ground_truth_checked=false rather than silently reporting no occurrence ambiguous — an available metric there is unverified on that one gate, not confirmed safe.

metrics (repeatable) restricts the report to those names; omit it for every trace metric visible in the dataset's workspace. A requested name that resolves to no trace metric is reported OpNotFound rather than dropped — with a distinct explanation when the name is registered under a different op kind — so a stale client selection gets a real answer.

The response carries verdicts only — never span payloads. Answering "can this metric run here" must not double as a way to read trace content.

Request​

Responses​

Successful Response