Evaluate an agent

This guide scores a set of already-recorded agent traces end to end. You land the traces into a dataset, attach ground truth if needed, then score them with platform metrics. Unlike the LLM, RAG, and CV guides, AIP never calls a live system here. The agent already ran. This guide evaluates the transcripts it left behind.

This is currently the only way to evaluate an agent on AIP. There is no live-invocation path. So the one thing you need first is your agent's traces, already exported. land_traces() understands three conventions: OpenInference, the OpenTelemetry GenAI semantic conventions, and LangSmith. These are existing, widely used tracing standards, not something specific to AIP. If your agent already runs on a framework like LangChain, LlamaIndex, or AutoGen, it may already emit one of them with little or no extra setup. If it doesn't yet, instrument it against whichever convention fits your stack, capture a run, export the spans, then come back here.

The example in this guide is a research agent that answers a model-monitoring question. Its final answer mixes a claim its evidence actually supports with a fabricated one. The guide also covers a coordinator agent that delegates drafting work to a sub-agent. One delegation completes normally. Another times out. We score the trace for hallucinated claims, and we score the sub-agent's responses for validity.

The complete executable version is aip_v2_demo_agent_trace_e2e.ipynb, in libs/aip-sdk/examples/notebooks/ of the SDK repository. It reads credentials only from environment variables. Start Jupyter from libs/aip-sdk after exporting the values shown in examples/.env.example. Judge-backed metrics (agent.hallucination, and the reused LLM metrics later in this guide) are opt-in. Set AIP_DEMO_ENABLE_JUDGE_METRICS=1, with a judge model configured server-side, to run them. The notebook still completes top to bottom without one.

Connect and create the project

Initialize the SDK with one supported authentication method, then resolve a workspace and create the trace project.

import os
import aip_sdk as aip

aip.init(
    os.environ["AIP_BASE_URL"],
    token=os.environ.get("AIP_TOKEN"),
    api_key=os.environ.get("AIP_API_KEY"),
    username=os.environ.get("AIP_USERNAME"),
    password=os.environ.get("AIP_PASSWORD"),
)
workspace_name = os.environ.get("AIP_WORKSPACE_NAME")
ws = aip.Workspace.get_by_name(workspace_name) if workspace_name else aip.Workspace.default()

project, _ = aip.Project.get_or_create(
    name="Agent Trace Demo",
    schema="agent_trace_v1",
    workspace_id=ws.id,
)

A trace project's task type is fixed to agentic automatically when it's created. You don't set it yourself, and passing anything else is rejected. No dimensions are declared here, unlike other, tabular, dataset schemas.

See Agent trace (agent_trace_v1) in Core Concepts to better understand how a trace is structured, and how it links to other related concepts like a span and a session.

Land the traces

Gather the traces your agent already produced. These must be exported spans in one of three conventions: openinference, genai, or langsmith. Upload them with the land_traces function.

If your agent doesn't emit traces in one of these conventions yet, see Instrument your agent for traces first. Landing is asynchronous. The land_traces call returns as soon as the run is accepted. The next step in this guide polls AIP until the run finishes, and checks for failures.

The shape below is a minimal but real OpenInference export. It covers three traces.

trace-1 is a single research agent (researcher) that calls a retriever and a tool. Its final answer mixes one claim the tool's output actually supports with one fabricated claim. A groundedness judge should split its verdict rather than call the whole answer clean or hallucinated. trace-1 also carries two LLM-kind spans on the same final-answer text. Those are used later in this guide.

trace-2 and trace-3 both show a coordinator agent delegating drafting work to a sub-agent (drafting_subagent). trace-2's sub-agent times out before producing a draft. trace-3's sub-agent completes normally. Scoring both together later gives a mixed verdict too.

Replace this with your own data if you wish. Two things to know about the shape: the span's semantic kind is the openinference.span.kind attribute, not a top-level field. AIP orders a trace's steps by start_time, not by their position in the export.

See agent_trace_v1 in GDI Schemas for more on the required fields in the trace schema.

payloads = [
    {
        "span_id": "span-agent-1",
        "parent_id": None,
        "trace_id": "trace-1",
        "name": "researcher",
        "attributes": {
            "openinference.span.kind": "AGENT",
            "session.id": "session-1",
            "input.value": "Which controls does NIST AI RMF recommend for model monitoring?",
            "output.value": (
                "NIST AI RMF recommends MEASURE 2.11 controls for ongoing monitoring of deployed "
                "models. It also mandates a 90-day third-party audit certified under ISO/IEC 42001."
            ),
        },
    },
    {
        "span_id": "span-retriever-1",
        "parent_id": "span-agent-1",
        "trace_id": "trace-1",
        "name": "vector_db",
        "attributes": {
            "openinference.span.kind": "RETRIEVER",
            "session.id": "session-1",
            "input.value": "NIST AI RMF model monitoring controls",
            "retrieval.documents.0.document.id": "doc-1",
            "retrieval.documents.0.document.content": "MEASURE 2.11: ongoing monitoring of deployed models.",
            "retrieval.documents.0.document.score": 0.91,
        },
    },
    {
        "span_id": "span-tool-1",
        "parent_id": "span-agent-1",
        "trace_id": "trace-1",
        "name": "summarise",
        "attributes": {
            "openinference.span.kind": "TOOL",
            "session.id": "session-1",
            "tool.name": "summarise",
            "input.value": '{"text": "MEASURE 2.11: ongoing monitoring of deployed models."}',
            "output.value": "Monitor deployed models continuously under MEASURE 2.11.",
        },
    },
    {
        # A metric like `llm.toxicity` scores LLM calls. It is only available on a
        # partition that resolves to at least one LLM-kind span. This is that span.
        "span_id": "span-llm-1",
        "parent_id": "span-agent-1",
        "trace_id": "trace-1",
        "name": "gpt-4",
        "attributes": {
            "openinference.span.kind": "LLM",
            "session.id": "session-1",
            "input.value": "What does NIST AI RMF recommend for model monitoring?",
            "output.value": (
                "NIST AI RMF recommends MEASURE 2.11 controls for ongoing monitoring of deployed "
                "models. It also mandates a 90-day third-party audit certified under ISO/IEC 42001."
            ),
        },
    },
    {
        # A second LLM-kind span on the same trace. It is a rambling, hedging non-answer
        # that never states what was asked. Used later in this guide to show that
        # relevance, conciseness, and helpfulness judges mark this down, while a
        # groundedness judge does not.
        "span_id": "span-llm-2",
        "parent_id": "span-agent-1",
        "trace_id": "trace-1",
        "name": "gpt-4",
        "attributes": {
            "openinference.span.kind": "LLM",
            "session.id": "session-1",
            "input.value": "What does NIST AI RMF recommend for model monitoring?",
            "output.value": (
                "That's a broad question, and there are a lot of angles to consider depending "
                "on context, since regulatory frameworks vary, organizational risk tolerance varies, "
                "and honestly it's hard to say anything definitive without more detail, but "
                "generally speaking monitoring is an important topic worth taking seriously in "
                "most AI deployments."
            ),
        },
    },
    {
        "span_id": "span-agent-2",
        "parent_id": None,
        "trace_id": "trace-2",
        "name": "coordinator",
        "attributes": {
            "openinference.span.kind": "AGENT",
            "session.id": "session-2",
            "input.value": "Draft a one-line monitoring policy summary.",
            "output.value": "Delegated drafting to the sub-agent, but it timed out; no draft was returned.",
        },
    },
    {
        "span_id": "span-agent-3",
        "parent_id": "span-agent-2",
        "trace_id": "trace-2",
        "name": "drafting_subagent",
        "status": {"code": "ERROR", "message": "Sub-agent timed out before producing a draft"},
        "attributes": {
            "openinference.span.kind": "AGENT",
            "session.id": "session-2",
            "input.value": "Draft a one-line monitoring policy summary.",
        },
    },
    {
        "span_id": "span-agent-4",
        "parent_id": None,
        "trace_id": "trace-3",
        "name": "coordinator",
        "attributes": {
            "openinference.span.kind": "AGENT",
            "session.id": "session-3",
            "input.value": "Draft a one-line monitoring policy summary.",
            "output.value": "Delegated drafting to the sub-agent and returned its answer.",
        },
    },
    {
        "span_id": "span-agent-5",
        "parent_id": "span-agent-4",
        "trace_id": "trace-3",
        "name": "drafting_subagent",
        "attributes": {
            "openinference.span.kind": "AGENT",
            "session.id": "session-3",
            "input.value": "Draft a one-line monitoring policy summary.",
            "output.value": "Monitor deployed models continuously under NIST AI RMF MEASURE 2.11.",
        },
    },
]  # your own export replaces this list. Same shape, real spans.

ingest = aip.land_traces(
    payloads,
    project_id=project.id,
    workspace_id=ws.id,
    dataset_name="agent-trace-demo",
    # ingest_convention is omitted. land_traces() auto-detects which of the three
    # conventions each record matches, by checking attribute keys (a bare
    # `openinference.span.kind` key is enough). Pass it explicitly only if you want to
    # force one convention instead of relying on detection. A record whose attributes
    # don't match any known convention gets dropped instead of landing. See the
    # polling step below for how to check for drops.
)

If your traces are already sitting in an object store, pull them instead of posting them inline. Create a saved connection to the bucket once, then trigger a pull from it. Name your own store with endpoint plus an access_key/secret_key pair. The secret key is write-only. It is stored encrypted and returned as ***. The access key is an identifier, not a secret, so it is returned in full.

connection = aip.create_source_connection(
    label="agent-trace-demo",
    endpoint="s3.us-east-1.amazonaws.com",
    bucket="my-agent-traces-bucket",
    prefix="agent-traces/",
    access_key=os.environ["MY_STORE_ACCESS_KEY"],
    secret_key=os.environ["MY_STORE_SECRET_KEY"],
    region="us-east-1",
    project_id=project.id,
    workspace_id=ws.id,
)

ingest = aip.land_traces_from_connection(connection.id)

Creation checks what it can check without your credentials. So if the connection saves, the platform is allowed to pull from it. A ForbiddenError here names exactly what failed, instead of leaving you to discover it later at pull time.

Omitting endpoint means "pull from a bucket the platform already runs." That is not your own store. This path only accepts the platform's own buckets, and only a prefix under your workspace's namespace. If either is wrong, the error tells you exactly what to change: the allowed buckets, or the required prefix.

An endpoint rejection is different. It only tells you the host is off the allowlist or is an internal address. It does not tell you which hosts would work instead. That's because the allowed hosts are set per deployment, so there's no fixed list to check in advance.

land_traces_from_connection() returns the same TraceIngestRun type as the inline call above, so the polling step below works identically regardless of which path you used to land your traces.

Poll the ingest run and check for drops

Wait for ingestion to finish, then check whether it completed successfully or failed.

Outcome	What happened	What to do next
COMPLETED with no drops	A dataset version was created and every record landed.	Continue to Annotate a trace with a trace tag below.
COMPLETED with drops	A dataset version was created, but some records were excluded.	Inspect each drop's detail. If the remaining data is sufficient for the evaluation, continue with the new version. Otherwise, correct the rejected records and ingest the export again.
FAILED because every record was rejected	No dataset version was created.	Inspect the drop details, correct the payload shape, fields, or ingestion convention they identify, and ingest the corrected export again.
FAILED for another reason	Ingestion stopped because of a connection, storage, worker, or platform error. No dataset version was created.	Read error. Correct any connection or storage problem it identifies, or retry after a cancellation or temporary failure. If the error persists, report the run ID and error message to your administrator or support team.

is_terminal tells you when to stop polling. It becomes True when ingestion completes or fails. Check failed to see which one happened.

Only a COMPLETED run creates a dataset version you can evaluate. On a failed run, the counters may be missing, or may only describe work done before the failure. They do not mean a usable dataset version was created.

Always read drop.detail. reason names the general category of a rejected record, like no_convention or frame_unsupported. detail explains what was wrong with that specific record. That's usually all you need to fix the export.

ingest = ingest.wait_until_terminal(interval=5, timeout=300)

if ingest.failed:
    print("Ingestion failed:", ingest.error)
    print("Run ID:", ingest.id)
    raise SystemExit(1)

dataset_id, version_id = ingest.dataset_id, ingest.dataset_version_id
print(ingest.landed_rows, "landed,", ingest.total_drops, "dropped")

if ingest.total_drops:
    unrecognised = 0

    for drop in aip.iter_ingest_drops(version_id):
        print(drop.stage, drop.reason, drop.record_id, drop.detail)

        if drop.reason is aip.TraceDropReason.NO_CONVENTION:
            unrecognised += 1

    if unrecognised:
        print(
            f"{unrecognised} record(s) matched no source convention. "
            "Check the export's attributes or set ingest_convention explicitly."
        )

Diagnose a failed ingest

A failed run does not create a dataset version. Diagnose it using the ingest run itself.

Start with error. It explains why the run stopped. If individual records were rejected, total_drops is greater than zero, and the run-scoped drop methods give you record-level details. Connection, worker, or platform failures may have no drop records at all. In those cases, error is your main diagnostic.

ingest = aip.get_ingest_run(ingest.id)

if ingest.failed:
    print("Ingestion failed:", ingest.error)
    print("Run ID:", ingest.id)

    if ingest.total_drops:
        page = ingest.list_drops(limit=1)

        if page.available < page.total_drops:
            print(
                f"showing {page.available} example(s) of "
                f"{page.total_drops} dropped record(s)"
            )

        for drop in ingest.iter_drops():
            print(drop.stage, drop.reason, drop.record_id, drop.detail)

These drop reasons account for almost every failed upload. Here's what each one means, and what to change:

reason	What the detail will tell you	What to fix
frame_unsupported	The object isn't a shape spans can be read from. Most often the export was nested one level too deep: a list[list[dict]] instead of a list[dict].	The export's shape, before it reaches land_traces()
no_convention	No source convention matched the record. The detail lists the attribute names the record actually carried, so you can compare them to what each convention expects.	The attribute names your telemetry emits, or pass ingest_convention= to force one. Seeing span_kind where openinference.span.kind was expected is the usual cause.
convention_mismatch	The ingest_convention you named doesn't match the payload's shape	Drop the argument and let detection choose, or name the convention the export actually uses
handler_error	The record matched a convention but could not be converted. The detail carries the converter's own error, naming the offending field and value.	The attribute the error names. A type mismatch, like an int where a string is expected, is the common case. It survives detection because detection only reads attribute names, not their values.

A record being recognised is not a promise that it converts. Detection only checks which attribute names an export carries. Conversion also has to satisfy each field's type. So an export that detects cleanly as one convention can still drop every record with handler_error.

A run that lands rows but scores nothing is a different problem. The records converted fine, but they carry no tool calls or span kinds to measure. Check the landed spans, not the drops.

Annotate a trace with a trace tag

Trace tags are per-trace key -> value annotations. They're overlaid onto the canonical view at read time. Tagging never rewrites the landed, immutable version. Use tags to mark a trace as reviewed, flagged, or triaged, without deriving a new dataset version.

tags = aip.upsert_trace_tags(
    dataset_id,
    version_id,
    [aip.TraceTagUpsert(target_ref="trace-1", key="reviewed", value="yes")],
)

Inspect the inferred topology

aip.get_topology() returns a summary pooled across every trace in the dataset version. It's lossy, but it shows which semantic components were observed, and which direct parent-to-child relationships happened between them. Use it as a quick sanity check that your export landed the shape you expected, before you author partitions against it.

graph = aip.get_topology(dataset_id, version_id)

for node in graph.nodes:
    print(node.entity.kind, node.entity.name, node.span_count)

Author and resolve partitions

A partition is a saved selector, not a set of rows. It's a named, reusable rule. You call resolve_partition() against it instead of redefining a target from scratch every run.

One thing to know: a partition is pinned to the dataset version it was created on. Deriving a new version (see Attach ground truth below) carries each partition across automatically, under a new id. It does not stay usable as-is against the new version.

Three partitions are useful for this example, one per evaluation target:

• A SPAN partition matching every AGENT span named drafting_subagent, across all traces. It resolves to both trace-2's errored sub-agent span and trace-3's healthy one, so agent.subagent_validity scores two occurrences instead of one.
• A TRACE partition, with no target (its scope is already the enclosing trace). It resolves to one occurrence per trace in the version: trace-1, trace-2, and trace-3.
• A SPAN partition matching every LLM-kind span. In this fixture, that's trace-1's two LLM-kind spans.

partition = aip.create_partition(
    dataset_id,
    version_id,
    name="drafting-subagent",
    selector=aip.PartitionSelector(
        partition_type=aip.PartitionType.SPAN,
        target=aip.EntitySelector(kind=aip.SpanKind.AGENT, name="drafting_subagent"),
    ),
)

trace_partition = aip.create_partition(
    dataset_id,
    version_id,
    name="whole-trace",
    selector=aip.PartitionSelector(partition_type=aip.PartitionType.TRACE),
)

llm_partition = aip.create_partition(
    dataset_id,
    version_id,
    name="final-answer-llm",
    selector=aip.PartitionSelector(
        partition_type=aip.PartitionType.SPAN,
        target=aip.EntitySelector(kind=aip.SpanKind.LLM),
    ),
)

resolve_partition() runs the saved selector and returns the concrete spans, traces, or sessions it matched. Use it to see exactly what a partition matched.

resolved = aip.resolve_partition(dataset_id, version_id, partition.id)

for item in resolved:
    if isinstance(item, aip.SpanPartition):
        print(item.span.span_id, item.span.kind, item.span.name)

A SESSION or TRACE partition, like whole-trace above, doesn't actually narrow anything. Every session or trace matches it already. So its available_metrics (next section) tells you what your data supports. It is not a guarantee that a run will accept it, because run creation checks the whole dataset version, not just the partition.

A SPAN partition, like drafting-subagent or final-answer-llm, doesn't have this gap. It genuinely narrows which spans get scored, so its available_metrics is reliable.

Handle UnprocessableEntityError from aip.run(), regardless of partition type. Run creation also requires the version you name to be the one this dataset's metrics currently score against. Attaching ground truth (see below) moves that to the version it derives. If you attach ground truth, score that derived version, not the one you originally landed into.

Report per-partition metric availability

Each metric is judged against the span kinds a partition's occurrences actually carry, and against the rows it would be scored on. So the verdict reflects the partition, not the whole dataset. If you want the full catalog of trace metrics instead of one partition's verdict, use aip.ops.list_metrics(kind="trace_metric").

for scope in (partition, trace_partition):
    availability = aip.get_partition_metric_availability(dataset_id, version_id, scope.id)
    print(scope.name, "can be scored by:", availability.available_metrics)
    for metric in availability.metrics:
        if not metric.available:
            reasons = "; ".join(f"{r.reason_code}: {r.detail}" for r in metric.reasons)
            print(" ", metric.op_name, "unavailable:", reasons)

agent.step_accuracy and agent.tool_selection_accuracy report OpMissingColumns here. Ground truth needs to be attached before these two metrics can be used. See Attach ground truth below.

Metrics from the metric catalogue under Reference, like llm.toxicity, work the same way. Just add their name to the metrics= list:

for scope in (partition, llm_partition):
    verdict = aip.get_partition_metric_availability(
        dataset_id, version_id, scope.id, metrics=["llm.toxicity"]
    ).metrics[0]
    print(scope.name, "llm.toxicity available:", verdict.available)

llm.toxicity can only run on LLM-kind spans, so it's available on final-answer-llm and not on drafting-subagent.

Score the partitions

agent.subagent_validity is rule-based (no judge) and always runs. agent.hallucination is LLM-judge-backed. It needs a judge model configured server-side to run. See Runs & Results for how judge credentials are configured.

partition_id=partition.id names the drafting-subagent partition specifically. Without it, partition_type=SPAN alone doesn't error, and it doesn't guess which partition you meant either. It falls back to scoring every span of the metric's own required kind, across the whole dataset version. agent.subagent_validity requires an AGENT-kind span, and this fixture has 5 of those, not just the 2 named drafting_subagent that the partition narrows to. Naming partition_id is what limits scoring to just those 2.

with aip.run(
    project=project.id,
    dataset=dataset_id,
    pipeline="trace_metric_invoke",
    targets=[
        aip.RunTarget(
            partition_id=partition.id,
            partition_type=partition.partition_type,
            metrics=["agent.subagent_validity"],
        )
    ],
) as run:
    run.poll_status(interval=5, timeout=600)
    page = run.results()

for m in page.metrics:
    print(m.scorer, "mean:", m.mean, "over", m.count, "row(s)")

The agent.subagent_validity metric scored 2 sub-agent runs: one passed, one failed. So the mean score is 0.5.

agent.hallucination is scored at TRACE granularity instead, so no partition_id is needed. A TRACE or SESSION partition's selector never narrows anything. The whole trace is already the scope, so there's nothing for a partition_id to pick out. partition_type=aip.PartitionType.TRACE alone scores every trace in the dataset version:

with aip.run(
    project=project.id,
    dataset=dataset_id,
    pipeline="trace_metric_invoke",
    targets=[
        aip.RunTarget(
            partition_type=aip.PartitionType.TRACE,
            metrics=["agent.hallucination"],
        )
    ],
) as run:
    run.poll_status(interval=5, timeout=600)
    page = run.results()

agent.hallucination is applied to every trace, but only trace-1 is a single-agent trace it can judge. Its final answer mixes a real claim with a fabricated one, so it also scores 0.5.

Score the reused LLM-judge metrics

llm.answer_relevance, llm.bias, llm.conciseness, llm.helpfulness, and llm.toxicity are the same metrics documented under LLM metrics in Reference. They also work on agent-trace data, the same way the availability check above showed. There's no separate SDK surface for this. Pass their names in the same metrics= list any trace metric uses, scoped to a SPAN partition that resolves to LLM-kind spans.

llm_partition is that SPAN partition. Its selector matches every LLM-kind span, with no name filter narrowing it further. These five metrics only ever score LLM-kind spans anyway, so partition_id doesn't narrow anything more here than omitting it would. It's still named explicitly, like every partition in this guide, so the run's results tie back to a reusable partition instead of an unnamed one.

with aip.run(
    project=project.id,
    dataset=dataset_id,
    pipeline="trace_metric_invoke",
    targets=[
        aip.RunTarget(
            partition_id=llm_partition.id,
            partition_type=llm_partition.partition_type,
            metrics=[
                "llm.answer_relevance",
                "llm.bias",
                "llm.conciseness",
                "llm.helpfulness",
                "llm.toxicity",
            ],
        )
    ],
) as run:
    run.poll_status(interval=5, timeout=600)
    page = run.results()

for m in page.metrics:
    print(m.scorer, "mean:", m.mean, "over", m.count, "row(s)")

The LLM-kind-span partition resolves to two occurrences: a clean final answer, and a rambling, hedging draft that never actually answers what was asked. The mean scores show what each metric catches on its own.

llm.answer_relevance and llm.conciseness land at mean=0.5. One occurrence is on-topic and tight, the other is not.

llm.helpfulness lands at mean=0.625. That's partial credit: the rambling draft isn't useless, just incomplete.

llm.bias and llm.toxicity both stay at a clean mean=1.0. Neither occurrence contains biased or hostile language, so there's nothing for those two metrics to flag.

None of these five check a claim against evidence, though. The clean-looking final answer still repeats the fabricated claim from Land the traces above. It still scores a clean 1.0 across all five metrics. That's because these five judge relevance, tone, and completeness, not groundedness.

agent.hallucination, scored in the previous section, is the metric built to catch that. The two metric families are complementary, not redundant. Use the LLM-judge catalog metrics to check how an agent's language reads. Use agent.hallucination to check whether what it claims is actually true.

---

Attach ground truth (optional)

agent.step_accuracy and agent.tool_selection_accuracy reported OpMissingColumns earlier. Both need ground truth attached before a run can score them: an expected step sequence, and an expected tool-call set. Ground truth is attached per session or trace, after landing, not carried in the raw export. aip.attach_ground_truth() does this. It takes the expected values as an in-memory mapping. See the metric catalogue in Reference for the full list of metrics that support ground truth, and what each one needs.

Each key in rows is the session_id a trace carries. If there is no session, use the trace's own trace_id instead. The call is the same either way.

Every identifier you upload must belong to a session or trace that actually exists in the dataset version. If one doesn't match, the whole upload is rejected, not just that row.

expected_steps is positional, so order matters. expected_tools is not positional. agent.step_accuracy compares expected_steps against a trace's own non-root TOOL and AGENT spans, in the order they actually happened.

rows= is the in-memory form. You can pass a file path instead, like aip.attach_ground_truth(version_id, "ground_truth.csv"), for ground truth a reviewer authored in a .csv, .json, or .jsonl file. The file extension names the format. The key column can be named session_id, trace_id, or parent_id.

derived = aip.attach_ground_truth(
    version_id,
    # matches trace-1's own TOOL span, "summarise". Its session_id is "session-1".
    rows={"session-1": {"expected_tools": ["summarise"], "expected_steps": ["summarise"]}},
)

print(derived.ground_truth_columns, derived.ground_truth_row_count)  # ['expected_tools', 'expected_steps'] 1

Re-submitting the identical ground truth against the same base is idempotent, tracked by derived.replayed. Submitting different ground truth against the version you just derived merges over it instead of replaying. So a retry must name the same base version as the first attempt.

Why this guide attaches ground truth last

You don't have to attach ground truth last. Starting a dataset fresh, you could attach it first, author your partitions against the derived version, and score everything in one pass.

This guide can't, because partition, trace_partition, and llm_partition were already created and scored earlier, against the version you landed into. Attaching ground truth derives a new version. It doesn't touch the one you landed into, so those earlier runs stay valid on their own.

The catch is dataset=dataset_id. Every aip.run() call in this guide passes it with no @vN suffix, so it always resolves to the latest version. If ground truth had been attached before those earlier runs, dataset_id would have silently pointed at the new derived version. partition, trace_partition, and llm_partition would still point at the old one. A run naming both would fail with UnprocessableEntityError, because a partition is pinned to one specific dataset version.

Deriving a version does carry each existing partition across, just under a new id. derived.migrated_id(partition.id) looks up that new id. Scoring agent.step_accuracy, agent.tool_selection_accuracy, or anything else against the version ground truth just derived reuses that lookup, instead of re-authoring partitions from scratch.

With ground truth attached, agent.step_accuracy and agent.tool_selection_accuracy are unblocked on the trace partition. Score them the same way as every other trace metric in this guide, against the migrated partition id:

migrated_trace_id = derived.migrated_id(trace_partition.id)

with aip.run(
    project=project.id,
    dataset=dataset_id,
    pipeline="trace_metric_invoke",
    targets=[
        aip.RunTarget(
            partition_id=migrated_trace_id,
            partition_type=trace_partition.partition_type,
            metrics=["agent.step_accuracy", "agent.tool_selection_accuracy"],
        )
    ],
) as run:
    run.poll_status(interval=5, timeout=600)
    page = run.results()

for m in page.metrics:
    print(m.scorer, "mean:", m.mean, "over", m.count, "row(s)")

Both land at a clean mean=1.0. trace-1's real summarise tool call exactly matches the expected_steps and expected_tools attached above.