Step Accuracy

Available

agent.step_accuracy

Measures trajectory adherence — whether the agent's steps matched the expected sequence, including any unordered concurrent-step groups — via edit-distance alignment by default or strict position via config. Supports trace-level and session-level scoring.

Contract

Field	Value
version	2.0.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_traces_agent.eval.step_accuracy.get_trace_metric
partition_types	["trace", "session"]
required_kinds	[{"anyOf": ["TOOL", "AGENT", "LLM"]}]

Required columns

• expected_steps

Accepted schemas

[
  {
    "name": "canonical_partition_v1"
  }
]

Methodology

1. For a trace or session resolved partition, the metric collects the executed steps in best-effort execution order, binding to the canonical model rather than any wire-format key. A step is normally a non-root TOOL or AGENT span; a trace with zero TOOL/AGENT spans (every action embedded in an LLM completion rather than split into its own span) instead contributes each embedded call as its own ordered step. Steps are sorted by start timestamp, then alphabetically by span name to break ties — the alphabetic tiebreak is not execution-meaningful and exists only for determinism.
2. Each observed step is identified by name. TOOL steps use the invoked tool's bare name. AGENT steps use aip.agent.name when explicitly provided, otherwise a convention-native identity such as gen_ai.agent.name, and finally the raw span name as a fallback. Write expected_steps using bare names such as "assess_collateral" and "financial_analyst".
3. The observed sequence is compared against that resolved partition's own expected_steps (from the canonical_partition_v1 expected_steps column, or falling back to config) using whichever scoring_method the config selects. Unlike tool_selection_accuracy, order is the whole point. Tool-derived step names are case-normalised and with -/_ stripped; agent names are compared exactly, since the customer controls both sides of that comparison.
4. scoring_method: "edit_distance" (default) aligns the two sequences via edit distance (insertions, deletions, and substitutions each cost one point) rather than comparing position by position — a single early extra or missing step no longer desyncs every later position, and an extra step costs the same wherever it falls in the sequence. This does not distinguish a skipped optional step from a skipped load-bearing one — every mismatch costs the same regardless of how consequential it was.
5. scoring_method: "strict_position" is the original exact comparison — the i-th observed step against the i-th expected step, with no tolerance for a shifted index. A single early insertion or omission cascades into every later position reading as wrong, and an extra step at the very end of the trajectory is never penalised (reported in extra_steps only) even though the identical extra step earlier in the sequence would be. Choose this over edit_distance for a task where any positional deviation should read as an outright failure.
6. An expected_steps element can also be a JSON list — an unordered GROUP, for concurrent subagent fan-out where the scheduler's own interleaving of steps carries no signal (a set of 12 concurrent steps across 3 subagents is a real motivating example). Order among a group's own members carries no signal; the group's position relative to surrounding bare steps still does. Only edit_distance can score a group — it's rejected under strict_position, at config validation for the run-level fallback and again at scoring time for annotation-supplied ground truth. A group cannot contain a group (one level of nesting only) — author expected_steps for a simpler agent shape when a subagent's own fan-out also needs to be unordered.
7. A group is scored as a windowed multiset match, not a permutation search — trying every ordering of a 12-member group is already computationally infeasible (12! is about 480 million), and a group has no internal order at all, so scoring it is a set-membership question, not a sequence-alignment one. A missing member and an unclaimed window step are paired and charged as ONE substitution (one point per pair) — the same paired-cost convention plain-token substitutions already use — and only the leftover size difference beyond the pairable overlap, when the group and window sizes differ, is a genuine, unpaired insertion or deletion (also one point each). This mirrors the group's own cost formula exactly — max(len(group), len(window)) - matched; reporting every missing member and every extra step as separate, always-one-point entries would imply double the actual cost whenever a group has both. metadata.steps expands a group into one match entry per claimed window step, one substitute entry per paired miss/extra, and one insert/delete entry per unpaired leftover, all sharing a group key equal to that group's own index in expected_steps — the same indexing metadata.expected_steps uses. total_steps and correct_steps count a group's own members individually (a 12-member group counts as 12, not 1), so the denominator and metadata.edit_distance stay in the same per-step units regardless of how many groups expected_steps contains.
8. The score is normalised to 0-1 either way. Under edit_distance it is 1 - edit_distance / max(total_steps, len(actual_steps)); the raw edit distance itself is reported in metadata.edit_distance since the score cannot exceed 1.0. Under strict_position it is the fraction of positions that matched, and observed steps beyond the expected trajectory's length are reported as extra_steps without affecting the score.

Score semantics

Scores range 0-1. Under the default edit_distance method, a high score means the observed trajectory closely matches the expected one after accounting for insertions/deletions/substitutions (and, for any group, membership regardless of order); a low score means it diverged substantially, in any combination of wrong, missing, or extra steps — see metadata.edit_distance for the raw, unnormalised divergence count (this field is only present in edit_distance results). Under strict_position, a high score means the agent did the expected thing at most positions and a low score means it deviated, skipped, or reordered steps. Higher is better either way. For session-level scoring, every trace's steps are merged into ONE execution-ordered trajectory and compared against a single session-spanning expected_steps — not a per-trace score averaged across the session.

Worked example

edit_distance (default), plain steps: a task expects [search, open_document]; the agent executed [preamble_call, search, open_document] -> the extra leading step costs one insertion, so score = 1 - 1/3 = 0.67, and both real steps still show as matched in metadata.steps. strict_position on the same example would score 0/2 = 0.0, since the leading extra step shifts every later position out of alignment. edit_distance, with a group: a task expects [authenticate, [fetch_a, fetch_b, fetch_c], finalize]; the agent executed [authenticate, fetch_c, fetch_a, fetch_b, finalize] (the group members in a different order than authored) -> every member still matched, so score = 1.0. Had fetch_b never run, that member would show as missing and score = 1 - 1/5 = 0.8 (total_steps = 5, one per plain step plus one per group member).

Configuration schema

{
  "description": "Parameters for step_accuracy.\n\n``expected_steps`` is the ordered reference trajectory: the observed steps are compared\nagainst it \u2014 a TOOL/AGENT step normally, or an LLM span's own embedded call when a trace has\nnone of those. When scoring pools multiple traces together, the same one trajectory is\nchecked across the combined step sequence rather than per trace.\n\nEach entry is either a bare step name (an ordered step, unchanged) or a JSON list of step\nnames (an unordered GROUP \u2014 order among the group's own members carries no signal, but the\ngroup's position relative to surrounding bare steps still does; see ``method.py``'s\n``_group_cost`` for how a group is scored). One level of nesting only \u2014 a group cannot itself\ncontain a group, enforced by the type here rather than hand-written validation. Meant for\nconcurrent subagent fan-out where the scheduler's own interleaving carries no signal, not for\ngenuinely open-ended step ordering.\n\n``expected_steps`` is a run-level fallback; per-occurrence ``annotations`` take precedence.\nBoth are optional here, but every occurrence must resolve a trajectory from one of them or\nscoring fails \u2014 a task with no expected steps gives this metric nothing to score against.\n\n``scoring_method`` picks how ``expected_steps`` is compared against the observed steps:\n``\"edit_distance\"`` (default) is an alignment-based comparison that gives graduated credit\nproportional to how far the trajectory actually diverged (insertions, deletions, and\nsubstitutions each cost one point); ``\"strict_position\"`` is the original exact\n``expected[i]`` vs ``observed[i]`` comparison, kept as an explicit opt-in for a task where any\npositional deviation should read as an outright failure rather than a smoothed score. See\n``method.py``'s module docstring for the tradeoffs between the two. Groups have no meaning\nunder ``strict_position`` and are rejected here (config-level ``expected_steps``) and again at\nscoring time (annotation-supplied ``expected_steps``, which bypasses this validator entirely).",
  "properties": {
    "annotations": {
      "anyOf": [
        {
          "additionalProperties": {
            "additionalProperties": {
              "type": "string"
            },
            "type": "object"
          },
          "type": "object"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Annotations",
      "x-aip-param-role": "run_input"
    },
    "expected_steps": {
      "anyOf": [
        {
          "items": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "items": {
                  "type": "string"
                },
                "type": "array"
              }
            ]
          },
          "minItems": 1,
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Expected Steps",
      "x-aip-param-role": "run_input"
    },
    "metric_name": {
      "default": "step_accuracy",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "scoring_method": {
      "default": "edit_distance",
      "enum": [
        "strict_position",
        "edit_distance"
      ],
      "title": "Scoring Method",
      "type": "string"
    }
  },
  "title": "StepAccuracyConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 4,
  "processing_kind": "cpu",
  "timeout_seconds": 300
}

Complete manifest

accepts:
- name: canonical_partition_v1
config_schema:
  description: 'Parameters for step_accuracy.


    ``expected_steps`` is the ordered reference trajectory: the observed steps are
    compared

    against it — a TOOL/AGENT step normally, or an LLM span''s own embedded call when
    a trace has

    none of those. When scoring pools multiple traces together, the same one trajectory
    is

    checked across the combined step sequence rather than per trace.


    Each entry is either a bare step name (an ordered step, unchanged) or a JSON list
    of step

    names (an unordered GROUP — order among the group''s own members carries no signal,
    but the

    group''s position relative to surrounding bare steps still does; see ``method.py``''s

    ``_group_cost`` for how a group is scored). One level of nesting only — a group
    cannot itself

    contain a group, enforced by the type here rather than hand-written validation.
    Meant for

    concurrent subagent fan-out where the scheduler''s own interleaving carries no
    signal, not for

    genuinely open-ended step ordering.


    ``expected_steps`` is a run-level fallback; per-occurrence ``annotations`` take
    precedence.

    Both are optional here, but every occurrence must resolve a trajectory from one
    of them or

    scoring fails — a task with no expected steps gives this metric nothing to score
    against.


    ``scoring_method`` picks how ``expected_steps`` is compared against the observed
    steps:

    ``"edit_distance"`` (default) is an alignment-based comparison that gives graduated
    credit

    proportional to how far the trajectory actually diverged (insertions, deletions,
    and

    substitutions each cost one point); ``"strict_position"`` is the original exact

    ``expected[i]`` vs ``observed[i]`` comparison, kept as an explicit opt-in for
    a task where any

    positional deviation should read as an outright failure rather than a smoothed
    score. See

    ``method.py``''s module docstring for the tradeoffs between the two. Groups have
    no meaning

    under ``strict_position`` and are rejected here (config-level ``expected_steps``)
    and again at

    scoring time (annotation-supplied ``expected_steps``, which bypasses this validator
    entirely).'
  properties:
    annotations:
      anyOf:
      - additionalProperties:
          additionalProperties:
            type: string
          type: object
        type: object
      - type: 'null'
      default: null
      title: Annotations
      x-aip-param-role: run_input
    expected_steps:
      anyOf:
      - items:
          anyOf:
          - type: string
          - items:
              type: string
            type: array
        minItems: 1
        type: array
      - type: 'null'
      default: null
      title: Expected Steps
      x-aip-param-role: run_input
    metric_name:
      default: step_accuracy
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    scoring_method:
      default: edit_distance
      enum:
      - strict_position
      - edit_distance
      title: Scoring Method
      type: string
  title: StepAccuracyConfig
  type: object
dependencies: []
description: Does the observed step sequence match the expected trajectory, including
  any unordered concurrent-step groups? Scored by edit-distance alignment by default,
  or by strict position via `scoring_method`.
direction: higher_is_better
display_name: Step Accuracy
entrypoint: aip_traces_agent.eval.step_accuracy.get_trace_metric
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 4
  processing_kind: cpu
  timeout_seconds: 300
kind: trace_metric
manifest_version: '1'
metric_metadata:
  methodology:
  - For a trace or session resolved partition, the metric collects the executed steps
    in best-effort execution order, binding to the canonical model rather than any
    wire-format key. A step is normally a non-root TOOL or AGENT span; a trace with
    zero TOOL/AGENT spans (every action embedded in an LLM completion rather than
    split into its own span) instead contributes each embedded call as its own ordered
    step. Steps are sorted by start timestamp, then alphabetically by span name to
    break ties — the alphabetic tiebreak is not execution-meaningful and exists only
    for determinism.
  - Each observed step is identified by name. TOOL steps use the invoked tool's bare
    name. AGENT steps use aip.agent.name when explicitly provided, otherwise a convention-native
    identity such as gen_ai.agent.name, and finally the raw span name as a fallback.
    Write expected_steps using bare names such as "assess_collateral" and "financial_analyst".
  - The observed sequence is compared against that resolved partition's own expected_steps
    (from the canonical_partition_v1 expected_steps column, or falling back to config)
    using whichever `scoring_method` the config selects. Unlike tool_selection_accuracy,
    order is the whole point. Tool-derived step names are case-normalised and with
    -/_ stripped; agent names are compared exactly, since the customer controls both
    sides of that comparison.
  - '`scoring_method: "edit_distance"` (default) aligns the two sequences via edit
    distance (insertions, deletions, and substitutions each cost one point) rather
    than comparing position by position — a single early extra or missing step no
    longer desyncs every later position, and an extra step costs the same wherever
    it falls in the sequence. This does not distinguish a skipped optional step from
    a skipped load-bearing one — every mismatch costs the same regardless of how consequential
    it was.'
  - '`scoring_method: "strict_position"` is the original exact comparison — the i-th
    observed step against the i-th expected step, with no tolerance for a shifted
    index. A single early insertion or omission cascades into every later position
    reading as wrong, and an extra step at the very end of the trajectory is never
    penalised (reported in extra_steps only) even though the identical extra step
    earlier in the sequence would be. Choose this over edit_distance for a task where
    any positional deviation should read as an outright failure.'
  - An expected_steps element can also be a JSON list — an unordered GROUP, for concurrent
    subagent fan-out where the scheduler's own interleaving of steps carries no signal
    (a set of 12 concurrent steps across 3 subagents is a real motivating example).
    Order among a group's own members carries no signal; the group's position relative
    to surrounding bare steps still does. Only edit_distance can score a group — it's
    rejected under strict_position, at config validation for the run-level fallback
    and again at scoring time for annotation-supplied ground truth. A group cannot
    contain a group (one level of nesting only) — author expected_steps for a simpler
    agent shape when a subagent's own fan-out also needs to be unordered.
  - A group is scored as a windowed multiset match, not a permutation search — trying
    every ordering of a 12-member group is already computationally infeasible (12!
    is about 480 million), and a group has no internal order at all, so scoring it
    is a set-membership question, not a sequence-alignment one. A missing member and
    an unclaimed window step are paired and charged as ONE substitution (one point
    per pair) — the same paired-cost convention plain-token substitutions already
    use — and only the leftover size difference beyond the pairable overlap, when
    the group and window sizes differ, is a genuine, unpaired insertion or deletion
    (also one point each). This mirrors the group's own cost formula exactly — `max(len(group),
    len(window)) - matched`; reporting every missing member and every extra step as
    separate, always-one-point entries would imply double the actual cost whenever
    a group has both. metadata.steps expands a group into one `match` entry per claimed
    window step, one `substitute` entry per paired miss/extra, and one `insert`/`delete`
    entry per unpaired leftover, all sharing a `group` key equal to that group's own
    index in expected_steps — the same indexing metadata.expected_steps uses. total_steps
    and correct_steps count a group's own members individually (a 12-member group
    counts as 12, not 1), so the denominator and metadata.edit_distance stay in the
    same per-step units regardless of how many groups expected_steps contains.
  - The score is normalised to 0-1 either way. Under edit_distance it is `1 - edit_distance
    / max(total_steps, len(actual_steps))`; the raw edit distance itself is reported
    in metadata.edit_distance since the score cannot exceed 1.0. Under strict_position
    it is the fraction of positions that matched, and observed steps beyond the expected
    trajectory's length are reported as extra_steps without affecting the score.
  score_semantics: Scores range 0-1. Under the default edit_distance method, a high
    score means the observed trajectory closely matches the expected one after accounting
    for insertions/deletions/substitutions (and, for any group, membership regardless
    of order); a low score means it diverged substantially, in any combination of
    wrong, missing, or extra steps — see metadata.edit_distance for the raw, unnormalised
    divergence count (this field is only present in edit_distance results). Under
    strict_position, a high score means the agent did the expected thing at most positions
    and a low score means it deviated, skipped, or reordered steps. Higher is better
    either way. For session-level scoring, every trace's steps are merged into ONE
    execution-ordered trajectory and compared against a single session-spanning expected_steps
    — not a per-trace score averaged across the session.
  summary: Measures trajectory adherence — whether the agent's steps matched the expected
    sequence, including any unordered concurrent-step groups — via edit-distance alignment
    by default or strict position via config. Supports trace-level and session-level
    scoring.
  worked_example: 'edit_distance (default), plain steps: a task expects [search, open_document];
    the agent executed [preamble_call, search, open_document] -> the extra leading
    step costs one insertion, so score = 1 - 1/3 = 0.67, and both real steps still
    show as matched in metadata.steps. strict_position on the same example would score
    0/2 = 0.0, since the leading extra step shifts every later position out of alignment.
    edit_distance, with a group: a task expects [authenticate, [fetch_a, fetch_b,
    fetch_c], finalize]; the agent executed [authenticate, fetch_c, fetch_a, fetch_b,
    finalize] (the group members in a different order than authored) -> every member
    still matched, so score = 1.0. Had fetch_b never run, that member would show as
    missing and score = 1 - 1/5 = 0.8 (total_steps = 5, one per plain step plus one
    per group member).'
metric_type: pointwise
name: agent.step_accuracy
partition_types:
- trace
- session
required_columns:
- expected_steps
required_kinds:
- anyOf:
  - TOOL
  - AGENT
  - LLM
scorer_contract: per_row
unsupported_trace_shapes: []
version: 2.0.0