Tool Selection Accuracy

Available

agent.tool_selection_accuracy

Measures tool-selection coverage — how many of the required tools the agent called, anywhere in the trace or session, regardless of order or repeat count. A trace with no TOOL spans at all falls back to reading tool calls embedded in its LLM spans. Scored per trace or pooled across a session; there is no span-level scoring.

Contract

Field	Value
`version`	2.0.0
`metric_type`	pointwise
`scorer_contract`	per_row
`direction`	higher_is_better
`entrypoint`	aip_traces_agent.eval.tool_selection_accuracy.get_trace_metric
`partition_types`	`["trace", "session"]`
`required_kinds`	`[{"anyOf": ["TOOL", "LLM"]}]`

Required columns

expected_tools

Accepted schemas

[
  {
    "name": "canonical_partition_v1"
  }
]

Methodology

For a trace or session resolved partition, the metric collects the distinct tools actually executed as their own TOOL-span step, binding to the canonical model rather than any wire-format key. A trace with no TOOL spans at all falls back to reading the tool calls embedded directly in its LLM-kind spans' own completions instead - for datasets where a tool call is part of the model's own response rather than a separate execution step. The two shapes are never mixed for one trace.
Scored per trace (TRACE), or pooled across every trace in the session (SESSION) - there is no span-level scoring; a standalone per-span score was tried and dropped for adding little real signal over TRACE/SESSION.
The observed tool set is compared against that resolved partition's own expected_tools (from the canonical_partition_v1 expected_tools column, or falling back to config), case- and separator-normalised — presence only, independent of call order or repeat count.
The score is the fraction of expected tools that were observed, normalised to 0-1.

Score semantics

Scores range 0-1, the fraction of the required tools the agent actually called. A high score means the agent selected the tools the task required; a low score means it skipped required tools. Higher is better. TRACE scores one trace's own called tools; SESSION pools tool calls across every trace in the session instead, for tasks where the right tool may be called in a later turn.

Worked example

A task requires get_weather and get_time; only get_weather appears as an executed tool step -> 1/2 = 0.5. For session scoring, it pools across all traces in the session. A trace with no TOOL spans at all instead reads that trace's LLM-kind spans' own embedded tool calls as the called set, scored the same way.

Configuration schema

{
  "description": "Parameters for tool_selection_accuracy.\n\n``expected_tools`` names the DISTINCT tools a task requires \u2014 checked against tools\nactually called anywhere in the trace, pooled across the session (session-level scoring).\nA trace with no distinct tool-execution steps at all (every call embedded in the model's\nown response instead) is reconstructed from those embedded calls instead.\nCase/separator-normalized, presence-only: independent of call order or count.\n``expected_tools`` is a run-level fallback only \u2014 the same value for every occurrence in\na run \u2014 while per-occurrence ``annotations`` take precedence whenever the current\noccurrence supplies its own value. Both fields are optional here, but every occurrence\nmust resolve an expected-tools set from one of them, or scoring fails.",
  "properties": {
    "annotations": {
      "anyOf": [
        {
          "additionalProperties": {
            "additionalProperties": {
              "type": "string"
            },
            "type": "object"
          },
          "type": "object"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Annotations",
      "x-aip-param-role": "run_input"
    },
    "expected_tools": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "minItems": 1,
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Expected Tools",
      "x-aip-param-role": "run_input"
    },
    "metric_name": {
      "default": "tool_selection_accuracy",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "treat_no_calls_as_zero": {
      "default": false,
      "description": "Score an occurrence with no observed tool calls as 0.0 when it has an occurrence-specific expected_tools annotation. Run-level fallback values do not qualify.",
      "title": "Treat No Calls As Zero",
      "type": "boolean"
    }
  },
  "title": "ToolSelectionAccuracyConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 4,
  "processing_kind": "cpu",
  "timeout_seconds": 300
}

Complete manifest

accepts:
- name: canonical_partition_v1
config_schema:
  description: 'Parameters for tool_selection_accuracy.


    ``expected_tools`` names the DISTINCT tools a task requires — checked against
    tools

    actually called anywhere in the trace, pooled across the session (session-level
    scoring).

    A trace with no distinct tool-execution steps at all (every call embedded in the
    model''s

    own response instead) is reconstructed from those embedded calls instead.

    Case/separator-normalized, presence-only: independent of call order or count.

    ``expected_tools`` is a run-level fallback only — the same value for every occurrence
    in

    a run — while per-occurrence ``annotations`` take precedence whenever the current

    occurrence supplies its own value. Both fields are optional here, but every occurrence

    must resolve an expected-tools set from one of them, or scoring fails.'
  properties:
    annotations:
      anyOf:
      - additionalProperties:
          additionalProperties:
            type: string
          type: object
        type: object
      - type: 'null'
      default: null
      title: Annotations
      x-aip-param-role: run_input
    expected_tools:
      anyOf:
      - items:
          type: string
        minItems: 1
        type: array
      - type: 'null'
      default: null
      title: Expected Tools
      x-aip-param-role: run_input
    metric_name:
      default: tool_selection_accuracy
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    treat_no_calls_as_zero:
      default: false
      description: Score an occurrence with no observed tool calls as 0.0 when it
        has an occurrence-specific expected_tools annotation. Run-level fallback values
        do not qualify.
      title: Treat No Calls As Zero
      type: boolean
  title: ToolSelectionAccuracyConfig
  type: object
dependencies: []
description: Were the required tools called, anywhere in the trace (or pooled across
  the session), regardless of order or count? A trace with no TOOL spans at all (every
  call embedded in the model's own completion instead) is reconstructed from those
  embedded calls instead.
direction: higher_is_better
display_name: Tool Selection Accuracy
entrypoint: aip_traces_agent.eval.tool_selection_accuracy.get_trace_metric
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 4
  processing_kind: cpu
  timeout_seconds: 300
kind: trace_metric
manifest_version: '1'
metric_metadata:
  methodology:
  - For a trace or session resolved partition, the metric collects the distinct tools
    actually executed as their own TOOL-span step, binding to the canonical model
    rather than any wire-format key. A trace with no TOOL spans at all falls back
    to reading the tool calls embedded directly in its LLM-kind spans' own completions
    instead - for datasets where a tool call is part of the model's own response rather
    than a separate execution step. The two shapes are never mixed for one trace.
  - Scored per trace (TRACE), or pooled across every trace in the session (SESSION)
    - there is no span-level scoring; a standalone per-span score was tried and dropped
    for adding little real signal over TRACE/SESSION.
  - The observed tool set is compared against that resolved partition's own expected_tools
    (from the canonical_partition_v1 expected_tools column, or falling back to config),
    case- and separator-normalised — presence only, independent of call order or repeat
    count.
  - The score is the fraction of expected tools that were observed, normalised to
    0-1.
  score_semantics: Scores range 0-1, the fraction of the required tools the agent
    actually called. A high score means the agent selected the tools the task required;
    a low score means it skipped required tools. Higher is better. TRACE scores one
    trace's own called tools; SESSION pools tool calls across every trace in the session
    instead, for tasks where the right tool may be called in a later turn.
  summary: Measures tool-selection coverage — how many of the required tools the agent
    called, anywhere in the trace or session, regardless of order or repeat count.
    A trace with no TOOL spans at all falls back to reading tool calls embedded in
    its LLM spans. Scored per trace or pooled across a session; there is no span-level
    scoring.
  worked_example: A task requires get_weather and get_time; only get_weather appears
    as an executed tool step -> 1/2 = 0.5. For session scoring, it pools across all
    traces in the session. A trace with no TOOL spans at all instead reads that trace's
    LLM-kind spans' own embedded tool calls as the called set, scored the same way.
metric_type: pointwise
name: agent.tool_selection_accuracy
partition_types:
- trace
- session
required_columns:
- expected_tools
required_kinds:
- anyOf:
  - TOOL
  - LLM
scorer_contract: per_row
unsupported_trace_shapes: []
version: 2.0.0

Contract​

Required columns​

Accepted schemas​

Methodology​

Score semantics​

Worked example​

Configuration schema​

Execution​

Complete manifest​