Sub-agent Validity

Available

agent.subagent_validity

Rule-based (no LLM judge) validity classification for every sub-agent (AGENT-kind) span's response - valid, empty, short, or error - reduced per partition to whether every sub-agent behaved validly this turn (TRACE) or the ratio of valid sub-agents across the session (SESSION).

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_traces_agent.eval.subagent_validity.get_trace_metric
target_kind	AGENT
partition_types	["span", "trace", "session"]
required_kinds	["AGENT"]

Required columns

No required columns are declared in this manifest.

Accepted schemas

[
  {
    "name": "canonical_partition_v1"
  }
]

Methodology

1. Checks run in a fixed order per AGENT span - error, then empty, then short - and the first match wins; only a span that fails none of them is 'valid'.
2. 'error' is a span whose status is an error status.
3. 'empty' is a span with no text output AND no non-text output (no tool call, no non-text message part) - a subagent that produced nothing at all.
4. 'short' is a span with non-empty text output shorter than min_response_length characters (default 10) — set per run via metric_configs, e.g. metric_configs={"agent.subagent_validity": {"min_response_length": 15}} (see Target-scoped configs for a full example). A tool-call-only response with no prose is exempt from this check (it passed Check 2 already) and scores 'valid' instead.
5. 'valid' spans (all checks pass) score 1.0; every other label scores 0.0.
6. The per-span verdict is reduced per partition type - SPAN scores one sub-agent span's verdict in isolation. TRACE takes min(per-span scores) across the trace's sub-agent spans - 1.0 only if every one is valid. SESSION takes the ratio of valid sub-agent spans across every trace in the session.

Score semantics

SPAN scores each sub-agent span 1.0 (valid) or 0.0 (empty/short/error). TRACE scores 1.0 only if every sub-agent span in the trace is valid, else 0.0. SESSION scores the fraction of valid sub-agent spans across the whole session. Higher is better throughout. min_response_length is configurable per run (not fixed platform-wide) for subagents that legitimately return short answers, e.g. a coordinate or an ID.

Worked example

An AGENT span has status=OK and output text 'Refund of $42.50 processed.' (27 chars, above the default min_response_length=10) -> label='valid', score=1.0. A sibling AGENT span has status=OK but empty output_messages and no tool_calls -> label='empty', score=0.0. At TRACE scope, a trace containing those same two sub-agent spans scores 0.0 (not every sub-agent was valid). Pooling two such traces at SESSION scope (4 sub-agent spans total, 2 valid) scores 2/4 = 0.5.

Configuration schema

{
  "description": "Parameters for SubagentValidityMetric.",
  "properties": {
    "metric_name": {
      "default": "subagent_validity",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min_response_length": {
      "default": 10,
      "description": "Responses shorter than this (chars) are flagged as suspiciously short. Raise this for deployments where subagents legitimately return short answers (e.g. a coordinate or an ID).",
      "title": "Min Response Length",
      "type": "integer"
    }
  },
  "title": "SubagentValidityConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 4,
  "processing_kind": "cpu",
  "timeout_seconds": 300
}

Complete manifest

accepts:
- name: canonical_partition_v1
config_schema:
  description: Parameters for SubagentValidityMetric.
  properties:
    metric_name:
      default: subagent_validity
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min_response_length:
      default: 10
      description: Responses shorter than this (chars) are flagged as suspiciously
        short. Raise this for deployments where subagents legitimately return short
        answers (e.g. a coordinate or an ID).
      title: Min Response Length
      type: integer
  title: SubagentValidityConfig
  type: object
dependencies: []
description: Rule-based validity classification (valid/empty/short/error) for each
  sub-agent (AGENT) span, reduced to TRACE (did every sub-agent behave validly this
  turn) and SESSION (ratio of valid sub-agents across the session) scope.
direction: higher_is_better
display_name: Sub-agent Validity
entrypoint: aip_traces_agent.eval.subagent_validity.get_trace_metric
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 4
  processing_kind: cpu
  timeout_seconds: 300
kind: trace_metric
manifest_version: '1'
metric_metadata:
  methodology:
  - Checks run in a fixed order per AGENT span - error, then empty, then short - and
    the first match wins; only a span that fails none of them is 'valid'.
  - '''error'' is a span whose status is an error status.'
  - '''empty'' is a span with no text output AND no non-text output (no tool call,
    no non-text message part) - a subagent that produced nothing at all.'
  - '''short'' is a span with non-empty text output shorter than `min_response_length`
    characters (default 10) — set per run via `metric_configs`, e.g. `metric_configs={"agent.subagent_validity":
    {"min_response_length": 15}}` (see [Target-scoped configs](#target-scoped-configs-schema-v2)
    for a full example). A tool-call-only response with no prose is exempt from this
    check (it passed Check 2 already) and scores ''valid'' instead.'
  - '''valid'' spans (all checks pass) score 1.0; every other label scores 0.0.'
  - The per-span verdict is reduced per partition type - SPAN scores one sub-agent
    span's verdict in isolation. TRACE takes min(per-span scores) across the trace's
    sub-agent spans - 1.0 only if every one is valid. SESSION takes the ratio of valid
    sub-agent spans across every trace in the session.
  score_semantics: SPAN scores each sub-agent span 1.0 (valid) or 0.0 (empty/short/error).
    TRACE scores 1.0 only if every sub-agent span in the trace is valid, else 0.0.
    SESSION scores the fraction of valid sub-agent spans across the whole session.
    Higher is better throughout. `min_response_length` is configurable per run (not
    fixed platform-wide) for subagents that legitimately return short answers, e.g.
    a coordinate or an ID.
  summary: Rule-based (no LLM judge) validity classification for every sub-agent (AGENT-kind)
    span's response - valid, empty, short, or error - reduced per partition to whether
    every sub-agent behaved validly this turn (TRACE) or the ratio of valid sub-agents
    across the session (SESSION).
  worked_example: An AGENT span has status=OK and output text 'Refund of $42.50 processed.'
    (27 chars, above the default min_response_length=10) -> label='valid', score=1.0.
    A sibling AGENT span has status=OK but empty output_messages and no tool_calls
    -> label='empty', score=0.0. At TRACE scope, a trace containing those same two
    sub-agent spans scores 0.0 (not every sub-agent was valid). Pooling two such traces
    at SESSION scope (4 sub-agent spans total, 2 valid) scores 2/4 = 0.5.
metric_type: pointwise
name: agent.subagent_validity
partition_types:
- span
- trace
- session
required_columns: []
required_kinds:
- AGENT
scorer_contract: per_row
target_kind: AGENT
unsupported_trace_shapes: []
version: 1.0.0