Hallucination
agent.hallucination
Measures how well agent text is grounded in real evidence, via LLM-judge scoring at either single-completion (span) or whole-trace granularity.
Contract
| Field | Value |
|---|---|
version | 1.0.0 |
metric_type | pointwise |
scorer_contract | per_row |
direction | higher_is_better |
entrypoint | aip_traces_agent.eval.hallucination.get_trace_metric |
target_kind | LLM |
partition_types | ["span", "trace"] |
required_kinds | [{"anyOf": ["TOOL", "LLM"]}] |
Required columns
No required columns are declared in this manifest.
Accepted schemas
[
{
"name": "canonical_partition_v1"
}
]
Methodology
- One metric spanning two granularities, each with its own judging path. At SPAN granularity, this metric's own definition restricts scoring to LLM-kind spans only — that's fixed by the metric, not something you configure — so only those are ever materialized as resolved partitions. Each is judged for faithfulness to data — the span's own text output against the tool_call_response evidence carried on that same span's input messages, scored over four weighted dimensions — numeric accuracy 35%, categorical accuracy 25%, no fabrication 30%, completeness 10%.
- A TRACE-level resolved partition is judged for final-answer traceability. The root span's final answer is split into claims, pre-filtered to drop self-references, confirmations, next-steps and filler, then each surviving claim is labelled supported, inferred, model_knowledge or hallucinated against the trace's tool-call evidence. Inferred and model_knowledge earn partial credit — legitimate degraded-mode output, not failures.
- The TRACE path requires a tree-shaped single-agent trace — exactly one root span, no non-root AGENT spans (no orchestrator/subagent hierarchy), and at least one descendant TOOL span. That last check looks for
TOOLspecifically — aRETRIEVERspan does not satisfy it, even when the retrieval call is itself the tool your agent invoked; kind that spanTOOLif you want it to count. A trace outside that shape is reported as a per-partition validation error rather than scored. - Both paths call a shared OpenAI-compatible judge model, configured the same way as any other judge-backed metric — through the run's
metric_configs(judge_connection_idor an inlinejudge_model/judge_api_key) — see Per-metric judge model. Internally the resolved connection reaches the scorer viaop_config.systems_params.evaluator_system.params, but that's plumbing, not something you set directly.
Score semantics
Scores range 0-1. Higher is better — 1.0 means the agent's text was fully grounded in the evidence available to it, low scores mean claims were fabricated or unsupported. SPAN and TRACE scores answer different questions (faithfulness of one completion to its own data vs. traceability of a whole trace's final answer) and are not interchangeable, which is why the partition_type is retained on every result row. Judge-backed and therefore non-deterministic; the four truncation limits on the config are score-defining, so runs using different values are not comparable.
Worked example
A TRACE-level resolved partition whose final answer makes four claims — three traceable to tool results and one invented figure — scores 0.75 (this coincides with a raw 3/4 fraction here only because the example uses just the two binary-weight labels — supported=1.0 and hallucinated=0.0 — and doesn't exercise the partial-credit inferred=0.5 and model_knowledge=0.7 labels), with the offending claim labelled hallucinated in the result metadata. A SPAN-level resolved partition quoting a figure that contradicts its own tool_call_response loses the numeric-accuracy dimension (35%) while keeping the rest.
Configuration schema
{
"description": "Parameters for HallucinationMetric.\n\nShared by both of ``HallucinationMetric``'s internal scoring paths \u2014 SPAN\n(``_score_span``/``_evaluate_faithfulness``) and TRACE (``_score_trace``/``_filter_claims``/\n``_detect_hallucinations``) \u2014 which draw on the same judge client, semaphore, and config\nfields rather than each holding its own. ``max_response_length`` / ``max_total_context`` /\n``max_tool_calls_length`` only matter on the TRACE path (subagent-trace truncation);\n``max_tool_response_length`` only matters on the SPAN path (tool_call_response truncation).\nEach path ignores the other's fields.\n\nAll four truncation fields are score-defining, not merely operational: each caps evidence\ntext the judge is shown, so lowering one can turn a claim the judge would otherwise verify\ninto one it can't \u2014 e.g. truncating the very tool_call_response value a claim depends on can\nflip that claim from supported to hallucinated. Comparing scores across runs that used\ndifferent values here is comparing different metrics, the same way comparing runs across\ndifferent judge rubrics would be. Truncation itself can't be turned off: an unbounded\ntool_call_response or tool_calls blob would make the judge prompt's size (and cost) a\nfunction of trace content instead of a bounded quantity.\n\nJudge connection details (api_key, base_url) are resolved separately from\n``systems_params[\"evaluator_system\"][\"params\"]`` in the factory \u2014 see ``__init__.py``.",
"properties": {
"concurrency_limit": {
"default": 10,
"description": "Max concurrent judge API calls",
"title": "Concurrency Limit",
"type": "integer",
"x-aip-param-role": "operational"
},
"judge_temperature": {
"default": 0.0,
"description": "Sampling temperature for the judge model",
"title": "Judge Temperature",
"type": "number"
},
"language": {
"default": "en",
"description": "Prompt language",
"title": "Language",
"type": "string"
},
"max_response_length": {
"default": 2000,
"description": "TRACE path only: max chars kept per subagent response.",
"title": "Max Response Length",
"type": "integer"
},
"max_tool_calls_length": {
"default": 500,
"description": "TRACE path only: max chars kept for a subagent's serialized tool_calls JSON in its evidence block.",
"title": "Max Tool Calls Length",
"type": "integer"
},
"max_tool_response_length": {
"default": 400,
"description": "SPAN path only: max chars kept per string value inside a span's own tool_call_response data before it reaches the faithfulness judge.",
"title": "Max Tool Response Length",
"type": "integer"
},
"max_total_context": {
"default": 8000,
"description": "TRACE path only: max total chars across all subagent traces.",
"title": "Max Total Context",
"type": "integer"
},
"metric_name": {
"default": "hallucination",
"title": "Metric Name",
"type": "string",
"x-aip-param-role": "scoring_metadata"
},
"model": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Judge model id; falls back to the evaluator_system model if unset",
"title": "Model"
},
"timeout_seconds": {
"default": 60.0,
"description": "Timeout per judge API call, in seconds",
"title": "Timeout Seconds",
"type": "number",
"x-aip-param-role": "operational"
}
},
"title": "HallucinationConfig",
"type": "object"
}
Execution
{
"emits_metric_family": false,
"function_name": null,
"max_concurrency": 8,
"processing_kind": "network",
"timeout_seconds": 900
}
Complete manifest
accepts:
- name: canonical_partition_v1
config_schema:
description: 'Parameters for HallucinationMetric.
Shared by both of ``HallucinationMetric``''s internal scoring paths — SPAN
(``_score_span``/``_evaluate_faithfulness``) and TRACE (``_score_trace``/``_filter_claims``/
``_detect_hallucinations``) — which draw on the same judge client, semaphore,
and config
fields rather than each holding its own. ``max_response_length`` / ``max_total_context``
/
``max_tool_calls_length`` only matter on the TRACE path (subagent-trace truncation);
``max_tool_response_length`` only matters on the SPAN path (tool_call_response
truncation).
Each path ignores the other''s fields.
All four truncation fields are score-defining, not merely operational: each caps
evidence
text the judge is shown, so lowering one can turn a claim the judge would otherwise
verify
into one it can''t — e.g. truncating the very tool_call_response value a claim
depends on can
flip that claim from supported to hallucinated. Comparing scores across runs that
used
different values here is comparing different metrics, the same way comparing runs
across
different judge rubrics would be. Truncation itself can''t be turned off: an unbounded
tool_call_response or tool_calls blob would make the judge prompt''s size (and
cost) a
function of trace content instead of a bounded quantity.
Judge connection details (api_key, base_url) are resolved separately from
``systems_params["evaluator_system"]["params"]`` in the factory — see ``__init__.py``.'
properties:
concurrency_limit:
default: 10
description: Max concurrent judge API calls
title: Concurrency Limit
type: integer
x-aip-param-role: operational
judge_temperature:
default: 0.0
description: Sampling temperature for the judge model
title: Judge Temperature
type: number
language:
default: en
description: Prompt language
title: Language
type: string
max_response_length:
default: 2000
description: 'TRACE path only: max chars kept per subagent response.'
title: Max Response Length
type: integer
max_tool_calls_length:
default: 500
description: 'TRACE path only: max chars kept for a subagent''s serialized tool_calls
JSON in its evidence block.'
title: Max Tool Calls Length
type: integer
max_tool_response_length:
default: 400
description: 'SPAN path only: max chars kept per string value inside a span''s
own tool_call_response data before it reaches the faithfulness judge.'
title: Max Tool Response Length
type: integer
max_total_context:
default: 8000
description: 'TRACE path only: max total chars across all subagent traces.'
title: Max Total Context
type: integer
metric_name:
default: hallucination
title: Metric Name
type: string
x-aip-param-role: scoring_metadata
model:
anyOf:
- type: string
- type: 'null'
default: null
description: Judge model id; falls back to the evaluator_system model if unset
title: Model
timeout_seconds:
default: 60.0
description: Timeout per judge API call, in seconds
title: Timeout Seconds
type: number
x-aip-param-role: operational
title: HallucinationConfig
type: object
dependencies: []
description: Judges whether agent text is grounded in real evidence — a SPAN partition
via faithfulness-to-data judging, a TRACE partition via final-answer traceability
judging.
direction: higher_is_better
display_name: Hallucination
entrypoint: aip_traces_agent.eval.hallucination.get_trace_metric
execution:
emits_metric_family: false
function_name: null
max_concurrency: 8
processing_kind: network
timeout_seconds: 900
kind: trace_metric
manifest_version: '1'
metric_metadata:
methodology:
- One metric spanning two granularities, each with its own judging path. At SPAN
granularity, this metric's own definition restricts scoring to LLM-kind spans
only — that's fixed by the metric, not something you configure — so only those
are ever materialized as resolved partitions. Each is judged for faithfulness
to data — the span's own text output against the tool_call_response evidence carried
on that same span's input messages, scored over four weighted dimensions — numeric
accuracy 35%, categorical accuracy 25%, no fabrication 30%, completeness 10%.
- A TRACE-level resolved partition is judged for final-answer traceability. The
root span's final answer is split into claims, pre-filtered to drop self-references,
confirmations, next-steps and filler, then each surviving claim is labelled supported,
inferred, model_knowledge or hallucinated against the trace's tool-call evidence.
Inferred and model_knowledge earn partial credit — legitimate degraded-mode output,
not failures.
- The TRACE path requires a tree-shaped single-agent trace — exactly one root span,
no non-root AGENT spans (no orchestrator/subagent hierarchy), and at least one
descendant TOOL span. That last check looks for `TOOL` specifically — a `RETRIEVER`
span does not satisfy it, even when the retrieval call is itself the tool your
agent invoked; kind that span `TOOL` if you want it to count. A trace outside
that shape is reported as a per-partition validation error rather than scored.
- Both paths call a shared OpenAI-compatible judge model, configured the same way
as any other judge-backed metric — through the run's `metric_configs` (`judge_connection_id`
or an inline `judge_model`/`judge_api_key`) — see [Per-metric judge model](#per-metric-judge-model-run_configmetric_configs).
Internally the resolved connection reaches the scorer via `op_config.systems_params.evaluator_system.params`,
but that's plumbing, not something you set directly.
score_semantics: Scores range 0-1. Higher is better — 1.0 means the agent's text
was fully grounded in the evidence available to it, low scores mean claims were
fabricated or unsupported. SPAN and TRACE scores answer different questions (faithfulness
of one completion to its own data vs. traceability of a whole trace's final answer)
and are not interchangeable, which is why the partition_type is retained on every
result row. Judge-backed and therefore non-deterministic; the four truncation
limits on the config are score-defining, so runs using different values are not
comparable.
summary: Measures how well agent text is grounded in real evidence, via LLM-judge
scoring at either single-completion (span) or whole-trace granularity.
worked_example: A TRACE-level resolved partition whose final answer makes four claims
— three traceable to tool results and one invented figure — scores 0.75 (this
coincides with a raw 3/4 fraction here only because the example uses just the
two binary-weight labels — supported=1.0 and hallucinated=0.0 — and doesn't exercise
the partial-credit inferred=0.5 and model_knowledge=0.7 labels), with the offending
claim labelled hallucinated in the result metadata. A SPAN-level resolved partition
quoting a figure that contradicts its own tool_call_response loses the numeric-accuracy
dimension (35%) while keeping the rest.
metric_type: pointwise
name: agent.hallucination
partition_types:
- span
- trace
required_columns: []
required_kinds:
- anyOf:
- TOOL
- LLM
scorer_contract: per_row
target_kind: LLM
unsupported_trace_shapes:
- orchestrator_subagent
version: 1.0.0