Factual Consistency
llm.factual_consistency
Measures whether the system agrees on facts, numbers, dates, and named entities when its query is reformulated, compared against its original answer.
Contract
| Field | Value |
|---|---|
version | 1.0.0 |
metric_type | pointwise |
scorer_contract | per_row |
direction | higher_is_better |
entrypoint | aip_metrics_llm.unanswered.factual_consistency |
target_kind | None |
Required columns
input_idpromptsut_response
Accepted schemas
[
{
"name": "gdi_text_v1",
"task_types": [
"multi_turn_llm",
"multi_turn_rag",
"single_turn_llm",
"single_turn_rag"
]
}
]
Methodology
- Build a reformulated variant of the row's query by substituting it into a randomly chosen inquiry-framing template (verification request, citation request, third-party, formal, ...) that wraps the question in a different framing without altering its substance.
- Sample responses_per_query - 1 independent SUT responses to that reformulation variant; the row's existing sut_response is reused as the base response rather than resampled, so cost stays linear in responses_per_query.
- A judge LLM compares the base response against each reformulation response and labels their agreement on key facts, numbers, dates, and named entities as Consistent, Slightly Inconsistent, or Contradictory, with a 0-1 score.
- Average the scores of the successfully judged comparisons; one whose judge call fails is excluded, and the metric only errors out if every comparison failed.
Score semantics
Scores range 0-1, the mean agreement between the base response and each reformulated-query response on their facts, numbers, dates, and named entities. A high score means the system's facts stay stable regardless of how the question is asked; a low score means reformulation surfaces contradictory factual details. Higher is better.
Worked example
A row's query is reframed via a reformulation template into a differently-worded variant; with responses_per_query=3, 2 fresh responses to that variant are each compared against the row's existing response (2 comparisons) — 1 judged Consistent (1.0) and 1 judged Contradictory (0.4, a conflicting date) -> mean = 0.7.
Configuration schema
{
"properties": {
"concurrency_limit": {
"default": 5,
"description": "Maximum SUT and judge calls in flight at once for this metric. A slot is held per attempt, so a backing-off retry does not occupy one while it waits.",
"maximum": 64,
"minimum": 1,
"title": "Concurrency Limit",
"type": "integer",
"x-aip-param-role": "operational"
},
"inverted": {
"default": false,
"title": "Inverted",
"type": "boolean",
"x-aip-param-role": "scoring_metadata"
},
"max": {
"default": 1.0,
"title": "Max",
"type": "number",
"x-aip-param-role": "scoring_metadata"
},
"metric_name": {
"default": "factual_consistency",
"title": "Metric Name",
"type": "string",
"x-aip-param-role": "scoring_metadata"
},
"min": {
"default": 0.0,
"title": "Min",
"type": "number",
"x-aip-param-role": "scoring_metadata"
},
"n_judge_retries": {
"default": 2,
"description": "Retries for a single judge call before that comparison is counted as failed. Defaults above 0: a transient judge failure would otherwise silently shrink the denominator the score is averaged over, which is the failure mode this budget exists to prevent.",
"maximum": 5,
"minimum": 0,
"title": "N Judge Retries",
"type": "integer"
},
"n_sut_retries": {
"default": 2,
"description": "Retries for a single SUT call before it is treated as a hard failure that errors the whole row. Defaults above 0: this family fans out responses_per_query SUT calls per row, so a single transient call failure would otherwise lose the row entirely.",
"maximum": 5,
"minimum": 0,
"title": "N Sut Retries",
"type": "integer"
},
"responses_per_query": {
"default": 3,
"description": "Total responses compared for a row, counting the row's existing sut_response as the base. The metric samples responses_per_query - 1 fresh responses to the reformulation variant and judges each one against that base response, so cost stays linear in this value.",
"maximum": 8,
"minimum": 2,
"title": "Responses Per Query",
"type": "integer"
},
"sut_temperature": {
"default": 0.7,
"description": "Sampling temperature for the SUT calls that answer the reformulation variant. Above 0 so repeated samples can differ; the judge is always called at temperature 0.",
"maximum": 2.0,
"minimum": 0.0,
"title": "Sut Temperature",
"type": "number"
},
"timeout_seconds": {
"default": 60.0,
"description": "Per-call timeout in seconds, applied to each individual SUT and judge request.",
"exclusiveMinimum": 0,
"title": "Timeout Seconds",
"type": "number",
"x-aip-param-role": "operational"
},
"weight": {
"default": 1.0,
"title": "Weight",
"type": "number",
"x-aip-param-role": "scoring_metadata"
}
},
"title": "FactualConsistencyConfig",
"type": "object"
}
Execution
{
"emits_metric_family": false,
"function_name": null,
"max_concurrency": 8,
"processing_kind": "network",
"stream_batch_size": null,
"timeout_seconds": 900
}
Complete manifest
accepts:
- name: gdi_text_v1
task_types:
- multi_turn_llm
- multi_turn_rag
- single_turn_llm
- single_turn_rag
config_schema:
properties:
concurrency_limit:
default: 5
description: Maximum SUT and judge calls in flight at once for this metric.
A slot is held per attempt, so a backing-off retry does not occupy one while
it waits.
maximum: 64
minimum: 1
title: Concurrency Limit
type: integer
x-aip-param-role: operational
inverted:
default: false
title: Inverted
type: boolean
x-aip-param-role: scoring_metadata
max:
default: 1.0
title: Max
type: number
x-aip-param-role: scoring_metadata
metric_name:
default: factual_consistency
title: Metric Name
type: string
x-aip-param-role: scoring_metadata
min:
default: 0.0
title: Min
type: number
x-aip-param-role: scoring_metadata
n_judge_retries:
default: 2
description: 'Retries for a single judge call before that comparison is counted
as failed. Defaults above 0: a transient judge failure would otherwise silently
shrink the denominator the score is averaged over, which is the failure mode
this budget exists to prevent.'
maximum: 5
minimum: 0
title: N Judge Retries
type: integer
n_sut_retries:
default: 2
description: 'Retries for a single SUT call before it is treated as a hard failure
that errors the whole row. Defaults above 0: this family fans out responses_per_query
SUT calls per row, so a single transient call failure would otherwise lose
the row entirely.'
maximum: 5
minimum: 0
title: N Sut Retries
type: integer
responses_per_query:
default: 3
description: Total responses compared for a row, counting the row's existing
sut_response as the base. The metric samples responses_per_query - 1 fresh
responses to the reformulation variant and judges each one against that base
response, so cost stays linear in this value.
maximum: 8
minimum: 2
title: Responses Per Query
type: integer
sut_temperature:
default: 0.7
description: Sampling temperature for the SUT calls that answer the reformulation
variant. Above 0 so repeated samples can differ; the judge is always called
at temperature 0.
maximum: 2.0
minimum: 0.0
title: Sut Temperature
type: number
timeout_seconds:
default: 60.0
description: Per-call timeout in seconds, applied to each individual SUT and
judge request.
exclusiveMinimum: 0
title: Timeout Seconds
type: number
x-aip-param-role: operational
weight:
default: 1.0
title: Weight
type: number
x-aip-param-role: scoring_metadata
title: FactualConsistencyConfig
type: object
dependencies: []
description: Does the system agree on facts, numbers, dates, and named entities when
its query is reformulated, compared against its original answer?
direction: higher_is_better
display_name: Factual Consistency
entrypoint: aip_metrics_llm.unanswered.factual_consistency
execution:
emits_metric_family: false
function_name: null
max_concurrency: 8
processing_kind: network
stream_batch_size: null
timeout_seconds: 900
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
methodology:
- Build a reformulated variant of the row's query by substituting it into a randomly
chosen inquiry-framing template (verification request, citation request, third-party,
formal, ...) that wraps the question in a different framing without altering its
substance.
- Sample responses_per_query - 1 independent SUT responses to that reformulation
variant; the row's existing sut_response is reused as the base response rather
than resampled, so cost stays linear in responses_per_query.
- A judge LLM compares the base response against each reformulation response and
labels their agreement on key facts, numbers, dates, and named entities as Consistent,
Slightly Inconsistent, or Contradictory, with a 0-1 score.
- Average the scores of the successfully judged comparisons; one whose judge call
fails is excluded, and the metric only errors out if every comparison failed.
score_semantics: Scores range 0-1, the mean agreement between the base response
and each reformulated-query response on their facts, numbers, dates, and named
entities. A high score means the system's facts stay stable regardless of how
the question is asked; a low score means reformulation surfaces contradictory
factual details. Higher is better.
summary: Measures whether the system agrees on facts, numbers, dates, and named
entities when its query is reformulated, compared against its original answer.
worked_example: A row's query is reframed via a reformulation template into a differently-worded
variant; with responses_per_query=3, 2 fresh responses to that variant are each
compared against the row's existing response (2 comparisons) — 1 judged Consistent
(1.0) and 1 judged Contradictory (0.4, a conflicting date) -> mean = 0.7.
metric_type: pointwise
name: llm.factual_consistency
partition_types: []
required_columns:
- input_id
- prompt
- sut_response
required_kinds: []
scorer_contract: per_row
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0