Retrieval Correctness
rag.retrieval_correctness
Measures how well the retrieved results match the expected evidence, combining ranking quality (precision) with coverage (recall).
Contract
| Field | Value |
|---|---|
version | 1.0.0 |
metric_type | pointwise |
scorer_contract | per_row |
direction | higher_is_better |
entrypoint | aip_metrics_rag.answered.retrieval_correctness |
target_kind | None |
Required columns
input_idpromptreference_contextssut_responseretrieved_context
Accepted schemas
[
{
"name": "gdi_text_v1",
"task_types": [
"multi_turn_rag",
"single_turn_rag"
]
}
]
Methodology
- Deterministic - no LLM. Uses rapidfuzz two-stage matching between retrieved and reference chunks.
- Precision: walk the retrieved chunks in rank order, accumulating hits/rank so earlier matches count more.
- Recall: the fraction of reference chunks matched by at least one retrieved chunk.
- Combine as weight_precisionprecision + weight_recallrecall and normalise to 0-1.
Score semantics
Scores range 0-1, a weighted blend of rank-aware precision and reference recall (default 50/50). A high score means relevant chunks are retrieved early and all expected chunks are covered; a low score means noisy or incomplete retrieval. Higher is better.
Worked example
3 references, retrieved [relevant, noise, relevant] matching at ranks 1 and 3: precision = (1/1 + 2/3)/2 ~ 0.83, recall = 2/3 ~ 0.67 -> 0.50.83 + 0.50.67 = 0.75.
Configuration schema
{
"properties": {
"coverage": {
"default": 0.5,
"maximum": 1.0,
"minimum": 0.0,
"title": "Coverage",
"type": "number"
},
"inverted": {
"default": false,
"title": "Inverted",
"type": "boolean",
"x-aip-param-role": "scoring_metadata"
},
"language": {
"default": "en",
"description": "Language for output messages",
"enum": [
"en",
"de",
"fr"
],
"title": "Language",
"type": "string"
},
"max": {
"default": 1.0,
"title": "Max",
"type": "number",
"x-aip-param-role": "scoring_metadata"
},
"metric_name": {
"default": "retrieval_correctness",
"title": "Metric Name",
"type": "string",
"x-aip-param-role": "scoring_metadata"
},
"min": {
"default": 0.0,
"title": "Min",
"type": "number",
"x-aip-param-role": "scoring_metadata"
},
"t1": {
"default": 0.7,
"maximum": 1.0,
"minimum": 0.0,
"title": "T1",
"type": "number"
},
"t2": {
"default": 0.65,
"maximum": 1.0,
"minimum": 0.0,
"title": "T2",
"type": "number"
},
"weight": {
"default": 1.0,
"title": "Weight",
"type": "number",
"x-aip-param-role": "scoring_metadata"
},
"weight_precision": {
"default": 0.5,
"maximum": 1.0,
"minimum": 0.0,
"title": "Weight Precision",
"type": "number"
},
"weight_recall": {
"default": 0.5,
"maximum": 1.0,
"minimum": 0.0,
"title": "Weight Recall",
"type": "number"
}
},
"title": "RetrievalCorrectnessConfig",
"type": "object"
}
Execution
{
"emits_metric_family": false,
"function_name": null,
"max_concurrency": null,
"processing_kind": "cpu",
"stream_batch_size": null,
"timeout_seconds": 60
}
Complete manifest
accepts:
- name: gdi_text_v1
task_types:
- multi_turn_rag
- single_turn_rag
config_schema:
properties:
coverage:
default: 0.5
maximum: 1.0
minimum: 0.0
title: Coverage
type: number
inverted:
default: false
title: Inverted
type: boolean
x-aip-param-role: scoring_metadata
language:
default: en
description: Language for output messages
enum:
- en
- de
- fr
title: Language
type: string
max:
default: 1.0
title: Max
type: number
x-aip-param-role: scoring_metadata
metric_name:
default: retrieval_correctness
title: Metric Name
type: string
x-aip-param-role: scoring_metadata
min:
default: 0.0
title: Min
type: number
x-aip-param-role: scoring_metadata
t1:
default: 0.7
maximum: 1.0
minimum: 0.0
title: T1
type: number
t2:
default: 0.65
maximum: 1.0
minimum: 0.0
title: T2
type: number
weight:
default: 1.0
title: Weight
type: number
x-aip-param-role: scoring_metadata
weight_precision:
default: 0.5
maximum: 1.0
minimum: 0.0
title: Weight Precision
type: number
weight_recall:
default: 0.5
maximum: 1.0
minimum: 0.0
title: Weight Recall
type: number
title: RetrievalCorrectnessConfig
type: object
dependencies: []
description: How well do the retrieved results match the evidence expected for the
answer?
direction: higher_is_better
display_name: Retrieval Correctness
entrypoint: aip_metrics_rag.answered.retrieval_correctness
execution:
emits_metric_family: false
function_name: null
max_concurrency: null
processing_kind: cpu
stream_batch_size: null
timeout_seconds: 60
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
methodology:
- Deterministic - no LLM. Uses rapidfuzz two-stage matching between retrieved and
reference chunks.
- 'Precision: walk the retrieved chunks in rank order, accumulating hits/rank so
earlier matches count more.'
- 'Recall: the fraction of reference chunks matched by at least one retrieved chunk.'
- Combine as weight_precision*precision + weight_recall*recall and normalise to
0-1.
score_semantics: Scores range 0-1, a weighted blend of rank-aware precision and
reference recall (default 50/50). A high score means relevant chunks are retrieved
early and all expected chunks are covered; a low score means noisy or incomplete
retrieval. Higher is better.
summary: Measures how well the retrieved results match the expected evidence, combining
ranking quality (precision) with coverage (recall).
worked_example: '3 references, retrieved [relevant, noise, relevant] matching at
ranks 1 and 3: precision = (1/1 + 2/3)/2 ~ 0.83, recall = 2/3 ~ 0.67 -> 0.5*0.83
+ 0.5*0.67 = 0.75.'
metric_type: pointwise
name: rag.retrieval_correctness
partition_types: []
required_columns:
- input_id
- prompt
- reference_contexts
- sut_response
- retrieved_context
required_kinds: []
scorer_contract: per_row
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0