Answer Correctness

Available

llm.correctness

Measures how well the facts in the response agree with the reference answer, in both accuracy and coverage.

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_metrics_llm.answered.correctness
target_kind	None

Required columns

• input_id
• prompt
• expected_output
• sut_response

Accepted schemas

[
  {
    "name": "gdi_text_v1",
    "task_types": [
      "multi_turn_llm",
      "multi_turn_rag",
      "single_turn_llm",
      "single_turn_rag"
    ]
  }
]

Methodology

1. An LLM judge (Ragas factual correctness) decomposes the response and the reference into atomic claims.
2. Claims not relevant to the question are optionally filtered out.
3. Claims are matched both ways: response-vs-reference (precision) and reference-vs-response (recall).
4. True/false positives and negatives are counted and combined into an F1 score (default mode).

Score semantics

Scores range 0-1, an F1 balance of factual precision and recall. A high score means the response's claims match the reference with few errors or omissions; a low score means missing or incorrect facts. Higher is better.

Worked example

For a reference listing 6 EU founding members where the response names 3 correctly with no errors: precision 1.0, recall 0.5 -> F1 ~ 0.67.

Configuration schema

{
  "description": "Configuration for claim-based answer correctness metric (Ragas).",
  "properties": {
    "beta": {
      "default": 1.0,
      "title": "Beta",
      "type": "number"
    },
    "filter_irrelevant_claims": {
      "default": true,
      "title": "Filter Irrelevant Claims",
      "type": "boolean"
    },
    "inverted": {
      "default": false,
      "title": "Inverted",
      "type": "boolean",
      "x-aip-param-role": "scoring_metadata"
    },
    "judge_temperature": {
      "default": 0.0,
      "maximum": 2.0,
      "minimum": 0.0,
      "title": "Judge Temperature",
      "type": "number"
    },
    "lang": {
      "default": "en",
      "enum": [
        "en",
        "de",
        "fr"
      ],
      "title": "Lang",
      "type": "string"
    },
    "max": {
      "default": 1.0,
      "title": "Max",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "metric_name": {
      "default": "correctness",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min": {
      "default": 0.0,
      "title": "Min",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "mode": {
      "default": "f1",
      "enum": [
        "precision",
        "recall",
        "f1"
      ],
      "title": "Mode",
      "type": "string"
    },
    "model": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Model"
    },
    "timeout_seconds": {
      "default": 180.0,
      "exclusiveMinimum": 0,
      "title": "Timeout Seconds",
      "type": "number",
      "x-aip-param-role": "operational"
    },
    "weight": {
      "default": 1.0,
      "title": "Weight",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "InitializationConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 8,
  "processing_kind": "network",
  "stream_batch_size": 5000,
  "timeout_seconds": 21600
}

Complete manifest

accepts:
- name: gdi_text_v1
  task_types:
  - multi_turn_llm
  - multi_turn_rag
  - single_turn_llm
  - single_turn_rag
config_schema:
  description: Configuration for claim-based answer correctness metric (Ragas).
  properties:
    beta:
      default: 1.0
      title: Beta
      type: number
    filter_irrelevant_claims:
      default: true
      title: Filter Irrelevant Claims
      type: boolean
    inverted:
      default: false
      title: Inverted
      type: boolean
      x-aip-param-role: scoring_metadata
    judge_temperature:
      default: 0.0
      maximum: 2.0
      minimum: 0.0
      title: Judge Temperature
      type: number
    lang:
      default: en
      enum:
      - en
      - de
      - fr
      title: Lang
      type: string
    max:
      default: 1.0
      title: Max
      type: number
      x-aip-param-role: scoring_metadata
    metric_name:
      default: correctness
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min:
      default: 0.0
      title: Min
      type: number
      x-aip-param-role: scoring_metadata
    mode:
      default: f1
      enum:
      - precision
      - recall
      - f1
      title: Mode
      type: string
    model:
      anyOf:
      - type: string
      - type: 'null'
      default: null
      title: Model
    timeout_seconds:
      default: 180.0
      exclusiveMinimum: 0
      title: Timeout Seconds
      type: number
      x-aip-param-role: operational
    weight:
      default: 1.0
      title: Weight
      type: number
      x-aip-param-role: scoring_metadata
  title: InitializationConfig
  type: object
dependencies: []
description: How well do the response's facts match the reference, in accuracy and
  coverage?
direction: higher_is_better
display_name: Answer Correctness
entrypoint: aip_metrics_llm.answered.correctness
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 8
  processing_kind: network
  stream_batch_size: 5000
  timeout_seconds: 21600
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
  methodology:
  - An LLM judge (Ragas factual correctness) decomposes the response and the reference
    into atomic claims.
  - Claims not relevant to the question are optionally filtered out.
  - 'Claims are matched both ways: response-vs-reference (precision) and reference-vs-response
    (recall).'
  - True/false positives and negatives are counted and combined into an F1 score (default
    mode).
  score_semantics: Scores range 0-1, an F1 balance of factual precision and recall.
    A high score means the response's claims match the reference with few errors or
    omissions; a low score means missing or incorrect facts. Higher is better.
  summary: Measures how well the facts in the response agree with the reference answer,
    in both accuracy and coverage.
  worked_example: 'For a reference listing 6 EU founding members where the response
    names 3 correctly with no errors: precision 1.0, recall 0.5 -> F1 ~ 0.67.'
metric_type: pointwise
name: llm.correctness
partition_types: []
required_columns:
- input_id
- prompt
- expected_output
- sut_response
required_kinds: []
scorer_contract: per_row
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0