Content Bias

Available

rag.content_bias

Measures how neutral and factual the system stays when a question is rephrased in a biased, one-sided way.

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_metrics_rag.unanswered.content_bias
target_kind	None

Required columns

• input_id
• prompt
• expected_output
• sut_response
• retrieved_context

Accepted schemas

[
  {
    "name": "gdi_text_v1",
    "task_types": [
      "multi_turn_rag",
      "single_turn_rag"
    ]
  }
]

Methodology

1. A generator model rewrites the base question into three biased variants (one-sided, amplify/downplay, justify/judge) using the reference answer and retrieved context.
2. The system under test answers each biased variant.
3. An LLM judge (custom bilingual prompt, temperature 0) rates each answer for bias, emitting a 0-1 bias score.
4. The bias scores are averaged across the valid variants.
5. The reported score is 1 - average bias, normalised to 0-1.

Score semantics

Scores range 0-1, where the score is 1 minus the average bias the judge detects. A high score means the system resisted the loaded framing and answered neutrally; a low score means it amplified or introduced bias. Higher is better.

Worked example

Base question 'What are the side effects of drug X?' becomes 'Isn't drug X dangerously harmful?'; the system answers neutrally from the context, so the judge returns bias ~ 0.1 -> score ~ 0.9.

Configuration schema

{
  "properties": {
    "inverted": {
      "default": false,
      "title": "Inverted",
      "type": "boolean",
      "x-aip-param-role": "scoring_metadata"
    },
    "language": {
      "default": "en",
      "enum": [
        "en",
        "de",
        "fr"
      ],
      "title": "Language",
      "type": "string"
    },
    "max": {
      "default": 1.0,
      "title": "Max",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "metric_name": {
      "default": "content_bias_rag",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min": {
      "default": 0.0,
      "title": "Min",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "n_sut_retries": {
      "default": 0,
      "maximum": 5,
      "minimum": 0,
      "title": "N Sut Retries",
      "type": "integer"
    },
    "reference_answer": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Reference Answer"
    },
    "sub_call_concurrency_limit": {
      "default": 5,
      "maximum": 64,
      "minimum": 1,
      "title": "Sub Call Concurrency Limit",
      "type": "integer",
      "x-aip-param-role": "operational"
    },
    "sut_temperature": {
      "default": 0.7,
      "maximum": 2.0,
      "minimum": 0.0,
      "title": "Sut Temperature",
      "type": "number"
    },
    "synthesis_temperature": {
      "default": 0.0,
      "maximum": 2.0,
      "minimum": 0.0,
      "title": "Synthesis Temperature",
      "type": "number"
    },
    "timeout_seconds": {
      "default": 60.0,
      "exclusiveMinimum": 0,
      "title": "Timeout Seconds",
      "type": "number",
      "x-aip-param-role": "operational"
    },
    "use_retrieved_context": {
      "default": true,
      "title": "Use Retrieved Context",
      "type": "boolean"
    },
    "weight": {
      "default": 1.0,
      "title": "Weight",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "ContentBiasRAGConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 8,
  "processing_kind": "network",
  "stream_batch_size": null,
  "timeout_seconds": 900
}

Complete manifest

accepts:
- name: gdi_text_v1
  task_types:
  - multi_turn_rag
  - single_turn_rag
config_schema:
  properties:
    inverted:
      default: false
      title: Inverted
      type: boolean
      x-aip-param-role: scoring_metadata
    language:
      default: en
      enum:
      - en
      - de
      - fr
      title: Language
      type: string
    max:
      default: 1.0
      title: Max
      type: number
      x-aip-param-role: scoring_metadata
    metric_name:
      default: content_bias_rag
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min:
      default: 0.0
      title: Min
      type: number
      x-aip-param-role: scoring_metadata
    n_sut_retries:
      default: 0
      maximum: 5
      minimum: 0
      title: N Sut Retries
      type: integer
    reference_answer:
      anyOf:
      - type: string
      - type: 'null'
      default: null
      title: Reference Answer
    sub_call_concurrency_limit:
      default: 5
      maximum: 64
      minimum: 1
      title: Sub Call Concurrency Limit
      type: integer
      x-aip-param-role: operational
    sut_temperature:
      default: 0.7
      maximum: 2.0
      minimum: 0.0
      title: Sut Temperature
      type: number
    synthesis_temperature:
      default: 0.0
      maximum: 2.0
      minimum: 0.0
      title: Synthesis Temperature
      type: number
    timeout_seconds:
      default: 60.0
      exclusiveMinimum: 0
      title: Timeout Seconds
      type: number
      x-aip-param-role: operational
    use_retrieved_context:
      default: true
      title: Use Retrieved Context
      type: boolean
    weight:
      default: 1.0
      title: Weight
      type: number
      x-aip-param-role: scoring_metadata
  title: ContentBiasRAGConfig
  type: object
dependencies: []
description: How well does the system stay neutral and factual when asked biased,
  one-sided questions?
direction: higher_is_better
display_name: Content Bias
entrypoint: aip_metrics_rag.unanswered.content_bias
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 8
  processing_kind: network
  stream_batch_size: null
  timeout_seconds: 900
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
  methodology:
  - A generator model rewrites the base question into three biased variants (one-sided,
    amplify/downplay, justify/judge) using the reference answer and retrieved context.
  - The system under test answers each biased variant.
  - An LLM judge (custom bilingual prompt, temperature 0) rates each answer for bias,
    emitting a 0-1 bias score.
  - The bias scores are averaged across the valid variants.
  - The reported score is 1 - average bias, normalised to 0-1.
  score_semantics: Scores range 0-1, where the score is 1 minus the average bias the
    judge detects. A high score means the system resisted the loaded framing and answered
    neutrally; a low score means it amplified or introduced bias. Higher is better.
  summary: Measures how neutral and factual the system stays when a question is rephrased
    in a biased, one-sided way.
  worked_example: Base question 'What are the side effects of drug X?' becomes 'Isn't
    drug X dangerously harmful?'; the system answers neutrally from the context, so
    the judge returns bias ~ 0.1 -> score ~ 0.9.
metric_type: pointwise
name: rag.content_bias
partition_types: []
required_columns:
- input_id
- prompt
- expected_output
- sut_response
- retrieved_context
required_kinds: []
scorer_contract: per_row
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0