Specificity

Available

object_detection.specificity

How reliably the model avoids raising detections for object classes that are not present in the image.

Contract

Field	Value
version	1.1.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_metrics_object_detection.answered.specificity
target_kind	None

Required columns

• image_id
• label
• predictions

Accepted schemas

[
  {
    "name": "gdi_image_v1",
    "task_types": [
      "detection"
    ]
  }
]

Methodology

1. Deterministic - no LLM. Uses Resaro's cv-eval library (custom Jaccard IoU, default threshold 0.5).
2. Build the dataset label vocabulary from classes observed in the full golden ground truth before unanswered rows are removed.
3. Use the configured label map only to translate labels and aliases. Unused mapped classes do not widen the denominator.
4. For each row, start with golden classes absent from that image and add any post-threshold prediction-only classes to that row's absent set.
5. Return null only when the row's absent set is empty. Otherwise count predicted absent classes as false positives and report specificity = true negatives / absent classes. Dataset rollups exclude null rows, and a rollup containing only null rows remains null.
6. An empty prediction set scores 1.0 when at least one class is absent because it creates no false alarms. Each distinct false-alarm class lowers the row score by 1/K, where K is the number of absent classes. One false alarm scores 0.0 at K=1 and 0.5 at K=2. Pair specificity with recall to measure missed objects.

Score semantics

Scores range 0-1, the true-negative rate over classes absent from each image. A high score means the model raises no false alarms for absent classes; a low score means it hallucinates objects. Empty predictions score 1.0 when an absent class exists. Rows with no absent class return null and are excluded from rollups. The score changes in steps of 1/K for K absent classes on that row, including any prediction-only classes emitted there. Higher is better.

Worked example

In a car and dog golden dataset, an image containing only a car where the model predicts the unseen class bird has two absent classes. Bird is a false positive and dog is a true negative, so specificity = 1/2 = 0.5.

Configuration schema

{
  "properties": {
    "inverted": {
      "default": false,
      "title": "Inverted",
      "type": "boolean",
      "x-aip-param-role": "scoring_metadata"
    },
    "iou_threshold": {
      "default": 0.5,
      "title": "Iou Threshold",
      "type": "number"
    },
    "label_map": {
      "anyOf": [
        {
          "additionalProperties": {
            "type": "integer"
          },
          "type": "object"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Label Map"
    },
    "max": {
      "default": 1.0,
      "title": "Max",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "metric_name": {
      "default": "specificity",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min": {
      "default": 0.0,
      "title": "Min",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "weight": {
      "default": 1.0,
      "title": "Weight",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "InitializationConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": null,
  "processing_kind": "cpu",
  "stream_batch_size": null,
  "timeout_seconds": 60
}

Complete manifest

accepts:
- name: gdi_image_v1
  task_types:
  - detection
config_schema:
  properties:
    inverted:
      default: false
      title: Inverted
      type: boolean
      x-aip-param-role: scoring_metadata
    iou_threshold:
      default: 0.5
      title: Iou Threshold
      type: number
    label_map:
      anyOf:
      - additionalProperties:
          type: integer
        type: object
      - type: 'null'
      default: null
      title: Label Map
    max:
      default: 1.0
      title: Max
      type: number
      x-aip-param-role: scoring_metadata
    metric_name:
      default: specificity
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min:
      default: 0.0
      title: Min
      type: number
      x-aip-param-role: scoring_metadata
    weight:
      default: 1.0
      title: Weight
      type: number
      x-aip-param-role: scoring_metadata
  title: InitializationConfig
  type: object
dependencies: []
description: How reliably does the model avoid flagging objects that aren't there?
direction: higher_is_better
display_name: Specificity
entrypoint: aip_metrics_object_detection.answered.specificity
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: null
  processing_kind: cpu
  stream_batch_size: null
  timeout_seconds: 60
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
  class_attribution:
    artifact_type: class_attribution
    version: 1
  methodology:
  - Deterministic - no LLM. Uses Resaro's cv-eval library (custom Jaccard IoU, default
    threshold 0.5).
  - Build the dataset label vocabulary from classes observed in the full golden ground
    truth before unanswered rows are removed.
  - Use the configured label map only to translate labels and aliases. Unused mapped
    classes do not widen the denominator.
  - For each row, start with golden classes absent from that image and add any post-threshold
    prediction-only classes to that row's absent set.
  - Return null only when the row's absent set is empty. Otherwise count predicted
    absent classes as false positives and report specificity = true negatives / absent
    classes. Dataset rollups exclude null rows, and a rollup containing only null
    rows remains null.
  - An empty prediction set scores 1.0 when at least one class is absent because it
    creates no false alarms. Each distinct false-alarm class lowers the row score
    by 1/K, where K is the number of absent classes. One false alarm scores 0.0 at
    K=1 and 0.5 at K=2. Pair specificity with recall to measure missed objects.
  score_semantics: Scores range 0-1, the true-negative rate over classes absent from
    each image. A high score means the model raises no false alarms for absent classes;
    a low score means it hallucinates objects. Empty predictions score 1.0 when an
    absent class exists. Rows with no absent class return null and are excluded from
    rollups. The score changes in steps of 1/K for K absent classes on that row, including
    any prediction-only classes emitted there. Higher is better.
  summary: How reliably the model avoids raising detections for object classes that
    are not present in the image.
  worked_example: In a car and dog golden dataset, an image containing only a car
    where the model predicts the unseen class bird has two absent classes. Bird is
    a false positive and dog is a true negative, so specificity = 1/2 = 0.5.
metric_type: pointwise
name: object_detection.specificity
partition_types: []
required_columns:
- image_id
- label
- predictions
required_kinds: []
scorer_contract: per_row
target_kind: null
unsupported_trace_shapes: []
version: 1.1.0