Confusion Matrix

Available

object_detection.confusion_matrix

How often real objects are found and given the correct class label, reported as a recall score plus a detection confusion-matrix artifact.

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	full_dataset
direction	higher_is_better
entrypoint	aip_scorers_object_detection.object_detection.confusion_matrix
target_kind	None

Required columns

• image_id
• label
• predictions

Accepted schemas

[
  {
    "name": "gdi_image_v1",
    "task_types": [
      "detection"
    ]
  }
]

Methodology

1. Deterministic - no LLM. Uses Resaro's cv-eval library (custom Jaccard IoU, default threshold 0.5).
2. Per image, greedily match same-class detections to ground-truth boxes above the IoU threshold, in descending confidence order.
3. Tally correct matches, missed ground-truth, and false positives into a confusion-matrix artifact.
4. Aggregate counts across the dataset and report recall = correct / total ground-truth (micro-average).

Score semantics

The scalar score is recall (0-1): correctly-classified detections divided by ground-truth objects. A high score means most objects are found and labelled correctly. Higher is better.

Worked example

3 ground-truth objects and 4 predictions, of which 2 match the correct class above IoU 0.5 -> recall = 2/3 ~ 0.667; the matrix records 2 hits, 1 missed object, and 2 false positives.

Configuration schema

{
  "properties": {
    "inverted": {
      "default": false,
      "title": "Inverted",
      "type": "boolean",
      "x-aip-param-role": "scoring_metadata"
    },
    "iou_threshold": {
      "default": 0.5,
      "title": "Iou Threshold",
      "type": "number"
    },
    "label_map": {
      "anyOf": [
        {
          "additionalProperties": {
            "type": "integer"
          },
          "type": "object"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Label Map"
    },
    "max": {
      "default": 1.0,
      "title": "Max",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "metric_name": {
      "default": "confusion_matrix",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min": {
      "default": 0.0,
      "title": "Min",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "weight": {
      "default": 1.0,
      "title": "Weight",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "InitializationConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": true,
  "function_name": "object_detection.confusion_matrix",
  "max_concurrency": null,
  "processing_kind": "cpu",
  "stream_batch_size": null,
  "timeout_seconds": 600
}

Complete manifest

accepts:
- name: gdi_image_v1
  task_types:
  - detection
config_schema:
  properties:
    inverted:
      default: false
      title: Inverted
      type: boolean
      x-aip-param-role: scoring_metadata
    iou_threshold:
      default: 0.5
      title: Iou Threshold
      type: number
    label_map:
      anyOf:
      - additionalProperties:
          type: integer
        type: object
      - type: 'null'
      default: null
      title: Label Map
    max:
      default: 1.0
      title: Max
      type: number
      x-aip-param-role: scoring_metadata
    metric_name:
      default: confusion_matrix
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min:
      default: 0.0
      title: Min
      type: number
      x-aip-param-role: scoring_metadata
    weight:
      default: 1.0
      title: Weight
      type: number
      x-aip-param-role: scoring_metadata
  title: InitializationConfig
  type: object
dependencies: []
description: How often are real objects found and given the correct class label?
direction: higher_is_better
display_name: Confusion Matrix
entrypoint: aip_scorers_object_detection.object_detection.confusion_matrix
execution:
  emits_metric_family: true
  function_name: object_detection.confusion_matrix
  max_concurrency: null
  processing_kind: cpu
  stream_batch_size: null
  timeout_seconds: 600
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
  methodology:
  - Deterministic - no LLM. Uses Resaro's cv-eval library (custom Jaccard IoU, default
    threshold 0.5).
  - Per image, greedily match same-class detections to ground-truth boxes above the
    IoU threshold, in descending confidence order.
  - Tally correct matches, missed ground-truth, and false positives into a confusion-matrix
    artifact.
  - Aggregate counts across the dataset and report recall = correct / total ground-truth
    (micro-average).
  score_semantics: 'The scalar score is recall (0-1): correctly-classified detections
    divided by ground-truth objects. A high score means most objects are found and
    labelled correctly. Higher is better.'
  summary: How often real objects are found and given the correct class label, reported
    as a recall score plus a detection confusion-matrix artifact.
  worked_example: 3 ground-truth objects and 4 predictions, of which 2 match the correct
    class above IoU 0.5 -> recall = 2/3 ~ 0.667; the matrix records 2 hits, 1 missed
    object, and 2 false positives.
metric_type: pointwise
name: object_detection.confusion_matrix
partition_types: []
required_columns:
- image_id
- label
- predictions
required_kinds: []
scorer_contract: full_dataset
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0