ROC/AUC

Available

image_classification.roc_auc

The area under the ROC curve, measuring how well the model's class scores separate the positive class from the rest across all thresholds.

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	full_dataset
direction	higher_is_better
entrypoint	aip_scorers_image_classification.answered.roc_auc
target_kind	None

Required columns

• image_id
• label
• predictions

Accepted schemas

[
  {
    "name": "gdi_image_v1",
    "task_types": [
      "classification"
    ]
  }
]

Methodology

1. Deterministic - no LLM. Uses scikit-learn roc_auc_score (with label_binarize for multiclass).
2. Require at least two observed ground-truth classes.
3. Binary case: resolve the configured positive_class and score its column against the binary truth.
4. Multiclass case: binarise the labels and compute micro-averaged one-vs-rest AUC over the observed classes.

Score semantics

Scores range 0-1, threshold-independent. A high score (near 1.0) means positive-class scores reliably rank above negatives; 0.5 is chance. Higher is better.

Worked example

Binary labels [cat, dog] with positive_class 'dog', truth [cat, cat, dog, dog] and dog-scores [0.2, 0.4, 0.6, 0.9]: both positives outrank both negatives -> AUC = 1.0.

Configuration schema

{
  "description": "Shared image-classification metric configuration.",
  "properties": {
    "inverted": {
      "default": false,
      "title": "Inverted",
      "type": "boolean",
      "x-aip-param-role": "scoring_metadata"
    },
    "max": {
      "default": 1.0,
      "title": "Max",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "metric_name": {
      "default": "image_classification_metric",
      "title": "Metric Name",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    },
    "min": {
      "default": 0.0,
      "title": "Min",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    },
    "positive_class": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Positive class label to use when ROC/AUC is evaluated on a binary classification dataset.",
      "title": "Positive Class"
    },
    "weight": {
      "default": 1.0,
      "title": "Weight",
      "type": "number",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "InitializationConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": null,
  "processing_kind": "cpu",
  "stream_batch_size": null,
  "timeout_seconds": 600
}

Complete manifest

accepts:
- name: gdi_image_v1
  task_types:
  - classification
config_schema:
  description: Shared image-classification metric configuration.
  properties:
    inverted:
      default: false
      title: Inverted
      type: boolean
      x-aip-param-role: scoring_metadata
    max:
      default: 1.0
      title: Max
      type: number
      x-aip-param-role: scoring_metadata
    metric_name:
      default: image_classification_metric
      title: Metric Name
      type: string
      x-aip-param-role: scoring_metadata
    min:
      default: 0.0
      title: Min
      type: number
      x-aip-param-role: scoring_metadata
    positive_class:
      anyOf:
      - type: string
      - type: 'null'
      default: null
      description: Positive class label to use when ROC/AUC is evaluated on a binary
        classification dataset.
      title: Positive Class
    weight:
      default: 1.0
      title: Weight
      type: number
      x-aip-param-role: scoring_metadata
  title: InitializationConfig
  type: object
dependencies: []
description: ROC area-under-curve computed from the full per-class score distribution.
direction: higher_is_better
display_name: ROC/AUC
entrypoint: aip_scorers_image_classification.answered.roc_auc
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: null
  processing_kind: cpu
  stream_batch_size: null
  timeout_seconds: 600
kind: metric
manifest_version: '1'
max_prompt_slots: 0
metric_metadata:
  methodology:
  - Deterministic - no LLM. Uses scikit-learn roc_auc_score (with label_binarize for
    multiclass).
  - Require at least two observed ground-truth classes.
  - 'Binary case: resolve the configured positive_class and score its column against
    the binary truth.'
  - 'Multiclass case: binarise the labels and compute micro-averaged one-vs-rest AUC
    over the observed classes.'
  score_semantics: Scores range 0-1, threshold-independent. A high score (near 1.0)
    means positive-class scores reliably rank above negatives; 0.5 is chance. Higher
    is better.
  summary: The area under the ROC curve, measuring how well the model's class scores
    separate the positive class from the rest across all thresholds.
  worked_example: 'Binary labels [cat, dog] with positive_class ''dog'', truth [cat,
    cat, dog, dog] and dog-scores [0.2, 0.4, 0.6, 0.9]: both positives outrank both
    negatives -> AUC = 1.0.'
metric_type: pointwise
name: image_classification.roc_auc
partition_types: []
required_columns:
- image_id
- label
- predictions
required_kinds: []
scorer_contract: full_dataset
target_kind: null
unsupported_trace_shapes: []
version: 1.0.0