Evaluate an image-classification model

This guide evaluates a hosted image classifier against a small, balanced EuroSAT sample. It follows the same governed lifecycle as object detection, but uses the classification task contract throughout: a gdi_image_v1 project with task_type="classification", one non-empty class-name string in each row's label, the huggingface_image_classification SUT protocol, and the image_classification.* metric family.

The complete executable version is aip_v2_demo_classification_e2e_satellite.ipynb. It reads credentials only from environment variables. Start Jupyter from libs/aip-sdk after exporting the values shown in examples/.env.example. The first cell fails immediately unless AIP_BASE_URL, SATELLITE_ENDPOINT_URL, HF_TOKEN, and one supported AIP authentication method are present; it never writes placeholder credentials into the process environment.

Connect and create the classification project

Initialize the SDK with one supported authentication method, then create the task-scoped image project. A dimension is optional; scenario is useful here for comparing the source train/test slices later.

import os
import aip_sdk as aip

aip.init(
    os.environ["AIP_BASE_URL"],
    token=os.environ.get("AIP_TOKEN"),
    api_key=os.environ.get("AIP_API_KEY"),
    username=os.environ.get("AIP_USERNAME"),
    password=os.environ.get("AIP_PASSWORD"),
)
workspace_name = os.environ.get("AIP_WORKSPACE_NAME")
workspace = aip.Workspace.get_by_name(workspace_name) if workspace_name else aip.Workspace.default()

project, _ = aip.Project.get_or_create(
    name="EuroSAT Image Classification",
    schema="gdi_image_v1",
    task_type="classification",
    dimensions=[
        aip.Dimension(
            name="scenario",
            column="scenario",
            values=["test", "train"],
            required=False,
        )
    ],
    workspace_id=workspace.id,
)

Build and validate the classification rows

Each row needs image_id, task_type="classification", a non-empty string label, and either embedded image bytes or an image_path. The hosted SUT populates predictions; each prediction is a {"label": <class>, "score": <probability>} entry. Keep a small example class-balanced so every class is represented in the confusion matrix. Five rows from each EuroSAT class gives a 50-row run that remains quick but exercises all ten classes.

import pandas as pd
from aip_core.schemas.image import GdiImageV1

df_full = pd.read_parquet("examples/data/eurosat_gdi_image_v1.parquet")
df = (
    df_full.groupby("label", group_keys=False)
    .sample(n=5, random_state=17)
    .sort_values(["label", "image_id"])
    .reset_index(drop=True)
)
GdiImageV1.validate(df)

assert set(df["task_type"]) == {"classification"}
assert df["label"].map(lambda value: isinstance(value, str) and bool(value.strip())).all()
assert df["label"].nunique() >= 2  # required by ROC/AUC

Classification labels are deliberately different from detection annotations. Nulls, empty strings, lists, and dictionaries are malformed. For example, this detection-shaped value fails the dataframe-level check_label_matches_task_type validation and raises Pandera SchemaError:

bad = df.head(1).copy()
bad["label"] = pd.Series([[{"class": "Forest"}]], index=bad.index)
GdiImageV1.validate(bad)  # SchemaError: classification label must be a non-empty string

Fix the row by using the class name itself:

bad["label"] = "Forest"
GdiImageV1.validate(bad)

Register the classifier and response adapter

Register the SUT under the project and add a connection using huggingface_image_classification. top_k must cover the class vocabulary so metrics receive the full score distribution needed by ROC/AUC, rather than only the winning label. The built-in cv_classification adapter maps the normalized endpoint response into the GDI predictions output.

sut, _ = aip.Sut.get_or_register(
    name="hf-eurosat-classifier",
    version="1.0",
    project_id=project.id,
)
conn = sut.add_connection(
    label="hf-eurosat",
    base_url=os.environ["SATELLITE_ENDPOINT_URL"],
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=os.environ["HF_TOKEN"],
    model_params={
        "top_k": int(df["label"].nunique()),
        "parameters": {"return_all_scores": True},
    },
    sut_protocol="huggingface_image_classification",
    gdi_schema="gdi_image_v1",
    task_type="classification",
)
conn.set_adapter(
    template_name="cv_classification",
    mapping_config={"predictions": "$.predictions"},
)

Upload, check, and promote the dataset

Upload the validated frame, apply the identity mapping, persist a local quality report, and promote the mapped version to golden. Promotion without a passing verdict is rejected; do not force it in the supported happy path.

import time

dataset = project.upload_dataset(
    df,
    name=f"eurosat-classification-{int(time.time())}",
)
version = dataset.latest_version()
version = dataset.map_version(
    version.id,
    column_mapping={
        "image_id": "image_id",
        "task_type": "task_type",
        "label": "label",
        "image": "image",
        "scenario": "scenario",
        "metadata": "metadata",
    },
)

report = dataset.run_checks(
    version.id,
    label="classification example",
    checks=["row_count", "duplicate_ids", "schema_conformance", "dimension_coverage"],
)
version = dataset.promote(version.id)

Select classification metrics

The classification family has five scalar metrics and four report artifacts. Confirm the names against the live registry before launching the run:

metrics = [
    "image_classification.accuracy",
    "image_classification.recall",
    "image_classification.specificity",
    "image_classification.fnr",
    "image_classification.per_class_recall",
    "image_classification.per_class_specificity",
    "image_classification.per_class_fnr",
    "image_classification.confusion_matrix",
    "image_classification.roc_auc",
]

available = {entry["name"].split("/", 1)[0] for entry in aip.ops.list_metrics(schema="gdi_image_v1")}
missing = set(metrics) - available
assert not missing, f"Missing live classification metrics: {sorted(missing)}"

Run and inspect the current report

Supplying the registered connection selects the hosted runner. poll_status raises on failure, cancellation, or timeout, so a failed example cannot look successful. Scalar summaries and report artifacts are available through the public results/analysis surfaces after completion.

with aip.run(
    project=project.id,
    dataset=f"{dataset.id}@v{version.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=metrics,
) as run:
    run.poll_status(interval=5, timeout=900)
    page = run.results(page=1, page_size=1000)
    analysis = run.analysis()
    print(run.url)

analysis["scorer_stats"] contains the five scalar summaries used by score cards. FNR is lower-is-better; its default classification score card passes at <= 0.2 and warns at <= 0.4. The remaining outputs are under analysis["metric_artifacts"]:

artifacts = analysis["metric_artifacts"]
matrix = artifacts["confusion_matrix"]

assert matrix["type"] == "matrix"
assert len(matrix["matrix"]) == len(matrix["labels"])
assert sum(sum(row) for row in matrix["matrix"]) == len(df)

for name in ("per_class_recall", "per_class_specificity", "per_class_fnr"):
    assert artifacts[name]["type"] == "per_class_scores"

Open run.url to verify the current report widgets: scalar score cards, classification confusion matrix, and per-class recall/specificity/FNR. These views come from the current widget system and do not require the interactive report engine.