Skip to main content

Evaluate an object-detection model

This guide walks the full lifecycle of a hosted object-detection evaluation on AIP: you create an image project, connect a detector as a system under test, convert a COCO sample into governed image rows, promote a golden dataset, select object-detection metrics, and run a hosted evaluation that calls your model and scores its predictions — all from the Python SDK.

The walkthrough uses a small navy-ship detector served on a Hugging Face inference endpoint, but every step is the same for any huggingface_object_detection SUT. Each stage links to the matching workflow reference, so you can drill into individual parameters there instead of re-reading them here. A closing variant shows how the drone-imagery notebook layers a raw baseline and an optional synthetic-augmentation pass on top of this same flow.

Connect and create the project​

Initialize the SDK once, then resolve a workspace to own everything you create. aip.init() caches the configured client for the rest of the session. Set AIP_API_KEY, AIP_WORKSPACE_ID, NAVY_ENDPOINT_URL, and HF_TOKEN in your environment, and place navy_ship_categories.json and navy_ship_detection_demo_v1_hf_coco.parquet in your working directory.

import base64, json, os, time
from pathlib import Path
import pandas as pd
import aip_sdk as aip

CLIENT = aip.init("https://api.trials.aip-v2.resarodev.ai", api_key=os.environ["AIP_API_KEY"])
ws = aip.Workspace.get(os.environ["AIP_WORKSPACE_ID"])
NAVY_ENDPOINT_URL = os.environ["NAVY_ENDPOINT_URL"]
HF_TOKEN = os.environ["HF_TOKEN"]
N_ROWS = int(os.environ.get("AIP_CV_ROWS", "20"))
CATEGORY_NAMES = json.loads(Path("navy_ship_categories.json").read_text())
SHIP_CLASSES = [CATEGORY_NAMES[key] for key in sorted(CATEGORY_NAMES, key=int)]

An object-detection project uses the gdi_image_v1 schema with the detection task type. Dimensions let you slice results later; here scenario carries the dominant ship class per image, which you compute when building the dataframe. get_or_create makes the notebook safe to re-run. See Projects for the full create/list/coverage surface.

od_project, _ = aip.Project.get_or_create(
    name="HF Navy Ship Detection",
    schema="gdi_image_v1",
    task_type="detection",
    dimensions=[aip.Dimension(name="scenario", column="scenario", values=SHIP_CLASSES)],
    workspace_id=ws.id,
)

Register the detector as a system under test​

Register the detector as a SUT, then attach a hosted connection that AIP calls during the run. For an image detector, use the huggingface_object_detection protocol with gdi_schema="gdi_image_v1" and task_type="detection". The bearer token is stored encrypted by the platform. The Systems Under Test reference covers auth options, connection testing, and the full protocol matrix.

sut, _ = aip.Sut.get_or_register(
    name="NavyShipDetector-HF",
    version="1.0",
    project_id=od_project.id,
)

conn = sut.add_connection(
    label="hf-navy-endpoint",
    base_url=NAVY_ENDPOINT_URL,
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=HF_TOKEN,
    sut_protocol="huggingface_object_detection",
    gdi_schema="gdi_image_v1",
    task_type="detection",
)

The endpoint returns raw detection JSON, so map it into the GDI predictions column with the built-in cv_detection adapter template. A single JSONPath mapping is enough when the endpoint already emits a predictions array.

conn.set_adapter(
    template_name="cv_detection",
    mapping_config={"predictions": "$.predictions"},
)

Confirm the wiring with a live request before you spend time building data — conn.test() reports success, HTTP status, and any error. If a scaled-to-zero endpoint returns 503, it is cold-starting; retry once it warms up.

result = conn.test()
assert result.success, result.error

Build the dataset from COCO​

The detector is fed GDI image-detection rows. Convert each COCO sample into a row that carries the image bytes, the ground-truth label boxes in xyxy form, an empty predictions list (the SUT fills these during the run), and the scenario dimension. Keep rows that have no ground-truth boxes — quality checks flag them rather than dropping them silently.

def coco_to_label(objects) -> list[dict]:
    labels = []
    for bbox, cat_id in zip(objects["bbox"], objects["category"]):
        x, y, w, h = (float(v) for v in bbox)
        labels.append({"class": CATEGORY_NAMES[str(int(cat_id))], "bbox": [x, y, x + w, y + h]})
    return labels


def dominant_class(objects) -> str:
    areas = [int(a) for a in objects["area"]]
    if not areas:
        return "unknown"
    biggest = max(range(len(areas)), key=lambda i: areas[i])
    return CATEGORY_NAMES[str(int(objects["category"][biggest]))]


df_raw = pd.read_parquet("navy_ship_detection_demo_v1_hf_coco.parquet").head(N_ROWS)

rows = []
for _, raw in df_raw.iterrows():
    rows.append(
        {
            "image_id": str(raw["image_id"]),
            "task_type": "detection",
            "label": coco_to_label(raw["objects"]),
            "image": {"bytes": raw["image"]["bytes"]},
            "scenario": dominant_class(raw["objects"]),
            "predictions": [],
            "collection_source": "open-source",
            "metadata": {"width": int(raw["width"]), "height": int(raw["height"])},
        }
    )
df = pd.DataFrame(rows)

Upload the dataframe through the project. This creates a versioned dataset in the gdi_image_v1 schema; a timestamped name keeps re-runs independent. See Datasets for versions, downloads, and row operations.

od_ds = od_project.upload_dataset(df, name=f"navy-ship-od-{int(time.time())}")
od_v = od_ds.latest_version()

Map, check, and profile the dataset​

Map your uploaded columns onto the schema columns before running checks. The dataframe here already uses GDI names, so the mapping is an identity, but the step is what advances the version's pipeline stage and lets the promote gate see a fresh verdict.

od_v = od_ds.map_version(
    od_v.id,
    column_mapping={
        "image_id": "image_id",
        "label": "label",
        "predictions": "predictions",
    },
)

Run quality checks against the mapped version. The run stores the verdict that gates promotion, and resolves the project's declared dimensions itself so coverage is reported per slice. annotation_completeness is the image-schema check that flags empty label lists.

print([option.check for option in od_ds.available_checks()])

report = od_ds.run_checks(
    od_v.id,
    label="post-mapping",
    checks=["row_count", "duplicate_ids", "schema_conformance", "annotation_completeness"],
)

Ask the platform for ranked metric suggestions for this dataset, then keep them as your working selection. Suggestions come back as bare metric names; you namespace them into object_detection.* op keys and validate them against the live registry in a later step.

sugg = aip.check_suggestions(od_ds.id)
selected_metrics = [f"object_detection.{s.metric}" for s in sorted(sugg.suggestions, key=lambda s: s.rank)]

Image datasets also support visual image-property analysis — brightness, contrast, and similar per-image statistics that help you understand the set without affecting the data-quality verdict. Trigger a run and poll it to completion; results are available only once the run reaches COMPLETED. See Datasets for the aggregate and per-row query surface.

analysis_run = od_ds.run_analysis(od_v.id)
detail = od_ds.analysis_run(od_v.id, analysis_run.id)
while detail.status in ("PENDING", "RUNNING"):
    time.sleep(2)
    detail = od_ds.analysis_run(od_v.id, analysis_run.id)
visual = od_ds.visual_analysis(od_v.id, analysis_run.id)

To slice results along more than one axis, add project dimensions backed by columns already present on every row — task_type and collection_source here — alongside the multi-class scenario dimension. Dimensions are replaced as a set through a project patch.

extra_dims = [
    aip.Dimension(name="task_type", column="task_type", values=["detection"], required=False),
    aip.Dimension(name="collection_source", column="collection_source", values=["open-source"], required=False),
]
names = {d.name for d in extra_dims}
new_dims = [d.to_dict() for d in od_project.dimensions if d.name not in names] + [d.to_dict() for d in extra_dims]
CLIENT.patch(f"/projects/{od_project.id}", json={"dimensions": new_dims})
od_project.dimensions = [aip.Dimension.from_dict(x) for x in new_dims]

Promote to golden and download​

Promote the mapped version to golden — the version the evaluation runs against. Use the quality report from the checks above: promote a PASS normally, or review a WARN and record the specific accepted warning and rationale with force=True. Other verdicts stop the workflow, and authentication, network, or other promotion errors propagate without a forced retry. Golden Datasets explains the promote/demote lifecycle and verdict gate.

if not od_v.is_golden:
    if report.status == "PASS":
        od_v = od_ds.promote(od_v.id)
    elif report.status == "WARN":
        od_v = od_ds.promote(
            od_v.id,
            force=True,
            reason="Reviewed object-detection warnings: <specific accepted warning and rationale>.",
        )
    else:
        raise RuntimeError(f"Quality verdict {report.status}; resolve the findings and re-run checks before promotion")

df_golden = od_ds.download()

Select object-detection metrics​

Discover the object-detection metrics registered on the platform, then confirm your suggested selection against them — dropping any suggestion that is not actually registered. list_metrics accepts a schema filter so you only see metrics that accept gdi_image_v1.

od_ops = [m for m in aip.ops.list_metrics(schema="gdi_image_v1") if m["name"].startswith("object_detection.")]
valid = {o["name"] for o in od_ops}
selected_metrics = [m for m in selected_metrics if m in valid]

Publish the evaluation config​

Bundle the project, the golden dataset version, the SUT and connection, and the metric selection into a reusable evaluation config, then publish it to the platform. Publishing returns a server-assigned id you can pull back by name or UUID in CI. See Evaluation Configs for the YAML schema and platform sync.

from aip_core.schemas.eval_config import EvalConfig

cfg = EvalConfig(
    project=str(od_project.id),
    dataset=f"{od_ds.id}@v{od_v.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
)
cfg_id = aip.publish_config(
    cfg,
    name=f"navy-ship-od-eval-{int(time.time())}",
    description="OD demo evaluation config",
    workspace_id=ws.id,
)

Run the hosted evaluation and read the scores​

Because you pass a connection and leave runner unset, aip.run() selects the hosted runner (sut_id is attribution-only): AIP calls the live detector through the Step-1 connection, maps its response into predictions, and computes your selected metrics — you only poll for completion. Pass the golden version pinned as @v{version} for a reproducible run.

with aip.run(
    project=od_project.id,
    dataset=f"{od_ds.id}@v{od_v.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results()

run.results() returns a RunResultsPage: per-row ScoreResult entries (one per row and metric) plus aggregated Metric summaries with mean, standard deviation, count, and pass rate. Page through larger golden sets by incrementing page. The Runs & Results reference documents the result types, run.analysis(group_by=...) for per-dimension breakdowns, and how to re-open a finished run with aip.get_run(run_id).

for r in page.results:
    print(r.input_id, r.scorer, r.score)
for m in page.metrics:
    print(f"{m.scorer}: mean={m.mean:.3f} pass_rate={m.pass_rate:.0%} (n={m.count})")

Variant: drone imagery with synthetic augmentation​

The drone-imagery walkthrough follows the identical create → connect → build → promote → evaluate flow against a YOLOv11 detector, with two additions.

First, it runs a raw baseline before any augmentation: it maps and promotes the freshly uploaded seed version and evaluates it once, so you have a reference score to compare against later. This is just the standard hosted run applied to the seed version — a fixed metric set (object_detection.precision, recall, f1, missed_detection_rate) rather than suggested metrics. The @v{od_v.version} pin passed to aip.run() here points at that raw seed version.

selected_metrics = [
    "object_detection.precision",
    "object_detection.recall",
    "object_detection.f1",
    "object_detection.missed_detection_rate",
]

with aip.run(
    project=od_project.id,
    dataset=f"{od_ds.id}@v{od_v.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
) as raw_run:
    raw_run.poll_status(interval=5, timeout=300)
    baseline = raw_run.results()

Second, it adds an optional Synthetic Data Generation pass. SDG expands the governed seed version on the platform into a new augmented version — here via a FLUX.2 Klein diffusion edit (for example, restyling the seed images to scenario="heavy snow"). Because FLUX is GPU-bound, the edit is offloaded to a generic ZeroGPU Hugging Face Space through the hf_space backend rather than requiring a CUDA GPU on the SDG worker. The seed lineage is preserved on the augmented version, and the remainder of the guide (map → check → promote → evaluate) then runs against that augmented version so you can compare its scores to the raw baseline.

seed DatasetVersion  ->  FLUX.2 Klein edit (ZeroGPU Space)  ->  GDI export  ->  augmented DatasetVersion

The full job-config wiring, adapter registration, Space deployment, and polling are covered in Guide: Synthetic Data Generation.