Skip to main content

Evaluate a semantic-segmentation model

This guide scores a semantic segmentation model end to end on AIP. The model assigns a class to every pixel, and the platform measures how closely those masks match the ground truth. The bundled ADE20K fixture already contains predictions, so this workflow uses pipeline="metric_invoke" and does not call a model endpoint.

The full executable walkthrough is in aip_v2_demo_semantic_segmentation_e2e.ipynb. It expects the local platform and all six semantic segmentation functions to be running. The SDK examples README lists the exact setup and deployment commands.

Create the project and load ADE20K​

Semantic segmentation uses the gdi_image_v1 schema and the semantic_segmentation task type. The fixture contains ten ADE20K scenes with embedded image bytes, ground-truth masks in label, and model masks in predictions.

from pathlib import Path

import pandas as pd
import aip_sdk as aip

aip.init("http://localhost:8010", username="admin", password="admin")
workspace = aip.Workspace.get_by_name("Default")
project, _ = aip.Project.get_or_create(
    name="Semantic Segmentation Demo",
    schema="gdi_image_v1",
    task_type="semantic_segmentation",
    workspace_id=workspace.id,
)

fixture = Path("libs/aip-sdk/examples/data/ade20k_semantic_segmentation_gdi_image_v1.parquet")
frame = pd.read_parquet(fixture)
assert len(frame) == 10
assert {"image_id", "image", "label", "predictions", "task_type"} <= set(frame.columns)
assert set(frame["task_type"]) == {"semantic_segmentation"}

Each annotation contains a class name and a compressed COCO RLE mask. When you prepare another dataset, preserve this shape. A dataset without stored predictions is also valid when a live SUT supplies them during the run.

Upload, check and promote​

Runs accept golden dataset versions. The fixture is intentionally smaller than the normal row-count threshold, so promotion requires an explicit acknowledgement after the quality report is stored.

dataset = project.upload_dataset(frame, name="ade20k-semantic-validation")
version = dataset.latest_version()

report = dataset.run_checks(version.id, checks=["row_count", "duplicate_ids", "schema_conformance"])
version = dataset.promote(
    version.id,
    force=True,
    reason="ADE20K validation fixture is intentionally small.",
)

Confirm the deployed metric surface​

The semantic surface contains three evaluation metrics and three dataset-quality scorers. Validate the live registry before submitting the run so a missing Nuclio deployment fails early with a useful list.

metrics = [
    "semantic_segmentation.mean_iou",
    "semantic_segmentation.mean_dice",
    "semantic_segmentation.per_sample_dice",
    "semantic_segmentation.label_format_check",
    "semantic_segmentation.label_coverage_check",
    "semantic_segmentation.prediction_coverage_check",
]

available = {entry["name"].split("/", 1)[0] for entry in aip.ops.list_metrics(schema="gdi_image_v1")}
missing = set(metrics) - available
assert not missing, f"Missing live semantic segmentation metrics: {sorted(missing)}"

Run and validate the ADE20K fixture​

The dataset-level metrics publish per-class artifacts. The headline score must equal the mean of the finite reported class scores, with the reserved background channel excluded. The three quality scorers must report full coverage for the bundled fixture.

import numpy as np

with aip.run(
    project=project.id,
    pipeline="metric_invoke",
    dataset=f"{dataset.id}@v{version.version}",
    metrics=metrics,
) as run:
    run.poll_status(interval=3, timeout=600)
    results = run.results()
    analysis = run.analysis()

stats = analysis["scorer_stats"]
artifacts = analysis["metric_artifacts"]
for scorer in ("mean_iou", "mean_dice"):
    scores = np.asarray(
        [score for score in artifacts[f"{scorer}_per_class"]["scores"].values() if score is not None],
        dtype=float,
    )
    assert abs(float(stats[scorer]["mean"]) - float(np.nanmean(scores))) < 1e-4

for scorer in (
    "label_format_check",
    "label_coverage_check",
    "prediction_coverage_check",
):
    assert abs(float(stats[scorer]["mean"]) - 1.0) < 1e-4

Open run.url to inspect the score cards and per-row Dice results. To evaluate a live model instead, register a SUT using the cv_semantic_segmentation template. The hosted pipeline then fills predictions before dispatching the same metrics.