Skip to main content

Datasets

A dataset is the evaluation input: a table of rows whose columns satisfy a GDI schema — the contract that says which columns are required, which are optional, and what scorers can consume the data. The build-and-validate flow is: shape a raw DataFrame to a schema, upload it, map its columns to the schema fields, run quality checks, read metric suggestions, and (optionally) edit individual rows. Promoting a validated version to golden and downloading the golden set are covered separately.

The GDI schema is what your data must satisfy, not something you author here. The full catalog of schemas, their columns, and scorer contracts lives in GDI Schemas in the Reference; the calls below reference schema names like gdi_text_v1 and gdi_image_v1 without redefining them.

Shaping a DataFrame to a schema​

Before upload you can align a raw DataFrame to a GDI schema client-side. from_dataframe applies a column mapping (raw column name → schema field) and returns a DataFrame tagged with the schema; detect_schema inspects the columns and returns ranked schema matches with a confidence score, so you can auto-detect rather than hard-code the target. detect_schema returns a list of matches, each shaped {schema_name, is_match, confidence, ...}, ranked by confidence.

Once a DataFrame carries a schema, the pandas .aip accessor exposes the schema metadata without another API call:

  • df.aip.schema — the schema name, e.g. "gdi_text_v1"
  • df.aip.family — the schema family, one of "llm", "rag", or "vlm"
  • df.aip.scorers — the scorers it supports, e.g. ["correctness", "faithfulness", ...]
  • df.aip.dimensions — the declared dimension metadata
  • df.aip.coverage() — per-dimension value counts
  • df.aip.describe() — the full summary dict

SDK

import aip_sdk as aip

df = aip.from_dataframe(df_raw, schema="gdi_text_v1", mapping={"question": "prompt"})

matches = aip.detect_schema(df_raw)

df.aip.schema
df.aip.family
df.aip.scorers
df.aip.dimensions
df.aip.coverage()
df.aip.describe()

Uploading a dataset​

Upload through a Project, which binds the new dataset to the project's schema and dimensions. project.upload_dataset(df, name=...) sends the DataFrame and returns a Dataset; the first version is created immediately. Each Dataset exposes .id, .name, .schema_name, .status, .project_id, .latest_version_number, and .latest_row_count. Names must be unique within scope, so demo notebooks append a timestamp.

For image datasets, local image_path values must be readable before upload. Relative paths resolve from the current process's working directory. The SDK validates each distinct path once and raises InvalidArgumentError with an affected row count and capped sample. Remote URIs and embedded bytes do not require a local file. The image decodability check cannot inspect a path-only row, so it reports that row as a warning.

To reload an existing dataset later, use aip.load_dataset(name) or aip.load_dataset(id=...) (with aip.aload_dataset(...) as the async variant); aip.list_datasets(include_deprecated=..., page=..., per_page=...) enumerates everything accessible. load_dataset raises DatasetNotFoundError for an unknown name and AuthError when credentials are missing.

SDK

dataset = project.upload_dataset(df, name="llm-qa-v1")
version = dataset.latest_version()
print(dataset.id, version.version, version.stage)

dataset = aip.load_dataset("llm-qa-v1")
dataset = aip.load_dataset(id="550e8400-e29b-41d4-a716-446655440000")

for d in aip.list_datasets(include_deprecated=False, page=1, per_page=50):
    print(d.id, d.name, d.schema_name, d.status)

API

Upload is a multipart request:

curl -sS -X POST "$AIP_API_URL/datasets/upload" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -F "file=@dataset.parquet" \
  -F "name=llm-qa-v1"

curl -sS "$AIP_API_URL/datasets?include_deprecated=false" \
  -H "Authorization: Bearer $AIP_TOKEN"

Versions and stages​

A dataset is a series of versions. Every upload or structural change produces a new DatasetVersion carrying .id, .version, .stage, .is_golden, and .row_count. The stage tracks where the version sits in the build pipeline — freshly uploaded, mapped, and so on — and advances as you map and validate it. Use dataset.versions() to list them and dataset.latest_version() to grab the newest.

Mapping is what turns raw uploaded columns into the schema's canonical fields. dataset.map_version(version_id, column_mapping) applies a {uploaded_column: schema_field} mapping and returns the updated version with its stage advanced. Run this before quality checks so the checks evaluate the schema-aligned columns.

SDK

for v in dataset.versions():
    print(v.id, v.version, v.stage, v.is_golden, v.row_count)

version = dataset.map_version(
    version.id,
    {
        "input_id": "input_id",
        "prompt": "prompt",
        "expected_output": "expected_output",
        "sut_response": "sut_response",
    },
)
print("mapped ->", version.stage)

API

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/map" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"column_mapping":{"question":"prompt","answer":"expected_output"}}'

Quality checks​

Quality checks produce a verdict (PASS / WARN / FAIL) over a dataset version, gating whether it is fit for evaluation. There is one way to run them: the platform dispatches the registered quality-check ops, persists per-check evidence, and refreshes the latest-verdict projection. Because that is the only path, the verdict the golden-promotion gate reads is always one the platform computed — a report produced anywhere else cannot certify a version.

dataset.run_checks(version_id, label=...) triggers a run and returns a report with .status and .row_count. dataset.quality_verdict(version_id) reads the latest verdict without starting a new run; dataset.check_runs(version_id, page=..., per_page=...) lists historical runs, and dataset.quality_check_run(version_id, run_id) returns one run with its per-check checks list (each carries operation_key, execution_status, and outcome).

Choosing which checks run. Omitting checks runs every check the dataset's schema admits, which is the default. Pass checks=[...] to run a subset — useful when one check is not meaningful for the data at hand, such as row_count on a deliberately small sample. dataset.available_checks() lists what can be named, derived from the dataset's schema and the checks registered for its workspace, so a caller picks from the catalogue rather than hardcoding. A selection naming something unknown, or a real check the schema does not admit, is rejected with a distinct error rather than quietly skipped — a run never dispatches a partial subset of what was asked for, so a rejection leaves the version's verdict untouched. In the SDK those arrive as UnknownQualityCheckError (.available), QualityCheckNotApplicableError (.admitted) and RemovedQualityCheckError (.replacements), all catchable as QualityCheckSelectionError. Each run records the set it set out to run as selected_checks, which distinguishes "the caller chose three checks" from "only three were applicable at the time".

The built-in suite is row_count, schema_conformance, required_field_null_rate, duplicate_ids, dimension_coverage, and class_imbalance on both schemas, plus annotation_completeness, image_decodability, and image_duplicate on gdi_image_v1. Custom checks are registered as ops rather than authored in the SDK — see the custom-op registration flow.

SDK

version_id = dataset.latest_version().id

for option in dataset.available_checks():
    print(option.check, option.op_version, option.description)

report = dataset.run_checks(version_id, label="post-mapping")
print(report.status, report.row_count)

# Or a subset — only these run.
report = dataset.run_checks(version_id, checks=["row_count", "duplicate_ids"])

latest = dataset.quality_verdict(version_id)
print(latest["verdict"], latest.get("run_id"))

runs = dataset.check_runs(version_id, page=1, per_page=20)
print(runs[0].selected_checks)
detail = dataset.quality_check_run(version_id, runs[0].id)
for check in detail.checks:
    print(check["operation_key"], check["execution_status"], check["outcome"])

API

Three calls: GET .../available-checks reads the selectable set, POST .../quality-checks:run runs them, and GET .../quality-checks reads the latest verdict without a run.

curl -sS "$AIP_API_URL/datasets/<dataset_id>/available-checks" \
  -H "Authorization: Bearer $AIP_TOKEN"

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/quality-checks:run" \
  -H "Authorization: Bearer $AIP_TOKEN" -H "Content-Type: application/json" \
  -d '{"label":"post-mapping","triggered_by":"api"}'

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/quality-checks:run" \
  -H "Authorization: Bearer $AIP_TOKEN" -H "Content-Type: application/json" \
  -d '{"label":"subset","checks":["row_count","duplicate_ids"]}'

curl -sS "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/quality-checks" \
  -H "Authorization: Bearer $AIP_TOKEN"

Quality-check suggestions​

Suggestions recommend which metrics to score against a dataset, ranked by relevance to its schema and contents. aip.check_suggestions(dataset.id) returns a response whose .suggestions each carry a rank, a metric, and a human-readable reason; .warnings is non-empty when the backend detects an edge case. A mixed-schema warning means the columns match more than one schema interpretation — inspect .source_family on the response (e.g. "gdi_text_v1") to see which rule family produced each suggestion. A small-dataset warning means statistical metrics may be unreliable and low-confidence suggestions can be suppressed. A source_family of None means no schema family was detected; results with warnings are still usable but the list may be incomplete.

SDK

response = aip.check_suggestions(dataset.id)
for warning in response.warnings:
    print("warning:", warning)
for s in sorted(response.suggestions, key=lambda s: s.rank):
    print(s.rank, s.metric, "-", s.reason)
print(response.source_family)

API

curl -sS "$AIP_API_URL/check-suggestions?dataset_id=<dataset_id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

Analysis dimensions​

Analysis produces exploratory, dataset-derived artifacts that do not affect the quality verdict — for example, per-image visual property profiling. dataset.run_analysis(version_id) starts a run and returns a record with .id, .status, and .analysis_type. Visual results exist only after the run reaches COMPLETED, so poll dataset.analysis_run(version_id, run_id) until it leaves PENDING/RUNNING before reading. dataset.visual_analysis(version_id, run_id) returns aggregate property_stats (each with key, mean, outlier_count); dataset.query_visual_analysis(version_id, run_id, filters=..., order_by=..., limit=...) returns per-row property values matching a filter.

SDK

import time

analysis_run = dataset.run_analysis(version_id)
detail = dataset.analysis_run(version_id, analysis_run.id)
while detail.status in ("PENDING", "RUNNING"):
    time.sleep(2)
    detail = dataset.analysis_run(version_id, analysis_run.id)
if detail.status != "COMPLETED":
    raise RuntimeError(f"Analysis did not complete: {detail.status}")

visual = dataset.visual_analysis(version_id, analysis_run.id)
for stat in visual["property_stats"]:
    print(stat["key"], stat["mean"], stat["outlier_count"])

rows = dataset.query_visual_analysis(
    version_id,
    analysis_run.id,
    filters=[{"column": "brightness", "op": ">", "value": 0.8}],
    order_by="brightness DESC",
    limit=20,
)

API

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/analysis-runs" \
  -H "Authorization: Bearer $AIP_TOKEN" -H "Content-Type: application/json" \
  -d '{"triggered_by":"api"}'

curl -sS "$AIP_API_URL/datasets/<dataset_id>/versions/<version_id>/analysis/visual?run_id=<run_id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

The distinct dimensions attached to a project (e.g. intent, category) drive coverage stratification rather than analysis. Dimensions live on the project, so add or replace them there; the change takes effect for the dataset's coverage reporting.

Editing dataset rows​

For targeted fixes you can mutate rows in place rather than re-uploading. aip.insert_rows(dataset_id, rows) returns the inserted rows including their auto-generated _row_id; aip.update_rows(dataset_id, {row_id: {column: value}}) patches specific cells; aip.delete_rows(dataset_id, [row_id, ...]) returns the deleted ids. aip.commit_row_batch(dataset_id, adds=..., updates=..., deletes=...) applies a mixed set atomically in one Iceberg snapshot and returns a RowOperationResponse. To guard against concurrent edits, read aip.get_current_snapshot_id(dataset_id) and pass it as current_snapshot_id; a conflicting mutation raises APIError with status_code == 409.

SDK

The final block guards against concurrent edits with optimistic concurrency, catching the 409 on a snapshot conflict:

inserted = aip.insert_rows(dataset_id, [{"prompt": "hello", "expected_output": "world"}])
print(inserted[0]["_row_id"])

aip.update_rows(dataset_id, {"row-abc": {"prompt": "hi"}})
aip.delete_rows(dataset_id, ["row-abc", "row-def"])

from aip_sdk import RowOperationResponse

result: RowOperationResponse = aip.commit_row_batch(
    dataset_id,
    adds=[{"prompt": "new row"}],
    updates={"row-xyz": {"expected_output": "fixed"}},
    deletes=["row-stale"],
)
print(result.adds, result.updates, result.deletes)

snapshot_id = aip.get_current_snapshot_id(dataset_id)
try:
    aip.insert_rows(dataset_id, [{"prompt": "safe"}], current_snapshot_id=snapshot_id)
except aip.APIError as exc:
    if exc.status_code == 409:
        print("Snapshot conflict — dataset modified concurrently")

API

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/rows/insert" \
  -H "Authorization: Bearer $AIP_TOKEN" -H "Content-Type: application/json" \
  -d '{"rows":[{"prompt":"hello","expected_output":"world"}]}'

curl -sS -X POST "$AIP_API_URL/datasets/<dataset_id>/rows/commit" \
  -H "Authorization: Bearer $AIP_TOKEN" -H "Content-Type: application/json" \
  -d '{"adds":[{"prompt":"new row"}],"deletes":["row-stale"]}'