The AIP Mental Model

AIP is an evaluation platform for AI systems: you register the model or service you want to test, point AIP at a curated dataset with ground-truth answers, choose the metrics that matter, and run an evaluation that scores every row and aggregates the results. The same flow covers single-turn LLMs, retrieval-augmented generation, and computer-vision models — only the schema and the connection protocol change. Everything is reachable two ways: the aip_sdk Python SDK and the HTTP API directly.

A complete evaluation moves through a fixed lifecycle. Each stage produces an artifact the next stage consumes:

Connect (SUT) ─▶ Anchor (Project) ─▶ Bring & Curate ─▶ Check Quality ─▶ Enrich (SDG) ─▶ Govern (Golden) ─▶ Configure (Metrics) ─▶ Run ─▶ Review & Report

Each stage plays the following role:

• Connect (SUT) — register the model/service under test and the HTTP endpoint, auth, and response adapter AIP calls.
• Anchor (Project) — declare the schema and analysis dimensions for a body of work.
• Bring & Curate — upload rows (prompts and expected outputs) against the schema and curate them into a dataset.
• Check Quality — validate the dataset against quality checks before it becomes a baseline.
• Enrich (SDG) — expand coverage with synthetic data generation.
• Govern (Golden) — promote the curated dataset to the evaluation baseline (golden dataset).
• Configure (Metrics) — assemble the reusable (project, dataset, SUT, metrics) evaluation config.
• Run — execute: generate SUT outputs, then score the metrics.
• Review & Report — per-row scores plus aggregated metrics, analysis, and the final report.

Domain vocabulary

These terms recur throughout the docs and the SDK surface.

Concept	Explanation
Project	A project is the top-level container that holds a body of evaluation work. It sets two fixed rules: a Golden Dataset Interface (GDI) schema and a task type (for gdi_text_v1: single_turn_llm or single_turn_rag; for gdi_image_v1: classification, detection). It also defines the dimensions used to slice your data and results. Datasets always belong to a project. Either you assign one, or a default project is used automatically. SUTs (Systems Under Test) work differently: they don't have to belong to a project at all. A SUT can be scoped just to a workspace, or not scoped to anything.
Golden Dataset Interface (GDI) schema	Identified by name. It sets out which dataset columns are required, which are optional, and the rules for scoring. You pick a schema when you create a project or upload a dataset: gdi_text_v1 for text, gdi_image_v1 for images, agent_trace_v1 for agent traces. canonical_partition_v1 is the one schema you never pick — AIP derives it from an agent-trace dataset when a run scores (see below).
Agent trace (agent_trace_v1)	A dataset of recorded agent activity, in a three-level hierarchy: a span is one operation the agent performed — an LLM call, a tool call, a sub-agent invocation — identified by its kind, and linked into a tree via each span's parent; a trace is one full run, the complete tree of spans it took; a session optionally groups several related traces together (e.g. the turns of one conversation). Uploaded through a dedicated trace-ingest path rather than the standard tabular upload. Metrics never score it directly; they score the partitions you carve out of it.
Partition	The evaluation target for trace metrics: a named, saved rule that picks out which part of a trace dataset to score (e.g. "every TOOL-kind span"). Create one with aip.create_partition() and list a version's partitions with aip.list_partitions(). It's keyed on stable semantic ids rather than a snapshot of raw span ids, so it still resolves correctly after a re-ingest. An eval-config target references one by partition_id; a target that names only a partition type scores that whole level ad hoc, without a saved partition.
Partition type (partition_type)	The granularity a partition scores at — session, trace, or span. Every metric declares the levels it supports in partition_types; agent.hallucination, for example, scores span and trace but not session. Note that partition_type and partition_id are independent: one is the level, the other identifies a saved partition.
Resolved partition (canonical_partition_v1)	One unit a partition resolves to — a single session, trace, or span — and the thing a metric is actually called on, once each. A span partition over a version with five matching spans resolves to five, and produces five scores. resolve_partition() returns one ResolvedPartition per unit, and scoring derives one canonical_partition_v1 row for each. The row holds no copy of the trace content: just target_ref, a join key back to the underlying spans, plus any ground truth the metric needs (e.g. expected_tools, expected_steps). The run's raw metric metadata (visible via the REST API; not yet a field on the SDK's typed Metric object) still counts these as occurrences_scored / occurrences_unscored.
Golden dataset	The specific dataset version promoted to be the evaluation baseline. Promotion requires a passing quality verdict (or an audited force=True acknowledgement of warnings). Runs evaluate against a golden version. An agent-trace dataset is the one exception: it has no quality-gated promote() step of its own — see Guide: Evaluate an Agent for how it keeps a current version instead.
Analysis dimensions	Categorical columns you define at the project level (like topic or category). AIP uses them to show how well your dataset covers different categories, and to group scores when analysing a run.
Dataset	A table of uploaded test case rows that follows a schema. Each row carries the inputs (e.g. prompt) and ground truth (e.g. expected_output) plus any dimension columns. Rows move through stages — raw, mapped, augmented, and golden. Datasets are immutable per version; new uploads and mappings produce new versions.
Quality check	An automated readiness check run against a dataset to see if it's good enough for testing. It looks at things like: does the data match the schema, is it complete, how many rows are there, are there duplicate IDs, how often are required fields left blank, are annotations complete, and does it cover the right range of dimensions. The set of check results produces the quality verdict (PASS/WARN/FAIL/ERROR/STALE/NOT_RUN).
Synthetic Data Generation (SDG)	A collection of Ops that can be used to create new dataset rows, variations, edge cases, or fresh examples, to broaden a dataset's coverage before it moves forward. This is the "Enrich" stage of the evaluation lifecycle.
System Under Test (SUT)	The model or service being evaluated. It's registered with a name, version, and owner. On its own, a SUT can't be used for testing — it needs at least one connection first.
Connection	The concrete HTTP endpoint AIP calls for a SUT: its base URL, auth (bearer / api_key / basic / none), a sut_protocol (e.g. openai_chat), and a response adapter that maps the raw endpoint JSON into GDI output columns via JSONPath.
Metric / Scorer	A single scoring function applied to each row, identified by a <family>.<name> op id (e.g. llm.bleu, llm.correctness, rag.faithfulness). Some metrics compare the system's response to an expected answer; others use an AI judge to evaluate the response directly.
Evaluation config	A reusable bundle of (project, dataset, sut_id, connection_id, metrics) — plus optional metric_configs, tags, parameters, and thresholds — that fully specifies a run. Publish it to reuse across runs and CI.
Run	A single run of an evaluation. It works one of two ways: hosted, where AIP calls your system and calculates the metrics for you, or external, where you run and score it yourself, then upload the results. Each run passes through these stages: pending, claimed, running, then completed — or it ends early as retryable_failed, failed, or cancelled.
Results vs. Analysis	Results are the raw, row-by-row scores from each metric, plus basic summaries (e.g. mean, std, count, pass_rate). Analysis goes further — it's the processed statistical view behind the charts, including score distributions, percentiles, and breakdowns by dimension.
Op	An Op is a single unit of work in the AIP’s registry e.g. a metric, a data transform, or a step in synthetic data generation. Ops are keyed by <family>.<name>, discoverable via aip.ops.list_ops() / list_metrics(), and each declares its own accepted schema(s) and config.