Evaluate a Model End-to-End

This guide runs a complete hosted evaluation of an OpenAI chat model end to end: you create a project, register the model as a system under test, upload a small question-and-answer dataset with ground-truth answers, promote it to a golden set, choose reference-based metrics, and let AIP call the live OpenAI endpoint and score the responses for you. Every step maps to a workflow reference section — Projects, Systems Under Test, Datasets, Golden Datasets, Evaluation Configs, and Runs & Results — so reach for those when you need the full parameter surface.

Download Example Notebooks

Choose an example below based on your model type:

Large Language Models (LLM)

Retrieval-Augmented Generation (RAG)

Agentic Systems

Computer Vision (CV)

The Walkthrough

The use case is a general-knowledge Q&A assistant. We hold ten questions with reference answers, point AIP at gpt-4o-mini, and measure how closely the model's answers match the references with BLEU, ROUGE, and exact-match.

Connect and pick a workspace

Initialize the SDK once with your trial host and API key, then resolve a workspace. Everything downstream is scoped to it.

import os
import pandas as pd
import aip_sdk as aip

aip.init("https://api.trials.aip-v2.resarodev.ai", api_key=os.environ["AIP_API_KEY"])

workspace_name = os.environ.get("AIP_WORKSPACE_NAME")
ws = aip.Workspace.get_by_name(workspace_name) if workspace_name else aip.Workspace.default()

You also need an OpenAI key in the environment — AIP will use it to authenticate the hosted calls to OpenAI on your behalf.

OPENAI_ENDPOINT = "https://api.openai.com/v1/chat/completions"
OPENAI_MODEL = "gpt-4o-mini"
assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY before running."

Create or reuse the project

The project anchors the SUT, the dataset, and the run. Because this is an LLM (not RAG) use case, the project is a gdi_text_v1 schema with the single_turn_llm task type. Declare a topic dimension so results can be sliced by it later. get_or_create is idempotent, so re-running the guide reuses the same project rather than creating duplicates. See Projects for the full create surface.

project, created = aip.Project.get_or_create(
    name="OpenAI LLM Demo",
    schema="gdi_text_v1",
    task_type="single_turn_llm",
    dimensions=[aip.Dimension(name="topic", column="topic", values=["general_knowledge"])],
    workspace_id=ws.id,
)

Register the OpenAI SUT and connection

The system under test is the OpenAI model. Register it against the project, then attach a connection describing the endpoint AIP should call: the Chat Completions URL, bearer auth carrying your OpenAI key (stored encrypted by the platform), the openai_chat protocol, and the model to invoke. The single_turn_llm task type must match the project. Full options are in Systems Under Test.

sut, _ = aip.Sut.get_or_register(
    name="openai-gpt4o-mini",
    version="1.0",
    owner="ml-platform",
    project_id=project.id,
)

conn = sut.add_connection(
    label="chat-prod",
    base_url=OPENAI_ENDPOINT,
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=os.environ["OPENAI_API_KEY"],
    sut_protocol="openai_chat",
    gdi_schema="gdi_text_v1",
    task_type="single_turn_llm",
    model_params={"model": OPENAI_MODEL},
)

The raw OpenAI response is JSON, so the connection needs an adapter that lifts the answer text into the GDI sut_response column. The openai_chat template maps $.choices[0].message.content for you.

conn.set_adapter(
    template_name="openai_chat",
    mapping_config={"sut_response": "$.choices[0].message.content"},
)

Before spending a real run on it, confirm the endpoint answers and the mapping resolves.

result = conn.test()
print(result.success, result.status_code, result.error)

Build and upload the Q&A dataset

Assemble the evaluation rows as a DataFrame. Each row carries an input_id, the prompt, the ground-truth expected_output, and the two dimension columns (topic, category). Leave sut_response empty — the hosted run fills it with the model's real answer. Because the answers are ground truth, this dataset drives reference-based scoring.

qa_data = [
    {
        "input_id": "q1",
        "prompt": "What is the capital of Japan?",
        "expected_output": "The capital of Japan is Tokyo.",
        "topic": "general_knowledge",
        "category": "geography",
    },
    {
        "input_id": "q2",
        "prompt": "How many continents are there on Earth?",
        "expected_output": "There are seven continents on Earth.",
        "topic": "general_knowledge",
        "category": "geography",
    },
    {
        "input_id": "q3",
        "prompt": "What is the chemical symbol for gold?",
        "expected_output": "The chemical symbol for gold is Au.",
        "topic": "general_knowledge",
        "category": "science",
    },
    {
        "input_id": "q4",
        "prompt": "Who developed the theory of general relativity, and when was it published?",
        "expected_output": "The theory of general relativity was developed by Albert Einstein and first published in 1915.",
        "topic": "general_knowledge",
        "category": "science",
    },
    {
        "input_id": "q9",
        "prompt": "In what year did the First World War end?",
        "expected_output": "The First World War ended in 1918.",
        "topic": "general_knowledge",
        "category": "history",
    },
    {
        "input_id": "q10",
        "prompt": "What is the smallest prime number and why is it considered prime?",
        "expected_output": "The smallest prime number is 2, and it is the only even prime number because it has exactly two distinct divisors: 1 and itself.",
        "topic": "general_knowledge",
        "category": "mathematics",
    },
]

df = pd.DataFrame(
    [
        {
            "input_id": r["input_id"],
            "task_type": "single_turn_llm",
            "prompt": r["prompt"],
            "expected_output": r["expected_output"],
            "sut_response": "",
            "topic": r["topic"],
            "category": r["category"],
        }
        for r in qa_data
    ]
)

Upload it through the project. AIP versions the dataset; the first upload becomes v1. See Datasets for versioning and download details.

dataset = project.upload_dataset(df, name="llm-qa")
version = dataset.latest_version()

Map to the schema, run checks, add dimensions

Declare how the uploaded columns satisfy the gdi_text_v1 contract. Here the column names already match, so the mapping is an identity map that pins the required columns.

version = dataset.map_version(
    version.id,
    column_mapping={
        "input_id": "input_id",
        "prompt": "prompt",
        "expected_output": "expected_output",
        "sut_response": "sut_response",
    },
)

Run server-side quality checks on the mapped version. The returned report carries the verdict (PASS / WARN / FAIL) and per-check evidence — this is the gate the golden promotion enforces next.

report = dataset.run_checks(version.id, label="post-mapping")
print(report.status, report.row_count)

You can also ask the platform which metrics suit this dataset. For a Q&A set with reference answers it steers you toward reference-based scorers, which is what we select in the next section.

suggestions = aip.check_suggestions(dataset.id)
for s in sorted(suggestions.suggestions, key=lambda s: s.rank):
    print(s.rank, s.metric, s.reason)

Dimensions let AIP stratify coverage and results. The project already has topic; add category (the knowledge domain) as a second analysis dimension. Dimensions live on the project, and patching replaces the whole dimension set — so send the existing dimensions plus the new one.

category_dim = aip.Dimension(
    name="category",
    column="category",
    values=["geography", "science", "history", "mathematics"],
    required=False,
)

client = aip._context.get_default_client()
new_dims = [d.to_dict() for d in project.dimensions] + [category_dim.to_dict()]
client.patch(f"/projects/{project.id}", json={"dimensions": new_dims})
project.dimensions = [aip.Dimension.from_dict(x) for x in new_dims]

Promote to golden and download

A run evaluates the golden version of a dataset, so promote the mapped version. A clean PASS promotes directly; for a small demo set that only raises warnings, acknowledge them with force=True and a reason (the promotion is audited). See Golden Datasets.

try:
    version = dataset.promote(version.id)
except aip.APIError:
    version = dataset.promote(version.id, force=True, reason="Demo dataset acknowledged for evaluation.")

print(version.is_golden, version.stage, version.version)

Download the golden set to confirm the rows that the evaluation will run against.

df_golden = dataset.download()
print(len(df_golden), list(df_golden.columns))

Discover and select metrics

List the LLM metrics that accept the gdi_text_v1 schema, then pick the ones that suit a Q&A set with ground-truth answers. llm.bleu, llm.rouge, and llm.exact_match are reference-based and run without a judge. Judge-backed metrics such as llm.correctness or llm.toxicity are also available and are scored server-side on the AIP host, which already holds the judge credential — add them to the list to include them.

metrics = aip.ops.list_metrics(schema="gdi_text_v1")
available = {m["name"] for m in metrics}

selected_metrics = [m for m in ["llm.bleu", "llm.rouge", "llm.exact_match"] if m in available]

Publish the evaluation config

Persist the run definition — project, pinned dataset version, SUT, connection, and metrics — as a reusable evaluation config. Publishing returns a server-assigned id you can pull later or share across runs. See Evaluation Configs.

from aip_core.schemas.eval_config import EvalConfig

cfg = EvalConfig(
    project=str(project.id),
    dataset=f"{dataset.id}@v{version.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
)

cfg_id = aip.publish_config(
    cfg,
    name="llm-qa-eval",
    description="LLM Q&A demo evaluation config",
    workspace_id=ws.id,
)

Run the hosted evaluation

Passing connection_id to aip.run() (here alongside sut_id, which is attribution-only) selects the hosted runner: AIP calls the live OpenAI endpoint through your connection to generate each answer, then computes the selected metrics — you only wait for it to finish. Poll until the run reaches a terminal state. See Runs & Results for runner modes and the result types.

with aip.run(
    project=project.id,
    dataset=f"{dataset.id}@v{version.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=selected_metrics,
) as run:
    run.poll_status(interval=5, timeout=600)
    run_id = run.id

Retrieve per-row scores and aggregates

Read the results back as a RunResultsPage. Each ScoreResult is one (input_id, scorer) pair with a numeric score and optional explanation; each Metric is the per-scorer aggregate across the whole run — mean, standard deviation, count, and pass rate.

run = aip.get_run(run_id)
page = run.results()

for r in page.results:
    print(r.input_id, r.scorer, r.score)

for m in page.metrics:
    print(f"{m.scorer}: mean={m.mean:.3f} pass_rate={m.pass_rate:.0%} (n={m.count})")

For descriptive statistics — including per-dimension breakdowns — call run.analysis() and group by one of the dimension columns you declared, for example category.

analysis = run.analysis(group_by="category")
print(analysis["scorer_stats"])

That closes the loop: a registered OpenAI SUT, a golden Q&A dataset, three reference-based metrics, and per-row plus aggregate scores from a fully hosted run — reproducible from the published evaluation config.