Quickstart: Your First Evaluation

Run a hosted evaluation against a small question-and-answer dataset, then inspect the metric scores.

Quick Start with Examples

Get started quickly with our example notebooks and sample data:

Before you begin

Complete installation and authentication. Your account needs an active workspace and permission to create projects, datasets, and systems under test. This example also needs an OpenAI API key with access to the configured model; model calls can incur charges. Set AIP_API_KEY and OPENAI_API_KEY in your environment.

Testing an agentic system? Go to Guide: Evaluate an Agent instead.

This end-to-end example evaluates OpenAI's Chat Completions API as an LLM under test: create a project, connect the SUT, upload and promote a golden Q&A dataset, then run a hosted evaluation and read the scores. It uses one modality (gdi_text_v1, single_turn_llm) and the hosted runner, so AIP calls OpenAI and computes the metrics for you.

1. Connect and resolve a workspace.

import os, time
import pandas as pd
import aip_sdk as aip

aip.init("https://api.trials.aip-v2.resarodev.ai", api_key=os.environ["AIP_API_KEY"])

workspace_name = os.environ.get("AIP_WORKSPACE_NAME")
ws = aip.Workspace.get_by_name(workspace_name) if workspace_name else aip.Workspace.default()

2. Create the project.

The schema and task_type fix the data contract; a Dimension declares a categorical column to slice results by. get_or_create is idempotent, so re-running the script reuses the project.

project, _ = aip.Project.get_or_create(
    name="OpenAI LLM Demo",
    schema="gdi_text_v1",
    task_type="single_turn_llm",
    dimensions=[aip.Dimension(name="category", column="category", values=["geography", "science", "history"])],
    workspace_id=ws.id,
)

3. Register the SUT and its connection.

This takes three steps — register the model, connect its endpoint, and map the response. The auth_header_value you pass is stored encrypted by the platform.

3a. Register the model.

sut, _ = aip.Sut.get_or_register(
    name="openai-gpt4o-mini",
    version="1.0",
    project_id=project.id,
)

3b. Add the OpenAI Chat Completions endpoint with bearer auth.

conn = sut.add_connection(
    label="chat-prod",
    base_url="https://api.openai.com/v1/chat/completions",
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=os.environ["OPENAI_API_KEY"],
    sut_protocol="openai_chat",
    gdi_schema="gdi_text_v1",
    task_type="single_turn_llm",
    model_params={"model": "gpt-4o-mini"},
)

3c. Set an adapter that maps the response JSON into the GDI sut_response column.

conn.set_adapter(
    template_name="openai_chat",
    mapping_config={"sut_response": "$.choices[0].message.content"},
)

4. Upload the dataset.

Build a DataFrame of prompts and ground-truth answers, then upload it under the project. Leave sut_response empty — the hosted run fills in the real predictions.

rows = [
    {
        "input_id": "q1",
        "task_type": "single_turn_llm",
        "prompt": "What is the capital of Japan?",
        "expected_output": "The capital of Japan is Tokyo.",
        "sut_response": "",
        "category": "geography",
    },
    {
        "input_id": "q2",
        "task_type": "single_turn_llm",
        "prompt": "What is the chemical symbol for gold?",
        "expected_output": "The chemical symbol for gold is Au.",
        "sut_response": "",
        "category": "science",
    },
]
dataset = project.upload_dataset(pd.DataFrame(rows), name=f"llm-qa-{int(time.time())}")
version = dataset.latest_version()

5. Run quality checks and promote to golden.

run_checks returns a verdict; promote marks the version as the golden baseline. A clean verdict promotes directly; pass force=True to acknowledge warnings on a small demo set.

report = dataset.run_checks(version.id, label="pre-promote")
if report.status == "PASS":
    version = dataset.promote(version.id)
elif report.status == "WARN":
    version = dataset.promote(version.id, force=True, reason="Reviewed small demo dataset warnings")
else:
    raise RuntimeError(f"Resolve dataset quality verdict {report.status} before promotion")

6. Choose metrics.

Reference-based LLM metrics compare the SUT output to the expected answer and need no external judge. Discover what's available with aip.ops.list_metrics(schema="gdi_text_v1") — see Discovering metrics in the Reference for the full SDK commands.

metrics = ["llm.correctness", "llm.bleu", "llm.rouge", "llm.exact_match"]

7. Run the hosted evaluation.

Passing connection_id (here alongside sut_id, which is attribution-only) puts aip.run() into hosted mode: AIP calls OpenAI through the connection, computes the metrics, and you only poll for completion.

• poll_status blocks until the run reaches a terminal state.
• results() returns a RunResultsPage.
• Keeping run.id lets you re-open the run later.

with aip.run(
    project=project.id,
    dataset=f"{dataset.id}@v{version.version}",
    sut_id=sut.id,
    connection_id=conn.id,
    metrics=metrics,
) as run:
    run.poll_status(interval=5, timeout=300)
    page = run.results(page=1, page_size=100)
    run_id = run.id

8. Read the results.

results() returns a RunResultsPage with per-row ScoreResults and per-scorer Metric aggregates.

for r in page.results:
    print(r.input_id, r.scorer, r.score)

for m in page.metrics:
    print(f"{m.scorer}: mean={m.mean:.3f} pass_rate={m.pass_rate:.0%} (n={m.count})")

To re-open a finished run later — for example in CI after the run script exits — attach to it by id with aip.get_run(run_id) and call .results() or .analysis() directly.

If the evaluation fails

If authentication returns 401, check the API host and credential. If no default workspace exists, set AIP_WORKSPACE_NAME to a workspace you can access. Review the quality report before acknowledging WARN; resolve FAIL or ERROR before promotion. If polling times out, retain run_id and inspect the run in the console before retrying.

Next, evaluate a model end to end or read results.