Systems under Test

A System Under Test (SUT) is the model or service you want AIP to evaluate. The SUT record itself is just metadata — a name, a version, and an owner — anchored to a project. What makes it callable is a Connection: the HTTP endpoint AIP invokes during a run, together with its authentication, the protocol AIP should speak, and the response mapping that lands the model's output into your GDI columns. A SUT can carry more than one connection (for example a staging and a production endpoint), and each run points at exactly one SUT plus one connection.

Registering a SUT is a two-part idea. First you declare the SUT and attach a connection. Then you tell AIP how to translate the raw JSON that endpoint returns into the canonical GDI output columns (sut_response, predictions, retrieved_context, …) that the scorers read. The sections below cover both, from the one-call shortcut down to the step-by-step flow.

Discovering schemas and templates

Before you register anything, it helps to see which GDI schemas and adapter templates are available. A template bundles a wire protocol, a target schema, and a default response mapping so you don't have to write the JSONPath by hand. aip.Sut.templates() returns the built-in templates keyed by name; each entry carries its sut_protocol, gdi_schema, params_schema, and default mapping_config. This is a read-only reference call.

schemas.list_names() lists the available GDI schemas, each of which can be used as a project schema or a SUT gdi_schema. aip.Sut.templates() lists the built-in SUT adapter templates, mapping each protocol to its schema and params. The built-in templates are:

• openai_chat — OpenAI ChatCompletion response parser
• rag — RAG (retrieval-augmented generation) parser
• cv_detection — Computer vision detection response parser
• cv_classification — Image classification response parser

SDK

Iterate both listings:

import aip_sdk as aip
import aip_core.schemas as schemas

for name in sorted(schemas.list_names()):
    print(name, schemas.get(name).family)

templates = aip.Sut.templates()
for name, info in templates.items():
    print(name, "-", info["description"], info.get("sut_protocol"), info.get("gdi_schema"))

API

GDI schemas ship with the SDK and are not served over the API; SUT templates are:

curl -sS "$AIP_API_URL/suts/templates" \
  -H "Authorization: Bearer $AIP_TOKEN"

Register Your SUT

aip.sut.register() creates the SUT record and its first connection in a single call. Pass the model metadata (name, version, owner, project_id), the endpoint AIP should call, the auth details, and the sut_protocol / gdi_schema pair that tells AIP how to speak to the endpoint. It returns the created SUT, whose .id you use everywhere downstream.

For task-scoped schemas — gdi_text_v1 distinguishes single_turn_llm from single_turn_rag, and gdi_image_v1 requires a detection / classification task — also pass task_type so AIP resolves the correct row shape. The second example below registers an image-detection SUT and passes task_type="detection" for exactly this reason.

register() raises DuplicateSutError if the name already exists; use aip.Sut.get_or_register() when you want an idempotent call.

SDK

register() handles both an OpenAI-compatible chat SUT and, in the second example, an image-detection SUT that passes task_type="detection" for its task-scoped schema:

import os
import aip_sdk as aip

sut = aip.sut.register(
    name="my-model",
    endpoint="https://api.openai.com/v1/chat/completions",
    project_id=project.id,
    label="production",
    version="1.0",
    owner="ml-team",
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=os.environ["OPENAI_API_KEY"],
    model_params={"model": "gpt-4o-mini", "temperature": 0.7},
    sut_protocol="openai_chat",
    gdi_schema="gdi_text_v1",
)
print(sut.id)

cv_sut = aip.sut.register(
    name="detector",
    endpoint="https://api.example.com/detect",
    project_id=project.id,
    auth_type="none",
    sut_protocol="huggingface_object_detection",
    gdi_schema="gdi_image_v1",
    task_type="detection",
)

API

POST the SUT metadata to /suts:

curl -sS -X POST "$AIP_API_URL/suts" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"My SUT","version":"1.0","owner":"ml-platform","project_id":"<project_id>"}'

Step-by-step registration

When you want more control — or you're wiring a non-standard endpoint — separate the four steps: register the SUT, add a connection, set the response mapping, then validate and test it. In the SDK this reads as get_or_register() → add_connection() → set_adapter() → validate() / test(). get_or_register() is idempotent and returns the SUT together with a created flag.

SDK

import os

sut, created = aip.Sut.get_or_register(
    name="OpenAI GPT-4o-mini",
    version="2024-01",
    owner="ml-platform",
    project_id=project.id,
)

Adding the connection

sut.add_connection() records the HTTP endpoint AIP calls for this SUT. The key parameters are:

• base_url — the full endpoint AIP POSTs to (it sends the request there directly; no path is appended).
• label — a human name for this connection (e.g. chat-prod), so you can tell staging from production.
• auth_type / auth_header_name / auth_header_value — how AIP authenticates. The value is stored encrypted and sent verbatim, so include any Bearer prefix the target requires yourself.
• sut_protocol — the request builder AIP uses to shape the outbound call (see the SUT Protocol Reference below).
• gdi_schema and, for task-scoped schemas, task_type — the row shape AIP resolves for this connection. task_type is required whenever gdi_schema is task-scoped (e.g. gdi_image_v1 detection).
• model_params — protocol-specific request options (e.g. {"model": "gpt-4o-mini"}, or a prompt_field rename for generic_json).

SDK

conn = sut.add_connection(
    label="chat-prod",
    base_url="https://api.openai.com/v1/chat/completions",
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=os.environ["OPENAI_API_KEY"],
    model_params={"model": "gpt-4o-mini", "temperature": 0.7},
    sut_protocol="openai_chat",
    gdi_schema="gdi_text_v1",
    task_type="single_turn_llm",
)

API

POST the connection to the SUT's /connections:

curl -sS -X POST "$AIP_API_URL/suts/<sut_id>/connections" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"label":"prod","base_url":"https://api.openai.com/v1/chat/completions","auth_type":"bearer","auth_header_name":"Authorization","sut_protocol":"openai_chat","gdi_schema":"gdi_text_v1","task_type":"single_turn_llm"}'

Pass credentials through an environment variable (--auth-header-value-env OPENAI_API_KEY) rather than --auth-header-value so the secret never lands in shell history. The available auth types are none (default), bearer, api_key, and basic.

Mapping the response into GDI output columns

The endpoint returns its own JSON shape; the scorers expect canonical GDI columns. The adapter bridges the two. conn.set_adapter() takes a template_name (an entry from aip.Sut.templates(), or None for a fully custom mapping) and a mapping_config — a dict of GDI column → JSONPath into the raw response. For an OpenAI chat endpoint the assistant text lives at $.choices[0].message.content, which maps to sut_response; you can pull more fields (token usage, citations, retrieved contexts) in the same call.

SDK

adapter = conn.set_adapter(
    template_name="openai_chat",
    mapping_config={
        "sut_response": "$.choices[0].message.content",
        "usage_prompt_tokens": "$.usage.prompt_tokens",
        "usage_completion_tokens": "$.usage.completion_tokens",
    },
)

API

POST the template and mapping to the connection's /adapter:

curl -sS -X POST "$AIP_API_URL/suts/<sut_id>/connections/<connection_id>/adapter" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"template_name":"openai_chat","mapping_config":{"sut_response":"$.choices[0].message.content"}}'

Supported JSONPath syntax

A mapped value can sit at any depth — write the full path to it. The mapper supports:

Syntax	Example	Resolves to
$	$	the whole response
.key	$.data.result.answer	a nested member
['key'], ["key"]	$['data']['result']['answer']	a nested member; the only way to name a key containing ., a space, or a bracket ($['odd.key'])
..key	$..answer	answer wherever it sits; one match resolves to the value, several to a list of all matches in document order (depth-first — a match nested inside an earlier sibling comes before a later sibling's)
[n], [-n]	$.choices[0].message.content, $.turns[-1].text	a list element, from the front or the back
[start:stop:step]	$.chunks[0:3].text	a slice of a list
[*], .*	$.sources[*].text, $.fields.*.value	every element of a list, or every value of an object

A wildcard, slice, or .. maps the rest of the path over each match, so $.sources[*].text yields a list of texts — with a null in place of any match that has no text. If no match has one, the path counts as unresolved and the field is reported as a mapping error, so a mistyped tail never reaches scoring as a list of nulls. A field that is present but null is an answer rather than a miss — it maps as null and keeps its position among its siblings. Filter expressions ([?(...)]) are not supported. A path the mapper cannot parse is rejected when you set the adapter, so a typo surfaces immediately rather than as an empty column after a run.

Validating the mapping

Before you spend a live call, check the mapping against a representative payload. adapter.validate() runs the JSONPath extraction over a sample response and reports whether every mapped column resolved. This catches the common ResponseMappingError (an expected field missing from the response) offline.

SDK

validation = adapter.validate(
    {
        "choices": [{"message": {"content": "Hello from the SUT"}}],
        "usage": {"prompt_tokens": 8, "completion_tokens": 6},
    }
)
assert validation.valid, validation.errors

API

POST a sample payload to the adapter's /validate:

curl -sS -X POST "$AIP_API_URL/suts/<sut_id>/connections/<connection_id>/adapter/validate" \
  -H "Authorization: Bearer $AIP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sample_payload":{"choices":[{"message":{"content":"Hello from the SUT"}}]}}'

Running a live connection test

Once the mapping validates offline, conn.test() makes a real HTTP request to the endpoint and reports .success, .status_code, and .error. This exercises auth, network reachability, and the mapping end to end. Hosted inference endpoints can cold-start, so a 503 or timeout on the first attempt often just means the endpoint is waking up — retry after a few seconds.

SDK

result = conn.test()
print(result.success, result.status_code, result.error)

API

POST to the connection's /test:

curl -sS -X POST "$AIP_API_URL/suts/<sut_id>/connections/<connection_id>/test" \
  -H "Authorization: Bearer $AIP_TOKEN"

Listing and inspecting SUTs

aip.list_suts() returns the SUT inventory (optionally filtered by project); each SutInstance carries .id, .name, .version, .owner, .project_id, and .connection_count. To see a connection's protocol, base URL, and status you must fetch the SUT with aip.get_sut() and read .connections() — connection-level detail is not returned by the list call.

SDK

suts = aip.list_suts(project_id=project.id, page=1, per_page=100)

sut = aip.get_sut("<sut-uuid>")
for conn in sut.connections():
    print(conn.id, conn.label, conn.sut_protocol, conn.base_url)

API

List SUTs by project and fetch one by id:

curl -sS "$AIP_API_URL/suts?project_id=<project_id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

curl -sS "$AIP_API_URL/suts/<sut_id>" \
  -H "Authorization: Bearer $AIP_TOKEN"

Adapter templates

The built-in templates cover the common wire formats. Each pairs a protocol with a schema and a default mapping, so you only override the JSONPath you need:

Template	Protocol	Schema	What it maps
openai_chat	openai_chat	gdi_text_v1	Builds messages from prompt; maps $.choices[0].message.content to sut_response.
rag	rag_api	gdi_text_v1 + task_type=single_turn_rag	OpenAI-compatible chat with optional context; maps answer, citations, and retrieved context.
cv_detection	huggingface_object_detection	gdi_image_v1 + task_type=detection	Sends image bytes when present; normalises detections into predictions.
cv_classification	huggingface_image_classification	gdi_image_v1 + task_type=classification	Sends image bytes when present; normalises per-class scores into predictions.

For endpoints that don't match a template — such as a RAG service with bespoke field names — use sut_protocol="generic_json" with template_name=None and a custom mapping_config. generic_json honours a prompt_field override in model_params, letting you rename the outbound prompt field (e.g. to message) and add extra request params. The example below wires a custom RAG endpoint through generic_json with no template, renaming the outbound field to message and mapping the bespoke response fields by hand. With the SDK:

conn = sut.add_connection(
    label="rag-service",
    base_url="https://rag.example.com/api/v1/workspace/nist/chat",
    auth_type="bearer",
    auth_header_name="Authorization",
    auth_header_value=f"Bearer {os.environ['RAG_API_KEY']}",
    sut_protocol="generic_json",
    gdi_schema="gdi_text_v1",
    task_type="single_turn_rag",
    model_params={"prompt_field": "message", "mode": "query"},
)
conn.set_adapter(
    template_name=None,
    mapping_config={
        "sut_response": "$.textResponse",
        "retrieved_context": "$.sources[*].text",
    },
)

SUT Protocol Reference

The sut_protocol picks the request builder; the gdi_schema (plus task_type where the schema is task-scoped) picks the row shape. Together they determine both what AIP sends and how it reads the reply.

Use case	gdi_schema	sut_protocol	Adapter template	Notes
OpenAI-compatible chat / LLM	gdi_text_v1	openai_chat	openai_chat	Builds messages from prompt; maps choices[0].message.content to sut_response.
RAG API	gdi_text_v1 + task_type=single_turn_rag	rag_api	rag	OpenAI-compatible chat with optional context; maps answer, citations, contexts, and retrieved context.
CV object detection	gdi_image_v1 + task_type=detection	huggingface_object_detection	cv_detection	Sends image bytes when present and normalises detections into predictions.
CV image classification	gdi_image_v1 + task_type=classification	huggingface_image_classification	cv_classification	Sends image bytes when present and normalises per-class scores into predictions.
Custom text JSON	gdi_text_v1	generic_json or custom	none	Sends JSON with prompt plus model_params; requires a custom mapping_config.
Anthropic Messages	gdi_text_v1	anthropic_messages	none	Builder exists; no built-in adapter template is exposed by Sut.templates().

model_params, mapping_config, and sample_payload each accept either inline JSON or a path to a JSON file.