Skip to main content

aip_sdk.judge

Judge connection SDK — register and manage the LLM judge used to score metrics.

aip_sdk.judge.JudgeConnection​

aip_sdk.judge.JudgeConnection(data: dict[str, Any], client: APIClient)

A registered judge connection — a named LLM judge (model, endpoint, key).

Register a judge once, then reference it when scoring metrics instead of pasting raw credentials into eval configs. The API key is write-only: it is sent at registration and never returned, so it is not exposed as an attribute.

Attributes

  • id str: Connection ID.
  • name str: Human-readable name, unique within a workspace.
  • model str: Judge model id, e.g. "gpt-4o-mini".
  • base_url str | None: OpenAI-compatible base URL as stored, or None to use the platform default judge. A trailing slash and a trailing /chat/completions are stripped on write, so this may differ from the string that was sent.
  • project_id str | None: Project this connection is scoped to, if any.
  • workspace_id str | None: Workspace this connection is scoped to, if any.
  • created_at datetime | None: When the connection was registered.
  • updated_at datetime | None: When the connection was last modified.

aip_sdk.judge.JudgeConnection.base_url​

aip_sdk.judge.JudgeConnection.base_url: str | None = data.get('base_url')

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.created_at​

aip_sdk.judge.JudgeConnection.created_at: datetime | None = parse_dt(data['created_at'])

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.delete​

aip_sdk.judge.JudgeConnection.delete() -> None

Delete this judge connection.

Raises

  • JudgeConnectionNotFoundError: If the connection no longer exists, or belongs to a workspace the caller is not a member of.
  • ForbiddenError: If the caller is not an editor or admin of the connection's workspace. A platform admin must be a member of that workspace too.
  • AuthError: If credentials are missing or invalid.
  • APIError: If the deletion otherwise fails.

aip_sdk.judge.JudgeConnection.id​

aip_sdk.judge.JudgeConnection.id: str = data['id']

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.model​

aip_sdk.judge.JudgeConnection.model: str = data['model']

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.name​

aip_sdk.judge.JudgeConnection.name: str = data['name']

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.project_id​

aip_sdk.judge.JudgeConnection.project_id: str | None = data.get('project_id')

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.test​

aip_sdk.judge.JudgeConnection.test() -> JudgeTestResult

Run a live test call through this connection's model, endpoint, and key.

Issues a minimal chat completion to confirm the judge is reachable, the credential is accepted, and the model is available. Never raises for a judge-side failure — inspect JudgeTestResult.success and reason.

Returns

  • JudgeTestResult JudgeTestResult: the outcome, with success and an actionable reason.

Raises

Example:

result = connection.test()
if not result.success:
    print(result.reason)

aip_sdk.judge.JudgeConnection.update​

aip_sdk.judge.JudgeConnection.update(*, name: str | None = None, model: str | None = None, base_url: str | None = None, api_key: str | None = None, project_id: str | None = None) -> JudgeConnection

Update fields on this connection, or rotate its API key.

Only the fields you pass are changed. Pass api_key to replace the stored credential; the previous key is discarded.

Parameters

  • name str | None: New name; must stay unique within the workspace.
  • model str | None: New judge model id.
  • base_url str | None: New OpenAI-compatible base URL. A trailing slash and a trailing /chat/completions are stripped before storage.
  • api_key str | None: New plaintext API key, encrypted at rest by the platform.
  • project_id str | None: Move the connection to this project.

Returns

Raises

aip_sdk.judge.JudgeConnection.updated_at​

aip_sdk.judge.JudgeConnection.updated_at: datetime | None = parse_dt(data['updated_at'])

No docstring is defined in the source.

aip_sdk.judge.JudgeConnection.workspace_id​

aip_sdk.judge.JudgeConnection.workspace_id: str | None = data.get('workspace_id')

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult​

aip_sdk.judge.JudgeTestResult(success: bool, status_code: int | None, reason: str | None, error: str | None, duration_ms: float | None, model: str | None, sample_response: str | None)

Outcome of a live test call against a judge connection.

Attributes

  • success bool: Whether the judge returned a usable response.
  • status_code int | None: HTTP status from the judge, when the call reached it.
  • reason str | None: Human-readable summary of the outcome.
  • error str | None: Underlying transport or API detail when the test failed.
  • duration_ms float | None: Round-trip duration in milliseconds, on success.
  • model str | None: The judge model the test call targeted.
  • sample_response str | None: Truncated judge output on success.

aip_sdk.judge.JudgeTestResult.duration_ms​

aip_sdk.judge.JudgeTestResult.duration_ms: float | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.error​

aip_sdk.judge.JudgeTestResult.error: str | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.model​

aip_sdk.judge.JudgeTestResult.model: str | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.reason​

aip_sdk.judge.JudgeTestResult.reason: str | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.sample_response​

aip_sdk.judge.JudgeTestResult.sample_response: str | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.status_code​

aip_sdk.judge.JudgeTestResult.status_code: int | None

No docstring is defined in the source.

aip_sdk.judge.JudgeTestResult.success​

aip_sdk.judge.JudgeTestResult.success: bool

No docstring is defined in the source.

aip_sdk.judge.get_judge_connection​

aip_sdk.judge.get_judge_connection(judge_id: str, *, client: APIClient | None = None) -> JudgeConnection

Fetch a registered judge connection by ID.

Parameters

  • judge_id str: Judge connection ID.
  • client APIClient | None: Optional pre-configured API client.

Returns

  • JudgeConnection JudgeConnection: the connection (its API key is never returned).

Raises

Example:

judge = aip.get_judge_connection("judge_abc123")

aip_sdk.judge.list_judge_connections​

aip_sdk.judge.list_judge_connections(*, project_id: str | None = None, workspace_id: str | None = None, page: int = 1, per_page: int = 100, client: APIClient | None = None, all_workspaces: bool = False) -> list[JudgeConnection]

List registered judge connections visible to the caller.

Reads your session's workspace unless you name one, and raises if none is set. Pass all_workspaces=True to read across every workspace you can access.

Parameters

  • project_id str | None: Scope the listing to this project, which determines the workspace. Pass this or workspace_id, not both.
  • workspace_id str | None: Restrict the listing to this workspace.
  • page int: 1-based page number.
  • per_page int: Page size.
  • client APIClient | None: Optional pre-configured API client.
  • all_workspaces bool: Read across every workspace you can access.

Returns

  • list[JudgeConnection]: list[JudgeConnection]: the connections on the requested page (keys masked).

Raises

Example:

for judge in aip.list_judge_connections(workspace_id="ws-abc123"):
    print(judge.name, judge.model)

aip_sdk.judge.logger​

aip_sdk.judge.logger = get_logger(__name__)

No docstring is defined in the source.

aip_sdk.judge.register_judge_connection​

aip_sdk.judge.register_judge_connection(name: str, model: str, *, base_url: str | None = None, api_key: str | None = None, project_id: str | None = None, workspace_id: str | None = None, client: APIClient | None = None) -> JudgeConnection

Register a judge connection so metrics can reference it by name.

Stores a named judge — model, optional OpenAI-compatible base URL, and API key — server-side. The key is encrypted at rest and never returned. Only name and model are required; omit api_key for an endpoint that authenticates by network rather than a token.

Parameters

  • name str: Connection name, unique within the workspace.
  • model str: Judge model id, e.g. "gpt-4o-mini".
  • base_url str | None: OpenAI-compatible base URL; omit to use the platform default judge. A trailing slash and a trailing /chat/completions are stripped before storage.
  • api_key str | None: Plaintext API key, encrypted at rest by the platform.
  • project_id str | None: Project to scope the connection to.
  • workspace_id str | None: Workspace to scope the connection to. Pass this or project_id, not both — a project-scoped connection inherits its workspace from the project. Falls back to the client's configured workspace.
  • client APIClient | None: Optional pre-configured API client.

Returns

  • JudgeConnection JudgeConnection: the registered connection, with its generated id.

Raises

Example:

import aip_sdk as aip

aip.init("https://aip.example.com", api_key="...")
judge = aip.register_judge_connection(
    name="approved-gpt4o",
    model="gpt-4o-mini",
    base_url="https://llm.internal.example.com/v1",
    api_key="sk-...",
    project_id="proj-abc123",
)
print(judge.id, judge.name)