Skip to main content

Troubleshooting & Debugging

Answers to common questions from customers about running evaluations, configuring metrics, comparing results, and deploying AIP. If you don't see your question here, reach out to your Resaro contact.

Contents:

Getting connected​

Q: aip.init raises a 401 Unauthorized — what's wrong?​

A: Your AIP_API_KEY is wrong or expired. Re-copy it from your credentials email and check for trailing spaces or accidental line breaks.

Q: Workspace.get_by_name raises a 404 / "Workspace not found" — what's wrong?​

A: Workspace.get_by_name() expects the exact, case-sensitive workspace name. If your credentials email supplies a workspace ID, use Workspace.get() instead:

import os

import aip_sdk as aip

aip.init()
workspace = aip.Workspace.get(os.environ["AIP_WORKSPACE_ID"])

For a name lookup, use aip.Workspace.get_by_name("<your-workspace-name>"). IDs and names are different values. The SDK reads AIP_WORKSPACE_ID automatically; AIP_WORKSPACE_NAME only works when your script explicitly reads it and calls get_by_name(). If the correct ID or name still cannot be found, check that your credentials belong to the same deployment and have access to that workspace.

Q: A run fails with Connection refused against my SUT — what's wrong?​

A: Your SUT isn't reachable from wherever it's being called from. In hosted mode, the AIP platform calls your SUT, so SUT_BASE_URL must be reachable from the platform, not just from your laptop — a URL that only resolves on a private network or behind a corporate VPN will fail even if it works locally. See Two run modes for hosted vs. external connectivity.

Q: My first object-detection run fails with a 503 — what's wrong?​

A: The Hugging Face CV endpoint was still starting up from idle. Re-run once the endpoint is ready.

Q: A run times out — what's wrong?​

A: Your SUT is slow per call. Lower max_tokens on the request, or raise the timeout passed to poll_status(timeout=...).

Workspaces & Deployment​

Q: What's the difference between a Trial and a Pilot?​

A: There are two deployment types:

  • Trials — run on Resaro-hosted AIP at trials.aip-v2.resarodev.ai. Each trial client gets one workspace and can add users to that workspace.
  • Pilot — you get your own dedicated deployment in your own environment. Because you have the full platform, you can assign your own staff Platform Admin and create or modify workspaces internally.

Q: Can I create additional workspaces during a Trial?​

A: Not during the Trial — Trial clients have a single workspace and can only add users to it. Multiple, self-managed workspaces become available once you move to a Pilot or Enterprise deployment.

Running Evaluations & Assessments​

Q: What run modes does AIP support?​

A: AIP supports two run modes:

  • Hosted — AIP calls your registered SUT connection for you: it pulls the golden data, invokes your endpoint for every row, and scores the responses. Your SUT keeps running wherever you deploy it — the platform only needs network access to its endpoint

  • External — you run inference yourself: you pull the golden data, invoke your SUT for every row on your own infrastructure, and upload the raw responses. AIP scores them server-side — your SUT never needs to be reachable from AIP, only the other way around

See Two run modes for the full walkthrough.

Q: How repeatable are my assessments, and what do I need to provide to run one?​

A: Yes, your assessments are repeatable — you define your test pipelines as code and keep them under version control, so you can re-run the same assessment reliably at any time. One nuance: if you use LLM-as-a-judge evaluators, you may see some natural variability in results, so not every outcome will be perfectly identical run to run. To run an assessment, you'll just need to provide:

  • Your dataset
  • A connection to your system under test
  • The metrics you want to evaluate against, plus evaluator configuration where relevant

Q: I registered my SUT in the UI but can't run it in external mode — why?​

A: This is expected. An SUT registered through the UI is automatically set to run in hosted mode. External mode is for SUTs that run locally, so registering in the UI and then trying to run it via the SDK in external mode isn't supported and returns an error. If you need external mode, register the SUT for local execution instead of through the UI.

Metrics & Test Configuration​

Q: Do you provide standardized evaluation criteria, or do I need to design my own tests?​

A: You author the criteria; AIP gives you the means to standardize them. A test plan publishes one methodology — metrics with their thresholds and configs, plus the dimensions — under a name, versioned and immutable per version. Projects bind to a version and can be locked to it, so a run cannot quietly deviate from what was agreed, and every result records the plan version that produced it. See Test Plans. Pre-packaged plans that we supply for common use cases are still on our roadmap.

Q: Can I customize or add my own metrics?​

A: Yes, you can define your own metrics alongside the built-in metrics libraries for LLM/RAG and CV tasks. For further standardization, registering these custom metrics for use across your organization will be available as part of the enterprise edition.

Q: Can I change the golden dataset for a SUT?​

A: Yes, the golden dataset attached to a SUT can be swapped or updated, for example as your operational scenarios evolve.

Comparisons & Drift​

Q: How do I detect drift, and can I compare results across time or across different SUTs?​

A: Yes — AIP's run comparison feature lets you compare results over time for the same system under test (e.g. to catch drift, or track regressions and improvements across versions), as well as across different SUTs (e.g. benchmarking alternative models against each other). Today, comparisons are something you trigger yourself rather than something that's surfaced automatically, so we'd recommend building a regular comparison check into your workflow, e.g. after each model update.

How AIP Compares to Other Tools​

Q: How does AIP differ from MLflow-type tools?​

A: AIP is an AI assurance and testing platform built for teams whose AI systems face production or regulatory scrutiny. It gives you:

  • Automated functional evaluation, synthetic data generation, and evidence packaging, so you're not assembling these manually for every release or audit.
  • A complete assurance loop — from defining your operational design domain, to analysing coverage gaps, to running scenario-based testing, to generating an audit-ready report — so you get one continuous workflow instead of stitching tools together.
  • Support for LLM, computer vision, and agentic pipelines under one framework, so you don't need separate tooling for each.

Key benefit: Instead of just being able to say "we ran some tests," AIP lets you show exactly what you tested, against what criteria, and by whom — the level of proof your stakeholders and regulators actually need.

Why generic evaluation tools may leave you exposed for mission-critical AI

If you're operating in a regulated or high-consequence context, out-of-the-box eval tools (Arize, Braintrust, MLflow) can leave gaps that matter:

  • You won't know what your test suite is missing — there's no systematic way to see which conditions, edge cases, or failure modes are underrepresented.
  • You end up testing against whatever data you happen to have, rather than against a clearly defined set of operational conditions.
  • You can't generate adversarial or rare-class scenarios to close coverage gaps — if you haven't seen a failure mode before, you have no way to test for it.
  • You have no built-in way to separate the team building the AI from the team grading it.
  • Your logs are mutable, not the kind of verifiable chain of custody an external party (auditor, regulator, customer) can independently check.

How AIP addresses this

AIP gives you an assurance layer that sits above your existing monitoring tools:

  • ODD (Operational Design Domain) decomposition: you define the full envelope of conditions your AI will face in deployment, then measure your coverage against it systematically.
  • Coverage gap analysis: you get a quantified view of which conditions are underrepresented in your current test set (e.g. "73% of operational conditions covered; here are the 27% gaps").
  • Synthetic scenario generation tied to those gaps: AIP generates test cases for your uncovered conditions automatically, including edge cases and rare-class events, so you're not manually hunting for them.
  • Metrics libraries by task: you get task-specific metrics relevant to your domain and functional requirements out of the box.
  • Custom evaluator framework: you can define your own metrics and thresholds for the domain-specific quality dimensions that generic tools don't encode.
  • Custom test plan builder: you can design your own data and evaluation configs, and codify evaluation best practices to scale consistently across your projects and teams.
  • Chain-of-custody record: you get full lineage of who ran each test, when, against what data, and what result was produced — structured so an external party can independently verify every step.

What this means for you: your AI system doesn't just pass tests — you can prove it passed a complete test suite, covering a defined operational domain, with scenario coverage that an independent party can audit against criteria that were locked in before the first result appeared.