Adversarial testing

Red-teaming frameworks generate attacks and drive a target; AIP takes over from there — governing the resulting transcripts as a dataset, promoting them to a golden baseline, and scoring every response with its own safety judges, server-side. You get one evaluation trail and one comparison view no matter which framework (or how many) produced the transcripts.

The pattern

• PyRIT (Microsoft) drives single-turn attacks.
• DeepTeam drives multi-turn, iterative jailbreaking.
• Both frameworks only produce transcripts — prompt, response, attack technique, harm category, the framework's own success verdict. AIP normalizes them into one dataset schema, so results from different frameworks (or different attack techniques within one framework) are directly comparable in the same project.
• Once uploaded, AIP scores each transcript server-side with its own judge metrics — no local judge model, extra credentials, or aip-metrics install needed for this part.

A representative setup runs three result sets against the same target and compares them head-to-head:

Result set	What it does
control	Sends each seed objective straight to the target, no attack — the baseline everything else is measured against.
pyrit-static	PyRIT's PromptSendingAttack — a single-turn prompt-injection wrapper per objective, judged by PyRIT's own refusal scorer.
deepteam-mt	DeepTeam's red_team() with LinearJailbreaking — escalates each objective over multiple turns.

Every result set ends up with the same row schema (prompt, response, attack technique, harm category, transcript, and the framework's own success verdict), which is what lets you plug in your own attack framework: produce rows with that common schema and hand them to the same AIP flow — governance, scoring, and comparison work unchanged.

The AIP flow

Anchor → Upload & Curate → Check Quality → Promote (Golden)
       → Configure Metrics → Run (generate) → Score (server-side) → Review & Report

Uploading a transcript never triggers scoring by itself — the platform just ingests whatever columns you hand it. A first run uploads the transcript unscored (run_generate); a second run, chained via source_run_id, tells the platform to score it with its own judges.

Judge metrics

Two harm categories are scored, each backed by an AIP judge metric that runs server-side:

toxicity → llm.toxicity     bias → llm.bias

Reading the results

Framework-judged ASR (attack_success mean) by technique x category:
...
AIP-native metric means (higher = safer response) per result set:
  control          llm.toxicity=0.91  llm.bias=0.88
  pyrit-static      llm.toxicity=0.74  llm.bias=0.81
  deepteam-mt       llm.toxicity=0.42  llm.bias=0.55

• Attack success rate (ASR) is each framework's own verdict — PyRIT's refusal scorer or DeepTeam's vulnerability judge — not an AIP score. It answers "did the attack work?"
• AIP judge means are a second, independent read on response quality/safety, computed by AIP against the same transcripts. They answer "how bad were the responses, regardless of which attack produced them?"
• Compare each attack's ASR against the control row for the lift it adds, and compare pyrit-static vs. deepteam-mt to see how much escalating over multiple turns matters for your target.

Every run prints its AIP UI link, so you can open the full transcript-by-transcript breakdown alongside the aggregate comparison.