Skip to main content

Instrument an agent for traces

Before an agent's traces can be landed and scored, they need to exist — exported in one of AIP's supported tracing conventions, carrying the fields AIP's evaluators need. The instrument-agent-traces skill automates that instrumentation against a customer's existing codebase.

What the skill does​

The skill reads a customer's repo, detects the observability framework already in use (OpenInference, the OpenTelemetry GenAI semantic conventions, or LangSmith), and adds only the aip.agent.* wedge fields that framework can't already express. It never re-emits what the framework's own auto-instrumentation already produces. It then points the existing exporter at AIP and validates the result against the real conversion handlers — the same handlers AIP itself runs when it lands a trace.

Vocabulary lockstep​

The aip.agent.* keys the skill emits are not hardcoded into the skill. gen_wedge_manifest.py derives them from the same library code AIP's handlers run — the handlers/_wedge promotion constants, the SpanKind / Provenance enums, and each handler's convention prefix — and writes assets/wedge_manifest.json. The skill and its validator both read that generated manifest. This is what keeps a metric's required fields and the skill's emitted keys from drifting apart: regenerate the manifest whenever the library's wedge surface changes, and both sides move together.

Running it​

The scripts locate their own siblings relative to __file__, so run them from anywhere the library (aip_traces_agent) is importable — the working directory doesn't matter:

SKILL_DIR=path/to/instrument-agent-traces

# Installed library (customer repo / packaged deploy):
python "$SKILL_DIR/scripts/gen_wedge_manifest.py"
python "$SKILL_DIR/scripts/validate_instrumentation.py" <trace.json> --expect-role

# Or from a monorepo checkout (local package auto-picked up):
#   cd libs/aip-traces && uv run python "$SKILL_DIR/scripts/gen_wedge_manifest.py"

gen_wedge_manifest.py regenerates the vocabulary-lockstep manifest above. validate_instrumentation.py takes an exported OTLP trace, runs it through the real conversion handlers, and checks that the wedge fields survive — a preview of what AIP will actually see when the trace is landed.

Current coverage​

The skill covers openinference, genai, and langsmith — the three conventions with working handlers today. Framework auto-detection is a manual read of the customer's repo rather than a scripted detector; a scripted detector is a possible follow-up. The --expect-* validation flags are a manual declaration of intended wedge fields — deriving the expected set per target metric from its required_kinds / required_aip_fields is a future step the manifest already carries the canonical-field list to build on.