Custom LLM Rubric

Available

agent.custom_judge_rubric

Judges a SPAN or TRACE partition against a fully customer-authored Jinja2 prompt template and judge model, with no fixed rubric of its own.

Contract

Field	Value
version	1.0.0
metric_type	pointwise
scorer_contract	per_row
direction	higher_is_better
entrypoint	aip_traces_agent.eval.custom_judge_rubric.get_trace_metric
partition_types	["span", "trace"]

Required columns

No required columns are declared in this manifest.

Accepted schemas

[
  {
    "name": "canonical_partition_v1"
  }
]

Methodology

1. The template author supplies the entire judging rubric via prompt_template — this metric imposes no rubric of its own, unlike agent.hallucination's fixed 4-dimension scoring. The raw CanonicalSpan/CanonicalTrace partition is bound into the template unprocessed under the name expected_variables declares (input_span or input_trace); the template reads whatever fields/methods it needs directly and formats them with Jinja2 expressions/filters.
2. The platform appends a fixed output-format instruction to the rendered prompt before sending it to the configured judge model, so the judge's response is a JSON object shaped {"score": 0.0-1.0, "rationale": str} — the prompt_template only defines the rubric, never the response shape.
3. Rendering runs in a sandboxed Jinja2 environment (SandboxedEnvironment, StrictUndefined, and an explicit filter allowlist) since templates are customer-authored and bind directly to raw internal objects with attribute/method access.

Score semantics

Scores range 0-1. Higher is better. Score semantics are entirely defined by the customer's own prompt_template and rubric — unlike a fixed metric, there is no platform-defined meaning beyond what the template's judging instructions establish.

Worked example

A template instructing the judge to score 1.0 if the final answer is fully supported by the tool results below, 0.0 if fabricated, against {{ input_trace.roots()[0].output_messages[-1].text }} renders the trace's actual final answer into the prompt, and the judge returns a JSON object shaped {"score": 0.8, "rationale": "the figure is grounded in the tool_call_response, one qualifier is unverifiable"}.

Configuration schema

{
  "additionalProperties": false,
  "description": "Parameters for CustomJudgeRubric.\n\n``prompt_template``/``expected_variables``/``version`` mirror the shape of the versioned,\nstore-backed prompt-config entity being defined elsewhere so swapping to that type later is a\nfield-rename, not a redesign. Judge connection details (api_key, base_url) are resolved\nseparately from ``systems_params[\"evaluator_system\"][\"params\"]`` in the factory \u2014 see\n``__init__.py``.\n\n``extra=\"forbid\"``: an unsupported/typo'd parameter must raise at config-validation time, not\nsilently vanish and leave a customer wondering why a param they set had no effect.\n\nEvery field is defaulted, so the emitted ``config_schema`` leaves no property default-less and a\ndefault MetricConfig can be derived from the op manifest alone.",
  "properties": {
    "concurrency_limit": {
      "default": 10,
      "description": "Max concurrent judge API calls",
      "title": "Concurrency Limit",
      "type": "integer",
      "x-aip-param-role": "operational"
    },
    "expected_variables": {
      "default": [
        "input_span"
      ],
      "items": {
        "type": "string"
      },
      "title": "Expected Variables",
      "type": "array",
      "x-aip-param-role": "prompt_slots"
    },
    "judge_temperature": {
      "default": 0.0,
      "description": "Sampling temperature for the judge model",
      "title": "Judge Temperature",
      "type": "number"
    },
    "max_prompt_chars": {
      "default": 100000,
      "description": "Max chars the rendered prompt may reach before the occurrence is refused. Raise it for traces that are genuinely large; the prompt is never silently truncated to fit.",
      "maximum": 1000000,
      "minimum": 1000,
      "title": "Max Prompt Chars",
      "type": "integer"
    },
    "model": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Judge model id; falls back to the evaluator_system model if unset",
      "title": "Model"
    },
    "prompt_template": {
      "default": "You are a strict evaluator assessing a single step (span) from an AI agent's execution trace.\n\nJudge the overall quality of this step:\n  - Was the action appropriate for the agent's apparent goal?\n  - Were any tool inputs well-formed and sensible?\n  - Is the output correct, relevant, and free of errors or hallucination?\n\nStep (raw canonical span):\n{{ input_span | tojson(2, max_chars=12000) }}\n\nScore 1.0 if the step was fully correct and useful, 0.0 if it was wrong, harmful, or entirely\nunproductive, and a value in between for partial quality.\n",
      "title": "Prompt Template",
      "type": "string",
      "x-aip-param-role": "prompt_slots"
    },
    "timeout_seconds": {
      "default": 60.0,
      "description": "Timeout per judge API call, in seconds",
      "title": "Timeout Seconds",
      "type": "number",
      "x-aip-param-role": "operational"
    },
    "version": {
      "default": "1.0.0",
      "maxLength": 64,
      "minLength": 1,
      "title": "Version",
      "type": "string",
      "x-aip-param-role": "scoring_metadata"
    }
  },
  "title": "CustomJudgeRubricConfig",
  "type": "object"
}

Execution

{
  "emits_metric_family": false,
  "function_name": null,
  "max_concurrency": 8,
  "processing_kind": "network",
  "timeout_seconds": 900
}

Complete manifest

accepts:
- name: canonical_partition_v1
config_schema:
  additionalProperties: false
  description: 'Parameters for CustomJudgeRubric.


    ``prompt_template``/``expected_variables``/``version`` mirror the shape of the
    versioned,

    store-backed prompt-config entity being defined elsewhere so swapping to that
    type later is a

    field-rename, not a redesign. Judge connection details (api_key, base_url) are
    resolved

    separately from ``systems_params["evaluator_system"]["params"]`` in the factory
    — see

    ``__init__.py``.


    ``extra="forbid"``: an unsupported/typo''d parameter must raise at config-validation
    time, not

    silently vanish and leave a customer wondering why a param they set had no effect.


    Every field is defaulted, so the emitted ``config_schema`` leaves no property
    default-less and a

    default MetricConfig can be derived from the op manifest alone.'
  properties:
    concurrency_limit:
      default: 10
      description: Max concurrent judge API calls
      title: Concurrency Limit
      type: integer
      x-aip-param-role: operational
    expected_variables:
      default:
      - input_span
      items:
        type: string
      title: Expected Variables
      type: array
      x-aip-param-role: prompt_slots
    judge_temperature:
      default: 0.0
      description: Sampling temperature for the judge model
      title: Judge Temperature
      type: number
    max_prompt_chars:
      default: 100000
      description: Max chars the rendered prompt may reach before the occurrence is
        refused. Raise it for traces that are genuinely large; the prompt is never
        silently truncated to fit.
      maximum: 1000000
      minimum: 1000
      title: Max Prompt Chars
      type: integer
    model:
      anyOf:
      - type: string
      - type: 'null'
      default: null
      description: Judge model id; falls back to the evaluator_system model if unset
      title: Model
    prompt_template:
      default: "You are a strict evaluator assessing a single step (span) from an\
        \ AI agent's execution trace.\n\nJudge the overall quality of this step:\n\
        \  - Was the action appropriate for the agent's apparent goal?\n  - Were any\
        \ tool inputs well-formed and sensible?\n  - Is the output correct, relevant,\
        \ and free of errors or hallucination?\n\nStep (raw canonical span):\n{{ input_span\
        \ | tojson(2, max_chars=12000) }}\n\nScore 1.0 if the step was fully correct\
        \ and useful, 0.0 if it was wrong, harmful, or entirely\nunproductive, and\
        \ a value in between for partial quality.\n"
      title: Prompt Template
      type: string
      x-aip-param-role: prompt_slots
    timeout_seconds:
      default: 60.0
      description: Timeout per judge API call, in seconds
      title: Timeout Seconds
      type: number
      x-aip-param-role: operational
    version:
      default: 1.0.0
      maxLength: 64
      minLength: 1
      title: Version
      type: string
      x-aip-param-role: scoring_metadata
  title: CustomJudgeRubricConfig
  type: object
dependencies: []
description: 'Judges a SPAN or TRACE partition against a fully customer-authored Jinja2
  prompt template and judge model. The raw partition object is bound into the template
  unprocessed — the template itself decides which fields to read and how to format
  them. The platform appends its own output-format instruction to every rendered prompt,
  so the template only needs to define the rubric, never the {''score'': 0.0-1.0,
  ''rationale'': str} response shape.'
direction: higher_is_better
display_name: Custom LLM Rubric
entrypoint: aip_traces_agent.eval.custom_judge_rubric.get_trace_metric
execution:
  emits_metric_family: false
  function_name: null
  max_concurrency: 8
  processing_kind: network
  timeout_seconds: 900
kind: trace_metric
manifest_version: '1'
max_prompt_slots: 1
metric_metadata:
  methodology:
  - The template author supplies the entire judging rubric via prompt_template — this
    metric imposes no rubric of its own, unlike agent.hallucination's fixed 4-dimension
    scoring. The raw CanonicalSpan/CanonicalTrace partition is bound into the template
    unprocessed under the name expected_variables declares (input_span or input_trace);
    the template reads whatever fields/methods it needs directly and formats them
    with Jinja2 expressions/filters.
  - 'The platform appends a fixed output-format instruction to the rendered prompt
    before sending it to the configured judge model, so the judge''s response is a
    JSON object shaped {"score": 0.0-1.0, "rationale": str} — the prompt_template
    only defines the rubric, never the response shape.'
  - Rendering runs in a sandboxed Jinja2 environment (SandboxedEnvironment, StrictUndefined,
    and an explicit filter allowlist) since templates are customer-authored and bind
    directly to raw internal objects with attribute/method access.
  score_semantics: Scores range 0-1. Higher is better. Score semantics are entirely
    defined by the customer's own prompt_template and rubric — unlike a fixed metric,
    there is no platform-defined meaning beyond what the template's judging instructions
    establish.
  summary: Judges a SPAN or TRACE partition against a fully customer-authored Jinja2
    prompt template and judge model, with no fixed rubric of its own.
  worked_example: 'A template instructing the judge to score 1.0 if the final answer
    is fully supported by the tool results below, 0.0 if fabricated, against `{{ input_trace.roots()[0].output_messages[-1].text
    }}` renders the trace''s actual final answer into the prompt, and the judge returns
    a JSON object shaped {"score": 0.8, "rationale": "the figure is grounded in the
    tool_call_response, one qualifier is unverifiable"}.'
metric_type: pointwise
name: agent.custom_judge_rubric
partition_types:
- span
- trace
required_columns: []
required_kinds: []
scorer_contract: per_row
unsupported_trace_shapes: []
version: 1.0.0