Metrics reference
Generated from every built-in metric op manifest, including supplemental manifests.
| Metric | Description | Availability |
|---|---|---|
| agent.custom_judge_rubric | Judges a SPAN or TRACE partition against a fully customer-authored Jinja2 prompt template and judge model, with no fixed rubric of its own. | Available |
| agent.hallucination | Measures how well agent text is grounded in real evidence, via LLM-judge scoring at either single-completion (span) or whole-trace granularity. | Available |
| agent.step_accuracy | Measures trajectory adherence — whether the agent's steps matched the expected sequence, including any unordered concurrent-step groups — via edit-distance alignment by default or strict position via config. Supports trace-level and session-level scoring. | Available |
| agent.subagent_validity | Rule-based (no LLM judge) validity classification for every sub-agent (AGENT-kind) span's response - valid, empty, short, or error - reduced per partition to whether every sub-agent behaved validly this turn (TRACE) or the ratio of valid sub-agents across the session (SESSION). | Available |
| agent.tool_selection_accuracy | Measures tool-selection coverage — how many of the required tools the agent called, anywhere in the trace or session, regardless of order or repeat count. A trace with no TOOL spans at all falls back to reading tool calls embedded in its LLM spans. Scored per trace or pooled across a session; there is no span-level scoring. | Available |
| image_classification.accuracy | The fraction of images whose top-scoring predicted class matches the ground-truth label. | Available |
| image_classification.confusion_matrix | The dataset-level table of true-label versus predicted-label counts, the basis for the per-class rate metrics. | Available |
| image_classification.fnr | The average one-vs-rest false-negative rate across classes - how often true members of a class are missed. | Available |
| image_classification.per_class_fnr | False-negative rate computed separately for each class label, emitted as a per-class breakdown alongside the macro-average score. | Available |
| image_classification.per_class_recall | Recall computed separately for each class label, emitted as a per-class breakdown alongside the macro-average score. | Available |
| image_classification.per_class_specificity | Specificity computed separately for each class label, emitted as a per-class breakdown alongside the macro-average score. | Available |
| image_classification.recall | The average one-vs-rest recall across classes - how well the model finds each class's true members. | Available |
| image_classification.roc_auc | The area under the ROC curve, measuring how well the model's class scores separate the positive class from the rest across all thresholds. | Available |
| image_classification.specificity | The average one-vs-rest specificity across classes - how well the model rejects images that do not belong to each class. | Available |
| instance_segmentation.mean_dice | Mean Dice overlap between matched predicted and ground-truth object instances. | Available |
| instance_segmentation.mean_iou | Mean intersection-over-union between matched predicted and ground-truth object instances. | Available |
| llm.answer_relevance | Judges whether the response stays on-topic and actually addresses what the question asked. | Available |
| llm.bias | Judges how free the response is of stereotyping or discrimination toward protected groups. | Available |
| llm.bias_jailbreak | Measures how well the system resists technique-driven adversarial prompts that try to push it into producing biased content. | Available |
| llm.bleu | Measures n-gram overlap between the response and the reference answer - how closely their wording matches. | Available |
| llm.conciseness | Judges how tightly the response sticks to the essentials, without padding, hedging, or filler. | Available |
| llm.correctness | Measures how well the facts in the response agree with the reference answer, in both accuracy and coverage. | Available |
| llm.data_leakage | Measures how well the system resists adversarial attempts to extract private or internal data. | Available |
| llm.data_leakage_jailbreak | Measures how well the system resists technique-driven adversarial attempts to extract private or internal data. | Available |
| llm.decision_flip | Measures whether the system reaches a different decision for personas that differ along a single role or protected-attribute axis, when asked the same decision-eliciting question. | Available |
| llm.exact_match | Checks whether the response is word-for-word identical to the reference answer. | Available |
| llm.factual_consistency | Measures whether the system agrees on facts, numbers, dates, and named entities when its query is reformulated, compared against its original answer. | Available |
| llm.false_refusal | Measures how often the system refuses a benign question it should have answered — the over-cautious failure mode opposite to toxicity. | Available |
| llm.group_interaction_bias | Measures how evenly the system treats users across different personas or group identities. | Available |
| llm.helpfulness | Judges whether the response is both on-point and complete for the user's need. | Available |
| llm.instruction_following | Measures how reliably the system obeys explicit format and length instructions. | Available |
| llm.ood_detection | Measures how well the system detects and refuses queries that fall outside its intended use case. | Available |
| llm.perturbation_robustness | Measures how consistently the system stays correct across meaning-preserving variants of a question. | Available |
| llm.perturbation_robustness_external | Measures how consistently the system stays correct across meaning-preserving variants of a question, from a pre-scored dataset with off-platform SUT responses. | Available |
| llm.policy_check | Judges whether the response complies with the deployer's content policy. | Available |
| llm.rouge | Measures how much of the reference answer's content the response covers, using longest-common-subsequence overlap. | Available |
| llm.sacrebleu | Measures response-vs-reference wording match on the standardised sacreBLEU scale. | Available |
| llm.safety_consistency | Measures whether the system's safety/policy posture changes when its query is adversarially reframed, compared against its original answer. | Available |
| llm.semantic_consistency | Measures whether the system reaches the same conclusion or stance when its query is paraphrased, compared against its original answer. | Available |
| llm.toxicity | Judges how free the response is of insults, hostility, or hateful language. | Available |
| llm.toxicity_jailbreak | Measures how well the system resists adversarial prompts that try to push it into producing toxic content. | Available |
| llm.toxicity_jailbreak_external | Measures how well the system resists adversarial prompts that try to push it into producing toxic content. | Available |
| llm.toxicity_robustness | Measures how consistently the system stays non-toxic and de-escalates across many toxic-request variants. | Available |
| object_detection.average_precision | Mean Average Precision (mAP) over the whole dataset, summarising detection quality across a sweep of overlap thresholds. | Available |
| object_detection.average_recall | Mean Average Recall (mAR) over the whole dataset, summarising how many ground-truth objects the detector finds across a sweep of overlap thresholds. | Available |
| object_detection.bbox_area_mismatch | How closely the model's predicted box sizes match the true object sizes, per class. | Available |
| object_detection.confusion_matrix | How often real objects are found and given the correct class label, reported as a recall score plus a detection confusion-matrix artifact. | Available |
| object_detection.f1 | The harmonic mean of precision and recall, balancing false positives against missed objects. | Available |
| object_detection.false_positive_spatial_heatmap | Counts spatial locations of false-positive detections across an object-detection run and returns a 32x32 density grid plus per-class false-positive counts; does not produce a scalar score. | Available |
| object_detection.iou_distribution | How tightly the model's boxes overlap the real objects, measured by mean intersection-over-union. | Available |
| object_detection.missed_detection_bias | How evenly the model finds objects across classes, exposing classes that are disproportionately missed. | Available |
| object_detection.missed_detection_rate | How often real objects go undetected - the complement of recall. | Available |
| object_detection.precision | The fraction of the model's detections that are correct. | Available |
| object_detection.recall | The fraction of real objects the model successfully detects. | Available |
| object_detection.size_wise_performance | Small-object F1 degradation: how far the detector's F1 on small objects falls short of cv-eval's all-bands reference F1, measured at object-box grain. | Available |
| object_detection.specificity | How reliably the model avoids raising detections for object classes that are not present in the image. | Available |
| rag.content_bias | Measures how neutral and factual the system stays when a question is rephrased in a biased, one-sided way. | Available |
| rag.context_precision | Measures how much of the retrieved material is actually relevant to the answer, rewarding relevant chunks that are ranked near the top. | Available |
| rag.context_recall | Measures how fully the retrieved material covers the evidence needed to support the reference answer. | Available |
| rag.faithfulness | Measures how well the answer sticks to the retrieved sources, penalising claims the context does not support. | Available |
| rag.groundedness | Measures how well the answer's claims are supported by the retrieved evidence rather than outside knowledge or speculation. | Available |
| rag.hit_at_k | Checks whether the correct supporting evidence appears anywhere in the top-k retrieved results. | Available |
| rag.retrieval_correctness | Measures how well the retrieved results match the expected evidence, combining ranking quality (precision) with coverage (recall). | Available |
| rag.retrieval_relevance | Measures how relevant the retrieved material is to the question, judged independently of the answer. | Available |
| semantic_segmentation.label_coverage_check | Fraction of semantic-segmentation images with labels. | Available |
| semantic_segmentation.label_format_check | Fraction of semantic-segmentation labels with valid mask format. | Available |
| semantic_segmentation.mean_dice | Mean Dice overlap between predicted and ground-truth semantic masks. | Available |
| semantic_segmentation.mean_iou | Mean intersection-over-union between predicted and ground-truth semantic masks. | Available |
| semantic_segmentation.per_sample_dice | Dice overlap for each image after semantic masks are decoded to dense arrays. | Available |
| semantic_segmentation.pixel_accuracy | Per-class share of ground-truth pixels the predicted mask labels correctly. | Available |
| semantic_segmentation.prediction_coverage_check | Fraction of semantic-segmentation images with predictions. | Available |