glossary

Terms, defined once

Plain definitions of the terms EvalShift uses: golden suite, paired evaluation, LLM judge, regression, capture, verdict, baseline, slice, and more.

Advisory evaluator: An evaluator configured with blocking: false. It still runs and still appears in the report, but its results are summarized separately and never flip the migration verdict.
Agent trace: The ordered record of what an agent did during one run: the model calls it made, the tools it invoked, and the arguments it passed. Evaluators that score output text cannot see any of it.
Baseline: A run pinned as the point of comparison for later runs of the same suite. Diffs and regression checks are measured against it rather than against the previous run.
Bundle: The immutable archive a CLI run uploads: cases, outputs, scores, and metadata in one versioned file. It is the unit the hosted app stores and diffs.
Capture: A file recorded by the capture SDK holding one real agent invocation — its inputs, model calls, and tool calls. Captures are the raw material a golden suite is promoted from.
Evaluator: A rule that scores one output. Evaluators are either deterministic, such as a schema or tool-selection check, or model-based, such as an LLM judge.
Golden suite: A frozen set of representative cases used to judge every model change. Because the cases do not move, a difference between two runs is attributable to the model rather than to the data.
Inconclusive verdict: The result when a suite is too small to settle the question. Rate budgets are Wilson-confidence-interval-aware, so a breach the interval still spans returns inconclusive rather than a failure someone would have overridden anyway.
LLM judge: A model asked to compare two outputs against written criteria. Useful for qualities that resist a deterministic rule, and vulnerable to position, verbosity, and self-preference bias.
Offline mode: Replaying a suite from recorded fixtures instead of calling a live model. It needs no API keys, costs nothing, and produces the same results on every run.
Paired evaluation: Running two models over the same cases with identical inputs so each case yields a matched pair of outputs. Pairing lets deltas be taken per example, so differences in case difficulty cancel instead of swamping the signal.
Pairwise judging: Asking a judge which of two responses is better rather than scoring each alone. A 1–5 rating is anchored on nothing and unstable between calls; a comparison carries its own reference point.
Position bias: A judge's tendency to prefer whichever response it reads first, because the first response sets the frame and the second reads as a revision of it. Randomizing the order converts that systematic shift into noise.
Redaction boundary: The point inside the capture SDK where sensitive values are removed, before anything is written to disk. Redaction applied after a write would not be a boundary.
Regression: A comparison that moved in the wrong direction with enough evidence to say so. Severity follows from the corrected p-value, the effect size, and the direction — not from the raw average.
Run: One execution of a suite against a source and target model, producing scores, a diff against the baseline, and a verdict. Runs are immutable once uploaded.
Severity floor: A per-evaluator setting that stops a regression being classified below a given severity regardless of effect size. It keeps a small-but-serious failure from being filed next to a formatting nit.
Slice: A labelled subset of a suite — by feature, customer, or difficulty — scored separately. Slices resolve worse overall into worse on these cases, flat everywhere else.
Trend: The movement of a suite's scores across successive runs over time. Trends catch slow drift that any single run-to-run diff is too small to show.
Verdict: The decision a run resolves to — pass, conditional_pass, fail, or inconclusive — derived from the migration policy written before the results were seen, rather than from a human reading a table.