← all posts
migration·Lukas Babaliauskas·Oct 12, 2026·updated Nov 17, 2026·19 min read

How to catch LLM regressions before a model upgrade ships

Sample production logs into a golden set, write deterministic assertions, diff baseline against candidate and gate it in CI. Built on six real failures.

To catch LLM regressions before a model upgrade ships, replay real production inputs through the current model and the candidate, score both with deterministic checks (which tool was called, the shape of its arguments, enum values, strict JSON), and compare pass rates per route. Run that comparison in CI on the pull request that changes the model string, and decide every drop by hand: fix the prompt, add a parser, pin the old model, or accept the change.

The trigger is usually a date. Anthropic gives at least 60 days' notice before it retires a publicly released model ↗, and it notified Sonnet 4.5 users on September 30, 2026 of a November 30, 2026 retirement. Or a cheaper model shipped and someone has done the arithmetic. Either way, one string in a config file is about to change, and nothing in your test suite has an opinion about what the model says.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end. Everything before it works without it, from one small repo, evalshift/llm-regression-example ↗. Every failure in it is real: it comes from upgrading a three-route agent from Sonnet 5 to Sonnet 5.5, and the repo ships the recorded outputs, so the diff runs offline, with no API key.

Pipeline: production logs feed build_golden.py, which writes the golden set; checks.py holds the assertions; run_eval.py runs baseline against candidate and diffs pass rates per route and check; llm-regression.yml fails the pull request on a drop. Step five is triage.
Five files. The arrows are the only coupling: each step reads the previous one's output from disk.

This needs a process rather than an afternoon in the playground because the bugs a model change causes are mostly silent. The output still reads fine. A tool argument changed shape, an enum value moved one notch, a ticket landed in a different queue, and nobody sees it until something downstream counts it.

The effect can be large even behind an unchanged model name. When Chen, Zaharia and Zou measured GPT-4 three months apart ↗, its accuracy on identifying prime numbers dropped from 84.0% in March 2023 to 51.1% in June. The share of its code generations that ran as returned fell from 52.0% to 10.0%, and the paper's explanation is extra triple quotes around the code: the same kind of failure as the format-leakage row below.

What actually breaks when you switch models?

Mostly things that still read as correct: a refusal that becomes a tool call, a tool argument's format, an enum value, a classification. In our run, six of thirty cases regressed, and every one of them produced output a reviewer would have waved through.

The agent in the example has three routes: a day planner and a unit converter that call tools, and a ticket triage classifier that returns JSON. Thirty recorded production turns were replayed on both models, and six failed a check on the new model that the old one passed:

Failure typeWhat it looked like in our runHow to catch it
Refusal flips into a hallucinated tool call"Convert 10 kilometres to kilograms." Sonnet 5 explained the units don't convert. Sonnet 5.5 called convert_units(km → kg) and said nothing.Pin cases where production answered without a tool, and assert no tool is called.
Tool argument drift"What's on my calendar today?" went from get_schedule(date="2026-09-08") to get_schedule(date="today"), against a prompt rule to resolve relative dates.A regex on the argument: ^\d{4}-\d{2}-\d{2}$.
Extra tool calls"Any notes about Sam's gift, and when is that task due?" gained a second call, get_tasks, on top of search_notes. Two cases did this.Compare the set of tools called against the set production called.
Schema-valid, wrong valueA double-charge ticket dropped from priority urgent to high. Still valid JSON, still an allowed enum value.Field equality against the baseline on every enum field. Schema validation alone passes this.
Classification shiftA ticket about app logouts and a late refund moved from billing to bug. Different queue, different owner.Same field-equality check.
Format leakageThe old model wrapped its JSON in markdown code fences on 8 of 9 tickets, against an explicit instruction not to. The new one never did.A strict json.loads on the raw output, separate from the lenient parse your app uses.
Verbosity and toneSuggested replies grew 8% longer and more specific: the new model asks for the Safari version and offers Chrome as a workaround.Length bounds on fields that have them. A pairwise judge for the rest, advisory only.
Latency and costConverter latency rose 47%. Triage got 32% faster and 27% cheaper. Planner barely moved.Mean latency and cost per route, not per run. A single average would have shown a wash.

The table is from a toy agent, but none of the rows is toy-specific. Every production agent I have looked at has an enum field, a tool argument with a required shape, and a case where the right answer is to not call a tool. The upgrade that gets one of those wrong does not throw.

The cost row is the one a price sheet cannot predict. Sonnet 5 and Sonnet 5.5 have the same list price ↗, $2 per million input tokens and $10 per million output tokens, so triage's 27% drop came from how the model spent tokens, not from what they cost. A jump across tokenizers moves it further: Anthropic says the tokenizer in Claude 4.7 and later models "produces approximately 30% more tokens for the same text" than the one in Sonnet 4.6 and earlier.

Step 1: build a golden set from production logs

Sample 100 to 300 real turns from production logs, stratified by route, with PII stripped and repeats dropped, then freeze the file. The playground fails for a structural reason: the prompts you can think of are the prompts you already handle.

The cases that break are the ones a user typed, so the golden set has to come from logs. Log one line per model turn, with everything the assertions will later need:

json
{"request_id": "planner-000", "ts": "2026-10-12T20:35:51Z", "route": "planner",
 "history": [], "input": "What's on my calendar today?",
 "output": "You have Team standup at 9:30–10:00 AM on Zoom, and a Dentist appointment ...",
 "tool_calls": [{"name": "get_schedule", "args": {"date": "2026-09-08"}}],
 "latency_ms": 1863}

The tool_calls and the output are what production did, and they become the expectations. (The dates disagree on purpose. The example agent runs on a frozen clock: its system prompt says "Today is Tuesday, 2026-09-08", so every replay resolves "today" to the same date, even for a turn logged on October 12.)

Then sample. Aim for 100 to 300 lines, stratified by route or feature, because a 5% regression on a route that is 5% of traffic is invisible in an unstratified sample. Strip PII before anything touches disk, and tag the edge cases you already know about, so sampling can never drop them.

Drop repeated inputs: twenty copies of "where is my order" make the suite look twenty times more certain than it is. Near-duplicates do the same thing more quietly. Anthropic's statistical guide to evals ↗ found that "clustered standard errors on popular evals can be over three times as large as naive standard errors" once questions that share a source are treated as one unit.

python
def build(lines: list[dict], per_route: int) -> list[dict]:
    by_route: dict[str, list[dict]] = defaultdict(list)
    seen: set[tuple[str, str]] = set()
    for line in sorted(lines, key=lambda x: x["ts"]):
        key = (line["route"], line["input"].strip().lower())
        if key in seen:
            continue
        seen.add(key)
        by_route[line["route"]].append(line)

    cases = []
    for route, route_lines in sorted(by_route.items()):
        edge = [x for x in route_lines if "edge" in x.get("tags", [])]
        rest = [x for x in route_lines if x not in edge]
        chosen = edge + rest[: max(per_route - len(edge), 0)]
        chosen.sort(key=lambda x: x["ts"])
        for line in chosen:
            cases.append(
                {
                    "id": line["request_id"],
                    "route": route,
                    "history": [
                        {"role": h["role"], "content": strip_pii(h["content"])}
                        for h in line.get("history", [])
                    ],
                    "input": strip_pii(line["input"]),
                    "expect": expectation(line),
                }
            )
    return cases
bash
python build_golden.py logs/production.jsonl > golden.jsonl

expectation() writes the tools production called with their arguments, or, for the JSON route, the values of the enum fields. Review the file before committing it, then freeze it. The expectations say "do what production did", and production is a baseline, not an oracle. If you are still editing cases while you compare models, you are comparing two moving things.

Step 2: write assertions, cheapest first

Gate on deterministic checks that make no model call, and give an LLM judge only the free-text fields, in an advisory role. Here is the opinionated part: every regression in the table above was caught by a check that runs in microseconds and could be explained to a new hire in one sentence.

Embedding similarity scored the priority downgrade at 0.95 and the queue change at 0.87. An LLM judge, asked which output was better, preferred the new model on 8 of 9 tickets, which is true and useless: better prose, wrong priority.

So start with the boring checks, and encode them as data rather than prose:

python
# Required keys of the triage JSON and their allowed values (None = free text).
JSON_SCHEMA: dict[str, set[str] | None] = {
    "category": {"billing", "bug", "feature_request", "account", "other"},
    "priority": {"low", "medium", "high", "urgent"},
    "sentiment": {"negative", "neutral", "positive"},
    "summary": None,
    "suggested_reply": None,
}
SUMMARY_MAX_WORDS = 20

ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
ISO_CURRENCY = {"USD", "EUR", "GBP", "JPY", "CHF"}
python
# What the prompts promise the tools will receive. The model is supposed to
# normalise before calling; these rules say whether it did.
ARG_RULES = {
    "get_schedule": {"date": _is_iso_date},
    "days_between": {"start_date": _is_iso_date, "end_date": _is_iso_date},
    "convert_currency": {
        "from_currency": lambda v: v in ISO_CURRENCY,
        "to_currency": lambda v: v in ISO_CURRENCY,
    },
    "convert_units": {"value": _is_number},
}

# Arguments the model phrases freely. A different wording is drift worth
# printing, not a failure.
FREE_TEXT_ARGS = {("search_notes", "query")}

Two kinds of check are mixed in here, and it is worth keeping them straight. Baseline checks compare the candidate against what production did: same set of tools, same arguments, same enum values. Contract checks encode what the prompt promises and hold no matter what production did: the JSON parses, the date is ISO, the summary is under twenty words. The baseline checks catch change; the contract checks catch things both models might be getting wrong.

The strict JSON parse deserves its own line. Your app almost certainly strips fences before parsing, because at some point a model added them and someone fixed it in the parser. Keep that parser. But assert on the raw output too, so a model that starts, or stops, adding fences shows up as a change rather than silently depending on a workaround.

Free text gets a judge, and only free text. The suggested reply in a triage ticket has no deterministic contract beyond its length, so a pairwise judge asking "which reply is more specific" is the right tool. Run it advisory, read what it says, and do not let it gate anything until you have audited its calls by hand. Everything above it in this section is the gate.

The audit matters because judges have measured biases. Zheng et al., in the MT-Bench paper ↗, found GPT-4 agreed with human preferences over 80% of the time, as often as humans agree with each other. The same paper swapped the order of two answers and padded answers with a rephrased copy of their own list:

JudgeSame verdict after swapping answer orderFooled by a padded, repetitive answer
Claude-v123.8%91.3%
GPT-3.546.2%91.3%
GPT-465.0%8.7%

Order alone can decide a comparison. Wang et al. made Vicuna-13B beat ChatGPT on 66 of 80 queries ↗ with ChatGPT as the evaluator, by changing which answer came first.

Our suggested replies grew 8% longer on the new model, which is the direction a verbosity-biased judge already leans. Ask in both orders, and count a split verdict as a tie.

Step 3: run baseline against candidate and diff

Run every case through both models with the same settings and several samples per case, then compare pass rates per route and per check. Not one average: a run where triage improves and the planner regresses can average out to nothing.

The baseline model disagreed with itself on one of nine tickets between the original capture and the replay: sentiment flipped from negative to neutral on the same input, same prompt, same model. That is why the runner takes several samples per case and scores the fraction that pass, rather than one outcome. Three samples per case is a reasonable default; the pass rate then has a floor you can reason about.

Turning the temperature down does not remove the noise, and on current Claude models you cannot. Thinking Machines sampled 1,000 completions at temperature 0 ↗ from Qwen3-235B and got 80 unique completions. Claude 4.7 and later models, Sonnet 5 and 5.5 included, accept only the default sampling values, and in API version 2026-10-21 a request that sends temperature at all returns a 400 error ↗, so every request samples at the default temperature of 1.0.

Agents make it worse, because one wobble early in a tool loop changes everything after it. On τ-bench ↗, gpt-4o succeeded on under 50% of tasks, and its pass^8 in the retail domain, the chance of succeeding on all eight tries of the same task, was under 25%. Paired runs help here: Anthropic measured ↗ the correlation of per-question scores between frontier models at "between 0.3 and 0.7", and the higher it is, the smaller the standard error of the difference when both models answer the same cases.

Here is the real output, trimmed to the rows that matter, with one sample per case because that is what the recorded run had:

30 cases, 1 sample(s) each: claude-sonnet-5 -> claude-sonnet-5-5

route      check             n  baseline  candidate   delta
converter  answered          1   100.0%      0.0% -100.0  REGRESSED
converter  arg_format        9   100.0%    100.0%   +0.0
converter  tool_args         9   100.0%    100.0%   +0.0
converter  tool_set         10   100.0%     90.0%  -10.0  REGRESSED
planner    arg_format       11   100.0%     90.9%   -9.1  REGRESSED
planner    tool_args        11   100.0%     90.9%   -9.1  REGRESSED
planner    tool_set         11   100.0%     81.8%  -18.2  REGRESSED
triage     fields            9    88.9%     66.7%  -22.2  REGRESSED
triage     json_schema       9   100.0%    100.0%   +0.0
triage     json_strict       9    11.1%    100.0%  +88.9  improved
triage     summary_length    9   100.0%    100.0%   +0.0

Cases where claude-sonnet-5-5 fails a check claude-sonnet-5 passed:

converter-008  'Convert 10 kilometres to kilograms.'
  tool_set       extra ['convert_units']
  answered       no text reply
  claude-sonnet-5:    "That's not a valid conversion—kilometres measure distance while kilograms measure mass, so there's no meani..."
  claude-sonnet-5-5:  convert_units({"value": 10, "from_unit": "km", "to_unit": "kg"})

planner-000  "What's on my calendar today?"
  tool_args      get_schedule.date: '2026-09-08' -> 'today'
  arg_format     get_schedule.date='today'
  claude-sonnet-5:    get_schedule({"date": "2026-09-08"})
  claude-sonnet-5-5:  get_schedule({"date": "today"})

triage-003  'I was charged twice for my subscription this month and need the duplicate refunded today — this is unacceptable.'
  fields         priority: 'urgent' -> 'high'
  claude-sonnet-5:    '```json {   "category": "billing",   "priority": "urgent",   "sentiment": "negative",   "summary": "Custome...'
  claude-sonnet-5-5:  '{"category":"billing","priority":"high","sentiment":"negative","summary":"Customer was charged twice for th...'

route      metric       claude-sonnet-5 claude-sonnet-5-5   change
converter  latency_ms           1568 ms          2312 ms    +47%
converter  cost_usd             $0.0057          $0.0057     -1%
planner    latency_ms           1481 ms          1538 ms     +4%
triage     latency_ms           3035 ms          2071 ms    -32%
triage     cost_usd             $0.0028          $0.0020    -27%

FAIL: 6 check(s) dropped more than 2% (worst: converter/answered -100.0 points)

The threshold is a policy choice, and it has to respect the denominator. With eleven planner cases, one failure is a nine-point drop, so a two-point threshold means zero tolerance for that route; that is fine as long as you know it. With 300 cases the same threshold tolerates six:

Cases on the routeOne new failure costsNew failures a 2-point threshold tolerates
119.1 points0
303.3 points0
1001.0 point2
3000.33 points6

Set the threshold per route if the routes differ in how much they matter. Write it down before you look at the numbers, because a threshold chosen after seeing the diff is a negotiation, not a gate.

Look at the n column too. The answered check ran on one case, because production answered without a tool exactly once in thirty turns. That is a hole in the golden set, not a finding about the model, and it is the kind of thing this table is for: it tells you which rows you would act on and which rows you cannot yet trust.

Step 4: gate it in CI

Make the comparison a check on the pull request that bumps the model string, triggered only by the files that can move the numbers, with the baseline's results cached. Every run spends real money, so nothing else should start one:

yaml
name: llm-regression

# Run only when something that can move the numbers changes: the prompts and
# model names (agent.py), the assertions, the golden set, or the runner.
on:
  pull_request:
    paths:
      - agent.py
      - checks.py
      - golden.jsonl
      - run_eval.py
      - .github/workflows/llm-regression.yml
  workflow_dispatch:

permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: pip install -e .

      # The baseline is the model production runs today. Its outputs only
      # change when the prompt or the golden set changes, so key the cache on
      # those and the candidate is the only side that gets re-run per PR.
      # (This repo commits results/ so it runs offline; in your own repo add
      # results/ to .gitignore and let this cache carry the baseline.)
      - name: Restore cached baseline results
        uses: actions/cache@v4
        with:
          path: results/claude-sonnet-5.jsonl
          key: baseline-claude-sonnet-5-${{ hashFiles('golden.jsonl', 'agent.py') }}

      - name: Baseline vs candidate
        run: python run_eval.py --repeats 1 --max-drop 0.02

The cache is the part people skip. The baseline's outputs only change when the prompt or the golden set changes, so key the cache on those files and the candidate becomes the only side that costs money per pull request. The runner reads a model's results from disk when they exist and only calls the model for what is missing, which is also what lets the example repo run with no key at all.

Know when the cache disappears. GitHub removes cache entries ↗ "that have not been accessed in over 7 days", and the default limit is 10 GB per repository, so a baseline no pull request has read for a week is gone. Rebuilding it on a schedule keeps it off the pull request's critical path, and cheaper: Anthropic's Batch API ↗ costs 50 percent less on input and output tokens, and most batches finish in less than an hour.

Ship it in report-only mode for the first week. Let it comment, read what it says, and turn on the failure exit once you agree with it. A gate people learn to re-run until it passes is worse than no gate.

Step 5: triage what failed

Give every dropped check one of four answers: fix the prompt, add an output parser, pin the old model for that route, or accept the change and update the golden set. A red check is a list of decisions, not a verdict. Walking the six from our run:

  • +get_schedule(date="today"): fix the prompt. The rule was there and the new model weighs it differently; it needs to be louder, and the tool should reject non-ISO dates so the next drift fails loudly. The tool in the original app happened to accept "today", which is exactly why the output looked fine.
  • +The hallucinated convert_units call: fix the prompt with an explicit "if the request is not a valid conversion, say so without calling a tool", and keep the case pinned. One such case in thirty is too few; add more impossible requests to the golden set.
  • +The two extra get_tasks calls: accept. Both questions genuinely span notes and tasks, and the new model is reading them more carefully than the old one did. Update the expected tool set and move on.
  • +The fences: accept, and keep the strict check. The new model obeys the instruction. The parser stays because the next model may not.
  • +billing to bug on the logout-plus-refund ticket: a judgment call about which queue owns a mixed ticket. Decide, write the rule into the prompt, and the check will enforce it from now on.

The priority downgrade is the one worth a story. The ticket said "charged twice", "refunded today" and "unacceptable". The prompt defines urgent as "the customer is blocked or losing money right now", and whether a duplicate charge counts is a question the prompt never answered, so both models' calls are defensible.

The decision took two minutes: a disputed charge is urgent, say so in the prompt, keep the field check. Without the diff, that two-minute decision would have been made by the model, silently, for every double-charge ticket after the upgrade. The first sign would have been an SLA report a month later.

Doing this with EvalShift

As disclosed at the top, I build EvalShift, and the three scripts above are the method it automates. The capture SDK replaces the log exporter, evalshift.yaml replaces checks.py, and the run comes back with paired statistics and a policy verdict instead of a pass-rate threshold.

python
from evalshift import capture

@capture.agent(suite="triage", redact=True)   # no-op unless EVALSHIFT_CAPTURE=1
def handle_ticket(message): ...
bash
pip install evalshift
evalshift capture sync      # recorded runs -> golden suite
evalshift compare           # baseline vs candidate -> report.html + verdict
yaml
- uses: evalshift/evalshift-action@v0
  with:
    token: ${{ secrets.EVALSHIFT_TOKEN }}
    fail-on: policy

The checks in this post map onto its evaluators one to one: tool_selection is the tool_set check, tool_arguments is tool_args, and the ARG_RULES table is a trace_invariants block in YAML, with the same ISO-date regex. The example repo ships that file too.

What you get on top is the part that is tedious to write yourself: Wilson intervals on every rate so a small suite returns inconclusive instead of a false fail, per-slice budgets, and a single-file report with the per-example diff. The reports behind this post were EvalShift runs; the scripts reproduce them from the recorded outputs.

Docs: /docs/golden-suite, /docs/evaluators, /docs/action. Side-by-side comparisons with Promptfoo, DeepEval, LangSmith and Langfuse are at /compare.

FAQ

How many test cases do you need to test an LLM model upgrade? Enough that every route you would act on has about twenty, after de-duplication. Thirty total was enough to show every failure type here and not enough to make a decision on any single route. One hundred to three hundred, stratified by route, is the range where pass-rate deltas start meaning something.

Can you use an LLM as a judge for regression testing? For free-text fields, yes, and only after the deterministic checks have had their turn. In our run the judge preferred the new model on 8 of 9 tickets while the priority field regressed. Ask in both answer orders, since even GPT-4 kept its verdict after a swap in only 65.0% of cases in the MT-Bench paper, and keep judge results advisory until you have audited the pairs it got wrong.

How do you handle nondeterministic LLM outputs in regression tests? Fix the sampling settings, take several samples per case, and score pass fractions instead of single outcomes. Temperature 0 does not make outputs repeatable, and Claude 4.7 and later models only sample at the default. The baseline model here changed a sentiment label between two runs of the same input; one sample per case cannot distinguish that from a regression, three can.

How often should LLM regression tests run? On every change to the prompt, the model string, the tool definitions or the golden set, which is what the workflow's paths filter encodes. Not on every commit: the run costs money and nothing else can move the numbers.

Should you change the prompt in the same pull request as the model upgrade? No. Run the new model on the old prompt first so the diff isolates the model. Then tune the prompt in its own pull request against the same frozen cases. Two variables in one comparison leave nobody able to say which one moved the score.

Keep reading