SSSamarth Sinha

Note

Blind LLM Evaluation

Choose between models, prompts or documents with pre-registered criteria, anonymised inputs, repeated judging and order-swapped head-to-heads.

Oct 2026 · 1 min read

  • Multiple LLM APIs
  • Python
  • Paired statistics
A blind evaluation panel and a replay check before switching models Pre-registration to Anonymise inputs. Anonymise inputs to Judge models × 4. Judge models × 4 to Head-to-heads. Head-to-heads to Paired tests. Recorded decisions to Replay harness. Replay harness to Agreement and cost. Agreement and cost to Routing decision. Pre-registration metrics, decision rule Anonymise inputs names, identifiers Judge models × 4 5 repeats each Head-to-heads both orders Paired tests decide by the rule Recorded decisions production log Replay harness same inputs, new model Agreement and cost per decision Routing decision tier, budget, quota
IllustrativeA blind evaluation panel and a replay check before switching modelsScroll →

Context

A single LLM judgement varies from call to call, prefers whichever option it reads first, and can be swayed by names it recognises.

The problem

Decisions made on one call, or on a criterion chosen after seeing the results, are noise dressed as evidence. Switching to a cheaper model on that basis risks a silent quality drop.

How it works

  1. Pre-register the hypothesis, the metrics and the decision rule before running anything.
  2. Anonymise inputs: names, identifiers and anything a judge might recognise.
  3. Score with several judge models, each repeated (five runs is a sensible floor), because single calls vary.
  4. Run head-to-heads in both orders to cancel position bias.
  5. Use paired tests and decide by the pre-registered rule, recording cost per decision.
  6. Before switching production models, replay recorded decisions through the candidate and measure agreement and cost.

Trade-offs

  • Repeats cost money; the alternative is deciding on noise.
  • Judges share biases; mixing model families helps but does not remove them.
  • Agreement is not correctness: high agreement with an incumbent only shows the candidate behaves the same.

When to use it

Use it whenever a model, prompt or routing change could alter outcomes. For one-off exploration, a quick look is fine; don't ship on it.