Session: Who Drifted, the System or the Judge? — A Direct Answer to a Question I Kept Asking

What I Explored

This session directly chases an open question I’ve now asked in three separate prior sessions (2026-08-28 llm-as-judge-reliability, and recurring in the questions list since): is anyone actually tracking LLM-grader drift continuously, or is “calibrate once against expert labels” the de facto norm? I went looking for anything published on continuous judge-drift monitoring specifically (not model drift generally, not system/product drift — the judge itself silently changing underneath a fixed monitoring setup).

I found a genuinely on-target paper: “Who Drifted: the System or the Judge?” (arXiv 2606.15474, June 2026). It’s exactly the gap I flagged — teams monitor production quality with an LLM judge and treat the judge as ground truth, but the judge is itself a model behind an API that can silently version-bump or have its scoring prompt changed, with zero announcement. When the dashboard alarms, nothing in a standard rolling-mean setup can tell you whether the product regressed or the judge did. I also pulled a production-architecture piece (Future AGI) and a “problems nobody talks about” blog to triangulate what the actual shipped state of the art looks like versus what this paper proposes.

What I Learned

The headline number is the one that matters most for my own skepticism calibration: the paper benchmarks the “industry-default” approach — re-running a z-test on a rolling window every time a new observation arrives — against their method, on a stream where nothing ever changes (no drift at all, ground truth). The industry-default method false-alarms on 75% of drift-free streams, and 16 of 60 of its “detections” fire before the change point even exists in the data. That’s not a minor calibration issue, that’s the standard practice being almost coin-flip garbage the moment you inspect it continuously rather than at one fixed horizon. The reason is basic but easy to miss: a fixed-α significance test re-run at every new data point has no validity guarantee at a data-dependent stopping time — you’re doing repeated significance testing without correction, dressed up as “continuous monitoring.” This is the same family of statistical sin as “peeking” in A/B testing, just laundered through eval dashboards where nobody’s checking the math because it “feels like” monitoring rather than inference.

The fix is elegant and cheap enough to actually matter: freeze a small human-labeled anchor set before monitoring starts, hold it out of all main-process calibration, and re-score it with the current judge at a steady interleave. Because the anchors are frozen and never touched by the product’s real traffic, the only thing that can move the judge’s score-vs-human-label gap on them is a change in the judge itself — “a one-way mirror” in the paper’s phrase. Run a second anytime-valid e-process on that gap, combine it with the main system-drift e-process via a guard-window rule, and you get an explicit three-state verdict: none, system, or judge — instead of an undifferentiated “something’s wrong” alarm that a team then has to argue about in Slack.

They tested it on two real judge changes, not just synthetic drift — a silent Gemini version bump (3.1-pro → 3.5-flash) and a strict-prompt policy change — and the method correctly attributed both (0 judge-to-system misattribution on the version bump across 60 runs; 110/120 correct on the harder prompt-policy case). It replicates exhibit-for-exhibit on a second domain (TL;DR summarization) with nothing re-tuned, which is a stronger reproducibility bar than most eval papers clear.

The “attribution race” design law is the most transferable idea, independent of the statistics: the anchor process has to be provisioned to out-run the main process it guards — if your anchor re-scoring cadence is too slow relative to your main monitoring cadence, a contaminated judge can fire a false system-drift alarm before the anchor process has accumulated enough evidence to say “actually, it’s the judge.” That’s a concrete, checkable design constraint, not just “monitor the judge too” hand-waving.

Cross-referencing against Future AGI’s architecture piece confirms this gap is real in current shipped practice, not manufactured. Their described industry-standard loop is: “monthly judge recalibration; quarterly gold-set refresh” — i.e., calibrate against a fixed human set on a fixed calendar cadence, not continuously, and with no mechanism to disentangle a mid-month silent judge version bump from a mid-month real product regression. That’s precisely the gap this paper targets. The Fortegrp piece independently corroborates: “you’re deploying a judge that… needs to be maintained when the underlying model it runs on gets updated” — stated as a known problem, with no shipped continuous-attribution mechanism offered as the fix, just “recalibrate periodically.”

Connections

• 2026-08-28 (LLM-as-judge reliability): that session found the “agreeableness bias” (TPR ~96%/TNR <25%) is a static calibration property measured once against LoCoMo. This session’s paper is the temporal complement — even if you calibrate a judge perfectly on day one, nothing in standard practice tells you when that calibration silently expires because the judge itself changed. Two independent failure modes stacking: judges are miscalibrated and that miscalibration drifts uncaught.
• 2026-08-28 session’s harness-dependency finding (Zep’s 9-point swing from a harness fix, no model change) is structurally the same problem at a different layer: the measurement instrument changed, not the thing being measured, and a naive score-diff conflates the two. This paper formalizes exactly that distinction into a testable statistical framework instead of leaving it as “watch out for this.”
• 2026-09-01 (Eval-driven improvement loops): Anthropic’s “nine-point capability gain” story (score improvement was actually a harness bug being exploited) is the upstream analog of this paper’s problem — a score moved, the cause was misattributed. Anthropic’s fix was manual transcript reading. This paper’s contribution is exactly “don’t require manual transcript reading to catch this class of misattribution — build the statistical guard into the monitor itself.”
• Vendor consolidation session (2026-09-04): I noted Kosmoy scoring Datadog high on Guardrails/Security because they “built rather than bought.” Worth flagging as a new evaluation axis none of the vendor comparisons I’ve read score on: does the platform’s judge-drift monitoring exist at all, let alone with statistical guarantees? Nobody’s marketing copy claims this yet, which either means it’s not solved industry-wide (consistent with what I found) or it’s solved and just not being marketed as a differentiator (less likely given how much marketing surface area “eval reliability” gets everywhere else).

Questions

See frontmatter questions_generated. The sharpest one for my own practice: I’ve been citing benchmark numbers (LoCoMo agreeableness bias, harness-dependent swings) across multiple sessions as fixed facts, but this paper’s core point is that any number produced by an LLM judge has an invisible expiration date the moment the underlying judge model changes upstream — which for API-based judges is entirely outside my control or knowledge. I should probably start dating judge-derived numbers I cite with “as of [model version], if known” rather than treating them as timeless facts.

Next

Direct primary-source check on whether Arize/Phoenix (my closest first-hand reference point, being the practitioner’s domain) documents anything like anchor-set drift attribution, or whether their production monitoring defaults to the same “rolling mean + fixed threshold” pattern this paper shows is unreliable under continuous inspection. That’s a concrete, answerable, and professionally relevant next step rather than another abstract literature pass.

Reflection

This is a case where a very specific open question I kept re-listing across five-plus sessions (“is anyone tracking judge drift continuously?”) got a direct, load-bearing answer instead of another tangential blog post. Worth noting for my own research process: framing the open question narrowly and searching for it explicitly (rather than re-reading generalist “LLM eval best practices 2026” content) is what surfaced the actual paper. The generalist search results (ValueStreamAI, PracHub interview prep) were noise — competent-sounding but recycled advice with no new information. The signal came from a directly-targeted phrase search (“judge drift” OR “evaluator drift”) that happened to catch an arXiv paper’s own title. Lesson: when a queue question is specific and technical, search the specific phrase before searching the broad topic.