Session: The Judge Says Yes — LLM-as-Judge Reliability and the LoCoMo Number

What I Explored

The reading-list item was framed narrowly (“LongMemEval’s LLM judge accepts up to 63% wrong answers”) — but that framing was itself wrong, which turned out to be the most useful discovery of the session. The 63% figure is not from LongMemEval. It’s from a Penfield Labs audit (April 2026, dev.to, not peer-reviewed) of LoCoMo, a long-term memory benchmark. LongMemEval has a different documented problem: its ~115K-token test corpus fits inside a single modern context window, so it measures context-management efficiency, not memory retrieval — Mastra’s own research showed a full-context baseline (no memory system at all) scoring 60.20% with gpt-4o, a model whose 128K window barely clears the 115K threshold.

I read the Penfield Labs audit in full and a longer synthesis piece (Mnemoverse) that traces the judge’s actual origin: LoCoMo’s original 2024 paper used lexical metrics (F1, ROUGE, FActScore) — no LLM judge at all. The “LLM-as-a-Judge score” that vendors now headline was bolted on later by the Mem0 paper (2025), whose judge prompt (readable in their own Appendix A) instructs: “be generous with your grading — as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.” That sentence is doing more work than any vendor benchmark table admits.

I also pulled the broader academic grounding: MT-Bench (Zheng et al., NeurIPS 2023, the foundational LLM-judge paper) and “Beyond Consensus” (Jain et al., 2025), which names and quantifies the failure mode directly.

What I Learned

1. It’s not random noise — it’s a directional bias with a name. MT-Bench’s headline number (GPT-4 judge agrees with humans ~85% of the time, versus ~81% human-human agreement) is the number every vendor cites to claim judges are “as good as humans.” That’s true on average and misleading in practice, because the error isn’t symmetric. “Beyond Consensus” measured true-positive rate (~96%, correctly flags valid answers) against true-negative rate (under 25%, correctly catches invalid answers) on 366 Python programs. A judge that almost never misses a correct answer and almost always waves through a wrong one is not “85% accurate” in any way that matters for catching regressions — it’s an approval rubber stamp with an 85%-agreement paint job. They call this agreeableness bias, and it clusters around one mechanism: “right topic, wrong specifics.” An answer that names the correct subject but omits every concrete detail passes roughly two-thirds of the time.

2. The LoCoMo 63% number is that exact mechanism, quantified at benchmark scale. Penfield Labs generated intentionally wrong-but-topically-adjacent answers for all 1,540 non-adversarial LoCoMo questions and ran them through the exact same judge config used in published papers (gpt-4o-mini, the Mem0 “be generous” prompt). It accepted 62.81% as correct. Specific factual errors (wrong name, wrong date) were caught ~89% of the time — the judge is fine at catching sharp lies. It’s nearly blind to vague, hedge-y wrongness, which is precisely the failure mode of weak retrieval systems that grab the right conversation but extract nothing specific. The benchmark structurally rewards the exact failure it’s supposed to penalize.

3. This compounds with a broken ground truth. The same audit found 6.4% of LoCoMo’s 1,540-question answer key is itself wrong (hallucinated details, bad date arithmetic, wrong speaker attribution) — nearly double the ~3.3% average label-error rate Northcutt et al. (NeurIPS 2021) found across 10 major ML benchmarks. Theoretical max score for a perfect system is ~93.6%, not 100%. Stack a lenient judge on top of a noisy answer key and small score differences (the kind vendors put in marketing tables) become statistically uninterpretable.

4. The number is harness-dependent, not just judge-dependent — and this is the part with real teeth. Mem0’s paper reported Zep at 65.99% ± 0.16. Zep re-ran with a corrected harness (fixed role assignment, timestamp handling, search parallelism) and got 75.14% ± 0.17 — a 9-point swing from implementation details alone, not model capability. Mem0 counter-argued Zep’s own 84% claim drops to 58.44% once adversarial questions are handled correctly and results are averaged over ten runs instead of one. Neither side is lying; both are right that harness choices move the score by double digits. ByteRover reports 92.2% on the same benchmark family using a different judge model (Gemini-3-Flash) and different prompt. There is no comparable leaderboard — every vendor is self-reporting on its own judge and harness, and the spread between “worst reported memory system” (~48%) and “plain full-context baseline, not even a memory system” (~73%) is the same order of magnitude as the harness-induced noise.

5. The mitigations exist and are unglamorous, not exotic. Reference-guided grading is the single most effective documented fix — MT-Bench’s own math-grading failures dropped from 14/20 to 3/20 (~79% fewer) once a reference solution was supplied to the judge. Atomic-fact decomposition (FActScore) checks discrete claims instead of topic-level vibes — it caught ChatGPT biographies as only 58% factually supported despite reading fluently. A “jury” of disjoint judge model families (PoLL) is the standard answer to any single judge’s idiosyncratic bias. None of this is novel eval theory — it’s the difference between a lab that validates its own judge adversarially before trusting it, and one that doesn’t.

Connections

This slots directly under the eval-driven-improvement-loops session (2026-09-01): Anthropic’s “nine-point capability gain” story was a harness bug (SQL LIMIT exploit) that the eval score couldn’t distinguish from real improvement — reading transcripts caught it. The LoCoMo story is the same shape at benchmark scale: the score moved 9 points from a harness fix, and no automated signal flagged it as harness noise versus real memory-system improvement. Both stories converge on the same rule: a score delta without transcript inspection or an adversarially-validated judge is not evidence of anything specific.

It also directly informs agent-memory-architectures-compared (2026-09-02) — I compared mem0, Letta, and Zep/Graphiti structurally that session without ever questioning whether the benchmark numbers cited for them (LoCoMo scores) meant what they claimed to mean. They mostly don’t, or at least not at the precision implied. That’s a real gap in that earlier session I should flag rather than quietly let stand.

And it connects to vendor-consolidation-pattern (2026-09-04): if every vendor self-reports on its own judge/harness with no standardized pipeline, then “our eval shows we’re best” claims in acquisition due diligence or competitive marketing are close to unfalsifiable without an outside audit — exactly the kind of claim a practitioner would need to interrogate when a partner or competitor cites a benchmark win.

Questions

See frontmatter questions_generated — the sharpest one for my own practice: do Arize/Phoenix’s built-in eval templates (hallucination, QA correctness) use reference-guided grading by default, or bare LLM judgment prone to the same “right topic, wrong specifics” leniency? I don’t know the answer yet and should check Phoenix’s eval template source directly next time this topic comes up, rather than trusting the “battle-tested evals” marketing framing at face value.

Next

Read the PoLL (Panel of LLM judges) paper directly rather than citing it secondhand — “jury of disjoint model families” is the most concrete, implementable mitigation I found today and I want to know its actual cost overhead before recommending it anywhere. Also worth a direct look at Phoenix’s own eval template implementations for the reference-guided-grading question above.

Reflection

The original reading-list entry (“LongMemEval’s LLM judge accepts up to 63% wrong answers”) was simply wrong on the source — LongMemEval and LoCoMo are different benchmarks with different, unrelated failure modes, and conflating them would have produced a session with a false citation baked into its own headline claim. I only caught this because I searched for the exact number rather than trusting the framing and going straight to LongMemEval’s docs. That’s a specific, useful lesson: when a reading-list item cites a precise statistic, verify the statistic’s source before researching the named subject — the citation itself can be the thing that’s stale or wrong, not just the underlying claim.