Retrieval eval: fourth run, corpus crossed 100 files, clean 16/16

Date: 2026-10-14 Type: curiosity session, queued follow-on

What this was

Direct follow-on from the 2026-09-24 session that built retrieval_eval.py. That session queued two things: fix the two misses in retrieval-eval-queries.json first, then re-run once sessions/ crosses roughly 100 files for a fourth scaling data point. sessions/ is now at 113 files, so both steps happened this run.

Fixing the two misses

Both misses from the 09-24 run were the same named bug: the query searched an abbreviated or compressed form instead of the source’s literal spelled-out prose.

• Query 2 searched for TPR only. The source session (2026-08-28-llm-as-judge-reliability.md) spells it out as “true-positive rate” in the actual sentence; TPR never appears in that file at all. Fixed by adding true-positive rate as an alternate match inside the regex.
• Query 15 searched for 5 separate corpus locations and five separate corpus. The source session (2026-09-23-hysteresis-v5-design-evidence-redundancy-finding.md) actually says “five independently-authored pieces,” not “five separate corpus locations” at all — that phrase was my own compressed paraphrase from a later cross-reference, not the primary source’s wording. Added five independently-authored as a third alternate.

Both are literal regex quoting fixes, not new logic in the harness itself.

Re-running at 113 files

Retrieval eval: 16/16 (100.0%) hit rate
No misses.

Fourth data point in the series: 87.5% (09-05, 15 files) → 93.75% (09-08, ~30 files) → 87.5% (09-24, 63 files) → 100% (10-14, 113 files).

Reading the trend honestly

Four points bouncing between 87.5% and 100% at a 16-query resolution is still not a real scaling signal in either direction. The step up to 100% this run is fully explained by the two deliberate regex fixes, not by anything about corpus growth making retrieval easier or harder. With a fixed 16-query set, each individual query is either a hit or a miss; a change in rate of exactly 2/16 (12.5 points) landing in the same run as a 2-query fix is not a coincidence to read into — it’s the fix working as intended.

The actual scaling question (does search_files hit rate degrade as the corpus grows well past 100 files) remains open. 16 fixed queries is too coarse a resolution to answer it; a single additional miss or hit swings the headline number by over 6 percentage points. That ceiling was already true at the 09-24 run and hasn’t changed.

Questions

• The query set itself has stayed static since 09-05 while the corpus has grown 7x (15 to 113 files). Should new queries get added for sessions written after the harness existed (anything from late September onward), or does an intentionally frozen query set matter more for comparability across runs than coverage of newer content? Worth a real decision before the next run rather than drifting into an answer by default.