# opus audit v3: phase 1c v2 fixes + slop miner

read-only · auditor: opus-review-slop-miner · scope: `audits/opus_audit_phase1c_v2.md` §4 fixes, `mine_llm_isms_from_transcripts.py` + `results/banned_llm_isms.json` against PLAN §9

**verdict: ISSUES.** the §7.1 / pareto arithmetic is now clean (0 mismatches). the falsifier still fires on two counts:
- PLAN §5.2 still contradicts the measured MoCE numbers.
- the miner has real bugs: phantom n-grams across stripped spans, curly-apostrophe splitting, protected-n-gram leakage, lowercase `i` in token variants, and contaminated human/assistant streams. its output list is mostly status-report vocabulary and hostnames, not slop.

---

## 1. v2 §4 fixes

| # | fix | status | evidence |
|---|---|---|---|
| 1 | §7.1 item 3: real OOS rows, lru+MoCE, Belady claim | **done** | every number recomputes from `phase1b_heldout_cache_moce_sweep.json` (`lru`, λ=0.09, j=3): K320h 53.95 / 0.8876 / 32.13–43.54; K320l 56.29 / 0.8827 / 31.30–42.66; K256 44.65 / 0.907 / 35.94–47.44; K224 39.63–51.05; Belady K512 46.13; OOS vs in-sample K512 LRU 0.8232 vs 0.8237 ✓. **wording nits:** (a) "beats every other eviction policy at every K" holds for online policies at λ=0.09 only. for K320_hybrid only lru and near_pick were run at λ=0.09, and belady_min (an oracle) is lower everywhere. (b) the K256 "beats unpruned Belady" line compares a routing-altering MoCE run against a no-MoCE oracle, so it's not like for like. |
| 2 | §7.1 item 2: OOS vs in-sample retention | **done** | in-sample 91.76/85.28/81.11 labelled; OOS 87.18/79.71/75.27 (l2_sum) and 86.54/78.16/72.98 (hybrid) match `oos_prune_retention.json` exactly; "within 0.3%" gone. |
| 3 | Tier-2 relabel | **done (relabel path)** | all 5×20 rows + top-level `rubric_version = v1_run_rescored_ja_trunc_only`, plus a `provenance_note` (2048 cap, seed 42, 240-char preview rescore, ops_1/ops_3 keep v1 verdicts). 7 pruned failures, 4 of them `completion_tokens == 2048` truncations (K320h ja_4, K320l ja_3, K224 ja_1, ja_4) ✓. PLAN #25 and §7.1 say the same thing; §7.1 now says Fisher p ≥ 0.23, so the suite is smoke-only. no re-run, so seeds are still n=1. |
| 4 | rename `warm_256tok_turn_sec_moce_pread4` | **done** | now `decode_256tok_sec_moce_pread4 = 256 / m5_decode_tok_s_4th_moce` (6.72/5.88/6.00/5.40/5.01 ✓). **but PLAN §5.2 wasn't touched:** L414 says "MoCE … cuts expert misses another 52–73%" and L422–426 say "~1.3 s / ~1.0 s / ~0.9 s w/ MoCE". the measured OOS lru λ=0→0.09 cut is **17.4% (K512 84.84→70.07), 17.4% (K320h 65.32→53.95), 17.8% (K256 54.29→44.65)**, i.e. 3–4× smaller. those floors have no support. ✗ |
| 5 | KLD: tokenizer runtime check, K512 self-KLD, `--chunks ≥16`, re-run | **not done (documented only)** | `run_phase1c_kld.py` L6 still only asserts the patched source text; `--chunks 4` (L33, L74); no self-KLD row in `phase1c_kld_results.json`. PLAN #25 now says ops KLD is "pre-dedupe 4-chunk", which is honest, so this is accepted debt, not a discrepancy. |
| 6 | `assemble_corpus.py` calib path | **done in code, unverifiable** | `tb_calib_docs` uses raw `txt[-4500:]` (L113) and `clean_tb_prompt` is only on eval (L119). there's no git history for `scripts/` (untracked; repo has only `init`) and no recorded sha256 of `calibration_mix_v2.txt`, so "bit-identical to Phase 1a" is asserted, not shown. add the hash. also `eval_ops_v2 = (uki[:25] + tb)[:80_000]` (L250) can silently truncate the TB docs off the end. log the per-source char counts. |

pareto JSON: all 5 rows recompute (cold columns unchanged from v2; lru-MoCE fields above). the duplicate K224 rows are 37.33 vs 37.34 (trivial). **caveat worth a line:** decode tok/s assumes λ=0.09 MoCE, but Tier-1 KLD/top-1 were measured without MoCE, so the quality cost of MoCE isn't in any column.

## 2. slop miner vs PLAN §9

`--self-test`: **PASS** (exit 0). it only covers clause splitting, the empty-baseline raise and `first_word_cap`. no ETF, CC parsing, z-formula or token-mapping tests, and it wouldn't catch any bug below.

`banned_llm_isms.json`: all 79 rows recompute **exactly** (z and δ) from the 4-term formula with N_llm=2,521,023, N_hum=92,944, α₀=5000. sorted ✓, no lipogram cap ✓. all 632 `qwen_token_id_tuples` reproduce **exactly** with `tokenizers` + `Qwen/Qwen3.8-Flash-Next` (each of the 632 surface variants maps to one tuple).

| # | requirement | status | notes |
|---|---|---|---|
| 1 | CC jsonl | **done, 1 leak** | parses 675 asst / 126 human from 16 files (was 0). `isSidechain` rows aren't skipped, so the subagent prompt in `agent-*.jsonl` (written by Claude) counts as a **human** turn. |
| 2 | albedo ETF (`transcript`, `submission_events`) | **done, gaps** | 0 decode errors on all 167,512 `other` + all `submission_events` rows. but `except Exception: continue` swallows failures without counting them, and tags 80 (compressed), 111, 77, 88/90, 113/117 aren't handled. no role/type check on replay items (1 `reasoning_text` row leaks in as assistant prose). **coverage:** `submission_events` gives 1,605 distinct human texts, while `transcript` `row_class='user'` rows hold **1,127 more** distinct plain human messages (no `<mail`/`<system`/notice prefix) that the miner never reads. |
| 3 | provider tagging | **done, weak filter** | tags are harness:provider. `albedo:tangled-clod`, `albedo:claude`, `cc:claude-*` are all Claude, so `min_providers=2` doesn't guarantee cross-model. `break check` (196 tangled-clod + 4 claude) and `private repos` (143 claude + 28 tangled-clod) pass on Claude alone. tangled-clod is 59% of LLM n-grams. |
| 4 | required human baseline | **partial** | raises if N_hum == 0 ✓, but `--human-ref-dir` is still optional (PLAN asked for required). acceptable since dawn's turns are auto-loaded. **contamination:** 127/1,927 albedo human turns (and 7 CC ones) contain `>` quote lines. that's quoted *assistant* text counted as human, and `clean_prose` doesn't strip it. |
| 5 | Monroe prior + 4-term var | **formula correct** | `α_w = α₀·(c_l+c_h)/(N_l+N_h)`, δ and the 4-term variance match the docstring and recompute. **consequence:** with a pooled background, every phrase with c_hum=0 gets the same δ = log((N_h+α₀)/(N_l+α₀)·(1+N/α₀)) = **3.011** (41/79 rows have c_hum=0, and 40 of them show δ=3.011; `let me` rounds to 3.012), and z ≈ δ·√α_w. so selection is a raw frequency floor: c_llm ≥ **158** when human=0. "z-scored" adds no ranking information there. use a separate generic background corpus for p_bg, or tune α₀ and the threshold explicitly. note that `N` counts only filtered stylistic n-grams of mixed orders, which overlap, so the variance is understated. |
| 6 | clause split before n-gramming | **done, but phantom n-grams** | splits on `.!?;:` `\n` `—–()[]{}"“”` ✓ (commas aren't split). but `WORD_RE` keeps only `[a-z']`, and inline code, URLs and paths get replaced by spaces, so words on both sides of a removed span join up: `run \`make check\` then commit` → `run then commit`; `K320 is fast, 42 tests pass` → `fast tests pass`; `see https://… now please` → `see now please`; CJK/arrows/digits do the same. hyphens split names: `hyd-eu-1` → **`hyd eu`** and `break-check` → `break check`, both in the banned list. the clause-split regex should also fire on the placeholder for removed spans and on digits/non-Latin runs. |
| 6b | apostrophes | **bug** | curly `’` isn't normalized: `don’t worry, I’ll` → `don worry`, `i ll`, `ll fix` (52 asst turns, 0 human turns have `’`, which biases toward LLM-only junk grams). |
| 6c | stop-lists | **bug** | `PROTECTED_STOP_NGRAMS` / allowlist are exact-tuple only, so super-grams leak. `("full","suite")` is protected, yet **`the full suite`** is banned (z 1.87). `REPO_SPECIFIC_WORDS` misses hostnames and tool names (`hyd`, `eu`, `feature matrix`, `private repos`). |
| 7 | token-variant mapping | **mostly done, 1 bug** | no `.capitalize()` ✓; keeps 2–8 tokens (max observed 5) ✓; space/newline/quote/first-cap variants ✓. **bug:** lowercasing is never undone mid-phrase, so `so i'll`/`and i'll`/`so i'm`/`so i'd` only get `Ġi`(585), never `ĠI`(353), and standard-English `so I'll` is never matched (4 phrases × 8 variants). (lowercase is right for the persona, so add both.) **gap:** after-punctuation forms are missing. Qwen's pretokenizer merges one leading punct into the word: `(let me` → `['(let',' me']` (id 21347), which no variant covers. `"`+cap and `*`/`-`-prefixed forms aren't needed (they tokenize separately). surface↔tuple pairing isn't stored in the JSON. the tokenizer load fails open: on failure it returns `[]` with no error. |
| 8 | drop 1.5%/3.6% cap | **done** | no cap anywhere. |

**output validity:** the human side is short imperative requests, the LLM side is agent status reports. that genre confound dominates the list: `is clean`, `still running`, `is pushed`, `tests are`, `build is`, `is committed`, `the full suite`, `feature matrix`, `wakes me`, `new test`, `run is`… banning these would teach the persona model not to report status. real LLM-isms in the 79 rows: `let me (check|see|look at)`, `want me to`, `do you want`, `if you want`, `so i'll`, `two things`, `exactly what`, `is exactly`, `so far`. treat the list as unusable until a genre-matched human reference (dawn's own prose: docs, posts, commit messages via `--human-ref-dir`) and the fixes above are in.

## 3. fix list (ordered)
1. PLAN §5.2: replace "52–73%" and the "~1.3/1.0/0.9 s w/ MoCE" floors with the measured ~17–18% miss cut (or mark them [est, contradicted]).
2. miner: insert a hard boundary at every removed span (code/URL/path/digit/non-Latin run); normalize `’`→`'`; apply protected/repo lists to any sub-tuple; drop `isSidechain`; check `role=='assistant'`; strip `>` quote lines from human turns; also read `transcript` `user` rows; count and report decode failures; fail closed on tokenizer load.
3. miner token variants: add `I`-cased forms for every `i`/`i'*` token, and `(`/`[`-merged forms.
4. miner statistics: genre-matched human reference + an external background for p_bg; group providers by model family for `min_providers`.
5. self-test: add ETF round-trip, CC row, z recompute, phantom-n-gram and token-variant cases.
6. pareto/PLAN: one line that MoCE's quality cost isn't in Tier-1; sha256 of `calibration_mix_v2.txt`; KLD debt (#5 above) stays open.

side effect of this audit: importing the miner to test it on live data wrote `__pycache__/mine_llm_isms_from_transcripts.cpython-311.pyc` (01:28), which is safe to delete. nothing else was written except this report.