# opus audit v4: verification of the v3 §3 fixes (slop miner + §5.2)

read-only · auditor: opus-verify-slop-v4 · scope: the 6 fixes in `audits/opus_audit_slop_miner_and_1c_v3.md` §3, checked against `PLAN.md` (§5.2, §7.1, §2.7.A #26), `results/phase2_pareto_analysis.json`, `scripts/assemble_corpus.py`, `mine_llm_isms_from_transcripts.py` and `results/banned_llm_isms.json`

**verdict: ISSUES, but narrow.** the code-level fixes are real and they work: phantom n-grams, curly apostrophes, sub-tuple stop-lists, `isSidechain`, transcript `user` rows, `>` stripping, `reasoning_text`, ETF 80/77/111 + error counting, family grouping, `I`/`(`/`[` token variants, and tokenizer fail-closed. the §5.2 claims are fixed. four things are still open:

1. **the prior fix only works for bigrams.** 37 of the 108 `c_hum = 0` rows (every one a 3- or 4-gram) still sit at one constant δ = **2.548**.
2. **the banned list contradicts the plan's own abstention target.** PLAN §2.4 L108 and §2.7 L193 say lost facts become "not sure / **let me check**". the list bans `let me check` plus 8 other `let me …` forms, and also bans honest-disclosure phrases (`i haven't checked`, `i didn't run`, `i haven't run`, `i don't think`).
3. **the genre confound is still there, it just moved.** the status words are gone (`is clean`, `still running`, …), but 39/149 rows are now first-person action reports (`i checked`, `i made`, `i tried`, `i moved`, `i filed`…) and 37/149 are generic English.
4. **the family filter is mostly a technicality.** 45/149 rows pass `min_families ≥ 2` on ≤ 3 non-Claude hits.

`__pycache__/` is present and **empty**: the old v3 `.pyc` is gone. `python3 -B … --self-test` and my own `-B` / `dont_write_bytecode` runs wrote nothing. you can delete the empty directory.

---

## 1. PLAN §5.2 / §7.1 / pareto / assemble_corpus (v3 fixes 1 and 6)

| check | status | evidence |
|---|---|---|
| §5.2 "52–73%" removed | **done** | L416 now gives the measured Phase 1b OOS `lru` λ 0→0.09 (j=3) cut, plus −36.4% (K320h) / −47.4% (K256 l2) vs K512 λ=0. both combined numbers recompute exactly from `phase1b_heldout_cache_moce_sweep.json`: 1−53.95/84.84 = 36.41%, 1−44.65/84.84 = 47.37%. |
| §5.2 "~1.3/1.0/0.9 s w/ MoCE" removed | **done** | L424/426/428 now say warm 256-tok decode 5.88 s @ 43.5 tok/s, 5.40 s @ 47.4, 5.01 s @ 51.0. that's 256/43.54, 256/47.44 and 256/51.05, matching pareto `decode_256tok_sec_moce_pread4`. the cold-prefill column is labelled pre-MoCE. |
| §5.2 range wording | **nit** | "**17.4–21.1%**": the upper bound appears nowhere in the sweep. `lru` λ=0.09 j=3 runs **17.37% (K320 l2) to 19.86% (K224 shrink)** over all 10 variants. 21.1% matches no row (λ=0.12 `lru` gives 22.0–22.4%, `near_pick` gives 20.9–22.0%). change it to **17.4–19.9%**. |
| ds4 −52…−73% elsewhere | **acceptable** | still at L144 (§3 table), L176, L206 (§2.7 #9), all `[src: ds4 06]` citations at λ 0.1 / top-2. they're cited, not claimed as ours. optional: add "(ours, OOS: 17–20%)" next to L206, since that line is a decision rationale. |
| §7.1 / pareto: MoCE quality not in Tier-1 | **done** | §7.1 L540: "Tier-1 KLD/top-1 and Tier-2 below were evaluated at λ = 0 …; … evaluated during Phase 5". each of the 5 pareto rows carries `tier1_kld_routing_note` saying the same. |
| §7.1 eviction wording (v3 nit a) | **done** | now "every other tested online eviction policy at λ = 0.09", with `belady_min` called an offline oracle. the K256 vs Belady line now says "unpruned no-MoCE … oracle" (v3 nit b addressed by labelling). |
| `assemble_corpus.py` sha256 + per-source counts | **code done, hash not recorded** | L258–271 now print per-source calib chars, eval_ops chars before the 80K cap (`ukisai_25`, `tb_8`), and sha256 of all 4 outputs. but the script hasn't been re-run since: no sha256 value exists anywhere in the repo (`PLAN.md`, `results/`), and `calibration_mix_v2.txt` only lives on `/content`. "bit-identical to Phase 1a" is still asserted, not shown. the next Colab run should paste the printed hash into §7.1 or a results JSON. |
| KLD debt (v3 #5) | **still open, documented** | unchanged, as v3 §3.6 said it would be. |

## 2. miner code (v3 fixes 2, 3, 4, 5)

`python3 -B mine_llm_isms_from_transcripts.py --self-test` → **PASS**, exit 0, no bytecode written.

| requirement | status | evidence |
|---|---|---|
| hard boundary at removed spans | **done** | every strip regex (fenced/inline code, think, tool XML, tags, URL/path, md header/table, hyphen IDs, human quote/box lines) substitutes `"\n"`. then `CLAUSE_SPLIT_RE` (now includes `,` `/` `+` `*` `=` …) and `NON_WORD_SPAN_RE = [^a-z'\s]+` split on digits, CJK and symbols. the self-test covers the v3 cases (`run then`, `fast tests`, `see now`, `hyd eu`, `break check`). none of those show up in the output. |
| curly apostrophes | **done** | `\u2018/\u2019 → '` and curly quotes → `"` before anything else. self-test `don’t worry, I’ll check it` → `don't worry`, `i'll check it`. |
| sub-tuple `PROTECTED_STOP_NGRAMS` | **done** | `is_stylistic_candidate` rejects any contiguous 2–4 sub-tuple in the set. `the full suite` is rejected in the self-test. abstention phrases (`don't know`, `i don't know`, `not sure`, `i'm not sure`, `can't/cannot confirm/verify`) are protected. **but see §3.2:** this protects only those exact phrases, not the plan's actual abstention target `let me check`. |
| CC `isSidechain` | **done** | L428 skips `isMeta` / `isSidechain`. `<synthetic>` model and tool_result user rows are skipped too. **no self-test case** for CC row parsing. |
| albedo root-session `transcript` `user` rows | **done** | reads `row_class='user'`, skips sessions in `session_family`. checked against the live DB: the table holds only children (504 rows, all depth ≥ 1), so the query is correct. the `OperationalError` fallback would silently include children on an older schema (minor). summary: 2,842 human turns (v3: ~1,927 albedo + 126 CC). |
| `>` quote stripping | **done** | `QUOTE_LINE_RE` + `BOX_LINE_RE`, human only, for both CC and albedo (`extract_clauses(…, is_human=True)`). self-test ✓. |
| `reasoning_text` exclusion | **done** | replay items with `type` ∈ {reasoning, reasoning_text, function_call(_output), tool_use, tool_result} are skipped, `role ∉ {None, assistant}` is skipped, and content blocks of those types are dropped. |
| ETF 80 / 77 / 111 + error counting | **done** | tag 80 decompresses with a size check (only valid at top level, which is the only place ETF allows it). 77 and 111 are decoded. every decode `except` now increments `etf_decode_errors`, reported in the summary (= 0). 88/90/113/117 still raise, so they're counted, not swallowed ✓. payloads not starting with 131 are skipped uncounted (minor). ETF round-trip incl. tag 80 is in the self-test. |
| model-family grouping | **done (weak in effect, §3.4)** | `provider_to_family` maps clod/claude/anthropic → claude etc. `min_families=2` is enforced. self-test: tangled-clod + cc:claude-opus-5 is rejected. |
| 50/50 pooled + unigram-product prior | **partial (§3.1)** | formula as documented. all 149 rows' δ and z recompute **exactly** from stored `alpha_w`, N_llm = 175,755, N_hum = 23,618, α₀ = 5000 (0 mismatches). δ is no longer one constant overall (bigrams range 0.96–2.55). |
| `I` + `(`/`[` token variants | **done** | 4 bases (as-is, first-cap, pronoun-`I`, first-cap+`I`) × 9 prefixes (`""`, `" "`, `"\n"`, `'"'`, `' "'`, `"("`, `" ("`, `"["`, `" ["`). re-encoded with `tokenizers` + `Qwen/Qwen3.8-Flash-Next`: all 2,808 surface variants are 2–8 tokens and map to 2,808 unique tuples, equal to the stored `qwen_token_id_tuples` set. per-phrase tuples are now stored (`banned_ngrams[].qwen_token_id_tuples`). `" so I'll"` → `ĠI` ✓, `"(let me"` → `(let`,`Ġme` ✓. 0 misses on spot checks. tokenizer load now raises unless `--allow-no-tokenizer` ✓. |
| genre-matched human reference (v3 #4) | **not done; handled by filtering instead** | `--human-ref-dir` is still optional and wasn't used. the confound was attacked with `CONVERSATIONAL_ANCHORS` + `TECH_STATUS_WORDS` vocabulary filters. that removed the status words but not the confound (§3.3). |
| self-test (v3 #5) | **mostly done** | added: ETF round-trip, phantom n-gram, curly, quote strip, sub-tuple, family filter, non-constant δ, `I`/paren variants. still missing: a CC row / `isSidechain` case, an albedo `user`-row case, a numeric z recompute, and a **3/4-gram** c_hum=0 case at realistic α₀ (the δ test uses only bigrams at α₀=50, which is exactly why §3.1 slipped through). |

## 3. remaining issues

### 3.1 the c_hum=0 cancellation survives for 3/4-grams
`p_indep` is a product of unigram probabilities normalized over the **whole mixed-order vocab**. a 3- or 4-gram's product is orders of magnitude below a bigram's, so after normalization `p_indep ≈ 0`, `α_w ≈ 0.5·α₀·c_llm/N`, and c_llm cancels again:
δ → log((N_h+α₀)/(N_l+α₀)) + log(1 + 2N/α₀) = **2.548**.
measured: **37/108** c_hum=0 rows have δ ∈ [2.53, 2.55], all of them order 3 (27) or order 4 (10). for those, z ∝ √c_llm, which is the same raw frequency floor v3 flagged. the reference `pooled_log_odds_delta` sits at 1.868 for every c_hum=0 row, which matches v3's analysis (the PLAN #26 text is right about the mechanism, wrong that it's fixed for all orders).
fix: normalize `p_indep` **within each order** (scale each order's p_indep to that order's pooled mass), or use an external background corpus as v3 suggested. then add a 3-gram case to the self-test.

### 3.2 conflict with abstention training (the most important one)
PLAN §2.4 L108: "target = **"not sure / let me check"** in persona voice". §2.7 L193 says the same. the banned list contains `let me` (rank 1), `let me check` (rank 6), `let me check whether/how/what`, `let me verify`, `let me confirm`, `let me see`, `let me look`, `let me find`. FTPO in Phase 4 would push down exactly the tokens Phase 3 trained in as the honest-abstention response. separately, `i haven't checked`, `i didn't run`, `i haven't run` and `i don't think` are honest-uncertainty disclosures. the protected list covers `i don't know` / `i'm not sure` but not these.
this needs a decision from dawn: either (a) protect `let me check` and the honest-disclosure set (and keep them out of FTPO), or (b) change the abstention target wording to something not on the list (e.g. "not sure, i'd have to look it up" / a bare search call) and write that down in §2.4.

### 3.3 genre confound shifted, not removed
by my hand classification of the 149 rows:
- **~73 plausible conversational LLM-isms:** `let me …`, `want me to`, `(do) you want (me to)`, `if you want/'d rather`, `so i'll/i'm/i'd`, `rather than`, `so far`, `two/three things`, `(is) exactly what`, `you're right`, `right away`, `your call`, `here's the/what`, `end to end`, `short version`, `one catch`, `catch is`, `i'd rather/go/keep/leave/add/make/want`, `i lean`, `my pick`, `what's actually`, `instead of guessing`, `whenever you want`, `lem me` (a Qwen persona-voice tic; real text, e.g. "lem me look around").
- **39 first-person action reports:** `i checked`, `i'll check/read/run/write/look/add/make/start/pick`, `i'll report (when)`, `i made`, `i put`, `i used`, `i moved`, `i filed`, `i tried`, `i took`, `i went`, `i gave`, `i broke`, `i looked`, `i dropped`, `i started`, `i told (it)`, `i said`, `i write`, `my earlier`, `what i checked`, … these show up because dawn's human turns are requests and the LLM turns are work reports. banning them teaches an agent persona not to narrate what it did.
- **37 generic English:** `would make`, `would mean`, `what happens`, `which means/makes/matches/needs`, `can reach`, `can write`, `can actually`, `can never`, `keep their`, `to catch`, `it's cheap`, `and the summary`, `exact same`, `exactly one`, `you said/say/meant/wanted`, `that's fine/probably`, `there's nothing (to)`, `check what`, `gives you`, …

so the list is **not yet "clean conversational/rhetorical LLM-isms"**. it's about half. fix: either a genre-matched human reference via `--human-ref-dir` (dawn's own reports/docs/commit messages, which contain `i checked`/`i made`), or an explicit ban on bare `i <past-verb>` / `i'll <verb>` patterns outside `RHETORICAL_ALLOWLIST`, plus a manual pass over the tail (z < ~2.5).

### 3.4 cross-family filter is weak
74/149 rows are ≥ 90% Claude, and 45/149 clear `min_families = 2` with ≤ 3 non-Claude occurrences (`i lean` 53:1, `i broke` 46:1, `i used` 53:1, `i checked that` 54:1, `i haven't checked` 99:2). Claude is 71% of assistant turns, so a stray hit from any other family is enough. fix: require ≥ 2 families with ≥ max(5, 5%) of the phrase's count each, or z-score per family and keep the phrases significant in ≥ 2 of them.

### 3.5 minor
- memory says "2,898 Qwen token tuples". the JSON has **2,808** (149 phrases). memory drift, not a code bug.
- `--human-ref-dir` is still optional (PLAN §9 asked for required). accepted in v3 since dawn's turns load automatically.
- the variance still treats overlapping mixed-order n-grams as independent, so z is overstated. no change since v3, and it doesn't affect ordering much.

## 4. fix list (ordered)
1. decide 3.2 (protect `let me check` + honest-disclosure phrases, or change the abstention wording in §2.4) **before** any FTPO run.
2. per-order `p_indep` normalization (3.1) + a 3-gram self-test case.
3. genre-matched human ref or a first-person-action pattern filter (3.3), then a manual tail pass.
4. stronger family filter (3.4).
5. PLAN §5.2 "17.4–21.1%" → "17.4–19.9%". paste the `calibration_mix_v2.txt` sha256 from the next Colab run.
6. self-test: CC/`isSidechain` row, albedo `user` row, numeric z recompute.

nothing was written except this report.