11 kB 53 lines
e68889be
separate persona/ subproject (docs, scripts, data, results, audits) from root engine
dawn
⋯
persona/audits/opus_audit_slop_miner_and_1c_v3.md
sourcerendered
download raw history
at main Markdown

opus audit v3: phase 1c v2 fixes + slop miner #

read-only · auditor: opus-review-slop-miner · scope: audits/opus_audit_phase1c_v2.md §4 fixes, mine_llm_isms_from_transcripts.py + results/banned_llm_isms.json against PLAN §9

verdict: ISSUES. the §7.1 / pareto arithmetic is now clean (0 mismatches). the falsifier still fires on two counts:

  • PLAN §5.2 still contradicts the measured MoCE numbers.
  • the miner has real bugs: phantom n-grams across stripped spans, curly-apostrophe splitting, protected-n-gram leakage, lowercase i in token variants, and contaminated human/assistant streams. its output list is mostly status-report vocabulary and hostnames, not slop.

1. v2 §4 fixes #

# fix status evidence
1 §7.1 item 3: real OOS rows, lru+MoCE, Belady claim done every number recomputes from phase1b_heldout_cache_moce_sweep.json (lru, λ=0.09, j=3): K320h 53.95 / 0.8876 / 32.13–43.54; K320l 56.29 / 0.8827 / 31.30–42.66; K256 44.65 / 0.907 / 35.94–47.44; K224 39.63–51.05; Belady K512 46.13; OOS vs in-sample K512 LRU 0.8232 vs 0.8237 ✓. wording nits: (a) "beats every other eviction policy at every K" holds for online policies at λ=0.09 only. for K320_hybrid only lru and near_pick were run at λ=0.09, and belady_min (an oracle) is lower everywhere. (b) the K256 "beats unpruned Belady" line compares a routing-altering MoCE run against a no-MoCE oracle, so it's not like for like.
2 §7.1 item 2: OOS vs in-sample retention done in-sample 91.76/85.28/81.11 labelled; OOS 87.18/79.71/75.27 (l2_sum) and 86.54/78.16/72.98 (hybrid) match oos_prune_retention.json exactly; "within 0.3%" gone.
3 Tier-2 relabel done (relabel path) all 5×20 rows + top-level rubric_version = v1_run_rescored_ja_trunc_only, plus a provenance_note (2048 cap, seed 42, 240-char preview rescore, ops_1/ops_3 keep v1 verdicts). 7 pruned failures, 4 of them completion_tokens == 2048 truncations (K320h ja_4, K320l ja_3, K224 ja_1, ja_4) ✓. PLAN #25 and §7.1 say the same thing; §7.1 now says Fisher p ≥ 0.23, so the suite is smoke-only. no re-run, so seeds are still n=1.
4 rename warm_256tok_turn_sec_moce_pread4 done now decode_256tok_sec_moce_pread4 = 256 / m5_decode_tok_s_4th_moce (6.72/5.88/6.00/5.40/5.01 ✓). but PLAN §5.2 wasn't touched: L414 says "MoCE … cuts expert misses another 52–73%" and L422–426 say "~1.3 s / ~1.0 s / ~0.9 s w/ MoCE". the measured OOS lru λ=0→0.09 cut is 17.4% (K512 84.84→70.07), 17.4% (K320h 65.32→53.95), 17.8% (K256 54.29→44.65), i.e. 3–4× smaller. those floors have no support. ✗
5 KLD: tokenizer runtime check, K512 self-KLD, --chunks ≥16, re-run not done (documented only) run_phase1c_kld.py L6 still only asserts the patched source text; --chunks 4 (L33, L74); no self-KLD row in phase1c_kld_results.json. PLAN #25 now says ops KLD is "pre-dedupe 4-chunk", which is honest, so this is accepted debt, not a discrepancy.
6 assemble_corpus.py calib path done in code, unverifiable tb_calib_docs uses raw txt[-4500:] (L113) and clean_tb_prompt is only on eval (L119). there's no git history for scripts/ (untracked; repo has only init) and no recorded sha256 of calibration_mix_v2.txt, so "bit-identical to Phase 1a" is asserted, not shown. add the hash. also eval_ops_v2 = (uki[:25] + tb)[:80_000] (L250) can silently truncate the TB docs off the end. log the per-source char counts.

pareto JSON: all 5 rows recompute (cold columns unchanged from v2; lru-MoCE fields above). the duplicate K224 rows are 37.33 vs 37.34 (trivial). caveat worth a line: decode tok/s assumes λ=0.09 MoCE, but Tier-1 KLD/top-1 were measured without MoCE, so the quality cost of MoCE isn't in any column.

2. slop miner vs PLAN §9 #

--self-test: PASS (exit 0). it only covers clause splitting, the empty-baseline raise and first_word_cap. no ETF, CC parsing, z-formula or token-mapping tests, and it wouldn't catch any bug below.

banned_llm_isms.json: all 79 rows recompute exactly (z and δ) from the 4-term formula with N_llm=2,521,023, N_hum=92,944, α₀=5000. sorted ✓, no lipogram cap ✓. all 632 qwen_token_id_tuples reproduce exactly with tokenizers + Qwen/Qwen3.8-Flash-Next (each of the 632 surface variants maps to one tuple).

# requirement status notes
1 CC jsonl done, 1 leak parses 675 asst / 126 human from 16 files (was 0). isSidechain rows aren't skipped, so the subagent prompt in agent-*.jsonl (written by Claude) counts as a human turn.
2 albedo ETF (transcript, submission_events) done, gaps 0 decode errors on all 167,512 other + all submission_events rows. but except Exception: continue swallows failures without counting them, and tags 80 (compressed), 111, 77, 88/90, 113/117 aren't handled. no role/type check on replay items (1 reasoning_text row leaks in as assistant prose). coverage: submission_events gives 1,605 distinct human texts, while transcript row_class='user' rows hold 1,127 more distinct plain human messages (no <mail/<system/notice prefix) that the miner never reads.
3 provider tagging done, weak filter tags are harness:provider. albedo:tangled-clod, albedo:claude, cc:claude-* are all Claude, so min_providers=2 doesn't guarantee cross-model. break check (196 tangled-clod + 4 claude) and private repos (143 claude + 28 tangled-clod) pass on Claude alone. tangled-clod is 59% of LLM n-grams.
4 required human baseline partial raises if N_hum == 0 ✓, but --human-ref-dir is still optional (PLAN asked for required). acceptable since dawn's turns are auto-loaded. contamination: 127/1,927 albedo human turns (and 7 CC ones) contain > quote lines. that's quoted assistant text counted as human, and clean_prose doesn't strip it.
5 Monroe prior + 4-term var formula correct α_w = α₀·(c_l+c_h)/(N_l+N_h), δ and the 4-term variance match the docstring and recompute. consequence: with a pooled background, every phrase with c_hum=0 gets the same δ = log((N_h+α₀)/(N_l+α₀)·(1+N/α₀)) = 3.011 (41/79 rows have c_hum=0, and 40 of them show δ=3.011; let me rounds to 3.012), and z ≈ δ·√α_w. so selection is a raw frequency floor: c_llm ≥ 158 when human=0. "z-scored" adds no ranking information there. use a separate generic background corpus for p_bg, or tune α₀ and the threshold explicitly. note that N counts only filtered stylistic n-grams of mixed orders, which overlap, so the variance is understated.
6 clause split before n-gramming done, but phantom n-grams splits on .!?;: \n —–()[]{}"“” ✓ (commas aren't split). but WORD_RE keeps only [a-z'], and inline code, URLs and paths get replaced by spaces, so words on both sides of a removed span join up: run \make check` then commit→run then commit; K320 is fast, 42 tests pass→fast tests pass; see https://… now please→see now please; CJK/arrows/digits do the same. hyphens split names: hyd-eu-1 → **hyd eu** and break-check→break check`, both in the banned list. the clause-split regex should also fire on the placeholder for removed spans and on digits/non-Latin runs.
6b apostrophes bug curly ’ isn't normalized: don’t worry, I’ll → don worry, i ll, ll fix (52 asst turns, 0 human turns have ’, which biases toward LLM-only junk grams).
6c stop-lists bug PROTECTED_STOP_NGRAMS / allowlist are exact-tuple only, so super-grams leak. ("full","suite") is protected, yet the full suite is banned (z 1.87). REPO_SPECIFIC_WORDS misses hostnames and tool names (hyd, eu, feature matrix, private repos).
7 token-variant mapping mostly done, 1 bug no .capitalize() ✓; keeps 2–8 tokens (max observed 5) ✓; space/newline/quote/first-cap variants ✓. bug: lowercasing is never undone mid-phrase, so so i'll/and i'll/so i'm/so i'd only get Ġi(585), never ĠI(353), and standard-English so I'll is never matched (4 phrases × 8 variants). (lowercase is right for the persona, so add both.) gap: after-punctuation forms are missing. Qwen's pretokenizer merges one leading punct into the word: (let me → ['(let',' me'] (id 21347), which no variant covers. "+cap and */--prefixed forms aren't needed (they tokenize separately). surface↔tuple pairing isn't stored in the JSON. the tokenizer load fails open: on failure it returns [] with no error.
8 drop 1.5%/3.6% cap done no cap anywhere.

output validity: the human side is short imperative requests, the LLM side is agent status reports. that genre confound dominates the list: is clean, still running, is pushed, tests are, build is, is committed, the full suite, feature matrix, wakes me, new test, run is… banning these would teach the persona model not to report status. real LLM-isms in the 79 rows: let me (check|see|look at), want me to, do you want, if you want, so i'll, two things, exactly what, is exactly, so far. treat the list as unusable until a genre-matched human reference (dawn's own prose: docs, posts, commit messages via --human-ref-dir) and the fixes above are in.

3. fix list (ordered) #

  1. PLAN §5.2: replace "52–73%" and the "~1.3/1.0/0.9 s w/ MoCE" floors with the measured ~17–18% miss cut (or mark them [est, contradicted]).
  2. miner: insert a hard boundary at every removed span (code/URL/path/digit/non-Latin run); normalize ’→'; apply protected/repo lists to any sub-tuple; drop isSidechain; check role=='assistant'; strip > quote lines from human turns; also read transcript user rows; count and report decode failures; fail closed on tokenizer load.
  3. miner token variants: add I-cased forms for every i/i'* token, and (/[-merged forms.
  4. miner statistics: genre-matched human reference + an external background for p_bg; group providers by model family for min_providers.
  5. self-test: add ETF round-trip, CC row, z recompute, phantom-n-gram and token-variant cases.
  6. pareto/PLAN: one line that MoCE's quality cost isn't in Tier-1; sha256 of calibration_mix_v2.txt; KLD debt (#5 above) stays open.

side effect of this audit: importing the miner to test it on live data wrote __pycache__/mine_llm_isms_from_transcripts.cpython-311.pyc (01:28), which is safe to delete. nothing else was written except this report.