14 kB 85 lines
e68889be
separate persona/ subproject (docs, scripts, data, results, audits) from root engine
dawn
⋯
persona/audits/opus_audit_slop_miner_v4.md
sourcerendered
download raw history
at main Markdown

opus audit v4: verification of the v3 §3 fixes (slop miner + §5.2) #

read-only · auditor: opus-verify-slop-v4 · scope: the 6 fixes in audits/opus_audit_slop_miner_and_1c_v3.md §3, checked against PLAN.md (§5.2, §7.1, §2.7.A #26), results/phase2_pareto_analysis.json, scripts/assemble_corpus.py, mine_llm_isms_from_transcripts.py and results/banned_llm_isms.json

verdict: ISSUES, but narrow. the code-level fixes are real and they work: phantom n-grams, curly apostrophes, sub-tuple stop-lists, isSidechain, transcript user rows, > stripping, reasoning_text, ETF 80/77/111 + error counting, family grouping, I/(/[ token variants, and tokenizer fail-closed. the §5.2 claims are fixed. four things are still open:

  1. the prior fix only works for bigrams. 37 of the 108 c_hum = 0 rows (every one a 3- or 4-gram) still sit at one constant δ = 2.548.
  2. the banned list contradicts the plan's own abstention target. PLAN §2.4 L108 and §2.7 L193 say lost facts become "not sure / let me check". the list bans let me check plus 8 other let me … forms, and also bans honest-disclosure phrases (i haven't checked, i didn't run, i haven't run, i don't think).
  3. the genre confound is still there, it just moved. the status words are gone (is clean, still running, …), but 39/149 rows are now first-person action reports (i checked, i made, i tried, i moved, i filed…) and 37/149 are generic English.
  4. the family filter is mostly a technicality. 45/149 rows pass min_families ≥ 2 on ≤ 3 non-Claude hits.

__pycache__/ is present and empty: the old v3 .pyc is gone. python3 -B … --self-test and my own -B / dont_write_bytecode runs wrote nothing. you can delete the empty directory.

1. PLAN §5.2 / §7.1 / pareto / assemble_corpus (v3 fixes 1 and 6) #

check status evidence
§5.2 "52–73%" removed done L416 now gives the measured Phase 1b OOS lru λ 0→0.09 (j=3) cut, plus −36.4% (K320h) / −47.4% (K256 l2) vs K512 λ=0. both combined numbers recompute exactly from phase1b_heldout_cache_moce_sweep.json: 1−53.95/84.84 = 36.41%, 1−44.65/84.84 = 47.37%.
§5.2 "~1.3/1.0/0.9 s w/ MoCE" removed done L424/426/428 now say warm 256-tok decode 5.88 s @ 43.5 tok/s, 5.40 s @ 47.4, 5.01 s @ 51.0. that's 256/43.54, 256/47.44 and 256/51.05, matching pareto decode_256tok_sec_moce_pread4. the cold-prefill column is labelled pre-MoCE.
§5.2 range wording nit "17.4–21.1%": the upper bound appears nowhere in the sweep. lru λ=0.09 j=3 runs 17.37% (K320 l2) to 19.86% (K224 shrink) over all 10 variants. 21.1% matches no row (λ=0.12 lru gives 22.0–22.4%, near_pick gives 20.9–22.0%). change it to 17.4–19.9%.
ds4 −52…−73% elsewhere acceptable still at L144 (§3 table), L176, L206 (§2.7 #9), all [src: ds4 06] citations at λ 0.1 / top-2. they're cited, not claimed as ours. optional: add "(ours, OOS: 17–20%)" next to L206, since that line is a decision rationale.
§7.1 / pareto: MoCE quality not in Tier-1 done §7.1 L540: "Tier-1 KLD/top-1 and Tier-2 below were evaluated at λ = 0 …; … evaluated during Phase 5". each of the 5 pareto rows carries tier1_kld_routing_note saying the same.
§7.1 eviction wording (v3 nit a) done now "every other tested online eviction policy at λ = 0.09", with belady_min called an offline oracle. the K256 vs Belady line now says "unpruned no-MoCE … oracle" (v3 nit b addressed by labelling).
assemble_corpus.py sha256 + per-source counts code done, hash not recorded L258–271 now print per-source calib chars, eval_ops chars before the 80K cap (ukisai_25, tb_8), and sha256 of all 4 outputs. but the script hasn't been re-run since: no sha256 value exists anywhere in the repo (PLAN.md, results/), and calibration_mix_v2.txt only lives on /content. "bit-identical to Phase 1a" is still asserted, not shown. the next Colab run should paste the printed hash into §7.1 or a results JSON.
KLD debt (v3 #5) still open, documented unchanged, as v3 §3.6 said it would be.

2. miner code (v3 fixes 2, 3, 4, 5) #

python3 -B mine_llm_isms_from_transcripts.py --self-test → PASS, exit 0, no bytecode written.

requirement status evidence
hard boundary at removed spans done every strip regex (fenced/inline code, think, tool XML, tags, URL/path, md header/table, hyphen IDs, human quote/box lines) substitutes "\n". then CLAUSE_SPLIT_RE (now includes , / + * = …) and NON_WORD_SPAN_RE = [^a-z'\s]+ split on digits, CJK and symbols. the self-test covers the v3 cases (run then, fast tests, see now, hyd eu, break check). none of those show up in the output.
curly apostrophes done \u2018/\u2019 → ' and curly quotes → " before anything else. self-test don’t worry, I’ll check it → don't worry, i'll check it.
sub-tuple PROTECTED_STOP_NGRAMS done is_stylistic_candidate rejects any contiguous 2–4 sub-tuple in the set. the full suite is rejected in the self-test. abstention phrases (don't know, i don't know, not sure, i'm not sure, can't/cannot confirm/verify) are protected. but see §3.2: this protects only those exact phrases, not the plan's actual abstention target let me check.
CC isSidechain done L428 skips isMeta / isSidechain. <synthetic> model and tool_result user rows are skipped too. no self-test case for CC row parsing.
albedo root-session transcript user rows done reads row_class='user', skips sessions in session_family. checked against the live DB: the table holds only children (504 rows, all depth ≥ 1), so the query is correct. the OperationalError fallback would silently include children on an older schema (minor). summary: 2,842 human turns (v3: ~1,927 albedo + 126 CC).
> quote stripping done QUOTE_LINE_RE + BOX_LINE_RE, human only, for both CC and albedo (extract_clauses(…, is_human=True)). self-test ✓.
reasoning_text exclusion done replay items with type ∈ {reasoning, reasoning_text, function_call(_output), tool_use, tool_result} are skipped, role ∉ {None, assistant} is skipped, and content blocks of those types are dropped.
ETF 80 / 77 / 111 + error counting done tag 80 decompresses with a size check (only valid at top level, which is the only place ETF allows it). 77 and 111 are decoded. every decode except now increments etf_decode_errors, reported in the summary (= 0). 88/90/113/117 still raise, so they're counted, not swallowed ✓. payloads not starting with 131 are skipped uncounted (minor). ETF round-trip incl. tag 80 is in the self-test.
model-family grouping done (weak in effect, §3.4) provider_to_family maps clod/claude/anthropic → claude etc. min_families=2 is enforced. self-test: tangled-clod + cc:claude-opus-5 is rejected.
50/50 pooled + unigram-product prior partial (§3.1) formula as documented. all 149 rows' δ and z recompute exactly from stored alpha_w, N_llm = 175,755, N_hum = 23,618, α₀ = 5000 (0 mismatches). δ is no longer one constant overall (bigrams range 0.96–2.55).
I + (/[ token variants done 4 bases (as-is, first-cap, pronoun-I, first-cap+I) × 9 prefixes ("", " ", "\n", '"', ' "', "(", " (", "[", " ["). re-encoded with tokenizers + Qwen/Qwen3.8-Flash-Next: all 2,808 surface variants are 2–8 tokens and map to 2,808 unique tuples, equal to the stored qwen_token_id_tuples set. per-phrase tuples are now stored (banned_ngrams[].qwen_token_id_tuples). " so I'll" → ĠI ✓, "(let me" → (let,Ġme ✓. 0 misses on spot checks. tokenizer load now raises unless --allow-no-tokenizer ✓.
genre-matched human reference (v3 #4) not done; handled by filtering instead --human-ref-dir is still optional and wasn't used. the confound was attacked with CONVERSATIONAL_ANCHORS + TECH_STATUS_WORDS vocabulary filters. that removed the status words but not the confound (§3.3).
self-test (v3 #5) mostly done added: ETF round-trip, phantom n-gram, curly, quote strip, sub-tuple, family filter, non-constant δ, I/paren variants. still missing: a CC row / isSidechain case, an albedo user-row case, a numeric z recompute, and a 3/4-gram c_hum=0 case at realistic α₀ (the δ test uses only bigrams at α₀=50, which is exactly why §3.1 slipped through).

3. remaining issues #

3.1 the c_hum=0 cancellation survives for 3/4-grams #

p_indep is a product of unigram probabilities normalized over the whole mixed-order vocab. a 3- or 4-gram's product is orders of magnitude below a bigram's, so after normalization p_indep ≈ 0, α_w ≈ 0.5·α₀·c_llm/N, and c_llm cancels again: δ → log((N_h+α₀)/(N_l+α₀)) + log(1 + 2N/α₀) = 2.548. measured: 37/108 c_hum=0 rows have δ ∈ [2.53, 2.55], all of them order 3 (27) or order 4 (10). for those, z ∝ √c_llm, which is the same raw frequency floor v3 flagged. the reference pooled_log_odds_delta sits at 1.868 for every c_hum=0 row, which matches v3's analysis (the PLAN #26 text is right about the mechanism, wrong that it's fixed for all orders). fix: normalize p_indep within each order (scale each order's p_indep to that order's pooled mass), or use an external background corpus as v3 suggested. then add a 3-gram case to the self-test.

3.2 conflict with abstention training (the most important one) #

PLAN §2.4 L108: "target = "not sure / let me check" in persona voice". §2.7 L193 says the same. the banned list contains let me (rank 1), let me check (rank 6), let me check whether/how/what, let me verify, let me confirm, let me see, let me look, let me find. FTPO in Phase 4 would push down exactly the tokens Phase 3 trained in as the honest-abstention response. separately, i haven't checked, i didn't run, i haven't run and i don't think are honest-uncertainty disclosures. the protected list covers i don't know / i'm not sure but not these. this needs a decision from dawn: either (a) protect let me check and the honest-disclosure set (and keep them out of FTPO), or (b) change the abstention target wording to something not on the list (e.g. "not sure, i'd have to look it up" / a bare search call) and write that down in §2.4.

3.3 genre confound shifted, not removed #

by my hand classification of the 149 rows:

  • ~73 plausible conversational LLM-isms: let me …, want me to, (do) you want (me to), if you want/'d rather, so i'll/i'm/i'd, rather than, so far, two/three things, (is) exactly what, you're right, right away, your call, here's the/what, end to end, short version, one catch, catch is, i'd rather/go/keep/leave/add/make/want, i lean, my pick, what's actually, instead of guessing, whenever you want, lem me (a Qwen persona-voice tic; real text, e.g. "lem me look around").
  • 39 first-person action reports: i checked, i'll check/read/run/write/look/add/make/start/pick, i'll report (when), i made, i put, i used, i moved, i filed, i tried, i took, i went, i gave, i broke, i looked, i dropped, i started, i told (it), i said, i write, my earlier, what i checked, … these show up because dawn's human turns are requests and the LLM turns are work reports. banning them teaches an agent persona not to narrate what it did.
  • 37 generic English: would make, would mean, what happens, which means/makes/matches/needs, can reach, can write, can actually, can never, keep their, to catch, it's cheap, and the summary, exact same, exactly one, you said/say/meant/wanted, that's fine/probably, there's nothing (to), check what, gives you, …

so the list is not yet "clean conversational/rhetorical LLM-isms". it's about half. fix: either a genre-matched human reference via --human-ref-dir (dawn's own reports/docs/commit messages, which contain i checked/i made), or an explicit ban on bare i <past-verb> / i'll <verb> patterns outside RHETORICAL_ALLOWLIST, plus a manual pass over the tail (z < ~2.5).

3.4 cross-family filter is weak #

74/149 rows are ≥ 90% Claude, and 45/149 clear min_families = 2 with ≤ 3 non-Claude occurrences (i lean 53:1, i broke 46:1, i used 53:1, i checked that 54:1, i haven't checked 99:2). Claude is 71% of assistant turns, so a stray hit from any other family is enough. fix: require ≥ 2 families with ≥ max(5, 5%) of the phrase's count each, or z-score per family and keep the phrases significant in ≥ 2 of them.

3.5 minor #

  • memory says "2,898 Qwen token tuples". the JSON has 2,808 (149 phrases). memory drift, not a code bug.
  • --human-ref-dir is still optional (PLAN §9 asked for required). accepted in v3 since dawn's turns load automatically.
  • the variance still treats overlapping mixed-order n-grams as independent, so z is overstated. no change since v3, and it doesn't affect ordering much.

4. fix list (ordered) #

  1. decide 3.2 (protect let me check + honest-disclosure phrases, or change the abstention wording in §2.4) before any FTPO run.
  2. per-order p_indep normalization (3.1) + a 3-gram self-test case.
  3. genre-matched human ref or a first-person-action pattern filter (3.3), then a manual tail pass.
  4. stronger family filter (3.4).
  5. PLAN §5.2 "17.4–21.1%" → "17.4–19.9%". paste the calibration_mix_v2.txt sha256 from the next Colab run.
  6. self-test: CC/isSidechain row, albedo user row, numeric z recompute.

nothing was written except this report.