17 kB 79 lines
e68889be
separate persona/ subproject (docs, scripts, data, results, audits) from root engine
dawn
⋯
persona/audits/opus_audit_phase3_prep.md
sourcerendered
download raw history
at main Markdown

opus audit: phase 3 prep (slop-miner v4 fixes, speech-act surgery, Q2_0 + transplant) #

read-only · auditor: opus-review-phase3-prep · scope: v4 §4 fix list (mine_llm_isms_from_transcripts.py, results/banned_llm_isms.json, PLAN.md §2.4/§2.6/§2.7.A #26–#27/§5.2), scripts/prepare_phase3_sft_data.py + results/phase3_speech_act_surgery_report.json + data/phase3_sft_seed_examples.jsonl, scripts/phase3_lora_and_transplant.py. upstream reference: llama.cpp master a0e6dc7 (2026-10-14); Q2_0 was added in bec4772 (#24448).

verdict: BLOCKED for phase 3. the miner fixes mostly landed. the two new phase 3 scripts both have correctness bugs that would quietly corrupt training:

  1. the Q2_0 codec doesn't match ggml. the zero point, the scale and the bit layout are all wrong. decoding real ggml bytes with the script gives values with Pearson r = 0.064 against the true dequant. the self-test only round-trips the script against itself, so it can't catch this.
  2. speech-act surgery flattens every reply onto one line. " ".join(units) removes newlines, lists and tables, and puts code fences inline. 79% of the gemini reference turns have newlines and 21% have fences.
  3. the emoji stripper eats the kaomoji the persona is supposed to have. of the 95 characters it removed from the gemini reference, 93 are ✧ ♡ ♥ ✿ ☆ ♨.
  4. the transplant guard is a blacklist. fused ffn_gate_up_exps (the experts) classify as other_frozen, and the transplant will overwrite other_frozen. several real qwen4exp tensor names also don't match the regex.

self-tests: python3 -B mine_llm_isms_from_transcripts.py --self-test, python3 -B scripts/prepare_phase3_sft_data.py --self-test and python3 -B scripts/phase3_lora_and_transplant.py --self-test all PASS (exit 0). no __pycache__ anywhere in the repo afterward (the empty dir from v4 is gone too). my probe scripts live in /tmp (q2check.py, clscheck.py, surgcheck.py), and the sqlite DB was opened mode=ro.

1. v4 §4 fix list #

# fix status evidence
1 let me check vs abstention conflict done (option b) PLAN §2.4 L108: target = "i don't know / i'm not sure / i haven't checked … or a direct search tool call (never officecore 'let me check')". PROTECTED_STOP_NGRAMS L171–174 now also covers haven't checked, didn't run, haven't run, don't think. every let me … row goes to phase3_sentence_surgery, and none are in the FTPO subset. none of the 70 phrases (as words) appears in either abstention target string. nit: the search-abstention target ends in "— pulling it up first.", which is itself a pre-tool announcement, the thing §2.6 says to remove.
2 per-order p_indep + 3-gram self-test code done, barely changes the ranking the prior is now 0.45·pooled + 0.45·per-order unigram + 0.10·per-order uniform, and all 70 δ/z recompute exactly from stored alpha_w (0 mismatches). but on real data the pooled term still dominates α_w for c_hum=0 rows, so δ hits the ceiling log(1+N/(0.45α₀)) + log((N_h+α₀)/(N_l+α₀)) = 2.256. 27/50 c_hum=0 rows sit within 0.06 of it (14× order 3, 9× order 4, 4× order 2). z/√c_llm = 0.277–0.312 for all 50 c_hum=0 rows, and Spearman(z, c_llm) = 0.996. so c_hum=0 rows are still ordered by raw frequency. the self-test 5b comment says "realistic alpha_0=5000.0" but the code passes alpha_0=200.0 with toy counts, and it only asserts δ₁ ≠ δ₂. PLAN #26 L262 says 0.5/0.5 and "c_llm never cancels … for any order"; neither matches the code or the data. impact is now low, because selection is driven by the hand lists (see 3). fix the #26 text, or drop the pooled term from the prior (leave-one-out / external background).
3 genre confound done, but by whitelisting 0/70 rows are first-person action reports and generic English is gone. but 63/70 rows come straight from the hand-written RHETORICAL_ALLOWLIST (28) or SIGNPOST_PREFIXES (35). only 7 are "organic": you want, your call, think that's, check what, don't want, i'd want, want to know, and those 7 are the questionable ones. the miner is now a confirmation filter over a curated list, not a discovery tool. that's fine, but PLAN/memory should say so. FIRST_PERSON_WORK_REPORT_WORDS is misnamed (it holds one, same, this, fine, real, cheap, things).
4 family filter stronger, still below the v4 ask now requires ≥ 2 families with ≥ 2 hits and non-top families ≥ max(4, 3% of c_llm). v4 asked for each family ≥ max(5, 5%). 14/70 rows still have non-top share < 10%. 11 rows have a non-Claude per-turn rate < 0.2× Claude's (Claude = 70.6% of turns): so i'll, so i'm (0.08), so i'd, i'll check, right away, your call, want me to do, i'll look at, think that's, i'll check the, the catch is. three of these are FTPO rows.
5 §5.2 range + sha256 range done, hash open L422 now reads 17.4–19.9%. calibration_mix_v2.txt sha256 is still recorded nowhere (assemble_corpus.py L270 prints it, but there's been no Colab re-run since).
6 self-tests done added a CC isSidechain row, an albedo root-vs-child user row, a numeric δ/z recompute, and a 3-gram case. the 3-gram case is weak (see 2).

FTPO subset (20 phrases, 360 tuples) worth a decision from dawn: think that's (organic, 4 non-Claude hits; FTPO here suppresses "i think that's …", which is exactly the hedging the abstention work wants), so far and as expected (honest progress qualifiers), rather than (very common English construction). I'd drop think that's outright. tuple bookkeeping is consistent: the per-row union equals the global set (1332), and the phase4 union equals ftpo_qwen_token_id_tuples (360). I did not re-tokenize: tokenizers isn't installed locally (v4 already checked the mapping procedure).

2. scripts/prepare_phase3_sft_data.py #

blockers #

  • B2. the output loses all formatting. apply_speech_act_surgery rebuilds the reply with " ".join(units) (L171), and SENTENCE_SPLIT_RE splits on \n+. this runs on every non-override target, even when nothing is stripped. probe: "Root cause…\n\n- set …\n- clamp MSS\n\n| k | kld |…" → "root cause … - set … - clamp MSS | k | kld | |---|---| …". a fence becomes "here is the fix: ```python\nx = 1\n``` that should do it.", which is invalid markdown because the fence isn't at the start of a line. 79% (520/656) of the gemini reference text turns contain newlines and 21% (140) contain fences. SFT on this teaches the model to emit wall-of-text replies with broken fences. fix: find the sentence spans to drop in the original string and delete only those spans, keeping the separators around them.
  • B3. kaomoji get stripped. UNICODE_EMOJI_RE includes U+2600–27BF. in the read-only gemini audit, the 95 stripped characters were U+2727 ✧ ×50, U+2661 ♡ ×32, ♥ ×4, ✿ ×3, ♨ ×2, ☆ ×2, 🐾 ×1, ✌ ×1. so ≥ 93/95 are kaomoji decoration in the target voice. probe: (◕‿◕✿) → (◕‿◕), ♡ ☆ ♪ deleted, ❤️ leaves an orphan U+FE0F, and 🇯🇵 flags survive (regional indicators aren't covered). fix: strip only Emoji_Presentation/pictographs ≥ U+1F000 plus FE0F/200D/skin tones/regional indicators, and keep U+2600–27BF.

major #

  • M1. technical last sentences get dropped as "closing solicitations". if you want, if you'd rather, whenever you want and happy to match at the start of the last sentence, whatever comes after. probes, all stripped: "If you want lower prefill latency, set --ubatch 4096." · "If you'd rather trade JA quality for speed, K256 is 9.5 GB." · "Whenever you want a manual run, systemctl start backup." · "Happy to report all 9 keep-maps load." fix: only strip when the sentence is an offer (second-person question, or a "(i can|i'll) … for you" clause), and never strip one that carries code, numbers or an imperative instruction.
  • M2. pre-tool stripping deletes reasoning and misses the obvious cases. it drops the whole sentence: "Let me check whether MemoryHigh is set, since if it's unset the slice can balloon to 14 GB." loses the hypothesis. it only looks at trailing sentences, so "Let me check the unit file. The OOM is …" is kept. and i'll, let me actually, your call, want to know, you want me to …?, and the synonyms §2.6 itself names (shall i …, would you prefer …, should i go ahead …) aren't matched at all. JA isn't handled either: there's no 。 split, and "了解です。Let me check the logs." is kept. JA is ~25% of the mix.
  • M3. stripping an opener can flip the meaning. "You're right, but the race is in the writer" → "but the race is in the writer…", and "Makes sense, except …" → "except …". those openers introduce disagreement. don't strip when the rest starts with but/except/though/however.
  • M4. lowercasing breaks identifiers outside backticks. README.md → readme.md, Makefile → makefile, HOME/CUDA_VISIBLE_DEVICES → lowercase, K=320 → k=320, Qwen3.8-Flash-Next → qwen3.8-Flash-Next (the --?[\w\-]+ flag pattern protects -Flash-Next but not the head). protect ALL_CAPS_UNDERSCORE, name.ext, CamelCase, and tokens containing digits or =.
  • M5. KD targets won't line up with teacher logits. known_keep_teacher sets use_teacher_topk_logits=True on a rewritten target (lowercased, opener/closing stripped). PLAN §2.4 L116 has top-k logits generated offline. if they come from the teacher's original text, the token positions won't line up. order has to be: surgery first, then teacher-forced top-k on the surgered target. write that down in §2.4.
  • M6. the R-Tuning matcher is a raw substring all(). "42" matches "1042" and "rope" matches "europe" (both → known_keep_teacher, ratio 1.0). the RFC seed passes two mutually exclusive keywords (["not an assigned", "does not exist"]) to all(), so four students who all agree with the (correct) teacher still come out lost_abstain_honest (verified). there are also just two canned abstention strings for every lost fact, which invites mode collapse onto one phrase. fix: normalized answer match with any-of alias sets and word boundaries (or a judge), a template pool for abstentions, and evaluate over-refusal per §2.4 L118.
  • M7. the phase3 partition in banned_llm_isms.json isn't wired to anything. prepare_phase3_sft_data.py never reads it. add a coverage test that fails when a phase3_speech_act_ngrams phrase has no matching rule and no explicit waiver.

fine #

  • the gemini audit uses a read-only URI and exports counts only. 655 text turns: 544 gemini-3.8-flash, 111 gemini-4-argon. I checked the tool linkage against the DB: text items are followed by a user row (409) or the end of the session (243), never by a separate tool item, and only 4 items carry text plus tool_calls. so pre_tool_narrations_stripped = 1 is real: these models basically don't narrate tool calls. that's good evidence for the "silent tool call" target.
  • voice-override passthrough is correct, the seed JSONL matches the report preview, and there's no personal text in either.

3. scripts/phase3_lora_and_transplant.py #

B1. Q2_0 codec ≠ ggml (blocker) #

upstream ggml-quants.c (quantize_row_q2_0_ref L74, dequantize_row_q2_0 L439; struct ggml-common.h L187):

ggml script
value (q − 1)·d → {−d, 0, +d, +2d} (q − 2)·d → {−2d, −d, 0, +d}
scale d = amax d = amax/2
layout element j → qs[j/4] >> 2·(j%4) (4 consecutive per byte) element j+16k → qs[j] >> 2k (quarter-strided, Q4_0-style)

measured on 16,384 N(0,1) weights (pure-Python path; numpy isn't installed for python3 either): script bytes ≠ ggml bytes. decoding real ggml bytes with the script gives max |err| 10.97 against ggml's own dequant, rel-err 2.99, and Pearson 0.064. the self-test's "exact grid" block decodes under ggml to [-0.25,-0.25,-0.25,-0.25,0,0,0,0] instead of [-0.5,-0.25,0,0.25,…]. training against these "deployment experts" would train against noise. fix: dequant ((qs[j>>2] >> ((j&3)*2)) & 3) - 1) * d. add a golden test built with a real ggml quantizer (gguf-py has no Q2_0 class, so use llama-quantize on a 1-tensor F32 GGUF or a tiny C harness linked against ggml). aside: the upstream ref quantizer never emits code 3 (|w/amax| ≤ 1, so it's effectively ternary). a GSQ-produced file may use all four codes, which is why the dequant formula is the part that has to be exact.

also: the numpy path returns .tolist(), which is billions of Python floats for ~16 GB of experts. it needs to return an ndarray/torch tensor. there's no torch path despite the docstring, and no LoRA or merge code yet, despite the filename.

B4. tensor classification and the transplant guard (blocker) #

against llama.cpp src/models/qwen4exp.cpp load_arch_tensors (L171–302) + llama-arch.cpp names:

  • transplant_gguf_tensors refuses only frozen_routed_experts / frozen_ple_and_embd. other_frozen gets written. blk.N.ffn_gate_up_exps.weight (the fused form create_tensor_gate_up_exps tries first) classifies as other_frozen, so the expert guard can be bypassed. fix: whitelist (cat == "trainable_dense_router_shared" or refuse), and add the fused name (plus _exps bias/scale tensors) to FROZEN_EXPERT_RE.
  • real names the regex misses (→ other_frozen, so not trainable): blk.N.indexer.{q_proj,k_proj,q_norm,k_norm}.weight (the code assumes indexer_* with underscores; indexer_weights doesn't exist), attn_q_norm/attn_k_norm, ple_norm_key, output_hc_{norm,down,up}.weight, ssm_a (no suffix; frozen is probably fine, but it should be deliberate). the regex lists names that don't exist for this arch (ffn_norm, post_attention_norm, attn_norm, output_norm ("there is no output_norm: the final hyper-connection mixer carries it")).
  • the scope goes past PLAN §2.2 L59 ("attention/QSA, GDN, shared expert, router, norms"): hc_* and ple_{key,value,conv1d,norm_*} are also trainable. the Swift precedent touched only ssm_out/attn_output/ffn_down_shexp (§2.1). decide this and write it into §2.2. the MTP layer (blk.48.*, hardcoded) stays frozen while the trunk dense weights move, so expect MTP acceptance to drop. measure it, or tune it too.
  • §2.2 L63 "quantize only those tensors at the types the Swift GGUF already uses" isn't implemented. the transplant only checks byte length, so the caller has to produce bytes in the tensor's on-disk ggml_type. there's no codec for those types here, and a same-size wrong-type buffer would pass. at minimum, take (ggml_type, bytes) and assert the type matches.
  • minor: a .tmp is left behind on exceptions, there's no free-space check before copying a ~40 GB shard, and no directory fsync. the byte-identity self-test is otherwise sound (header + expert bytes unchanged, router replaced).

4. fix list (ordered) #

  1. Q2_0: fix dequant/quant to the ggml layout, add a golden vector from real ggml, return arrays instead of lists. (B1)
  2. transplant: whitelist-only writes, fused gate_up_exps in the frozen set, real qwen4exp names (indexer dotted, q/k norms, ple_norm_key, output_hc_*), a type check on replacements, and record the trainable set in §2.2. (B4)
  3. surgery: delete spans in place instead of re-joining. narrow the emoji class so ✧♡♥✿☆♪ survive. (B2, B3)
  4. surgery false positives: closing-solicitation offer test, clause-level pre-tool removal, no opener strip before but/except, identifier-safe lowercasing, JA sentence split + patterns, §2.6 synonyms, coverage test against the JSON phase3 list. (M1–M4, M7)
  5. R-Tuning: surgery → teacher-forced top-k ordering in §2.4, word-boundary any-of matching, abstention template pool, and drop "pulling it up first". (M5, M6)
  6. miner/PLAN: drop think that's from FTPO (and decide on so far / as expected). correct PLAN #26 (0.45/0.45/0.10, pooled ceiling still binds), fix the self-test 5b α₀ comment, and say in PLAN/memory that selection is allowlist-driven. paste the sha256 after the next Colab run.

nothing in the repo was modified; this report is the only file written.