16 kB 175 lines
e68889be
separate persona/ subproject (docs, scripts, data, results, audits) from root engine
dawn
⋯
persona/audits/opus_audit_phase3_prep_v2.md
sourcerendered
download raw history
at main Markdown

opus audit v2: phase 3 prep fix verification #

read-only · auditor: opus-verify-phase3-v2 · scope: the 6 fix groups in audits/opus_audit_phase3_prep.md §4. upstream reference: llama.cpp master fetched 2026-10-15 (ggml/src/ggml-quants.c, ggml/src/ggml-common.h, ggml/include/ggml.h, src/models/qwen4exp.cpp, src/llama-arch.cpp, src/llama-model.cpp). probe scripts and fetched sources are in /tmp/v2audit/ (q2probe.py, clsprobe.py, surgprobe.py, repprobe.py, repprobe2.py, up_q2.c). the sqlite DB was opened mode=ro. the only file written in the repo is this report.

verdict: NOT PASS. one blocker left, in the surgery formatting path. the codec, the transplant guard and the miner/PLAN work all check out. surgery still changes the contents of code fences on real data:

  • the final double-space collapse (re.sub(r"(?<=\S)[ \t]{2,}(?=\S)", " ", ...), prepare_phase3_sft_data.py L351) runs over the whole reply, fences included. on the live gemini reference set it rewrites fence contents in 34 of 140 fenced turns (24%), and all 34 are explained by that one regex alone. aligned assignments, print(a, b), padded markdown tables, 4-space indented code blocks and string literals with double spaces all get changed. this is the B2 falsifier ("broken markdown"), just a smaller version of it.

everything else is major or minor (below).

self-tests: python3 -B mine_llm_isms_from_transcripts.py --self-test, python3 -B scripts/prepare_phase3_sft_data.py --self-test and python3 -B scripts/phase3_lora_and_transplant.py --self-test all PASS (exit 0, 0.07 s / 0.07 s / 0.72 s). there's no __pycache__ or *.pyc anywhere in the repo before or after.

1. Q2_0 codec (B1): PASS #

  • the upstream quantize_row_q2_0_ref and dequantize_row_q2_0 are byte-for-byte what the script's docstring says: d = amax, id = 1.0f/d computed from the unrounded float d, q = clamp((int)roundf(w*id)+1, 0, 3), qs[j/4] |= q << ((j%4)*2), and y = ((int)q-1)*d with d = FP16_TO_FP32. block_q2_0 = ggml_half d + uint8_t qs[16] (18 B, QK2_0 = 64), and GGML_TYPE_Q2_0 = 42.
  • independent golden: I pulled the two functions verbatim out of upstream ggml-quants.c (not the script's re-typed copy), compiled them with an fp16 shim (/tmp/v2audit/up_q2.c), and compared:
case quant bytes == C dequant == C
N(0,1) ×16,384 yes yes
heavy-tailed (x³·50) ×4,096 yes yes
exact halfway (±0.5·amax, 0.4999999, 0.5000001, scales 2⁻²⁰…2¹⁰) yes yes
all-zero blocks yes yes
fp16-subnormal scales (1e-6) yes yes
random bytes, all 4 codes, incl. inf/NaN fp16 d (512 blocks) n/a yes (NaN==NaN)

pure-Python path (system python3, no numpy) and the numpy path (uv run --with numpy) both match bit-exactly. the dequant return type is array('f') and np.ndarray (float32) respectively, so it's contiguous. no boxed lists.

  • the double-precision emulation is sound. float/float division and float×float products done in double and then rounded to float give exactly the IEEE float result, and c_roundf reproduces round-half-away.
  • minor:
    • (a) the self-test's "compiled C ggml-quants.c" is a hand-transcribed copy, not the upstream file. it's correct (my verbatim extraction agrees), but PLAN #28 L271 overstates it.
    • (b) the self-test hard-codes /usr/bin/clang and _Float16, so it will fail on a Colab/Linux box without clang. fall back to cc/gcc, or skip with a warning.
    • (c) quantize_q2_0_f32 raises OverflowError for amax > 65504 where C stores inf. that's irrelevant for real weights.
    • (d) the quantizer is pure-Python only. that's fine, since experts are never requantized.

2. tensor classification + transplant (B4): PASS (minor nits) #

  • I built every tensor name qwen4exp.cpp::load_arch_tensors can create (including the create_tensor_qkv / create_tensor_gate_up_exps alternatives from llama-model.cpp, and the nextn.* names in llama-arch.cpp) for blk.0..48 and classified them:
    • trainable (1,779): every trunk name in the PLAN §2.2 L61 list. that's output_hc_{norm,down,up}, attn_{qkv,q,k,v,output,gate,q_norm,k_norm}, dotted indexer.{q_proj,k_proj,q_norm,k_norm}, ssm_{conv1d,beta,alpha,norm,out}, ssm_dt.bias, ple_{key,value,norm_key,norm_query,norm_conv,conv1d}, hc_{attn,ffn}_{norm,down,up,inject}, ffn_gate_inp, ffn_gate_inp_shexp and ffn_{gate,up,down}_shexp.
    • no trainable name contains _exps, embd, or blk.48..
    • frozen_routed_experts: ffn_{gate,up,down,gate_up}_exps (+ .bias/.scale). fused gate_up_exps is now caught.
    • frozen_ple_and_embd: token_embd, output, per_layer_token_embd, ssm_a (bare name, deliberate) and all of blk.48.*.
    • other_frozen: only the optional attn_{q,k,v,qkv}.bias (absent in this arch's GGUFs, and frozen is safe).
  • transplant_gguf_tensors refuses anything with cat != "trainable_dense_router_shared" before copying. the self-test exercises the fused expert, an unknown other_frozen and a wrong-ggml_type tuple, and all three raise ValueError. the header and expert bytes stay identical after a legitimate write. .tmp gets unlinked on error, and there's a free-space check.
  • swift_writers_plus_router scope: only ssm_out, attn_output/attn_out, ffn_down_shexp and the two router gates in blk.0..47.
  • minor:
    • (a) bare bytes replacements are still accepted with no type check. the audit's "at minimum take (ggml_type, bytes)" is only optional. a same-size wrong-type buffer passed as bare bytes still goes through. make the tuple mandatory.
    • (b) the regexes use .match with $, so "blk.4.ssm_out.weight\n" classifies as trainable. GGUF names can't contain newlines, so it's cosmetic, but fullmatch is cleaner.
    • (c) there's no directory fsync after os.replace.
    • (d) attn_out isn't a qwen4exp name (LLM_TENSOR_ATTN_OUT → attn_output). it's harmless in the regex, but PLAN §2.2 L60 calls the family attn_out.weight.
    • (e) PLAN count: in qwen4exp each layer is either full-attention (attn_output) or GDN (ssm_out), via if (!hparams.is_recr(il)). so the writer families give 48 + 48 = 96 tensors on a 48-layer trunk, not "144" (L60). re-check against the SC117 card before quoting 144/240.
    • (f) the trunk range 0..47 is hard-coded, not read from block_count. it fails safe (anything else is frozen).

3. surgery formatting + kaomoji (B2, B3): PARTIAL, blocker remains #

fixed:

  • span deletion is now in place. newlines, bullet/numbered/nested lists, tables and fences survive opener, pre-tool and closing removal.
  • a trailing fence is never touched, and a solicitation after a fence is removed cleanly ("patch:\n```diff…```").
  • the seed JSONL (6 rows) equals both a fresh build_synthetic_phase3_seed_examples() and the report preview.

kaomoji:

  • ✧ ♡ ♥ ✿ ☆ ♪ ★ ♨ ❀, (◕‿◕✿), ٩(◕‿◕｡)۶, (´• ω •)and(｡•̀ᴗ-)✧` all survive.
  • on the live DB only 🐾 ✅ ❌ ⚠ U+FE0F ✨ get stripped (6 chars, 0 kaomoji). the report's count of 1 differs only because the DB grew by one argon turn (656 → 657 text turns, 27,042 → 27,046 silent tool calls).

still broken:

  • B2' (blocker): fence/table/indent whitespace collapse (L351, described above). measured: 34/140 fenced real turns changed. probes:

    • x = 1 → x = 1, print(a, b) → print(a, b) inside a ```python fence
    • | name | kld | → | name | kld |
    • make -j8 all → make -j8 all

    fix: run the collapse only on non-fence, non-inline-code segments, and ideally only around the positions where emoji were removed.

  • B3' (major, low frequency): the emoji-adjacent punctuation cleanup re.sub(r"[ \t]+([.!?,;:])", r"\1", part) (L325) runs on the whole non-fence part whenever it contains any emoji:

    • kaomoji: ( ; ω ; ) → (; ω; )
    • inline code: `a ; b` → `a; b`

    only the space right where the emoji was should be removed. it also leaves a trailing space ("fixed it 🚀\n" → "fixed it \n").

  • minor: BMP Emoji_Presentation characters outside U+2600–27BF survive (⭐ U+2B50, ⏰ U+23F0, ⌛).

4. surgery false positives (M1–M4, M7): MOSTLY FIXED #

fixed:

  • the 4 original M1 probes, and "If you want lower latency, use the smaller model." / "If you want the details, the root cause is …", are all kept.
  • M3:
    • "You're right, but…", "Makes sense, except…", "Exactly right, though…" and "Great question — however…" are kept.
    • "Absolutely not, …" and "You're right that …" are kept.
    • "Good catch! …" is stripped.
  • M2:
    • "Let me check X, since Y" → "if it's unset the slice can balloon to 14 GB."
    • "Let me check the unit file. The OOM is …" → "the OOM is …".
    • a trailing pre-tool sentence after an explanation is stripped.
    • 了解です。まずはログを確認します。 → "" and 原因はスワップです。ログを確認します。 → 原因はスワップです。.
  • M4: README.md, Makefile, HOME, CUDA_VISIBLE_DEVICES, K=320, Qwen3.8-Flash-Next, MemoryHigh, GitHub, PyTorch, Llama.cpp, ggml_type, x86_64, OOMPolicy=kill, WireGuard, JSON and URLs/POSIX paths are all preserved.
  • M7: the self-test iterates all 25 phase3_speech_act_ngrams (15 pre-tool, 9 closing, 1 opener) and asserts the right counter is 1 for each, so coverage is 100%.

remaining (major):

  • M1' informational / genuine-question tails still stripped (with a tool call absent):

    • "If you want determinism, I can confirm the seed is fixed at the top of main." (i can matches any statement)
    • "If you want it faster, I can't help much; the bottleneck is the disk." (\bi\s+can\b matches I can't)
    • "Should I write the parser in Rust or Go?" (genuine design fork; GENUINE_TECH_TRADEOFF_RE only knows project jargon)
    • "Would you like to keep the old schema or migrate everything now?" (an either/or decision the user owns)

    fixes:

    • require can(?!'t|not)
    • require the offer verb to act for the user (i can (apply|run|push|do|fix|write|…))
    • exempt X or Y? alternatives

    TECHNICAL_TOKEN_IN_SENT_RE (L132) is defined but never used. wiring it into the closing guard was part of the M1 fix.

  • M2' hypotheses still lost when there's no comma + since/because/as:

    • "Let me check the unit because the OOM killer usually targets the biggest cgroup." → ""
    • "I'll test this hypothesis: the fd leaks in the retry loop." → ""
    • "I'll run the tests first, then we can talk about the race." → ""
    • "Let me see if I understand: you want the cache cleared on boot?" → "" (a clarification question, deleted when a tool call follows)

    PRE_TOOL_NARRATION_SENT_RE ends in [^\n]*, so any sentence that starts with a pre-tool verb is deleted whole. fix: allow ,?\s*(since|because|as|so|:), and refuse to delete sentences containing ? or : followed by a clause.

minor:

  • PPPoE, iOS, macOS get lowercased (pppoe, ios, macos). the seed row 2 target contains pppoe.
  • Windows paths get lowercased.
  • the coverage test silently skips if the JSON is missing, has no waiver list, and has no else: fail for an unknown category.
  • module docstring L18–20 of the miner is unrelated. see §6.

5. R-Tuning (M5, M6): FIXED, two issues #

fixed:

  • word-boundary matching: 42≠1042, rope≠europe, 3.14≠3.141, 8080≠18080.
  • the RFC any-of group gives known_keep_teacher.
  • a template pool of 6.
  • a silent web_search (target_response == "", target_tool_call set).
  • kd_logit_extraction_order = "extract_teacher_topk_logits_after_surgery_on_target_response", and PLAN §2.4 L118 now mandates surgery → teacher-forced top-k.
  • "pulling it up first" is gone from the code and the targets.

remaining:

  • (major) ABSTENTION_TEMPLATES[0] ends in "want me to look it up or run a check?". want me to is literally a phase3_speech_act_ngrams closing-solicitation phrase. every lost fact whose prompt hashes to slot 0 (≈1/6) trains the exact servile closing that surgery removes everywhere else. surgery doesn't catch it, because there's no sentence boundary before it. drop the question, or use the silent-search target instead.
  • (minor) default-mode alias semantics depend on spaces. a flat list of single-token aliases is treated as all-of: ["42", "forty-two"] with every answer "42" → lost_abstain_honest. a flat list with any multi-word string is treated as any-of. that's a silent trap for whoever writes probe sets. make flat lists always any-of (or require explicit groups), and document it.
  • (nit) the helper is literally named _word_boundary_おそらく (L389/L404, "probably"), a pointless wrapper around _word_boundary_match. rename or inline it.

6. miner + JSON + PLAN: PASS #

  • prior: p_bg = 0.85·p_indep + 0.15·p_unif (L783), per-order normalized, with no pooled term in alpha_w (pooled_* fields are reference-only). summary.prior_type = "0.85_per_order_unigram_plus_0.15_per_order_uniform".
  • recompute: all 32 rows' log_odds_delta / z_score recompute from stored alpha_w with N_llm = 97,737, N_hum = 13,455, α₀ = 5000, and 0 mismatches.
  • c_hum = 0 rows:
    • 23 rows, 23 distinct deltas, range 0.626–2.814. this matches PLAN #26 L264 exactly, and there's no ceiling cluster anymore.
    • Spearman(z, c_llm) on those rows = 0.82, down from 0.996 in v1, so the ordering is no longer pure frequency.
  • family floor: max(5, int(0.05·c_llm)) with ≥ 2 families meeting it (L765–768). all 32 rows satisfy it.
  • FTPO: think that's, so far and as expected are absent from every list (and from the miner source). FTPO = rather than, right away, end to end, exactly what, here's what, what's actually, short version → 126 tuples, consistent with PLAN §2.6 L172, #27 L269 and summary.
  • self-test 5b now really uses alpha_0=5000.0 and asserts a > 0.2 delta gap plus a numeric δ/z recompute.
  • PLAN:
    • §2.6 L174 and #26 say selection is allowlist-driven.
    • §2.2 records both trainable scopes.
    • §2.4 has the any-of matcher, template pool, silent search and logit order.
    • #26–#28 are present.
  • minor:
    • the miner module docstring L18–20 still says "50/50 mixture of empirical pooled frequency and independent unigram-product". stale, fix it to 0.85/0.15.
    • the L781 comment calls the prior "external/leave-one-out", but the unigram table is built from the same corpus (each n-gram contributes to its own unigrams). it's weak, but it isn't leave-one-out.
    • the calibration_mix_v2.txt sha256 is still unrecorded (it waits on a Colab run, as expected).
    • rather than / exactly what in FTPO are still a judgment call for dawn (common English).

fix list (ordered) #

  1. (blocker) L351: limit the whitespace collapse to non-fence, non-inline-code segments, ideally only where emoji were removed. add a self-test with an aligned fence plus a padded table.
  2. L325: only remove the space that came from a deleted emoji. don't run the punctuation regex on kaomoji or inline code. add ( ; ω ; ) and `a ; b` + emoji to the self-test.
  3. remove "want me to …?" from ABSTENTION_TEMPLATES[0]. add a self-test asserting no template contains a phase3_speech_act_ngrams phrase.
  4. closing guard:
    • can(?!'t|not)
    • an action-verb offer requirement
    • exempt X or Y? forks
    • wire up or delete TECHNICAL_TOKEN_IN_SENT_RE
  5. pre-tool: keep because / :-clauses and questions.
  6. R-Tuning: make flat alias lists any-of. rename _word_boundary_おそらく.
  7. transplant: make (ggml_type, bytes) mandatory, use fullmatch, fsync the directory, and fix the "144" count in PLAN §2.2 L60.
  8. Q2_0 self-test: compiler fallback (cc/gcc) for Linux/Colab. word PLAN #28 as "transcribed from" ggml-quants.c.
  9. miner docstring L18–20.