opus audit v3: phase 3 prep fix verification #

read-only · auditor: opus-verify-phase3-v3 · scope: the 9 items in audits/opus_audit_phase3_prep_v2.md §fix list. the DB was opened mode=ro, probes ran in-kernel with sys.dont_write_bytecode=True, and no __pycache__/*.pyc exists anywhere in the repo afterwards. the only file written is this report.

verdict: NOT PASS. one new major false positive in the M2' hypothesis path. items 1–4 and 6–9 pass.

the new causal-clause regex PRE_TOOL_HYPOTHESIS_CLAUSE_RE (prepare_phase3_sft_data.py L83) has no word boundary before since|because|as|so that, and it treats bare as as causal. so any pre-tool sentence that contains the letters as␠ (was, has, alias, as root, as well) gets cut mid-word, and only the garbage tail is kept as the "hypothesis". this hits exactly the falsifier ("any false positive on M2' probes").

self-tests (python3 -B) #

1 & 2. emoji strip / layout (B2', B3'): PASS #

• the global (?<=\S)[ \t]{2,}(?=\S) collapse and the [ \t]+([.!?,;:]) punctuation regex are both gone. _strip_emoji_preserving_layout (L181) only touches the [ \t]* captured right next to each emoji match, and fences are split out and passed through untouched (L361–368).
• live data (fresh run, 658 text turns): code_fences_altered = 0 / 140 fenced turns. that's from both the script's own check and my own findall comparison. there are 0 unclosed-fence turns and 0 ~~~ fences.
• stricter check: on turns with no emoji and nothing stripped, the whitespace skeleton is identical in 100% of cases. the only indented lines that changed (2 turns) are nested bullet prose being lowercased by design, not code.
• the committed results/phase3_speech_act_surgery_report.json says code_fences_altered: 0 / 140 (657 turns, since the DB has grown by one since). the seed JSONL equals a fresh build_synthetic_phase3_seed_examples().
• probes, all with an emoji stripped in the same reply:
  • an aligned fence (x = 1, print(a, b)), a padded table (| name | kld |) and make -j8 all all come through byte-identical.
  • ( ; ω ; ) and `a ; b` survive, while 🚀 ✅ 🇯🇵 ⭐ are stripped.
  • "fixed it 🚀\n" → "fixed it\n", so there's no trailing space anymore.
• nits (not blocking):
  • (a) an emoji inside inline code is stripped: `echo 🚀` → `echo `. it happens in 1 of 658 live turns.
  • (b) a table cell padded around an emoji loses its padding: | name 🚀 | → | name |. that's only at the emoji position, which the v2 spec allowed.
  • (c) the final .strip() (L385) eats the indentation of a reply whose first line is 4-space code.
  • (d) ~~~ fences aren't protected from emoji stripping or lowercasing. there are 0 in the live data.
  • (e) hello🚀world → helloworld.

3. ABSTENTION_TEMPLATES: PASS #

• none of the 6 templates contains anything from banned_ngrams (32), phase3_speech_act_ngrams (25), phase4_lexical_ftpo_ngrams (7) or the 576 surface_variants. I checked both word-boundary and raw substring matches.
• there's no ? and no want me anywhere, and every template passes through surgery unchanged.
• the self-test asserts against banned_ngrams, which is a superset of phase3 ∪ phase4 (I checked), so the docstring's claim holds.

4. closing guard (M1'): PASS (false negatives noted) #

• can(?!'t|\u2019t|not) plus OFFER_ACTION_VERB_PAT are in place (L108–125). CHOICE_FORK_QUESTION_RE (L142) with (?!or\b) handles the leading Or.

• TECHNICAL_TOKEN_IN_SENT_RE was deleted, which is acceptable ("wire up or delete").

• kept (0 false positives in 14 probes):

  • I can confirm the seed…, I can't help much…, I cannot…, I can not…
  • Should I write the parser in Rust or Go?, Would you like to keep the old schema or migrate everything now?
  • Would you like Postgres or SQLite…?, Do you want the cache in Redis or in-process?, Want the tables sharded by user or by tenant?
  • If you want lower latency, use…, If you want the details, the root cause…, If you want to reproduce it, run…
  • If you'd rather avoid swap, I can see why…, Should I be worried about…?

• stripped as intended:

  • Do you want me to apply…?, Or do you want me to revert it instead? (sentence-initial Or), If you want, I can apply the fix now.
  • If you want, I'll push it., Want me to run the tests?, Let me know if…, Shall I commit this?, Should I go ahead and commit this?, Happy to help with anything else!

• live data: the fork exemption keeps 3 final sentences. 2 are real forks (branch vs main, extend-login vs trace-wire-format). 1 is borderline (…or were you just checking that permissions work?).

• false negatives (minor, not the falsifier): the fork regex matches any or, so binary permission asks survive:

  • Do you want me to push this or not?
  • Should I commit this, or not?
  • Would you like me to fix this or anything else?
  • Would you like me to apply the patch, or would you rather review it first? (the classic menu-offer)

  suggestion: don't count or (not|anything else|something else|would you rather|should i|do you want) as a fork. separately, If you want, I can apply it to main.py. survives because [^.!?\n] stops at the . in main.py.

5. pre-tool hypothesis / questions (M2'): FAIL (new false positive) #

• the v2 probes now pass:
  • …because the OOM killer usually targets the biggest cgroup. → the OOM killer usually targets the biggest cgroup.
  • I'll test this hypothesis: the fd leaks… → the fd leaks…
  • …first, then we can talk about the race. → we can talk about the race.
  • Let me see if I understand: you want the cache cleared on boot? is kept whole (the ? guard is at L256 and L311).
  • Let me check the config? It might be stale. is kept.
• new false positives. each is a single pre-tool sentence with has_following_tool_call=True:

• cause: in L83 (?:,?\s*(?:since|because|as|so\s+that)\s+|:\s+|,\s*then\s+), the \s* can match zero characters, so w·as␠, h·as␠ and ali·as␠ match. as is also ambiguous even as a whole word.
• these sentences should either be deleted whole (pure narration) or kept verbatim. a mid-word fragment becomes a broken SFT target.
• live impact today is ~0: only 1 pre-tool strip on 4 text+tool turns, and that one is correct ("no, i haven't. let me look." → "no, i haven't."). but this path will run over all teacher-generated phase 3 data.
• fix:
  • require (?:,\s*|\s+)\b(?:since|because)\b\s+ (or ,\s*as\s+ only after a comma), and drop so that.
  • add the 7 probes above to self-test §5, asserting the output is either "" or the unchanged input.

6. R-Tuning aliases + naming: PASS #

• a flat list is always any-of (L428–431). ["42","forty-two"] → known_keep_teacher at 1.0 (self-tested), and the teacher answer "forty-two" alone also passes.
• nested groups require every group: [["42","forty-two"],["seconds","sec"]] gives 0.75 when one sample says 42 s, which is correct.
• match_mode="all" is explicit opt-in.
• _word_boundary_おそらく is gone everywhere. only _word_boundary_match remains.

7. transplant: PASS (nits) #

• the (ggml_type:int, bytes) tuple is mandatory (L273), a bare-bytes call is self-tested to raise ValueError (L505), and the type is checked against the header (L278).
• all 4 regexes are used via .fullmatch (L113–128).
• the directory is fsync'd after os.replace (L312–316).
• PLAN §2.2 L60 now reads 12 attn_output + 36 ssm_out + 48 ffn_down_shexp = 96 writers, plus 96 routers = 192, with the SC117 144 explained.
• nits:
  • (a) the L93 comment still says "144 writers + 96 router gates = 240 tensors". it's stale, make it 96 + 96 = 192.
  • (b) isinstance(item[0], int) accepts bool, so (True, buf) passes as type 1 = F16. harmless in practice, but type(item[0]) is int is stricter.

8. Q2_0 compiler fallback + PLAN wording: PASS #

• shutil.which("clang") or which("gcc") or which("cc") (L436), with a clear RuntimeError if none is found.
• PLAN #28 L271 now says "compiled C reference transcribed from ggml-quants.c".
• nit: _Float16 needs GCC ≥ 12 on x86-64. older Colab images would fail to compile instead of skipping.

9. miner docstring: PASS #

L18–21 now describe the 85/15 mixture (0.85 * p_indep + 0.15 * p_unif, no self-pooled term).

remaining fix list #

1. (major) PRE_TOOL_HYPOTHESIS_CLAUSE_RE L83: add \b and require a real separator before since|because, drop bare as (or require , as) and so that. add the 7 probes from §5 to the self-test.
2. (minor) CHOICE_FORK_QUESTION_RE: don't exempt or not / or anything else / or would you rather permission asks.
3. (nit) L93 comment 144/240 → 96/96/192; type(...) is int; skip emoji stripping inside inline code; protect ~~~ fences.