strata metal opts v3 (opus partner): v2 verdict, last kernel polish, and the Phase 4 audit #

patch: /tmp/opus_metal_probes/opus_v3.diff (against the current vendor tree with v1+v2 in; 105 changed lines, git apply --check clean, window.cpp passes -fsyntax-only, the Metal library with the new gr instantiations compiles). not run on the GPU.

1. v2, interleaved (ops_1024, 7.0 GB, wired) #

arm	rep 1 ms/tok	rep 2 ms/tok	CB "GPU"	wait
stock (all off, PF 8)	58.11	59.30	49.7 / 51.0	7.2
v2, EXPERT_SPLIT=0, unpert PF 6	47.68	48.05	41.4 / 41.6	5.0
v2, EXPERT_SPLIT=1, unpert PF 6	46.00	46.18	39.6 / 39.6	5.0
+ DN tile 8,2,4 (1 rep)	45.75	—	39.6	4.7

• net: 58.7 → 46.1 ms/tok (−21%), 17.0 → 21.7 tok/s (+27%), repeatable across both reps (spread ≤ 0.4 ms). the resident-first split is worth a clean −1.7…−1.9 ms on top of the rest. that's about the ~3 ms of handshakes minus what v1's shared-expert overlap already hid.
• CB time (39.6) is now within 1.0 ms of KPROF's serialized kernel sum (38.6): the handshakes are essentially gone from the GPU's view.
• open item: the window is 46.05 ms but the CB is 39.6. if the 5.0 ms of SSD wait were inside the CB it couldn't be 39.6 (38.6 of kernels alone); phase 1 can hide at most ~140 µs per layer of it. so either the CB timestamps don't count event-wait idle, or ~6 ms sit outside the CB. one STRATA_METAL_PROFILE=1 line for the v2 arm (host before / run_graph / GPU / patch+finish) settles it. it matters because it says whether the SSD wait or something on the host is the next 5–6 ms.

2. v3 patch (all small) #

1. moe_down Q2_0 tile 8,2 on the M5 by default (8,1 stays on the M4). your sweep: 2.185 vs 2.431 ms (−0.25 ms).
2. exact BF16 router (−63 MB a token, ~−0.5 ms, zero change in weights). checked over the healed GGUF: all 48 ffn_gate_inp F32 tensors (655,360 values each) have zero low 16 bits, i.e. they're upcast BF16. the routing GEMV reads a BF16 copy built at init (63 MB), only if every value is exact (else it keeps F32); router_route gets the same buffer. STRATA_METAL_ROUTER_BF16=0 turns it off. note the Q8_0 alternative is not free: on the 1022-token trace it keeps 99.66% of top-10 sets and 99.46% of top-1, with 0.0049 mean routing-weight L1 change (rq.cpp). BF16 gives 100% / 100% / 0. check once that ids stay bitwise equal (gemv_30 vs gemv_0 tile choice is the same).
3. per-group HC precision. STRATA_HC_Q4 and new STRATA_HC_Q5 each take 1 | attn | ffn (Q4 wins where both):
   • Q5_0 = 22 B / 32 weights (0.647 × Q8_0's bytes). there's a host quantizer (ggml block_q5_0 layout, round-tripped through ref::dequant in q5test.cpp) and gr_down_q5_8_8 / gr_up_q5 instantiations.
   • on Gaussian blocks: rel-RMSE Q8_0 0.0054, Q5_0 0.0428, Q4_0 (your Lloyd) 0.0816. Q5's weight MSE is 0.275× Q4's.
   • expected, if KLD tracks weight MSE: ops_tools ~0.13 → ~0.035 all-Q5; HC time −1.6 ms (vs −2.14 for all-Q4); +240 MB → ~170 more arena slots.
   • suggested ladder: HC_Q5=1, then HC_Q4=attn HC_Q5=ffn, then HC_Q4=1. gate each by KLD ≤ 0.02 on all three domains (ops_tools is the one that moves).
   • your (a) hypothesis — that hc_ffn is the sensitive half because it feeds the router — is plausible, but hc_attn feeds the same residual one sublayer earlier, so measure HC_Q4=attn rather than assume it.
4. not in the patch, worth an experiment: the two-stage head (output.weight = 3.96 ms, 10.3%):
   • a ~2.6 bpw copy (~205 MB) picks the top 256; exact Q6_K on those rows; ~−2 ms net after the lost arena slots.
   • it's exact for greedy only when the true argmax is in the candidates, and for top-k/top-p only if the sampler normalizes over the top-k (llama.cpp order).
   • measure argmax/top-20 recall with STRATA_METAL_LOGPOS on all three domains before writing the kernel.

after v3: ~45.3 ms/tok with the router + tile (~22.1 tok/s); with HC_Q5=1 ~43.7 ms (~22.9 tok/s) if its KLD passes. what's left is ~36 ms of kernels at 122–145 GB/s, ~5 ms of SSD wait, and whatever §1's open item turns out to be.

3. Phase 4 audit (phase4-colab-persona-ftpo, results/phase4_colab_validation.json) #

verdict: reject the bundle; keep Phase 3's writers. your real-engine numbers (KLD vs Phase 3: 0.276 ops_tools, 0.408 chat_json; PPL 3.02 → 3.42, 1.20 → 1.81) are the only measurements of this artifact on the real model, and they fail every bar we've used. beyond the two proxy errors you found:

1. the reported metrics come from a proxy that isn't the model. "before" teacher-forced CE on the persona turns is 12.08 nats (top-1 7.5%), abstention 13.14, known facts 12.63. ln(248,320) = 12.42, so the proxy predicts these turns about as well as a uniform distribution. the real model is ~1–3 nats on such text. every "after" number (CE −1.2, win rate 50 → 93.75%) is measured on that proxy, and so is the 0.000897 calibration KLD.
2. the FTPO set measures non-events. the 126 tuples are 7 phrases × 18 two-token contexts (e.g. [1, 61745] → 1056). before training the rejected token already has p ≈ 1e-6…1e-9 (e.g. "end to end" 8.3e-9), so "−98.13% rejected probability" is a drop from almost-never to almost-never.
   • FTPO needs contexts mined from the model's own generations, where the slop token is actually likely (PLAN §7 phase 4: "slop mining on tuned outputs").
3. not reproducible from the repo. scripts/run_phase4_colab_ftpo.py holds only the FTPO loss reference and a self-test that checks the artifacts' format and the JSON's own numbers. the training code only exists in the Colab transcript. total runtime was 25.9 s.
4. its own table shows L40–47 MoE MSE got worse (11749 → 11833 after quantization), and scope: swift_writers_plus_router modified no router.
5. the 16 SFT pairs / 807 target tokens are far too few to move voice, even with a real forward.

3.1 the layer-47 zero-proxy formulation: correct, with these precise terms #

from window.cpp (decode) / prompt_chain.inc L326–366 (prefill), gr.metal, glue.metal, prompt.metal. per token t, streams c = 0..3, n = 2560:

hh   = silu(W_sg47 · XF47) ⊙ (W_su47 · XF47)            [640]   (dump: .hh; fp16 on the prompt path)
ys   = W_sd47 · hh                                      [2560]  <- the trainable W_sd47 = blk.47.ffn_down_shexp.weight (Q8_0)
bo   = yr47 + σ(sgv47) · ys                             [2560]  yr47 = Σ_k w_k·expert_k(XF47) (dump: .yr), sgv47 = w_gsh47·XF47 (.sgv)
R'_c = R_c + bo · 2σ(inj47_c / 4)                        [4×2560] R = the streams after layer 47's ATTENTION write (applied by
                                                          hc_ffn_47's gr_norm), inj47 = layer 47's hc_ffn inject (4 floats)
xn_c = R'_c ⊙ w_onorm_c · rsqrt(mean_d R'_c² + 1e-6)       w_onorm = output_hc_norm.weight (F32, 4×2560)
lo   = silu((W_odown · xn) / 4)                          [320]   W_odown = output_hc_down rows 0..319 [320×10240]
mix  = ¼ Σ_c xn_c ⊙ σ(W_oup[c·2560 + d] · lo)            [2560]  W_oup = output_hc_up [10240×320]
logits = W_out · mix                                      [248320] W_out = output.weight (Q6_K); no further norm

• yes, nothing sits between ffn_down_shexp.47 and the logits but moe_combine_sh → the output_hc read → output.weight. for a ΔW on blk.47.ffn_down_shexp (or on output_hc_* / output.weight) this is the real network, not a proxy. Δ(R'_c) = 2σ(inj_c/4) · σ(sgv) · ΔW·hh, and the routing and everything before it is untouched.
• your dump list is one term short: (R_post47, inj_47, hh_47, sgv_47) also needs yr_47, unless R_post47 means the streams after the base FFN write. then use R'_base + 2σ(inj/4)·σ(sgv)·(ΔW·hh). both work; pick one and name it in the dump. you also need R itself (4×2560 f32 per token), which STRATA_METAL_DUMP_MOE doesn't write today (.mixed is XF, not R). inj47 is the inject buffer produced by layer 47's hc_ffn gr_down, i.e. inj_[cur_inj_] at the output read.
• use the runtime's weights, not the file's:
  • output_hc_down/up are BF16 in the GGUF but the engine reads its Q8_0 copies (q8off_ covers every BF16 hc_* except norms; the Q4/Q5 options never touch output_hc). dequantize through ref::quantize_q8_0 → ref::dequant.
  • output.weight through Q6_K dequant.
  • train ΔW in float, then score the loss after re-quantizing blk.47.ffn_down_shexp to Q8_0, as it will ship.
• precision: the prompt path rounds hh (and its GEMM operands) to fp16, the decode window doesn't. differences are ~1e-3 relative. "identical" holds up to float rounding, not bitwise.
• routing: yr47 and everything upstream depend on MoCE residency when λ > 0. dump with STRATA_METAL_MOCE_BONUS=0 for a deterministic training/eval set; measure deployed λ separately.
• what it can and can't do:
  • it's exact only for L47 / output-side tensors. any ΔW at L40–46 goes through later attention, GDN, QSA and routed experts (routing has discontinuities). no closed form exists; those need the real forward.
  • capacity is one 2560×640 map, gated per token by σ(sgv47) and the stream gates. that's plausible for lexical FTPO (moving a few next-token logits in specific contexts) and weak for persona voice. keep persona/abstention in Phase 3's real-forward LoRA.
• acceptance for a re-run, all measured on the real engine (dawn's text bars aside):
  • KLD vs Phase 3 ≤ ~0.01 on the 3 eval domains,
  • p(rejected) dropping in mined contexts where it started ≥ 1e-2,
  • the transplanted GGUF's own LOGPOS, not the trainer's numbers.

4. v3 verification addendum (M5 Air, ops_1024, 7.0 GB, wired; parent's call_1809920, call_2299199) #

• v3 defaults (ROUTER_BF16=1, Q2_0 down tile 8,2, EXPERT_SPLIT=1, unpert PF 6): 45.28 ms/tok = 22.08 tok/s (predicted ~45.3 / ~22.1). hit 97.59%, 11.6 misses/tok, 479 late read-aheads, host 5.21 ms of which waiting 5.08 (+0.46 after the GPU went on). route_parity: 0 mismatches.
• §1's open item is closed; there's no hidden host gap. profile: window 45.298 = host before 0.225 + run_graph 45.027 (CB GPU span 44.068
  • encode/submit/wait 0.959) + patch+finish 0.047. the CB span does include the event waits: 44.07 ≈ 38.99 kernels + 5.08 SSD wait. the "39.6 ms" in the earlier table was the validation script's pure_gpu = GPU − host_wait, not the CB span. so the budget is ~39 ms of kernels, ~5 ms of SSD wait, ~1.2 ms of host/submit, and the SSD wait is the next item.
• ROUTER_BF16 is not bitwise in the end. the weights are identical, but gemv_30 and gemv_0 sum in a different order, and that flipped one rank-10 tie on ops_tools: KLD 0.0087, top-1 96.88%, PPL 3.0165 → 2.9914. kept as default. read the 0.0087 as ops_tools' noise floor for any change that only reorders floats: the HC rows below should be compared against ~0.01, not 0.
• HC ladder, KLD vs HC_Q8 (top-1):

config	ops_tools	chat_json	ja	macro KLD	macro top-1
HC_Q4=1 (earlier)	0.1298	0.00997	0.02755	—	—
HC_Q5=1	0.0786 (89.06%)	0.0029 (98.44%)	0.0123 (96.88%)	0.0313	94.79%
Q4 attn + Q5 ffn	0.0889 (90.62%)	0.0043 (98.44%)	0.0235 (96.88%)	0.0389	95.31%
Q4 attn + Q8 ffn	0.0656 (90.62%)	0.0042 (98.44%)	0.0197 (96.88%)	0.0298	95.31%
Q8 attn + Q5 ffn	0.0575 (93.75%)	0.0020 (98.44%)	0.0062 (96.88%)	0.0219	96.36%

• chat_json and ja tracked the weight-MSE prediction (Q5 ≈ 0.25–0.45× Q4). ops_tools didn't: Q5 is 0.61× Q4, not ~0.28×. it's dominated by discrete routing flips, not by the smooth error, and every row is still 6–9× its floor. so HC_Q8 stays the default, with HC_Q5/HC_Q4 opt-in.
• if one is ever wanted: Q8 attn + Q5 ffn has the best ratio (half the HC bytes saved of all-Q5, lowest KLD).
• a better route for ops_tools would make routing ties robust rather than the HC weights finer, but that's beyond the kernel pass.
• next: the 5 ms of SSD wait (479 late read-aheads at PF 6, up from 118 at PF 8 with MRU, so lateness, not recall, is now the cost). and the layer-47 Phase 4 bundle once it's back: verify with real-engine KLD vs Phase 3 and LOGPOS on mined slop contexts (§3.1).

5. Phase 4 v2 (zero-proxy L47) verification #

independent re-implementation (/tmp/opus_metal_probes/l47eval.cpp): my own C++ forward of §3.1 from the dumps, with W_sd47 from the Phase 3 GGUF vs from the bundle, through Q8_0-round-tripped output_hc_* and the exact Q6_K head over all 248,320 rows, on both 1,022-token traces. per-position output: /tmp/opus_metal_probes/l47eval.tsv.

what checks out:

• the bundle: 47 of 48 tensors are byte-identical to K256-q2_0-healed-metal.gguf; only blk.47.ffn_down_shexp.weight (Q8_0) differs. ‖ΔW‖/‖W‖ = 0.32 (large; it's the SVD filter that keeps it quiet on the background).
• the dumps + equations are the real model. base CE on the natural cal text is 2.5496 with top-1 53.91% (a proxy would sit near 12.4). my numbers reproduce the report: cal CE 2.5496 → 2.5510 (reported 2.5501 → 2.5515), cal KLD 0.000140 (0.00014), p4 OOS KLD 0.00202 (0.002039), and all 126 FTPO p(rejected) before/after within 0.0012. the trainer does what its JSON says.
• real-engine 3-domain KLD vs Phase 3 (parent's run): ops 0.0061, chat_json 0.0002, ja 0.0008. ops_tools sits under its ~0.009 float-order floor.

what doesn't hold yet (don't ship it as the anti-slop result):

1. every slop number is in-sample. the 126 FTPO tuples, the 8 "active" and the 95 "mined" positions are all training positions on the same trace. no held-out slop context was scored.
2. the training trace is synthetic word salad, not model output: "Rather instead place"rather instead place…, End to cleanly suite[End to cleanly suite\nend to cleanly suite…, what's what at[What's what at…. the 126 FTPO positions do end in the banned prefixes (126/126), but those contexts are nothing like generations.
3. the "mined" loss suppresses the six final tokens in every context — the objective is not FTPO.
   • mined_* takes any position where " end", " than", " what", " actually", " away" or " version" has p ≥ 0.005, with no check that the preceding tokens are the banned prefix.
   • of the 54 mined positions outside the FTPO/SFT sets, 0 of 54 follow a banned prefix. they're ordinary uses, and all were pushed down (mean p 0.030 → 0.0039, −87%).
   • examples: the kprof trace shows → " what" 0.016 → 0.0023; yeah, that's → " what" 0.014 → 0.0009; smap_ update into at → " end" 0.007 → 0.0008.
   • so it learns "say ' what' / ' than' / ' end' less", not "don't say 'here's what' / 'rather than' / 'end to end'".
4. the KLD pass can't see that damage: the three eval texts (223 / 520 / 308 tokens) contain zero occurrences of any of the six tokens, and the cal text has none as a target either.

to accept a v3 bundle:

• (a) restrict the mined loss to positions whose preceding tokens match a banned tuple's prefix (tuple[:-1]).
• (b) build the FTPO/mined set from the model's own generations (persona/ops prompts), not concatenated fragments.
• (c) hold out ≥ 30% of those slop contexts and report p(rejected) only on them.
• (d) add a collateral set: natural text with legit "more than", "what is", "the end of", "far away", "version 2", "actually" (≥ 200 occurrences). dump its L47 states with the Phase 3 GGUF; l47eval scores p(target) base vs new in ~20 s. bar: mean log-prob change on those targets ≥ −0.05 nats.
• (e) keep the real-engine 3-domain KLD ≤ 0.01.
• persona/abstention: in-sample CE 4.15 → 3.95 on 224 tokens is within what a 2560×640 map can do. not evidence of a voice change.

6. Phase 4 v3 (WFLD L47) verification #

same independent forward (l47eval2.cpp, Phase 3 GGUF W_sd47 vs the bundle's, exact Q6_K head over 248,320 rows), on cal_p3 (1,022 rows), p4_ftpo_sft (1,022) and p4_collateral (1,638). per-position output: /tmp/opus_metal_probes/l47eval_v3.tsv.

reproduced exactly:

check	mine	reported
bundle	47/48 byte-identical to Phase 3, ‖ΔW‖/‖W‖ = 0.0725	0.0725
cal KLD (CE 2.5496 → 2.5515, top-1 agreement 96.87%)	0.002952	0.002952
FTPO train p(rejected) (28/28 end in the exact prefix, rejected = actual next token)	0.63264 → 0.00546	same
FTPO held-out p(rejected) (14/14 prefix-exact)	0.87327 → 0.02889	same
collateral, all 207 Δlog p	+0.00690	+0.00690
collateral train_anchor (65)	+0.06797	+0.06797
collateral heldout_eval (142)	−0.02105	−0.02105
collateral occurrences after a banned prefix	0	0
whole collateral trace KLD (CE 2.9318 → 2.9299)	0.0071	—

so against the bars from §5 it passes. (a) prefix-exact, (b)/(c) natural sentences with a 33% held-out split, (d) held-out collateral mean −0.021 ≥ −0.05, (e) real-engine ops/chat_json/ja = 0.0099/0.0020/0.0013 (ops sits right at its ~0.009 float-order floor).

but the collateral mean hides a per-word problem. my bar was too loose:

held-out collateral word	n	mean Δlog p	worst
" actually"	16	−0.154	−0.844 (will actually: p 0.405 → 0.174)
" away"	23	−0.056	−0.628 (sit a directory away: 0.427 → 0.228)
" end"	24	−0.042	−0.231
" than"	20	−0.019	−0.390 (require more than: 0.423 → 0.287)
" version"	32	−0.010	−0.373
" what"	27	+0.091	−0.809 (If you wonder what: 0.704 → 0.314)

• 21.8% of held-out collateral occurrences lose more than 0.1 nats. " what" being raised on average (+0.09) is what pulls the mean over the bar. " actually" (what's actually) and " away" (right away) are suppressed in legit contexts too.
• the FTPO held-out is weaker than it looks: its 14 positions sit in the same document, after 4 earlier uses of the same phrase. that's why p_before is higher on held-out (0.87) than on train (0.63) — in-context copying. it's held-out positions, not a held-out distribution.

recommendation: close, not quite. before shipping:

1. tighten (d) to per word: each of the six words' held-out mean Δlog p ≥ −0.05, and ≤ 10% of occurrences below −0.1. a raised " what" shouldn't be allowed to offset a suppressed " actually".
2. to get there: give "what's actually" and "right away" a weaker margin, or add their collateral directions (more actually / away anchor sentences) to the zero-inner-product set. it's one or two extra constraints in the WFLD solve.
3. a held-out slop set from a different document, with no earlier uses of the phrase in context. ~6 sentences per phrase, dumped like the others; l47eval2 scores it in seconds.

if dawn wants it shipped as-is, it's a defensible improvement (−97% on the 14 held-out slop positions, macro KLD 0.0044), with a known "actually"/"away" cost.

6.1 Phase 4 v3.1 verification #

same independent forward (l47eval2), now also over the separate-document held-out slop trace (p4_ftpo_heldout_1024). per-position output: /tmp/opus_metal_probes/l47eval_v31.tsv.

reproduced exactly:

• bundle: 47/48 byte-identical, ‖ΔW‖/‖W‖ = 0.0363.
• held-out per-word collateral: than −0.0335, what +0.1632, end −0.0168, away +0.0141, version +0.0154, actually −0.0390. 13/142 below −0.1 (9.2%). the per-word bar passes.
• separate-document held-out slop (21, all prefix-exact, rejected = actual next token): p(rejected) 0.414 → 0.104. FTPO train 0.633 → 0.094, held-out-14 0.873 → 0.226. cal KLD 0.00114 (CE 2.5496 → 2.5511). collateral trace KLD 0.0040.

new problem: the suppressed mass goes to <|im_end|> (id 248046), and it shows up across chat-formatted text.

• at the 21 separate-document held-out slop positions the new argmax is <|im_end|> in 7/21. e.g. …reproduces the head forward pass end to → <|im_end|>; Regularize against the calibration trace\nRather → <|im_end|>. in generation that ends the assistant's turn mid-sentence, which is worse than the slop.

• it isn't confined to slop contexts. argmax flips to <|im_end|> at ordinary positions:

  | trace | argmax flips to <|im_end|> | |---|---| | held-out chat document (1,022 positions) | 28 | | SFT document | 23 | | cal (plain text) | 0 | | collateral (plain text) | 0 |

  examples: exactly what went wrong with → <|im_end|> (was " the"); Short version: we want to → <|im_end|> (was " know"); user turns too: why did the proxy → <|im_end|>.

• on the held-out chat document, ordinary targets (not slop, not <|im_end|>; n = 952) lose −0.060 nats on average. CE 2.583 → 2.693, whole-document KLD 0.0236, and 0.0080 at positions more than 5 tokens from any slop position.

• why the real-engine 3-domain KLD (ops 0.0075, chat_json 0.0003, ja 0.0008) doesn't show it: those texts aren't multi-turn chat with <think>/<|im_end|> structure, and the effect is concentrated where the L47 state looks like an open assistant or user turn. cal and the collateral trace are plain text: 0 flips.

• likely cause: delta_cho = 0 plus the per-word null-space removed the "raise a natural alternative" component, so lowering y_r frees mass that the head hands to the most generic continuation in chat context, <|im_end|>. the phrase directions also overlap with an "assistant/user turn in progress" component of z that cal (plain text) doesn't constrain.

verdict: don't ship v3.1. it fixed the per-word collateral and generalizes on the separate document, but it trades slop for early turn ends. bars to add, each checked on a held-out chat-formatted document:

• (f) 0 argmax flips to <|im_end|> / <|im_start|> / </think> at non-terminal positions, and no rise in their mean probability at slop positions. better: an explicit penalty on special-token logits in the objective.
• (g) ordinary-token Δlog p on the held-out chat document ≥ −0.01 nats, and KLD ≤ 0.005 there.
• fix ideas:
  • put the background constraint on chat-formatted anchors too (a separate chat document's z, not only plain-text cal);
  • restore a small delta_cho > 0 so the freed mass goes to the natural alternatives (each held-out slop position's top non-rejected token, not <|im_end|>);
  • add <|im_end|> as a second "rejected" logit at every FTPO position.

6.2 Phase 4 v3.2 verification #

same independent forward (l47eval2). the rewritten held-out chat trace has 1,098 rows. per-position output: /tmp/opus_metal_probes/l47eval_v32.tsv.

reproduced:

• bundle: 47/48 byte-identical to Phase 3. ‖ΔW‖/‖W‖ = 0.0978 (up from 0.0363 in v3.1; not in the report).
• (f) 0 argmax flips to <|im_end|>/<|im_start|>/<think>/</think>/<|endoftext|> on all four traces.
  • flips away from <|im_end|>: 2 (cal), 13 (SFT), 12 (held-out chat). all are mid-sentence positions where the base model wrongly preferred ending the turn (e.g. the limbo list keeps → base <|im_end|>, now " the").
  • real turn ends untouched: at the 57 positions whose actual next token is <|im_end|>, its argmax survives 56/56 (where it was the base argmax), and Δlog p(<|im_end|>) ≥ −0.002 on SFT/held-out (one cal end at −0.27).
  • so the side effect now leans toward fewer premature ends, not more. that's fine.
• (g) held-out chat document, 1,077 non-slop positions: KLD 0.00472. ordinary-token Δlog p +0.0208 (n = 1,034; median 0.000). the largest shifts are at user-turn openers (<|im_start|>user\n → "what"/"how"), which don't matter for generation.
• (d) per-word held-out collateral: end −0.0140, than −0.0343, what +0.1598, version +0.0071, away +0.0066, actually −0.0419; 13/142 below −0.1 (9.2%). passes.
• (a)–(c):
  • separate-document slop p(rejected) 0.470 → 0.143 (−70%), 21/21 prefix-exact. the new argmax is a real word in 20/21 ("start", "answer", "the", "how"…); one <|im_end|> was already the base argmax there.
  • FTPO train 0.633 → 0.100, same-doc held-out 0.873 → 0.262.
• cal KLD 0.0019 (CE 2.5496 → 2.5505). collateral trace KLD 0.0034.

verdict: v3.2 passes (a)–(g) and i'd accept it, with one caveat on how clean the evidence is.

• the held-out chat document was rewritten after v3.1's failure, and the solver was tuned over three rounds against these same held-out numbers (per-phrase scales, the special-token rows, a hand-picked tr_p4.z[705] constraint).
• so these held-out sets are now partly development sets. for final sign-off:
  • write one fresh chat document (~21 slop turns + ordinary turns) that nobody tuning the solver has looked at, dump it, and run l47eval2 once with no further changes;
  • if it lands within ~1.5× of the §6.2 numbers on (f)/(g)/slop reduction, ship v3.2.
• expected cost of the trade: slop is suppressed less than in v3 (−70% vs −97% held-out), in exchange for no turn-ending damage. that's the right side of the trade.

6.3 Blind chat test sign-off #

untouched v3.2 bundle (sha256 4f8d9de8…db10, ‖ΔW‖/‖W‖ = 0.0978, the same as §6.2). fresh 1,100-token multi-turn chat document (p4_blind_test_1100, 1,098 rows, 21 slop positions, 3 per phrase). one run of l47eval2, no changes. per-position output: /tmp/opus_metal_probes/l47eval_blind.tsv.

check	blind	§6.2 (dev)	within ~1.5×?
slop p(rejected), 21 positions (all prefix-exact, rejected = actual next token)	0.554 → 0.253 (−54%)	0.470 → 0.143 (−70%)	residual 1.77× — just outside
argmax no longer the rejected token	14/21 (base 8/21)	20/21	—
(g) non-slop KLD	0.0030	0.0047	yes, better
(g) ordinary-token Δlog p (n = 1,029)	+0.012 (3.0% below −0.1)	+0.021	yes
(f) argmax flips to a special token	2	0	see below
real turn ends (48), Δlog p(`<	im_end	>`)	+0.002 mean
whole-doc CE / KLD	1.677 → 1.696 / 0.014	—	—

per phrase (base → new mean p(rejected)), on the blind document:

phrase	base	new
end to end	0.984	0.263
short version	0.274	0.034
here's what	0.428	0.099
exactly what	0.272	0.063
what's actually	0.056	0.005
right away	0.893	0.540
rather than	0.974	0.767

• the two flips to a special token:
  • one is a fix: …lets readers proceed concurrently. → <|im_end|> is the true next token; the base predicted \n\n.
  • the other is at a user-turn opener (<|im_start|>user\n → <|im_end|> instead of "how"), a position the assistant never generates.
  • in the other direction there are 10 flips away from mid-sentence <|im_end|> (benign, as in §6.2), and one real turn end after a kaomoji now prefers another kaomoji glyph ((´• ω •)→´instead of<|im_end|>`). mild.
  • no assistant-side turn damage.
• the 3 other uses of the six words in the blind document are all " end" opening an "end-to-end" phrase (Running the integration suite → " end", Δ −0.08…−0.32). that's the banned phrase's first word getting less likely, which is fine.

sign-off: ship v3.2. it generalizes safely to unseen chat: KLD 0.003, ordinary tokens unharmed, no assistant turn-ending damage.

• its slop effect is partial on fresh text: −54% overall. strong on 5 phrases, weak on the two the model is surest about: "rather than" (−21%) and "right away" (−40%), where base p ≈ 0.9–0.97.
• don't push the L47 solve harder for those: that's what broke v3.1.
• cover the residual at the sampler instead: a prefix-conditioned logit penalty. when the last token(s) match a banned tuple's prefix, subtract a fixed bias (e.g. 4–6) from that tuple's final token.
  • it's exact, costs nothing, touches no other context, and stacks with v3.2.
  • in Strata it belongs where the sampler sees the token history, or as a 6-token bias applied to lm_ before argmax when the previous token is one of the prefix tokens.