# opus audit — Phase 3 local healing + stream.cpp, v1

scope: read-only audit of `scripts/heal_k256_q2_0.cpp`, `scripts/run_phase3_local_healing.py`, `scripts/patch_strata_metal.py` (`stream.cpp` `smap_` + `new_drop_block`, `window.cpp` `dump_moe_layer`), `results/phase3_local_healing_validation.json`; plus out-of-sample falsifiers on turns/prompts outside the 192-token eval set and the 2 speed prompts. nothing in the repo was edited; all probes, builds and outputs live in `/tmp/p3audit` and `/tmp/strata_noevict`.

## 0. tl;dr

- **falsifier not triggered.** on 22 unseen windows (17 docs, 2,686 scored tokens, ops / chat-json-tool / ja / persona), healed Q2_0 beats unhealed in **21/22 windows**: pooled KLD **−24.7%** [95% CI −28.7, −19.2] at MoCE 0 and **−24.5%** at MoCE 0.85, and top-1 goes **up** +2.5 / +2.0 pt (the −2.6 pt top-1 drop on the 192-token set was noise). but the 192-token set **overstated the win** (−35% → −25%; isolated MoCE −53% → −24%).
- **stream.cpp `new_drop_block` A/B** (shipped build vs. the same source with only that block reverted) over 9 diverse prompts: hit rate identical to 4 decimals everywhere (MoCE .85 mean 0.9327 vs 0.9327), decode speed within noise, and the patched build is the deterministic one. not triggered either; my theoretical worry about it (waiting on, then discarding, unused read-aheads) doesn't cost anything in practice, because unclaimed read-aheads were already sitting at the LRU tail.
- **Part A (α_l)**: with 4 experts, sampling SD is only ≈0.01, so that's not the problem. the problem is the Gaussian inputs: on real activations the shipped α is worse than α=1 in 22/48 layers (most of layers 24–44 overshoot), and on an unseen dump it removes just 0.4% of routed residual. an α fit on the real cal trace gets 1.5% and transfers fine.
- **Part B ridge (T=510 < D=640)**: rank(Z) 369–510, effective dof 5–63. in-sample gain is 4–40× the held-out gain, which is the classic T<D overfit signature, but the shipped (ρ=6, η=0.15) never hurts out of sample. requantization then eats most of what's left: the shipped tensors give −0.36% residual on unseen data. it's safe, and it's basically a no-op.
- **the "AWQ" weighting is uninformative**: hc_ffn_norm-derived s² vs the real E[mixed²] has mean log-corr −0.03 (range −0.50…+0.41). healing works *despite* it.
- **eval hygiene**: the calibration trace is drawn from the same 3 eval files and templates as the KLD set (no lexical leakage in the scored tokens, but it's close). persona text has the highest residual KLD and isn't in the eval at all. multi-turn tool calling is the only window type where healing regresses, and the calibration trace has zero tool-call content.

## 1. math & code audit

### 1.1 `heal_k256_q2_0.cpp --quantize` (signed-d LS Q2_0)

what it does per 64-weight block: 10 seeded scales `d = ±{0.42..0.94}·amax`, nearest-code q∈{-1,0,1,2} under the scale, one LS refit `d = Σwqx/Σwq²`, re-round, keep the best `score = (Σwqx)²/Σwq²` (= weighted energy captured), then inflate `d_ls` by `gain = max(r²,0.25)^(-gain_exp/2)` where `r² = score/Σwx²` (weighted cos² of the block). codes packed `qs[i/4] >> 2(i%4)`, value `(code-1)·d` with signed fp16 d. matches the C-verified codec from §2.7 #27, and a negative d really does give the mirrored grid `{-2|d|,-|d|,0,|d|}` with no kernel change. the math of the search is fine.

findings:

1. **the "AWQ" importance is not an activation statistic** (medium severity, affects gate/up quantization and the Part A inputs). `col_imp[d] = mean_c(hc_ffn_norm.weight[c·N+d]²)` is the per-channel RMS of the 4 hyper-connection norm gains. the experts don't consume that; they consume `mixed` = output of `gemm_hc_up_64_64_4` (HC read + low-rank up-mix), see `window.cpp` L2336-2343. i measured the correlation between `log s²` and the empirical `log E[mixed_d²]` from the 510-token teacher dump, per layer: **mean −0.03, range −0.50 … +0.41** (per-layer table in §1.2). so the weighting is essentially uninformative and in some layers anti-correlated: gate/up blocks are "protected" on channels that are not actually large. the gain still helps because the signed-d LS search does the heavy lifting, not the weighting. fix: use `E[mixed_d²]` (or better, the per-layer diagonal of `XᵀX` from a few thousand calibration tokens) — it's already on disk in `cal_cache/*.mixed`.
2. **two stacked norm-inflation knobs on W_down**: `gain_d=0.30` inflates every down block by `cos_block^-0.3`, then Part A multiplies all down d's by `α_l` which was itself computed on the already-inflated weights. the two are confounded; there is no ablation in the repo showing either is needed (vs. plain LS d, gain=0, with an MSE-optimal α). `gain` is MSE-suboptimal by construction (LS d is the MSE optimum per block), so it's a bet on "norm preservation helps KLD", which needs a held-out KLD ablation, not the 192-token set.
3. W_down is quantized with no input importance at all (`col_imp=nullptr`). the down input `h = silu(g)·u` is very heavy-tailed; per-expert `E[h²]` is cheap to collect from the same dump (per routed token). this is the most obvious unexploited win if more healing is wanted.
4. systems: no error checks on any `open/pread/fopen` (an unopened fd reads 0 bytes silently, a missing d_bak in `--calibrate-and-heal` falls back to `/tmp/healed_down_d_bak.bin`, and a null `FILE*` would crash). header patching searches the first 16 MiB for the tensor name and writes the type in place — fine for this file, fragile in general.

### 1.2 `--calibrate-and-heal` Part A: per-layer `α_l = cos^(1-γ)/norm_ratio`

- γ semantics: γ=0 → `α = cos/ratio` = the MSE-optimal scalar `⟨y_ref,y_q⟩/‖y_q‖²`; γ=1 → `α = 1/ratio` = norm matching. γ=0.75 sits 75% of the way (in log space) toward norm matching. that's a deliberate bias away from MSE; fine as a hypothesis, but again only testable on held-out KLD.
- **4-expert sampling variance is not the problem.** i recomputed Part A for all 256 experts per layer (reproduces the shipped α to 1.3e-5 for the {0,32,64,128} subset, so the probe is faithful). random 4-expert subsets give **SD(α) ≈ 0.010 per layer (max 0.027, L45)**, and the fixed 4-subset differs from the all-256 Gaussian α by 0.007 on average. per-expert α spans ~1.00–1.26, but averaging 4 already kills most of that.
- **the input distribution is the problem.** Part A feeds iid Gaussian `x = z·s` with the (uninformative, §1.1) `s` proxy, 16 tokens. on the *real* teacher trace (all routed experts, routing-weighted, 510 tokens) the MSE-optimal α per layer looks completely different: early layers want much more gain (L0: 1.33 real vs 1.12 used), deep layers want ~none (L24-L44: 0.97–1.06 real vs 1.08–1.10 used). the real-data γ=0.75 α is on average +0.074 higher than the Gaussian one, and the depth pattern differs.
- consequence (routed-MoE relative residual `‖y_ref − α·y_q‖²/‖y_ref‖²`, pooled over 48 layers):

| | α = 1 | α_used (Gaussian, shipped) | α* fit on cal trace | α* oracle on that data | α_used + shipped ridge |
|---|---|---|---|---|---|
| cal trace (in-sample, 510 tok) | 21.17% | 20.27% | 19.83% | 19.83% | 19.93% |
| **unseen 512-tok dump** (ops14 + cj20 + ja22 + persona1 + cj15) | 24.31% | 24.21% | **23.96%** | 23.81% | 24.13% |

  per-layer (every 4th layer + L47; Δres = relative change of that layer's routed residual):

| layer | α_used (Gauss, 4 exp) | α* real cal | α* real held-out | Δres α_used vs α=1 (cal) | (held-out) | Δres ridge (cal) | (held-out) | rank(Z) | df@λ | corr(log s², log E[x²]) |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.120 | 1.325 | 1.337 | -22.28% | -23.14% | -0.01% | -0.00% | 369 | 5.5 | +0.08 |
| 4 | 1.120 | 1.240 | 1.173 | -17.08% | -8.10% | -2.44% | -0.15% | 510 | 45.0 | -0.50 |
| 8 | 1.065 | 1.107 | 1.108 | -6.49% | -6.04% | -0.04% | -0.01% | 510 | 53.1 | +0.25 |
| 12 | 1.047 | 1.074 | 1.080 | -2.72% | -3.12% | -0.02% | -0.01% | 510 | 31.4 | +0.23 |
| 16 | 1.095 | 1.130 | 1.060 | -4.75% | -0.67% | -0.85% | -0.18% | 510 | 57.0 | +0.12 |
| 20 | 1.053 | 1.045 | 1.040 | -0.94% | -0.66% | -0.20% | -0.02% | 510 | 56.6 | -0.21 |
| 24 | 1.103 | 1.044 | 1.021 | +0.43% | +1.68% | -0.24% | -0.04% | 510 | 51.7 | +0.16 |
| 28 | 1.081 | 1.011 | 1.009 | +1.25% | +1.34% | -0.26% | -0.04% | 510 | 47.5 | -0.01 |
| 32 | 1.095 | 1.064 | 0.988 | -0.87% | +2.91% | -0.76% | -0.07% | 510 | 57.3 | -0.07 |
| 36 | 1.051 | 1.062 | 1.041 | -1.69% | -0.71% | -0.18% | -0.01% | 510 | 43.7 | -0.16 |
| 40 | 1.084 | 1.042 | 0.992 | +0.00% | +2.06% | -0.73% | -0.02% | 510 | 39.1 | -0.26 |
| 44 | 1.082 | 0.996 | 0.968 | +1.89% | +3.00% | -0.18% | -0.01% | 510 | 32.3 | -0.05 |
| 47 | 1.107 | 1.213 | 1.134 | -10.69% | -5.10% | -4.62% | -1.81% | 485 | 14.6 | +0.16 |

  on the unseen dump the shipped α buys only −0.4% relative (vs α=1), and **22/48 layers get worse than α=1** (most of the deep layers 24–44 overshoot, by up to +3.0%). α* fit on the cal trace transfers fine (−1.5% held-out, ~70% of the oracle gain). so: replace the Gaussian Part A with `α_l = ⟨y_ref,y_q⟩/‖y_q‖²` (or the γ-variant if you want norm bias) computed from the real dump you already have; it's one dot product per layer and strictly better in-sample and out-of-sample on MSE.
- caveat i want to be loud about: MSE on the routed sum is not KLD. the shipped α still *might* be better for KLD than the MSE α if norm preservation matters more than direction. that's exactly the ablation to run on the multi-prompt KLD suite (§3), not on 192 tokens.

### 1.3 Part B: ridge on `ffn_down_shexp` with T=510 < D=640

- `G = ZᵀZ` (640×640) from 510 tokens is rank-deficient: measured **rank 369–510** (L0 is 369, L47 485). λ = `ridge_rel · tr(G)/D` = 6× the *mean* eigenvalue, so the effective degrees of freedom `Σσ/(σ+λ)` are only **5–63 of 640**. the solution lies in the row space of the 510 calibration z's and is shrunk hard; then η=0.15 scales it further. so the T<D concern doesn't produce a blow-up — the regularization is doing its job, and the Cholesky has a positive diagonal (λ>0) so it's well-posed.
- cross-validated on the cal trace (float ΔW, before requant), relative change of the shexp-target residual:

| split | ρ=0.25 η=.15/.5/1 | ρ=1 | ρ=6 (shipped η=.15) | ρ=24 |
|---|---|---|---|---|
| in-sample | −15.1 / −43.6 / −67.5% | −9.7 / −28.8 / −47.3% | **−4.2** / −12.9 / −22.4% | −1.9 / −5.9 / −10.5% |
| leave-one-block-out (4 contiguous ~domain blocks) | −1.7 / +0.8 / **+19.8%** | −1.5 / −2.3 / +3.2% | **−1.0** / −2.4 / −2.2% | −0.6 / −1.5 / −1.9% |
| even/odd tokens | −4.1 / −7.0 / +5.2% | −3.3 / −7.7 / −5.7% | −2.0 / −5.3 / −7.2% | −1.1 / −3.1 / −4.8% |

  the in-sample numbers are 4–40× the held-out ones (classic T<D overfit signature), and low-ρ/high-η settings go *positive* (harmful) out of block. the shipped (ρ=6, η=0.15) is on the safe side: all 4 LOBO folds improve (−0.5…−1.6%).
- **requantization eats most of it.** the monotonic IQ4_NL search can only rescale d by 7 multipliers and re-pick nearest indices, so the realized in-sample gain is −1.68% (vs −4.2% float), and on the unseen dump the *actual transplanted tensors* give **−0.36%** (no layer worse than +0.004%). ΔW is 0.03%–5.9% of ‖W‖ per layer.
- verdict: Part B is safe (doesn't overfit into harm) and also ~a no-op. it is not where the 35% KLD gain comes from. either drop it (fewer moving parts, 48 fewer transplanted tensors) or do it properly: thousands of calibration tokens (T ≫ D), quantization-aware fitting for IQ4_NL, and pick (ρ,η) by LOBO, not by in-sample MSE.
- smaller notes: the stdout "MoE MSE before → after" metric is in-sample; `eta_router` is documented but never parsed; the writers file re-emits `ffn_gate_inp` and `ffn_gate_inp_shexp` byte-identical to the original, so "144 transplanted tensors" is really 48 changed + 96 no-op copies (harmless, but the report language overstates it).
- cache hazard: `cal_teacher_yr_q2.bin` is reused if it exists, keyed by nothing. if anyone re-runs `--quantize` with different `gain_gu/gain_d` (i.e. a sweep), Part B silently fits to the *old* Q2 outputs. same for `d_bak.bin` / stage-1 skip in `run_phase3_local_healing.py`. currently consistent (d_bak 17:18 < yr_q2 18:00 < calibrate 18:52), but any sweep must hash the experts file into the cache name.

### 1.4 `window.cpp` `dump_moe_layer`

checked against the prompt graph: at cut `s` the host sees layer s's `mixed`/`ids`/`wts` (router done, experts not yet run) and layer s−1's `yr` (`mm_combine` output = routed sum only; shared expert goes through `ys`/`sgv` into `moe_out`, which uses `sigmoid(sgv)` exactly as the healer models). so `.yr` is routed-only and `dy = y_ref − α·y_q` is the right ridge target. caveats: `mixed` is dumped fp32 while the experts consume `mixh` fp16 (negligible); the file stride is `l·T·N` with `T = seg_T_` of the *current chunk* and layer 0 opens with `"wb"` — a prompt longer than one chunk (1024) silently keeps only the last chunk, and `T_CAL=510` is hard-coded in the healer. fine for the 511-token dump, fragile otherwise.

### 1.5 `stream.cpp`: the `smap_` change and the `new_drop_block` read-ahead eviction

what changed (patch_strata_metal.py §6):
- (a) background I/O completion no longer publishes `smap_` for read-ahead (`x.pre`) slots; `smap_` is written only at claim time in `at_layer` (`if (st == 0) smap_[key] = code_`) or for urgent reads. this removes a real CPU→GPU race: with MoCE on, `moe_route` reads `smap_` to add the residency bonus, so a read-ahead finishing mid-graph could change routing nondeterministically. **this part is correct and good.**
- (b) `new_drop_block`: at every `at_layer(l)`, after cancelling queued-but-unstarted read-aheads, it **blocks on `done_cv_` until every in-flight unused read-ahead of layer l completes, then evicts all of them** (`where_=-1`, `smap_=FFFF`, `lru_back`).

theoretical concerns with (b) (measured right below and in §5: they don't materialize):
1. it puts SSD latency for experts the token *does not use* on the decode critical path (the wait happens under the layer handshake, before the GPU can continue). that's pure `host_wait` cost.
2. it throws away the bytes it just waited for. routing has strong token-to-token locality; a read-ahead that missed token t at layer l is a good candidate for token t+1 at layer l. before the patch those slots stayed resident (and a later use counted as a read-ahead hit, `x.pre` still true); now they never can.
3. the stated reason is determinism of `where_/smap_/LRU`. but after (a), `smap_` (the only thing the GPU sees) is already only set at claim time, so leaving a completed-but-unclaimed pre slot resident is *correctness*-safe: if it's used later it's a hit and gets published at claim; if not, it ages out via LRU. what remains nondeterministic without (b) is only *which* slots get evicted later (LRU order depends on whether a read-ahead finished before the cancel), i.e. residency → MoCE bonus → routing. so (b) buys run-to-run bit-reproducibility *only when MoCE bonus > 0*; at `MOCE_BONUS=0` routing ignores `smap_` and (b) is pure cost.
4. cheaper deterministic alternatives exist: (i) keep completed unclaimed pre slots but give them a deterministic LRU position independent of completion time (e.g. always insert read-aheads at a fixed position at *submit* time, and on cancel leave the slot where it is but mark it empty), so the eviction order is a function of the submit sequence only; or (ii) don't wait — mark the in-flight unused ones "evict on completion" (the I/O thread already holds the slot), which keeps determinism without stalling; or (iii) gate (b) behind `MOCE_BONUS > 0`.

**measured (A/B, §5):** none of 1–2 shows up in the numbers. combined hit rate is identical to 4 decimals on all 9 prompts at both MoCE settings (MoCE .85 mean 0.9327 vs 0.9327, misses/tok 32.29 vs 32.31; MoCE 0 0.8480 vs 0.8480), decode tok/s differences are inside rep-to-rep noise (±7%), and host_wait is not higher with the patch. the reason is in the base code: `submit()` never moves a read-ahead slot off the LRU tail (only a claim calls `lru_front`), and `victim()` scans from the tail, so an unclaimed read-ahead was already the next thing evicted. the patch just makes that eviction explicit and timing-independent. meanwhile the determinism claim **does** hold empirically: with the patch, 9/9 prompts produce identical GEN_IDS across reps at MoCE 0.85; without it, `prompt_43tok_wal` diverges between two runs of the same binary and 2 more prompts (math43, coldshift) settle on a different continuation than the patched build. so my concerns above are theoretical; verdict: **keep it**. the cheap improvement is (iii): skip the wait when `moce_bonus_ == 0` since routing can't see `smap_` then.

## 2. what's actually in the eval files, and what's already burned

i segmented the files the way `scripts/assemble_corpus.py` joins them (`"\n\n".join(docs)`) and located every existing prompt by substring search:

| artifact | source doc(s) |
|---|---|
| `eval_tf_ops_tools.txt` (scored 64 tok) | `eval_ops_tools` doc 13 (InferredBugs versions-maven-plugin, last turn) |
| `eval_tf_chat_json.txt` | `eval_chat_json` doc 11 ("Launch retro" JSON extraction) |
| `eval_tf_ja.txt` | `eval_ja` doc 28 (PNOZmulti) |
| `cal_disjoint_512tok.txt` (healing calibration) | `eval_ops_tools` doc 8 (getDBConfig), `eval_chat_json` doc 4 (JSON extraction, same template as doc 11), `eval_ja` doc 2 (胃), 3 `phase3_sft_seed_examples.jsonl` rows |
| `prompt_1024tok_ops.txt` / `prompt_2048tok_ops.txt` | `eval_ops_tools` docs 8 → 9 (→ 10, 11) with a generic system prompt + "Summarize the root cause" |
| `prompt_43tok_wal.txt` | synthetic, not from any file |

so the calibration trace is drawn from the *same three eval files and the same templates* as the KLD set (the self-test only checks 16-grams on the last 96 tokens). i checked token-level leakage of the 64 scored tokens: 0/64 complete an 8-gram seen in cal for all three domains (ops has 2/64 4-gram hits) — so it's not lexical leakage, but it is distributional proximity, which flatters a calibration-fitted method. the long speed prompt also shares doc 8 with the calibration trace.

the eval files actually hold (by doc, not by "turns"): `eval_ops_tools` = 8 Terminal-Bench task prompts (docs 0–7, user-only, no assistant turn) + 14 ukisai multi-turn docs (8–21; 21 has no assistant turn); `eval_chat_json` = 23 whittle docs (math, chat, code review, JSON extraction ×4, tool-calling ×7 incl. a 5-turn one, Italian); `eval_ja` = 35 hotchpotch QA docs; `calibration_corpus.txt` = 4 persona conversations (2/2/2/5 assistant turns, the actual target voice); `phase3_sft_seed_examples.jsonl` = 6 rows (3 already in the cal trace).

**unused and safe to evaluate on** (not in cal, not in eval_tf, not in speed prompts): ops docs 0–7 (prefill/speed only), 12, 14–20; chat_json all except 4, 11 (and treat 7, 8 as "template-near" controls); ja all except 2, 28; all 4 persona docs. that's ~60 docs / ~80 assistant turns.

## 3. benchmark spec (concrete)

### 3.1 teacher-forced KLD suite v2 (what i ran, §4; reproducible)

- **22 windows × ≤128 scored tokens = 2,686 scored tokens** (14× the current 192), 17 distinct unseen docs, two window types: `think` (window starts at `<|im_start|>assistant\n`, scores the reasoning prefix) and `answer` (starts right after `</think>`, scores the answer body — JSON, tool-call XML, persona prose, japanese answer).
- strata: ops 5 (C# InferredBugs last turn, 2-turn terminal session, terminal task, Java turn-1 mid-conversation, JSON command answer), chat/json/tool 9 (tool-call w/ tools system prompt, 5-turn tool convo at turn 3 + its tool-call answer, PHP review, Italian, Dostoevsky summary, math, JSON extraction think + answer), ja 5 (4 think + 1 answer), persona 3 (2 think, 1 answer; one at turn 4 of 5 with ~1.9k-token context).
- token files + manifest: `/tmp/p3audit/turns/*.txt`, `/tmp/p3audit/turns/manifest.json` (built with the HF `tokenizer.json`, round-trip-verified against `eval_tf_ops_tools.txt`). same CLI flags as `run_phase3_local_healing.py` (`--n-prompt N --n-gen G --logits-out/--logits-ref/--logits-ref2`).
- configs: IQ3_S MoCE0 (ref), unhealed Q2_0 MoCE0/0.85, healed Q2_0 MoCE0/0.85.
- report: token-pooled mean KLD, per-stratum macro, top-1 agree, and the **per-window paired win count** (healed vs unhealed). decision rule i'd adopt: a healing variant is accepted only if (i) pooled KLD improves with a paired bootstrap 95% CI over windows excluding 0, (ii) no stratum regresses by > 10% relative, (iii) top-1 non-inferior (Δ ≥ −1.0 pt pooled).

### 3.2 hyperparameter discipline

- split the ~60 unused docs by doc index parity: **odd = dev** (tune γ, gain_gu, gain_d, η, ρ, MoCE bonus), **even = test** (touch once per milestone). never tune on eval_tf_* again; keep them only as a regression canary.
- calibration for α/ridge: dump from `calibration_mix_v2.txt` (the Phase 1a corpus, disjoint by construction from all eval docs), ≥ 4 × 511-token chunks (T ≥ 2,040 > D=640) once the dump supports multi-chunk (§1.4).
- the cheapest high-value ablation: {shipped α, real-trace MSE α, α=1} × {ridge on, off} × {gain_d 0.3, 0} = 12 configs × the 22-window suite ≈ 2.5 h on the Air. that tells you which of the five knobs earns its keep.

### 3.3 speed / hit-rate suite

single-prompt CLI runs, `--n-gen 128` free decode, MoCE 0.85 (and 0 as control), 2 reps, A/B interleaved, files in `/tmp/p3audit/speed/`:

| id | prompt | n_prompt | what it stresses |
|---|---|---|---|
| wal43 | existing `prompt_43tok_wal` | 43 | cold short prompt (baseline) |
| ops1024 | existing `prompt_1024tok_ops` | 1024 | long code prompt (baseline) |
| math43 | chat_json doc 17 | 43 | cold short, math reasoning |
| json163 | chat_json doc 7 | 163 | schema/JSON |
| ja398 | ja doc 22 | 398 | japanese (different expert population) |
| persona458 | persona doc 0, turn 2 | 458 | target persona voice |
| tool588 | chat_json doc 20, turn 3 | 588 | multi-turn tool use |
| ops713 | ops doc 14 | 713 | multi-turn debugging (C#) |
| coldshift668 | ja doc 12 + "write a rust ISO-8601 duration parser" | 668 | in-context domain shift ja → code |

metrics: combined hit rate, misses/tok, MB read/tok, late read-aheads, host_wait ms/tok, decode tok/s, prefill tok/s, and GEN_IDS equality across reps (determinism). gaps this suite can't close: multi-turn *sessions* (the `hot` slot protection in `hold_region` only matters across turns in one process; the CLI is one prompt per process) — that needs a server-mode harness replaying 3–5 turns of persona doc 3 / chat_json doc 20 in one process.

## 4. unseen-turn falsifier: teacher-forced KLD (22 windows, 2,686 scored tokens)

all runs: `vendor/strata/build-metal/strata_window_cli` (the shipped build, eviction patch in), same env as `run_phase3_local_healing.py`, reference = IQ3_S MoCE 0, K256 healed = `K256-q2_0-healed-metal.gguf`, unhealed = `K256-q2_0-metal.gguf`. raw per-run JSON: `/tmp/p3audit/kld_results.json`.

### 4.1 headline vs the 192-token set

| metric | 192-tok eval_tf set (shipped report) | unseen suite, pooled (95% CI by window bootstrap) |
|---|---|---|
| KLD reduction, MoCE 0 (healed vs unhealed) | −35.05% | **−24.7%** [−28.7, −19.2]; healed wins 21/22 windows |
| KLD reduction, MoCE 0.85 | −37.46% | **−24.5%** [−28.8, −19.8]; wins 20/22 |
| isolated MoCE KLD (m.85 vs own m0) reduction | −53.48% | **−24.3%** [−37.4, −8.6]; wins 15/22 |
| absolute KLD unhealed → healed, MoCE 0 | 0.2032 → 0.1320 | 0.2029 → 0.1527 |
| absolute KLD unhealed → healed, MoCE 0.85 | 0.2182 → 0.1365 | 0.2123 → 0.1602 |
| top-1 agree unhealed → healed, MoCE 0 | 90.63 → 88.02 (−2.6) | 85.22 → 87.75 (**+2.53** [+1.36, +3.83]; 16 up / 3 down) |
| top-1 agree, MoCE 0.85 | 91.67 → 89.06 (−2.6) | 85.55 → 87.60 (**+2.05** [+0.56, +3.66]; 14 up / 6 down) |

reading: **the falsifier is not triggered** — healing generalizes to unseen docs in every stratum, and the top-1 regression on the 192-token set was noise (it flips sign with 14× more tokens). but the 192-token set overstated the size of the win by ~10 points (35% → 25%), and overstated the isolated-MoCE improvement by more than 2× (53% → 24%, with a wide CI). treat −25% as the number to quote.

### 4.2 per stratum (token-pooled)

| stratum | tokens | KLD unhealed → healed (m0) | top-1 (m0) | KLD unhealed → healed (m.85) | top-1 (m.85) |
|---|---|---|---|---|---|
| ops | 640 | 0.1650 → 0.1199 (-27.3%) | 89.7 → 90.9 | 0.1589 → 0.1175 (-26.1%) | 91.4 → 90.0 |
| chat_json | 1128 | 0.1554 → 0.1243 (-20.0%) | 89.5 → 90.4 | 0.1702 → 0.1341 (-21.2%) | 89.5 → 90.2 |
| ja | 534 | 0.2374 → 0.1723 (-27.4%) | 80.2 → 85.0 | 0.2513 → 0.1887 (-24.9%) | 79.6 → 84.1 |
| persona | 384 | 0.3576 → 0.2637 (-26.3%) | 72.1 → 78.4 | 0.3702 → 0.2683 (-27.5%) | 72.7 → 80.7 |

persona (the actual target voice, not represented in the 192-token set at all) has the *highest* absolute KLD of any stratum (0.36 unhealed, 0.26 healed) and the lowest top-1 (72–80%). japanese is second. that's where the remaining Q2_0 damage lives, and it's invisible to the current eval.

### 4.3 per window

| window | stratum | n_prompt | n_scored | IQ3_S ppl | KLD unhealed m0 | KLD healed m0 | Δ | top1 u→h (m0) | KLD unhealed m.85 | KLD healed m.85 | top1 u→h (m.85) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ops14_csharp_bug | ops | 713 | 128 | 2.6873 | 0.0797 | 0.0767 | -3.8% | 89.8→92.2 | 0.0776 | 0.0834 | 93.0→92.2 |
| ops17_term_multiturn | ops | 1186 | 128 | 1.5712 | 0.0627 | 0.0331 | -47.3% | 95.3→95.3 | 0.0595 | 0.0437 | 96.9→93.8 |
| ops19_term | ops | 489 | 128 | 1.4673 | 0.0581 | 0.0513 | -11.7% | 93.0→91.4 | 0.0603 | 0.0332 | 93.8→95.3 |
| ops12_java_turn1 | ops | 244 | 128 | 3.9154 | 0.3805 | 0.2778 | -27.0% | 79.7→82.8 | 0.3384 | 0.2758 | 80.5→78.1 |
| cj03_toolcall | chat_json | 951 | 128 | 1.0482 | 0.0978 | 0.0631 | -35.5% | 96.1→98.4 | 0.1349 | 0.0780 | 95.3→98.4 |
| cj20_tool_5turn_t3 | chat_json | 588 | 128 | 1.2628 | 0.1440 | 0.1783 | +23.8% | 91.4→89.8 | 0.1534 | 0.1363 | 91.4→92.2 |
| cj15_php_review | chat_json | 1153 | 128 | 3.0428 | 0.0947 | 0.0703 | -25.8% | 85.9→89.1 | 0.0959 | 0.0908 | 86.7→89.1 |
| cj19_italian | chat_json | 27 | 128 | 2.7238 | 0.1779 | 0.1493 | -16.1% | 82.8→81.2 | 0.1982 | 0.1839 | 82.0→81.2 |
| cj06_dostoevsky | chat_json | 26 | 128 | 5.5487 | 0.4949 | 0.3406 | -31.2% | 78.9→82.0 | 0.5289 | 0.3677 | 78.1→80.5 |
| cj07_json_extract_near | chat_json | 163 | 128 | 1.2885 | 0.0420 | 0.0348 | -17.0% | 95.3→95.3 | 0.0442 | 0.0322 | 96.1→93.8 |
| cj17_math | chat_json | 43 | 128 | 3.2414 | 0.2147 | 0.1663 | -22.6% | 82.0→83.6 | 0.2379 | 0.1839 | 81.2→82.0 |
| ja05 | ja | 154 | 128 | 5.2946 | 0.2284 | 0.1627 | -28.8% | 79.7→82.8 | 0.2378 | 0.1666 | 82.0→85.2 |
| ja12_table | ja | 370 | 128 | 5.2017 | 0.2305 | 0.1847 | -19.9% | 77.3→85.9 | 0.2354 | 0.2060 | 76.6→83.6 |
| ja22_pedal | ja | 398 | 128 | 7.9757 | 0.2965 | 0.1856 | -37.4% | 80.5→85.2 | 0.3175 | 0.2114 | 76.6→82.0 |
| ja30_climate | ja | 568 | 128 | 7.8123 | 0.2173 | 0.1750 | -19.5% | 82.0→85.2 | 0.2397 | 0.1880 | 82.0→85.2 |
| persona0_t2 | persona | 458 | 128 | 6.2793 | 0.4100 | 0.2858 | -30.3% | 68.8→76.6 | 0.4163 | 0.2945 | 70.3→82.0 |
| persona3_t4 | persona | 1861 | 128 | 14.6586 | 0.3872 | 0.2856 | -26.3% | 73.4→76.6 | 0.4340 | 0.3357 | 68.8→74.2 |
| cj07_answer_json | chat_json | 697 | 104 | 1.0017 | 0.0006 | 0.0001 | -85.4% | 100.0→100.0 | 0.0019 | 0.0001 | 100.0→100.0 |
| cj20_t3_answer_tool | chat_json | 629 | 128 | 1.1132 | 0.1030 | 0.0927 | -10.1% | 95.3→96.1 | 0.1053 | 0.1090 | 96.1→96.9 |
| ja05_answer | ja | 632 | 22 | 1.9475 | 0.1030 | 0.0625 | -39.4% | 86.4→90.9 | 0.1058 | 0.0885 | 86.4→86.4 |
| persona0_t2_answer | persona | 531 | 128 | 2.7925 | 0.2755 | 0.2197 | -20.3% | 74.2→82.0 | 0.2603 | 0.1746 | 78.9→85.9 |
| ops19_answer_json | ops | 644 | 128 | 1.79 | 0.2441 | 0.1606 | -34.2% | 90.6→93.0 | 0.2587 | 0.1515 | 93.0→90.6 |

the one consistent loser is **multi-turn tool calling** (`cj20_tool_5turn_t3`: +23.8% KLD at MoCE 0; its answer window and `ops14` lose slightly at MoCE 0.85). the calibration trace has no `<tool_call>`/`<tools>` content at all; that's the first thing to add to the dump.

## 5. unseen-prompt falsifier: `stream.cpp` read-ahead eviction A/B

- **new** = shipped `vendor/strata/build-metal/strata_window_cli` (with `new_drop_block`); **old** = identical source in `/tmp/strata_noevict` with only `new_drop_block` reverted to `old_drop_block` (everything else, including the `smap_` claim-time publishing, kept). built out-of-tree, repo untouched.
- healed Q2_0, `--n-gen 128` free decode, 9 prompts (§3.3), MoCE 0.85 × 2 reps + MoCE 0 × 1 rep, new/old interleaved with alternating order. raw: `/tmp/p3audit/speed_results.json`.

| prompt | MoCE | combined hit new / old | misses/tok new / old | MB/tok new / old | decode tok/s new / old | host_wait ms/tok new / old | late RA new / old |
|---|---|---|---|---|---|---|---|
| ja_pedal_398 | 0.85 | 0.9276 / 0.9276 | 34.7 / 34.8 | 119.5 / 121.0 | 10.34 / 10.09 | 15.63 / 15.90 | 158 / 254 |
| ja_pedal_398 | 0.0 | 0.8422 / 0.8422 | 75.7 / 75.7 | 276.9 / 276.9 | 8.59 / 8.57 | 28.79 / 29.25 | 446 / 625 |
| json_extract_163 | 0.85 | 0.9560 / 0.9556 | 21.1 / 21.2 | 79.4 / 79.6 | 9.41 / 9.86 | 10.48 / 10.50 | 33 / 40 |
| json_extract_163 | 0.0 | 0.8663 / 0.8663 | 64.2 / 64.2 | 228.3 / 228.3 | 8.92 / 9.15 | 25.01 / 25.79 | 236 / 504 |
| tool_multiturn_588 | 0.85 | 0.9098 / 0.9098 | 43.3 / 43.3 | 140.8 / 140.8 | 9.79 / 9.72 | 19.06 / 21.84 | 186 / 520 |
| tool_multiturn_588 | 0.0 | 0.8290 / 0.8290 | 82.1 / 82.1 | 297.2 / 297.2 | 8.53 / 8.66 | 30.80 / 31.33 | 638 / 1219 |
| persona_458 | 0.85 | 0.9304 / 0.9304 | 33.4 / 33.3 | 109.8 / 109.2 | 10.00 / 10.31 | 16.45 / 15.58 | 249 / 285 |
| persona_458 | 0.0 | 0.8453 / 0.8452 | 74.3 / 74.3 | 266.9 / 266.9 | 8.77 / 8.84 | 28.64 / 28.97 | 587 / 1112 |
| math_43 | 0.85 | 0.9443 / 0.9446 | 26.8 / 26.6 | 90.0 / 90.0 | 9.97 / 10.15 | 12.59 / 12.68 | 36 / 70 |
| math_43 | 0.0 | 0.8454 / 0.8454 | 74.2 / 74.2 | 255.7 / 255.4 | 8.09 / 8.07 | 31.83 / 35.65 | 491 / 1047 |
| ops_csharp_713 | 0.85 | 0.9249 / 0.9249 | 36.0 / 36.0 | 123.0 / 123.0 | 10.07 / 10.75 | 16.70 / 16.77 | 211 / 329 |
| ops_csharp_713 | 0.0 | 0.8409 / 0.8409 | 76.4 / 76.4 | 280.9 / 280.6 | 8.89 / 7.54 | 29.22 / 42.35 | 780 / 1889 |
| coldshift_ja2rust | 0.85 | 0.9394 / 0.9400 | 29.1 / 28.8 | 98.4 / 96.4 | 10.19 / 9.87 | 13.96 / 13.57 | 192 / 194 |
| coldshift_ja2rust | 0.0 | 0.8690 / 0.8690 | 62.9 / 62.9 | 227.4 / 227.5 | 9.13 / 9.38 | 24.70 / 25.47 | 527 / 896 |
| prompt_43tok_wal | 0.85 | 0.9313 / 0.9302 | 33.0 / 33.5 | 109.3 / 110.5 | 9.71 / 10.16 | 16.80 / 15.04 | 143 / 114 |
| prompt_43tok_wal | 0.0 | 0.8494 / 0.8493 | 72.3 / 72.3 | 264.1 / 263.8 | 8.55 / 8.60 | 27.00 / 27.77 | 258 / 529 |
| prompt_1024tok_ops | 0.85 | 0.9309 / 0.9309 | 33.2 / 33.2 | 112.9 / 112.9 | 9.82 / 10.49 | 17.38 / 15.74 | 257 / 212 |
| prompt_1024tok_ops | 0.0 | 0.8448 / 0.8448 | 74.5 / 74.5 | 273.1 / 273.0 | 8.52 / 8.33 | 28.68 / 28.87 | 546 / 715 |

means over the 9 prompts — MoCE 0.85: hit 0.9327 / 0.9327, misses/tok 32.29 / 32.31, MB/tok 109.2 / 109.3, decode 9.92 / 10.16 tok/s, host_wait 15.45 / 15.29 ms; MoCE 0: hit 0.8480 / 0.8480, decode 8.67 / 8.57, host_wait 28.30 / 30.61 ms.

- **hit rate: falsifier not triggered.** zero measurable hit-rate or miss cost on any prompt type (japanese, JSON, multi-turn tool, persona, math, in-context ja→rust shift, long ops).
- **decode speed: no signal above noise.** same-binary reps differ by up to ±7% (e.g. json163 new 8.78 vs 10.05), so ±2% means are noise.
- **determinism: verified for the patched build** (9/9 prompts bit-identical GEN_IDS across reps at MoCE 0.85); the unpatched build diverged run-to-run on `prompt_43tok_wal`.
- **generalization of the speed numbers themselves:** MoCE-0.85 hit rate across the 9 prompts spans **0.910 (multi-turn tool) – 0.956 (JSON)**, misses/tok 21–43; MoCE 0 spans 0.829–0.869. so the two shipped speed prompts (0.913 / 0.931) sit in the middle of the range and aren't cherry-picked, but tool-use prompts are the worst case and should be in the headline.
- **measurement caveat:** my absolute decode rates (≈10 tok/s at 1024 tokens, pure GPU 89 ms/tok) are well below the shipped report's 13.8 tok/s (57 ms/tok) for the *same prompt and the same hit rate*. this ran after ~1.5 h of continuous load on a fanless Air, so it's thermal/DVFS. even in the shipped report, the 43-token run (80 ms GPU/tok) vs the 1024-token run (57 ms) gap is probably GPU clock ramp-up after a long prefill, not a property of the prompt. report `pure_gpu_ms_per_tok` next to tok/s, cool down between runs, interleave A/B, and use medians over the 9-prompt suite.

## 6. recommendations, in priority order

1. **keep the healed Q2_0 and the stream.cpp change** — both survive the falsifier on unseen docs/prompts. quote healing as **≈ −25% KLD, +2–2.5 pt top-1** (CI in §4.1), not −35%/−53%.
2. **stop tuning on `eval_tf_*` and the two speed prompts.** use the §3 suites: tune on odd-indexed unused docs, confirm on even. every knob (γ, gain_gu, gain_d, η, ρ, MoCE bonus) was picked on the 192-token set; none has been ablated out-of-sample.
3. **replace Part A's Gaussian α with the real-trace MSE α** `⟨y_ref,y_q⟩/‖y_q‖²` per layer (or its γ-variant). it's one dot product on data already on disk, better in-sample and out-of-sample on MSE (§1.2). then check with the KLD suite whether norm bias (γ) helps KLD at all.
4. **fix the AWQ importance**: `E[mixed_d²]` from the dump instead of the HC-norm-gain proxy (corr ≈ 0 measured), and add input importance for W_down. this is the most likely place for further real gains.
5. **Part B ridge: drop it or redo it** with T ≫ D (multi-chunk dump from `calibration_mix_v2.txt`), (ρ, η) chosen by leave-block-out, and quantization-aware fitting. as shipped it's safe but worth −0.36% of routed residual out of sample.
6. **add tool-calling and persona text to the calibration dump** — the only regressing window is multi-turn tool calling, and persona/ja carry the highest residual KLD.
7. hygiene: key `cal_teacher_yr_q2.bin` and `d_bak.bin` by a hash of the experts file and gains; add error checks to every open/pread in the healer; make `dump_moe_layer` multi-chunk safe; fix the self-test docstring (says 32-gram, checks 16-gram on 96 tokens) and broaden it to the whole eval doc set.
8. stream.cpp: optional — skip the wait in `new_drop_block` when `moce_bonus_ == 0`.

## 7. artifacts (all outside the repo)

- probe source/binary: `/tmp/p3audit/probe.cpp`, `/tmp/p3audit/probe` (links `vendor/strata/build-metal/libmtl_rt.a`; reproduces shipped α to 1.3e-5)
- per-layer probe outputs: `/tmp/p3audit/eval_cal.tsv` (+`.cv`), `/tmp/p3audit/eval_heldout.tsv` (+`.cv`), `/tmp/p3audit/experts.tsv` (48×256 per-expert Gaussian + real stats)
- unseen 512-token teacher dump: `/tmp/p3audit/heldout_iq3s_512.*`, token file `/tmp/p3audit/heldout_512tok.txt`, its Q2 routed output `/tmp/p3audit/heldout_yq2.bin`
- KLD suite token files + manifest: `/tmp/p3audit/turns/`; results `/tmp/p3audit/kld_results.json`
- speed suite token files: `/tmp/p3audit/speed/`; results `/tmp/p3audit/speed_results.json`
- no-eviction build: `/tmp/strata_noevict/build/strata_window_cli`
- logits files deleted after scoring (~8 GB).