# opus audit v2: verification of the 6 fixes from `audits/opus_audit_phase0_2_m5.md` §4

scope: read-only re-audit of `PLAN.md` §2.7.A #29–#30 and §7.1 item 5, `results/phase0_phase2_m5_validation.json`, `scripts/run_phase0_phase2_m5_validation.py`, `scripts/patch_strata_metal.py`, `data/m5_val_prompts/*.txt`, the patched `vendor/strata/metal` tree and `vendor/strata/build-metal/{route_parity,strata_window_cli}`. the only things i ran: `route_parity` (3×), `run_phase0_phase2_m5_validation.py --self-test`, and a GGUF header parse of `K256-q2_0-metal.gguf`. no repo files were edited.

**verdict: MOSTLY FIXED, falsifier hit (PLAN only).**

- every number in `summary_comparison` recomputes exactly from the archived raw tails: 0 mismatches across 14 strata runs, 3×3 KLD blocks, and 2 llama blocks.
- the provenance chain is now real: the hashes match, the cmd and env are archived, and the prompt files are archived and hashed.
- the MoCE race is fixed and `ahead()` mirrors the top-3 protection.
- **but PLAN §7.1 #5 / §2.7.A still quote 12 numbers that are in no archived run.** they look like they came from an earlier run: arena/read-ahead hit splits, host-wait/GPU ms, a fill rate, a slot count, the run count, and the `route_parity` errors.
- two hard-coded or stale log strings remain in the prompt-scratch path (§5).

---

## 1. PLAN / `summary_comparison` vs raw runs

method: i re-parsed every `output_tail` myself (`STRATA_RESULT`, `expert stream`, `window T1`, `expert arena`, `process footprint`, `STRATA_KLD`, `STRATA_KLD_REF2`, llama `common_perf_print` / `[ Prompt | Generation ]`). then i recomputed `perf`, `summary_comparison` and the PLAN strings from that.

### 1a. `summary_comparison`: PASS (0 mismatches)

- all 5 speed runs and 9 KLD runs: the `perf` dicts equal the tails, including the derived fields (`combined = hit + ra`, `pure_gpu = T1 GPU − host_ms`, `misses/tok = miss_rate × 480`).
- `strata_metal_iq3s_moce0_43tok`, `strata_metal_q2_0_moce0_43tok` and all 30 fields of `strata_metal_optimized_k256_q2_0_moce085` equal the source runs.
- miss reductions: `1 − 40.1/83.2 = 51.80%` ✓ and `1 − 40.1/108.9 = 63.18%` ✓.
- llama: `105,851.73/65 = 1628.49` ✓, `33,428.34/65 = 514.28` ✓, `46,808.58` ✓, `0.28 tok/s` (`--moe-cache-mib 1536`, `221,796.57 ms / 63`) ✓, `3.22` ✓, `14.3/21.7/1.4` ✓, `38.11/63.27` on `1264` tokens ✓. the relabelling to "1,264-tok" and "whole-run disk0 / 65 tok" is correct.
- `0.0 MiB` swapout in all 22 runs ✓. `min_avail` is 2.85–5.50 GiB, against the 2.2 GiB fuse.

### 1b. PLAN text: FAIL, 12 numbers have no raw source

| PLAN claim | where | archived raw | status |
|---|---|---|---|
| 43-tok MOCE=0.85: `84.22%` arena + `7.43%` RA | §7.1 | **82.53% + 9.12%** (sum 91.65 ✓) | ✗ |
| 43-tok: `18.69 ms` wait + `55.07 ms` pure GPU | §7.1 | **16.01 ms + 59.57 ms** | ✗ |
| 1024-tok: `87.30%` + `5.49%` | §7.1 | **84.23% + 8.56%** (sum 92.79 ✓) | ✗ |
| 1024-tok: `16.81 ms` wait + `53.68 ms` GPU | §7.1 | **14.11 ms + 58.52 ms** | ✗ |
| 2048-tok: `89.38%` + `4.65%` | §7.1 | **86.81% + 7.22%** (sum 94.03 ✓) | ✗ |
| arena fill `6.36–6.85 GB/s` | §7.1 | Q2_0: **3.06** (moce0 speed run), 6.36–6.43. IQ3_S: 5.49, 6.90–7.07 | ✗ (6.85 is in no run) |
| IQ3_S `1,397` slots across 3 classes in a `3.40 GB` arena | #30 | **1550 slots** (1440 filled), 3 classes, 3.40 GB | ✗ |
| "all **21** archived runs", "0.0 MiB across all 21 runs" | §7.1 | **22** runs (5 + 9 + 4 + 4) | ✗ count |
| `route_parity` `2.15e-07` (K=512) / `4.00e-07` (K=256+MoCE) | #30 | **2.16e-07 / 3.06e-07**, deterministic over 3 runs (seeded `Rng(9)`) | ✗ |
| upstream `PromptGemm` wiped `644–976` slots (`0.90–1.36 GB`) | #30 | no upstream-region run archived | ✗ unsupported |
| init `0.34–0.36 s (345–359 ms)` | §7.1 | moce085 speed runs 344.67–359.27 ✓. the same bullet's slot range `2,441–2,513` mixes in the KLD runs, whose init is 346–433 ms | ⚠ inconsistent scope |

everything else in #29, #30 and §7.1 #5 matches:
- decode `7.22/9.75/12.97/13.46/12.44` tok/s, the ms/tok, combined hit rates, misses/tok and MB/tok.
- prefill `319.39` (3.21 s) / `342.87` (5.97 s).
- `2.90/0.34/1.28/4.53 GB`. i rechecked these against the GGUF header: dense non-`hc` = 2.904 GB, `token_embd` = 0.338 GB, 290 BF16 `hc_*` = 1.279 GB, all dense = 4.521 GB.
- footprint `4.83 -> 4.77`.
- all KLD/top-1 macros and per-domain values (§3).

**fix:** replace the 12 cells with the raw values above, or archive the runs they came from.

## 2. provenance: PASS, with notes

- all 5 SHA-256s in `provenance` match the files on disk: runner, `patch_strata_metal.py`, `patch_m5_lazy_moe.py`, `prepare_strata_k256.py`, `strata_window_cli`. all 6 prompt-file hashes match too. token counts: 43, 1024 and 2048 for the speed prompts, 96 for each of the three `eval_tf_96tok_*` files.
- mtime order:
  - prompts 07:25:22
  - `patch_strata_metal.py` 07:34:49.890
  - patched vendor files 07:34:49.98
  - `libstrata_metal_kernels.a` 07:34:53.54
  - `strata_window_cli` 07:34:53.60
  - runner 07:36:34.34
  - JSON 07:38:52.72 (`generated_at_utc` 04:38:52.716Z = same instant, UTC+3)

  that is consistent with build → run → write. **tight:** the runs sum to 137.89 s, plus about 0.6 s of `vm_stat`/`iostat` overhead (measured at 19 ms per pair), which puts launch at about 07:36:34.1 ± 0.3 s. so the runner was saved roughly at the moment it was launched. this can't be resolved further. the schema, env keys and summary keys exactly match the code that is in the tree.
- the metal kernels are compiled at run time from `vendor/strata/metal/kernels` (`STRATA_METAL_KERNELS_DEFAULT`), so the benchmark used the 07:34:49 shaders.
- the `cmd`/`env` per strata run match the runner's construction. note that IQ3_S uses `ARENA_GB=3.4` but the Q2_0 speed runs use 3.5 (1550 vs 2513 slots), so the "−63.2% vs IQ3_S" figure mixes format, slot count and MoCE.
- **the llama runs were not re-run.** the runner copies them from the previous JSON. they have `env: null`, and `patch_m5_lazy_moe.py` (05:22) predates them only as far as anyone can tell. the llama blocks of `summary_comparison` are **hard-coded literals** in the runner (lines 647–664), not computed. they happen to equal the raw. `/tmp/m5_prompt_1600tok.txt`, the llama prompt, is not archived.
- `route_parity` binary 07:23:30 vs `route_parity.cpp` 07:34:49: the source mtime moved because of the patch's `git checkout -- .` rewrite. the binary's format strings equal the current source, and the kernel under test is compiled at run time from the current `moe.metal`.
- **`evict_file_cache()` is a silent no-op.** `(c_char*1).from_buffer()` on a `PROT_READ` mmap raises `TypeError: underlying buffer is not writable` (reproduced on a /tmp probe), and `except Exception: pass` swallows it. the effect: `strata_q2_0_moce0_short_43tok` ran with a cold Q2_0 dense file (init 1038 ms, fill 3.06 GB/s), while the MOCE=0.85 runs ran warm. misses/tok is a count, so the −51.8% holds, but the tok/s pair isn't like for like. n = 1 everywhere.
- `--self-test`: `self-test: PASS` (exit 0).

## 3. quality measurement: PASS on the arithmetic, the caveats matter

- `strata_window_cli.cpp` teacher-forcing logic is correct.
  - positions: logits after the last prompt token predict `all_ids[n_prompt]`. each decode step feeds `all_ids[n_prompt+i]` and scores `all_ids[n_prompt+i+1]`. that's 1 + 63 = 64 positions = `n_gen` per domain, 192 in total.
  - the FP64 log-sum-exp and `Σ p·(log p − log q)` run over the full `n_vocab`, which the header gives as `token_embd [2560, 248320]` ✓.
  - `--logits-ref2` is scored against the same candidate logits ✓.
  - PPL = `exp(mean NLL)` ✓.
  - the moce085 run is TF too, so its arena state follows the same tokens as the reference.
- every per-domain value and macro recomputes exactly from the tails:
  - `Q2_0 m0 vs IQ3_S m0` 0.56976 / 72.40%
  - `Q2_0 m0.85 vs Q2_0 m0` 0.056163 / 92.71%
  - `Q2_0 m0.85 vs IQ3_S m0` 0.589351 / 71.36%
- caveats that PLAN doesn't state:
  1. **the TF text is a user turn.** every eval file starts `248045 846 198` (`<|im_start|>user\n`). it is not assistant output, so this measures prompt-text modelling, not the response distribution.
  2. the sample is n = 64 tokens × 1 document per domain. no CI.
  3. **the `ja` PPL goes `53.87 -> 406.29` (7.5×) under Q2_0.** PLAN quotes only KLD 0.96 and top-1 59%. the PPL blowup is the louder signal, and it belongs in §7.1.
  4. `chat_json` PPL *improves* (52.0 → 30.6) at KLD 0.18. with 64 tokens, PPL is noise-dominated.
  5. the reference is IQ3_S K256 (itself pruned and quantized), so these are increments, not distance to unpruned.
  6. the KLD runs use a 3.4 GB arena (2441 slots), the speed runs 3.5 GB (2513), so MoCE's routing-change rate differs slightly between the quality and speed numbers.
  7. `p90` takes index `floor(0.9·63)` = p≈89, and the median is the upper median. both are minor.

## 4. MoCE `smap` race, `ahead()`, `route_parity`: PASS

- `moe.metal` `moe_route` and `router.metal` `route_body`:
  - `smap[idx]` is read **once**, at `rank == 3`, and the result goes into a per-lane `ushort scaled_mask`.
  - the restoration for picks at rank ≥ 3 uses `simd_broadcast(scaled_mask, ex & 31)` and bit `ex >> 5`, with no second `smap` read. so an I/O-thread flip can no longer desync scale and unscale. the race is fixed ✓.
  - ranks 0..2 are picked from unscaled `v` ✓. lanes ≥ `n_expert` are `-INF` ✓.
- the decode path actually used is `moe_route`: `routef_` needs `STRATA_METAL_ROUTE=1`, and `ffn_gate_inp` is F32. that's the kernel `route_parity` covers. `router_route` carries the identical body but has no parity test.
- `stream.cpp` `ahead()`: the first `min(3,K)` come from raw `lg` (unperturbed), and the rest are ranked by `eff_lg = lg + bonus·[smap resident]`. it now reads `smap` (completed reads), not `where_`. top-3 get prio `+1e3` so they're submitted first ✓.
  - nit: `eff_lg` re-reads `sm[e]` racily during the selection, which is harmless for a prefetch heuristic.
- `route_parity`, 3 runs, identical output each time:
  - `K=512 (pad=0, 200 windows): mismatches 0, worst weight rel err 2.16e-07`
  - `K=256 + MoCE=0.85 (random smap + 256..511 traps, 200 windows): mismatches 0, worst weight rel err 3.06e-07 PASS`

  limits: smap is static per dispatch, so the test can't exercise the race, and the race fix rests on code review. `route_topk` (prompt, K=256) and `router_route` aren't covered.

## 5. separate `PromptGemm` scratch: logic PASS; hard-coded/stale strings FAIL

- ✓ `region()` (`SEPARATE_PROMPT_BUF=1`): fresh `MAP_ANON` with `region_bytes_ = 0`, so `hold_region(true)`'s `memset` is skipped. there's no memset on fresh pages.
- ✓ `prompt_region(false)` order:
  1. `proc_mem()` (`task_vm_info.phys_footprint`)
  2. `cx_->free` on every `all_bufs` wrap
  3. `delete pg_`
  4. `release_region()` → `munmap`
  5. `purge_device_allocations`
  6. `proc_mem()` and the log
- ✓ `load_dense`:
  - `locked_bytes` sums the `mlock` successes (page-rounded), and `unpinned_embd_bytes` / `skipped_hc_bytes` sum `t.bytes`. all three are computed, and they match the header.
  - the `"dropped 1.60 GB"` literal is gone.
  - the ≈0.34 GB saving vs upstream is confirmed. `git show HEAD:` `load_dense` skips `hc_*` and anon-copies the remaining 3.241 GB. patched locks 2.904 GB, so 3.241 − 2.904 = 0.338 GB = `token_embd`.
- ✗ **hard-coded plan value:** `window.cpp` `const double pg_scratch_est = (pg_on && sep_pg) ? 1.36 * gb : 0.0;` is a literal (`patch_strata_metal.py` L888). the real `need` depends on `PROMPT_CHUNK`, `max_cells` (`scores = MT·MB`) and `ple_on_`. also, it's only *printed* ("6.6 GB peak prefill …") plus a "above cap" suffix. it doesn't enter `budget`, `footprint_plan` or the cap check, so the scratch still isn't *counted* in the plan. use the measured `need`, and feed it into the cap check.
- ✗ **stale log string:** `prompt_gemm_init` still prints `"…1.36 GB of buffers, in the expert arena while a prompt runs"` (patch L984 replaces it with itself). with the separate buffer that's false, and it appears in every archived tail.
- ⚠ tautological "measurements":
  - `"0 arena slots stolen"` = `region_slots_.size()` right after `region_slots_.clear()`.
  - `"0.00 GB anon"` = the `cx.allocated()` delta across a loop that only wraps with `count_allocated=false`.

  both are computed, but they can't be anything except 0.
- ⚠ **the logged reclaim doesn't show 1.36 GB.** the 1024- and 2048-tok runs show `4.83 -> 4.77 GB` (0.06 GB). the 43-tok and 32-tok runs show `4.55 -> 4.55` / `4.67 -> 4.67` (0.00). footprint after the fill was 4.7 GB, so prefill added only about 0.13 GB. either most of the 1.36 GB scratch is never resident, which would make "6.6 GB peak prefill" an overestimate, or `phys_footprint` misses GPU-touched no-copy pages. PLAN's "`munmap`ped immediately after prefill (`4.83 -> 4.77`)" is literally true. it just isn't evidence of a 1.36 GB peak or reclaim.
- ⚠ `mtp_src_ = nullptr` on release. that's fine with MTP off (these runs), but an MTP decode after a prompt would lose its source. the "tight" fallback in `region()` mmaps without setting `sep_region_bytes_`, so it's never unmapped (`SEPARATE_PROMPT_BUF=0` small-arena path only). the comments in `stream.cpp` L799 and `stream.h` L67 ("inside the arena", "places them (once)") are stale.

## 6. router GEMM over `n_exp_` + multi-turn `slot.hot`: PASS

- `prompt_chain.inc` L275 and `window.cpp` L2359 run `gemm(…, Wr, logits, T, n_exp_, N)`. `route_topk` gets `T | n_exp_<<16`, and `prompt.metal` indexes `logits + ulong(t) * ulong(n_exp)`, masking `idx >= n_exp`. that's consistent with the GEMM's `Nn = n_exp_` row stride. `PgPlan`/`PcMmPlanArgs` and `mm_scan` use `n_exp_` ✓. the buffers are still sized `NEXP` (over-allocation, harmless).
- `hold_region(true)` now re-marks `hot` on the first `n − (ne_+32)` slots of each class's LRU list (MRU first) on **every** prompt. `hold_region(false)` no longer clears the flags, and `victim()` honours `hot` only while `in_prompt_`. on the separate-buffer path, `pg_` is deleted after each prompt, so the next prompt re-runs `prompt_gemm_init` → `region()` → `hold_region(true)`, and the refresh happens. by code review that's correct. it's untested, since the CLI is single-prompt.

## falsifier summary

| falsifier | result |
|---|---|
| PLAN / `summary_comparison` vs raw | `summary_comparison`: 0 mismatches. **PLAN: 12 mismatches** (§1b) |
| remaining `smap` race | none (single read + register mask) |
| hard-coded log string | **yes**: `pg_scratch_est = 1.36 GB` literal and the stale "in the expert arena while a prompt runs" text. "0 slots stolen" / "0.00 GB anon" are tautological |
| stale mtime/hash | none. all hashes match and the build/run/write order holds (runner launch within ~0.3 s of its save). llama runs are carried over, not re-run |

## what to fix

1. PLAN §7.1 #5 / §2.7.A #30: replace the 12 numbers in §1b with the raw values. add the `ja` PPL 53.9 → 406 and the "user-turn TF, 64 tok/domain" caveat.
2. `window.cpp`: log and plan the measured prompt `need` instead of `1.36 * gb`, include it in the cap check, and fix the "in the expert arena" string. drop or label the tautological "0 slots stolen" / "0.00 GB anon".
3. runner: make `evict_file_cache` actually work, or remove it. compute the llama summary from `prev_llama_*` instead of literals. archive `/tmp/m5_prompt_1600tok.txt`. repeat runs (n ≥ 3), and use the same arena size for the IQ3_S and Q2_0 comparison.
4. explain the 0.06 GB footprint delta: either the scratch is mostly non-resident, so lower the peak estimate, or `phys_footprint` misses it, so measure differently.