# opus audit v3: verification of the v2 fixes (`audits/opus_audit_phase0_2_m5_v2.md`)

scope: read-only. i re-parsed every `output_tail` in `results/phase0_phase2_m5_validation.json` (generated 2026-10-15T05:03:32Z, a fresh run, so every strata number changed since v2) with my own regexes, and recomputed `perf`, `summary_comparison` and every number in `PLAN.md` §2.7.A #29–#30 and §7.1 #5 from that.

what i ran:
- `route_parity` 3× default math and 3× `STRATA_METAL_MATH=fast`
- `python3 -B scripts/run_phase0_phase2_m5_validation.py --self-test`
- a `mincore` probe of `evict_file_cache` / `warm_file_cache` on throwaway 256 MiB files in `/tmp`
- a GGUF-vocab detokenization of the three `eval_tf_96tok_*` files
- a Python mirror of `prompt_gemm_need_bytes()`

no repo files were edited except this report.

**verdict: FALSIFIER HIT.**

- `summary_comparison`: 0 mismatches against the raw runs.
- code fixes (#2): done.
- PLAN still quotes **9 numbers that are in no archived run**: two hit-rate splits, two prefill ms values, and the fill-rate/fill-time ranges.
- `evict_file_cache` returns `True` but **evicts nothing** (`mincore` shows 100% still resident), and the self-test only checks the return value.
- **the archived llama prompt is not the prompt the llama runs used.** it's the first 1,600 chars of the 5,600-char `/tmp` file.
- the patched-vs-unpatched llama `disk0` comparison in #29 isn't like for like.

---

## 1. PLAN / `summary_comparison` vs raw

### 1a. `summary_comparison`: PASS (0 mismatches)

- `perf` dicts equal my independent parse for all 14 strata runs. the only differences are float rounding of `pure_gpu = T1 GPU − host` to 2 dp.
- all 44 derived fields of `strata_metal_optimized_k256_q2_0_moce085` equal the raw. the other 4 (`locked 2.90`, `unpinned 0.34`, `hc 1.28`, `scratch 1.36`) equal the log line.
- both 43-tok MoCE=0 blocks are exact.
- KLD macros, recomputed:
  - `0.327365 / 75.52`
  - `0.064461 / 90.11` (90.107)
  - `0.356695 / 75.00`
- miss reductions: `1−40.6/84.3 = 51.84%` → 51.8 ✓ and `1−40.6/108.9 = 62.72%` → 62.7 ✓.
- llama blocks are now computed from `prev_llama_*` (runner L665–692), and the values equal the archived llama tails:
  - unpatched: 14.3 / 21.7 / 1.4 / 105,851.73 / 1,628.5
  - patched: 38.11 / 63.27 / 3.22 / 0.28 / 33,428.34 / 514.3 / 46,808.58
- `0.0 MiB` swapout in all 22 runs ✓. `min_avail` 2.85–3.36 GiB ✓.

### 1b. the 12 v2 mismatches (re-checked against the *new* raw)

| v2 item | PLAN now | raw now | status |
|---|---|---|---|
| 43-tok arena + RA | 82.17 + 9.38 | 82.17 + 9.38 | ✓ |
| 43-tok wait + GPU | 16.11 + 56.29 | 16.11 + 56.29 | ✓ |
| 1024 arena + RA | 84.23 + 8.50 | 84.23 + 8.50 | ✓ |
| 1024 wait + GPU | 14.10 + 57.61 | 14.10 + 57.612 | ✓ |
| 2048 arena + RA | 86.50 + 7.36 | 86.50 + 7.36 | ✓ |
| fill rate range | `6.39–6.44` Q2_0, `6.82–7.07` IQ3_S | Q2_0 **3.66**, 6.39–6.44. IQ3_S **6.10**, 7.03–7.07 | ✗ (6.82 in no run; both ranges drop the first/cold run) |
| IQ3_S slots | 1,550 / 1,440 filled | 1550 / 1440 | ✓ |
| run count | 22 | 5 + 9 + 4 + 4 = 22 | ✓ |
| `route_parity` | 2.16e-07 / 3.06e-07 default, 2.15e-07 / 4.00e-07 fast | identical, deterministic ×3 each | ✓ |
| upstream 644–976 slots | removed | n/a | ✓ |
| init range scope | 339.50–356.06 (MoCE speed), 335.05–418.56 (all 14) | exact | ✓ |
| ja PPL / user-turn caveat | TF switched to assistant tokens. ja PPL is now 25.57 → 29.52, so the blowup is gone | exact | ✓ (see §3 for new caveats) |

### 1c. NEW PLAN numbers with no raw source (falsifier)

| PLAN claim | line | raw | |
|---|---|---|---|
| IQ3_S 43-tok split `69.43%` arena + `7.89%` RA | §7.1 L580 | **52.60% + 24.72%** (sum 77.32 ✓) | ✗ |
| Q2_0 MoCE=0 split `71.09%` arena + `11.35%` RA | §7.1 L581 | **63.86% + 18.58%** (sum 82.44 ✓) | ✗ |
| 1024-tok prefill `3,194.23 ms` | §7.1 L578 | **3,194.19** | ✗ |
| 2048-tok prefill `5,793.62 ms` | §7.1 L578 | **5,793.65** | ✗ |
| `6.39–6.44 GB/s` Q2_0 fill | §7.1 L577 | **3.66**–6.44 (`strata_q2_0_moce0_short_43tok`) | ✗ |
| `6.82–7.07 GB/s` IQ3_S fill | §7.1 L577 | **6.10**–7.07 (6.82 appears nowhere) | ✗ |
| `0.5s` profile fill | §7.1 L577 | **0.9 s** on `q2_0_moce0` speed, 0.4 s on IQ3_S TF runs | ✗ (range is 0.4–0.9 s) |

this is the same failure mode as v2. the sums match, which suggests the splits were written to make the sums right, or were carried over from a non-archived run. none of the 7 strings occurs in any tail.

everything else in #29 / #30 / §7.1 #5 recomputes:
- every tok/s, ms/tok, combined hit rate, misses/tok, MB/tok, host wait and pure GPU value
- prefill 320.58 / 353.49
- `2.90 / 0.34 / 1.28 / 4.53` (dense GGUF = 4.220 GiB = 4.53 GB ✓)
- `phys_footprint 4.73 → 4.67` and `RSS 7.65 → 7.60` (identical in the 1024 and 2048 runs)
- the 2441 / 1550 / 1440 slot counts and 3.40 GB in all 14 runs
- `23.42 GiB` / `15.82 GiB` expert files (`.experts` sizes 23.419 / 15.820 GiB ✓). `85` pages per Q2_0 slot: 15.82 GiB / 12288 = 1.38 MB = 84.4 × 16 KiB ✓, and 3.40 GB / (85 × 16 KiB) = 2441 ✓
- all KLD / PPL / top-1 values

### 1d. #29 llama comparison: numbers recompute, but the comparison is invalid

- **patched `step2_ngl99_cmoe_ub1024` decoded nothing.** it shows `eval time = 0.00 ms / 1 runs`, and the echo ends `…You are solving a Termina [end of text]`. unpatched `step2_ub1024` (llama-cli chat mode) generated ~64 thinking tokens at 1.4 tok/s.
  - so "whole-run `disk0` drops from 105,851.73 MB to 46,808.58 MB on the 1,264-token -ub 1024 run" compares prefill-only against prefill plus about 64 decode steps.
  - the same goes for 85,725.73 vs 28,272.94 at ub2048.
- **different binaries and modes.**
  - unpatched is `llama-cli` (chat template applied, `-st`, no `-t`).
  - patched is `llama-completion -no-cnv -t 8` (raw text).
  - so the unpatched prompt isn't 1,264 tokens (template tokens are added, and llama-cli never prints a count), and the 14.3 → 38.11 prefill ratio mixes binary, template and threads.
- "`1.40 → 3.22 tok/s` decode" compares an unpatched 1,264-token-context chat run against a patched 22-token raw run.
- the `/ 65.0` divisor in `whole_run_disk0_mb_per_65tok*` (runner L680, L689) is a literal, and the runs' disk0 includes load + prefill, so "MB / 65 tok" isn't a per-token rate.

## 2. code fixes: PASS, with two notes

- ✓ `window.cpp` L1207: `pg_scratch_need = prompt_gemm_need_bytes()`. no `1.36 * gb` literal remains anywhere in `vendor/strata/metal`, `patch_strata_metal.py` or the runner (the only `1.36` is in the self-test sample).
  - my Python mirror of the function gives **1.3586 GB** for C = 1024, 4224 cells, PLE on, flash. that matches the logged `1.36`, and both the plan line and the `prompt_gemm_init` line print 1.36.
- ✓ the stale string is fixed. `prompt_gemm_init` (L2140–2142) prints "in a separate transient mmap while a prompt runs" when `SEPARATE_PROMPT_BUF != 0`. the "in the expert arena" text survives only in the `=0` branch, and it appears in no tail.
- ✓ "0 arena slots stolen" / "0.00 GB anon": no match in source, patch or any tail.
- ✓ `prompt_region(false)` (L2203–2215):
  1. `proc_mem()` (`phys_footprint` + `resident_size` via `TASK_VM_INFO`)
  2. `cx_->free`
  3. `delete pg_`
  4. `release_region()` → `munmap`
  5. `purge_device_allocations`
  6. `proc_mem()`
  7. the log line with both pairs

  all 14 tails carry it.
- ⚠ **`prompt_gemm_need_bytes()` is a hand-maintained duplicate of `make()`** (L2101–2129 vs L2166–2191).
  - it re-derives `maxrows` with a hard-coded `16` for `p.TM` (L2159, L2178), where `make()` uses `p.TM`, which defaults to 16.
  - nothing asserts that it equals the measured `need`. an edit to `make()` would silently desync the plan.
  - suggest: `assert(need == prompt_gemm_need_bytes())` in `prompt_gemm_init`, or run `make()` in measure mode from both places.
- ⚠ the cap check is still log-only when `STRATA_METAL_ARENA_GB` is set, which it is in all 14 runs. the budget is reduced by `need` only in auto mode (L1208–1210). with an explicit arena, the scratch only adds a "peak prefill above cap" suffix. that's fine for these runs (6.5 < 18.8 GB cap), but not enforced.
- ⚠ PLAN #30 says "XNU bills only the ~0.06 GB CPU-faulted `p.R`+`p.emb` slice". `p.R + p.emb` = 1024 × (4·2560 + 2560) × 4 B = **0.052 GB**, consistent with the 2-dp delta. **RSS also only drops 0.05 GB**, though (7.65 → 7.60), so the 1.36 GB scratch was never resident by either metric. the mechanism is a hypothesis, not measured. label it so, or measure with `vmmap`/`footprint`.

## 3. runner / data

### `evict_file_cache` / `warm_file_cache`: FAIL (evict) / PASS (warm)

- the implementation is now `ctypes` `libc.mmap(PROT_READ, MAP_SHARED)` + `madvise` + `munmap` (L33–64). no more `from_buffer` `TypeError` swallow. ✓
- `mincore` probe (separate process, 16 KiB pages, 256 MiB file):
  - `evict_file_cache` (`MADV_DONTNEED`=4) on a fully cached file → returns `True`, residency **1.00 → 1.00**. on Darwin, `MADV_DONTNEED` deactivates pages; it does not drop clean UBC pages. **evict is a no-op.**
  - `warm_file_cache` (`MADV_WILLNEED`=3) on a file written with `F_NOCACHE` (residency 0.00) → **1.00**. warm works.
- **the self-test is tautological.** `assert evict_file_cache(__file__) is True` checks only the `madvise` return code. it can't detect the no-op.
- **it targets the wrong file anyway.**
  - evict/warm touch `K256-*-metal.gguf`, the dense 4.53 GB file. that one is `mlock`ed per run, so init is warm in every run (335–419 ms).
  - the arena fill reads `K256-*-metal.gguf.experts` with `F_NOCACHE` preads, and neither function touches it.
  - the first run per expert file is still the slow one: `q2_0_moce0` speed run 0.9 s / 3.66 GB/s, and `iq3s` speed run 6.10 GB/s vs 7.03–7.07 later. so the cold/warm asymmetry v2 flagged on the MoCE=0 vs MoCE=0.85 pair remains, now on the fill rather than on init.
  - decode tok/s is measured after the fill, so the 9.51 → 13.52 pair is less affected than in v2, but n = 1.
- fix suggestions:
  - drop evict, or use `F_NOCACHE` + `fcntl(F_PURGE…)` / `purge` (needs root). or accept warm-for-all and warm the `.experts` file region the profile fills.
  - make the self-test assert residency via `mincore`.

### uniform arena: PASS

all 14 strata runs have `env.STRATA_METAL_ARENA_GB = "3.4"` and log `expert arena 3.40 GB`. Q2_0 runs have 2441 slots / 1 class, IQ3_S runs 1550 slots (1440 filled) / 3 classes.

### `eval_tf_96tok_*`: structure PASS, content caveats

- all three are 96 ids:
  - ids[0..28] = `<|im_start|>user\n` … `<|im_end|>\n` (29 tokens)
  - ids[29..31] = `<|im_start|>assistant\n`
  - ids[32..95] = the 64 scored assistant tokens
- `--n-prompt 32 --n-gen 64` matches. all 9 KLD runs report `eval_tokens=64`. hashes match provenance.
- caveats, from decoding with the GGUF vocab:
  1. **the user turns are left-truncated fragments.**
     - ops_tools: `);\n df.erase();\n\nroot@f71c07bf8043:/app#`
     - chat_json: `{19}~?$$\n\nSolve this step by step…`
     - ja: `カイブ\n36\n\n質問: 5G医療救急車…`

     the responses refer to context that isn't there. ja says 「文章の内容を理解する」 about a document that's absent, and ops_tools says "The fix looks correct… `testValidate()`". so this is off-context teacher forcing.
  2. **all 64 scored tokens are `<think>` reasoning.** the first scored token is the `<think>` template token (248068), which is near-deterministic: 1/64 per domain is free agreement.
  3. **`chat_json` is a modular-arithmetic math problem**, not chat/JSON. the domain label is wrong.
  4. ops_tools has `<think>` with no following `\n`, unlike the other two and the chat template.
  5. still n = 64 tokens × 1 doc/domain, and no CI.

### `llama_prompt_1264tok.txt`: FAIL (wrong file archived)

- the archived file is 1,600 bytes (sha `334a7331…`, matching provenance).
- every llama run's `cmd` uses `-f /tmp/m5_prompt_1600tok.txt`. that file still exists: **5,600 bytes**, mtime 05:24:16, sha `2549439506dd…`.
- the archived file is exactly its first 1,600 chars.
- the patched run's prompt echo ends with "…Stop only when you believe the task is complete…\n\n\nYou are solving a Termina". that text is in the `/tmp` file's last 200 chars and **absent from the archived file**.
- so the runs used the 5,600-char file. the name "1600tok" was the char count of an earlier version, and "1264tok" now labels a ~350-token prefix.
- fix: copy `/tmp/m5_prompt_1600tok.txt` into `data/m5_val_prompts/` before /tmp is cleared, and re-hash it.

### `llama_cpp_*` from `prev_llama_*`: PASS

- runner L665–692 indexes `prev_llama_unpatched` / `prev_llama_patched` by step. no literals remain except the `/ 65.0` divisor (§1d).
- the llama runs are still carried over (`env: null`), not re-run.

## 4. provenance / tests

- all 5 script/binary SHA-256s and all 7 prompt hashes match the disk.
- mtimes:
  - patch 07:54:46
  - `moe.metal` 07:54:46.45
  - `route_parity` 07:54:47.65
  - `strata_window_cli` 07:54:49.78
  - runner 08:01:11.43
  - JSON 08:03:32.52 (`generated_at_utc` 05:03:32.517Z)

  runs sum to 138.55 s, leaving ~2.5 s for `vm_stat`/`iostat` and launch. consistent with build → edit runner → run → write.
- `route_parity` (6 runs, all exit 0, deterministic):
  - default math: `K=512 … 2.16e-07`, `K=256 + MoCE=0.85 … 3.06e-07 PASS`
  - `STRATA_METAL_MATH=fast`: `2.15e-07`, `4.00e-07 PASS`
  - matches PLAN #30 exactly.
- `--self-test`: exit 0, but the PASS line has a garbled prefix: `self-test: 0 scripts/run_phase0_phase2_m5_validation.py self-test: PASS`. the evict assertion is tautological (§3).

## falsifier summary

| falsifier | result |
|---|---|
| PLAN vs raw | **7 cells / 9 numbers unsourced** (§1c): IQ3_S and Q2_0-m0 arena/RA splits, 2 prefill ms, the fill-rate ranges, the fill time |
| `summary_comparison` vs raw | 0 mismatches |
| hard-coded literal | `1.36 * gb` gone ✓. remaining: the `/ 65.0` llama divisor, and the hard-coded `16` (TM) in the `prompt_gemm_need_bytes()` mirror |
| archived inputs | **the llama prompt archive is the wrong file** (1,600-char prefix of the 5,600-char prompt used) |
| cache control | **`evict_file_cache` is a verified no-op**, and it targets the dense file, not `.experts` |

## what to fix

1. PLAN §7.1 L577–581: put in the raw splits (`52.60 + 24.72`, `63.86 + 18.58`), the prefill ms (`3,194.19`, `5,793.65`) and the fill ranges (`3.66–6.44` Q2_0 incl. the cold first run, `6.10–7.07` IQ3_S, `0.4–0.9 s`).
2. PLAN #29: state that the patched ub1024/ub2048 runs generated 0 tokens and used `llama-completion` raw vs `llama-cli` chat, so the disk0/prefill/decode ratios aren't like for like. or re-run both with the same binary, mode and `-n`.
3. archive the real `/tmp/m5_prompt_1600tok.txt` (5,600 B, sha `2549439506dd…`).
4. replace or drop `evict_file_cache`, warm the `.experts` file if cold/warm parity matters, and test residency with `mincore` in `--self-test`.
5. assert `prompt_gemm_need_bytes() == need` in `prompt_gemm_init`.
6. TF eval: use untruncated user turns (or a document-free prompt), relabel `chat_json`, and exclude the template `<think>` token from scoring. n ≥ 3 docs/domain.