# opus audit v4: verification of the v3 fixes (`audits/opus_audit_phase0_2_m5_v3.md`)

scope: read-only. the JSON is a fresh run (`generated_at_utc 2026-10-15T13:00:39Z`, file mtime 16:00:39 local), so every strata number changed again since v3. i re-parsed all 22 `output_tail`s with my own regexes (`STRATA_RESULT`, `STRATA_KLD`, `STRATA_KLD_REF2`, `STRATA_PPL`, `expert stream`, `window T1`, `expert arena`, `locked`, `prompt scratch … unmapped`, llama `common_perf_print` / `[ Prompt: … ]`). from that i recomputed every `perf` dict, every `summary_comparison` field, and every number on PLAN L274–282 (#29, #30) and L571–586 (§7.1 #5).

what i ran:
- `vendor/strata/build-metal/route_parity`: 3× default math, 3× `STRATA_METAL_MATH=fast`
- `python3 -B scripts/run_phase0_phase2_m5_validation.py --self-test`
- a mutation check of the new self-test: `_madvise_file` with advice 0 / 3 / 4 on `F_NOCACHE`-written files (256 KiB and 64 MiB), measured with the script's own `file_cache_residency`
- a GGUF-vocab detokenizer, plus a byte-level BPE using the GGUF merges, on the three `eval_tf_*` files and the unpatched llama output. the BPE round-trips the chat_json ids exactly.
- SHA-256 of the 7 prompt files and 5 scripts/binaries against `provenance`

no repo files were edited except this report.

**verdict: FALSIFIER HIT (PLAN provenance only).**

- all 4 requested fix items **pass**.
- `summary_comparison`: **0 mismatches** (80 scalar fields + 3 KLD blocks).
- both tests pass.
- but PLAN still carries **8 numbers attributed to `v3` that appear in no file on disk**, plus 4 older #29/#30 numbers with no archived source (§1c).

---

## 1. PLAN / `summary_comparison` vs raw

### 1a. `perf` + `summary_comparison`: PASS (0 mismatches)

- **all 14 strata `perf` dicts** equal my parse field for field, including `ref2` = `STRATA_KLD_REF2` and `pure_gpu = T1 GPU − host`.
- **`strata_metal_iq3s_moce0_43tok` / `strata_metal_q2_0_moce0_43tok`:** 8 + 8 fields exact.
- **`strata_metal_optimized_k256_q2_0_moce085`:** all 50 fields exact.
  - miss reductions recompute: `1 − 40.9/84.3 = 51.48%` → 51.5 ✓, and `1 − 40.9/108.9 = 62.44%` → 62.4 ✓
  - footprint / RSS before→after recompute from the 1024 run: `4.73→4.68`, `7.41→7.36` ✓
  - `swapouts_mib` = the max over all 22 runs = 0.0 ✓
- **quality block:** all per-domain KLD / PPL / top-1 / perf values exact. macros:

  | block | macro mean KLD | macro top-1 |
  |---|---|---|
  | q2_0_moce0 vs iq3s | 0.203236 | 90.63 (90.627) |
  | moce085 vs q2_0_moce0 | 0.058768 | 96.88 (96.877) |
  | moce085 vs iq3s | 0.238592 | 89.58 (89.583) |

  all ✓.
- **llama blocks:** the 6 + 8 fields equal the archived tails:
  - unpatched `[ Prompt: 14.3 | Generation: 1.4 ]`, `21.7` at ub2048, disk0 `105,851.73` / `85,725.73`
  - patched `38.11` / `63.27` / `3.22` / `0.28`, disk0 `33,428.34` / `46,808.58` / `28,272.94`
- **no `65.0` / `65tok` string** anywhere in the JSON or the runner. the per-token divisor is gone ✓.
- nit: the quality block's per-domain `eval_tokens: 64` sits next to `eval_ms` / `eval_tok_per_s`, which are over the 63 decode steps (`10746.76 / 63 = 170.58`). `perf.eval_tokens` is overwritten by the `STRATA_KLD` count. label clash only, nothing is wrong numerically.

### 1b. PLAN numbers that recompute (✓)

- **#29:** `1.40` (= `1.4`), `14.30` / `21.70` (= `14.3` / `21.7`; the trailing zeros are false precision, cosmetic), `105,851.73`, `85,725.73`, `3.22`, `22`-token, `33,428.34`, `63` decode (the patched run's "63 runs"), `5,600`-byte, `1,264`-token, `38.11`, `63.27`, `0` decode tokens (`eval time = 0.00 ms / 1 runs` + `[end of text]`), `46,808.58`, `28,272.94`, `0.28`.
- **#30:**
  - route_parity: `200` windows, `2.16e-07` / `2.15e-07`, `3.06e-07` / `4.00e-07`
  - memory: `2.90` / `0.34` / `1.28` / `4.53`; `3.24` = 2.90 + 0.34 (derived)
  - prompt scratch: `1.36`, `4.73→4.68`, `7.41→7.36`, `0.052` (labelled hypothesis ✓)
  - expert files: `23.42` / `15.82` GiB (`.experts` = 23.419 / 15.820 GiB ✓). `85` pages: 15.82 GiB / 12288 = 84.4 × 16 KiB → 85 ✓
  - arena: `1,550` / `1,440` / `2,441` / `3.40`
  - misses: `51.5%`, `84.3→40.9`, `304.5→127.2`, `82.44→91.48`, `62.4%`, `108.9`
  - soaked speeds: `229.99` / `268.23`, `8.04–9.33` (the soaked MoCE decodes are 8.04 / 9.33 / 8.37)
  - KLD: `0.0588` / `96.88` / `0.2032` / `90.63` / `0.2386`, per-domain top-1, `192 = 3 × 64`, `244 / 159 / 456`
  - cool: `320.58` / `353.49` are in v3 ✓
- **§7.1 #5:**
  - run count and watchdog: `22 = 5 + 9 + 4 + 4`; watchdog `0.25 s` / `2.2 GiB` / `64.0 MiB` = `watchdog_policy`; 0.0 MiB swap ✓
  - init: `424.45–504.58` (MoCE speed: 469.70 / 424.45 / 504.58) ✓, `424.45–506.90` (all 14) ✓
  - fill: `0.6–1.3 s` ✓; `2.69–5.92 GB/s` over the 10 Q2_0 runs ✓; `3.98–5.04` over the 4 IQ3_S runs ✓
  - prefill: `4,452.38` / `7,635.20` ✓
  - every soaked tok/s, ms/tok, wait, GPU, hit split, misses/tok and MB/tok on L579–582 ✓. the splits now sum correctly: `52.62 + 24.70 = 77.32`, `63.86 + 18.58 = 82.44`, `82.34 + 9.14 = 91.48`, `84.38 + 8.25 = 92.63`, `86.70 + 7.14 = 93.84`.
  - KLD per domain is rounded to 4 dp, all ✓. `248,320` = GGUF `tokenizer.ggml.tokens` length ✓.
  - the v3-cited cool values that *are* in v3: `339.50–356.06`, `3,194.19`, `5,793.65`, `9.51`, `13.52`, `16.11 + 56.29` ✓.
- the 7 v3 §1c offenders (69.43/7.89, 71.09/11.35, 3,194.23, 5,793.62, 6.39–6.44, 6.82–7.07, 0.5 s) are **all gone** ✓.

### 1c. unsourced PLAN numbers (falsifier)

| PLAN claim | line | status |
|---|---|---|
| IQ3_S cool `7.27 tok/s / 137.56 ms/tok` "cool in `v3`" | L579 | **not in v3 audit, not in any file.** `7.27` occurs nowhere in v3, and `137.56` only in an unrelated upstream bench JSON. |
| Q2_0 m0 cool `105.12 ms/tok` | L580 | **not in v3** (v3 has only `9.51`) |
| MoCE 43-tok cool `73.94 ms/tok` | L581 | **not in v3** (v3 has `13.52`, `16.11`, `56.29`) |
| 1024 / 2048 cool `13.63 / 73.37`, `13.22 / 75.62` | L582, L282 (`13.22–13.63`) | **not in v3 or on disk** |
| `17.0 GiB` repack buffer at `src/llama-model.cpp L1914` | L275 | **not in any archived output.** every unpatched run uses `-nr`, so no tail can show it. in the patched tree, L1914 is the mmap-vs-alloc branch condition, which is plausible but not the allocation itself. |
| "96 graph splits/token" | L276 | **not in any tail.** v1 already flagged this as unverified. |
| IQ3_S size classes "`93..163` pages" | L281 | not in any tail/artifact. my recompute from the GGUF tensor types gives 121/133/143/163 pages for the 39 layers whose types i decode. the max of 163 matches; the min of 93 depends on the 9 layers with `ffn_down` type 42, which i didn't decode. |
| unpatched "generating 63 tokens" / "63 generated tokens" | L276, L573 | llama-cli prints no count. the visible thinking text tokenizes to ≈64–65 tokens with `-n 64` (the smoke run gives 32 at `-n 32`, moecache gives 64). that's 64 generated tokens = 63 decode steps, so the wording is off by one and not printed. minor. |

**where the 8 cool numbers came from:**
- the parent's transcript row 176426 holds the v3-era `summary_comparison` (`decode_ms_per_tok` 137.56 / 105.12 / 73.94 / 73.37 / 75.62).
- they were real values of the 05:03Z run that v3 recomputed, and each pair is internally consistent (`1000/137.56 = 7.27` etc.).
- but that JSON was **overwritten** by this run: the runner writes the same `--output` path, and the file isn't in git.
- the cited audit file only quotes 6 of the cool values, so the citation can't be checked from the repo.

fix, either of:
1. re-archive the v3-era summary as a separate file (raw tails are lost), or
2. drop the 8 values / quote only what v3 contains, or
3. re-run cool and soaked and archive both to timestamped outputs.

### 1d. "thermal soak" attribution: unmeasured, and the runner contradicts it

- PLAN labels all soaked numbers "after sustained fanless thermal soak".
- but the runner itself says (L506): "Phase A: Run all 5 speed benchmark runs first while the fanless M5 Air is thermally unthrottled". in this run the speed runs came **first**. in the v3-era run they were interleaved with the TF refs.
- no temperature, `pmset -g therm` or `powermetrics` data is archived.
- both sides of the decode time grew, and pure GPU grew too, so this isn't just I/O:
  - host wait **doubled** at near-identical misses/tok and MB/tok (43-tok MoCE: `16.11 → 34.15 ms` at 126.2 vs 127.2 MB/tok). the implied SSD rate fell from ~7.8 to ~3.7 GB/s.
  - pure GPU rose `56.29 → 88.35`.
- that is consistent with thermal throttling, but also with the binary rebuild in between (window.cpp / route_parity / CLI rebuilt 15:56–15:57) or with background load. label it a hypothesis or log thermals.

## 2. llama prompt archive + #29 / §7.1 caveats: PASS

- `data/m5_val_prompts/llama_prompt_1264tok.txt`: **5,600 bytes, sha256 `2549439506dda855058b64b48167d9b87aeb0fe5ab4fdfb1e191782218a9ac79`**. that equals `provenance.prompt_files_sha256` and is byte-identical to `/tmp/m5_prompt_1600tok.txt`, which still exists.
- the patched echo tail "…You are solving a Termina [end of text]" equals the file's last bytes ✓. the unpatched echo begins with the file's first task ✓.
- **#29 (L276)** explicitly states:
  - unpatched = `llama-cli` chat mode
  - patched = `llama-completion -no-cnv -t 8` raw
  - the patched ub1024/2048 runs hit EOS after prefill with `0` decode tokens, so their disk0 is load + prefill only
  - "the two runs also differ in `llama-cli` chat vs `llama-completion -t 8` raw mode"

  ✓
- **§7.1 #5 (L573–574)**: names both modes and says "hitting EOS after prefill with `0` decode tokens" / "prefill only" ✓. it has no explicit "not like-for-like" sentence the way #29 does, but the modes and the prefill-only status are on the lines. acceptable.
- `summary_comparison` key `whole_run_disk0_mb_1264tok_ub*_prefill_only` carries the caveat in the name. no `/ 65.0` ✓.
- data caveats (not falsifiers):
  - the 5,600-byte file is cut **mid-word** ("Termina"). that's the likely reason the raw-mode run emits EOS immediately.
  - the unpatched keys say `1264tok`, but llama-cli wraps the text in the chat template, so its prompt is 1,264 + template tokens (never printed).

## 3. `eval_tf_{ops_tools,chat_json,ja}.txt`: PASS (with content caveats)

| file | ids | `--n-prompt` / `--n-gen` | starts with | `<think>` | scored span (last 64) |
|---|---|---|---|---|---|
| ops_tools | 308 | 244 / 64 | `<|im_start|>user\n` (248045 846 198) | in prompt only | tail of the visible JSON reply (`"…is null, the code skips…" … "task_complete": true\n}`) |
| chat_json | 223 | 159 / 64 | `<|im_start|>user\n` | in prompt only | `2:00", attendees alphabetical… JSON only.\n</think>\n\n{"title":"Launch retro","date":"2026-07-11","start":"12:00","attendees":["Dara","Lena","Priya"]}` |
| ja | 520 | 456 / 64 | `<|im_start|>user\n` | in prompt only | tail of the reasoning + `</think>` + the full Japanese answer |

- all three contain exactly 2 `<|im_start|>`: one complete user turn (ending in `<|im_end|>\n`) plus the assistant turn. 244 + 64, 159 + 64 and 456 + 64 match the cmds, and all 9 runs report `eval_tokens=64`. **no `<think>` (248068) in any scored span** ✓.
- `chat_json` is now a genuine schema-extraction task: title / date / start / alphabetical attendees. the scored JSON is correct ✓.
- in all 9 runs the per-run `GEN_IDS` (argmax) align position-for-position with the forced tokens (ops 49/64, chat 61/64, ja 54/64 on the ref). so teacher forcing is aligned and off-by-one is ruled out ✓.
- caveats:
  1. **ops_tools' scored span has no reasoning and no `</think>`.** the `</think>` is in the prompt, and the 64 tokens are visible JSON only. so PLAN L583's "scoring final `<think>` reasoning + `</think>` + complete visible JSON/Japanese output" holds for chat_json and ja, not ops_tools.
  2. ops_tools still has `<think>` with no following `\n` (v3 caveat 4), and `</think>\n\n\n{` has a triple newline. this doesn't match the chat template.
  3. ops_tools' user turn is complete but is a mid-trajectory `New Terminal Output` observation. the task statement from earlier turns is absent.
  4. ja's reasoning is a keyword-extraction / question-*writing* trace (「質問文を作成する」), a data-synthesis artifact that doesn't match the user's QA request. its tail is in the scored span.
  5. still 1 turn × 64 tokens per domain, no CI. v3 fix 6's "n ≥ 3 docs/domain" is not done.

## 4. code fixes: PASS

- **`prompt_gemm_need_bytes()`** (window.cpp L2148–2195):
  - declares `PromptGemm p_tm;` and uses `p_tm.TM` for both `mr = C*TOPK + NEXP*p_tm.TM` (L2161) and the tiles buffer `8*(mr/p_tm.TM + 1)` (L2180). no hard-coded `16` remains ✓.
- **`prompt_gemm_init()`** L2137:
  - `if (need != prompt_gemm_need_bytes()) { err = "metal window: prompt_gemm_need_bytes() != need"; return false; }`, after `make()` in measure mode.
  - this is a hard runtime check, stronger than `assert` (which `NDEBUG` would compile out).
  - the string is in the `strata_window_cli` binary (sha matches provenance, built 15:57:05 after window.cpp 15:56:59). all 14 runs completed, so the check passed at `C = 1024`, PLE on, flash.
  - nit: on mismatch it returns after `st_->region(need)` has already mapped the region, and leaves it mapped.
- **`evict_file_cache` removed:** no occurrence in `scripts/`, `results/` or PLAN ✓.
- **`file_cache_residency`:** ctypes `mmap(PROT_READ, MAP_SHARED)` + `mincore`, counting `vec[i] & 1` (L62–90).
- **self-test** (L419–434): writes 16 × 16 KiB with `F_NOCACHE`, asserts residency `== 0.0`, calls `warm_file_cache`, asserts `== 1.0` ✓.
  - **the mutation check shows it discriminates:**
    - advice 0 (no-op) → `0.0 → 0.0`, which the test would fail
    - advice 3 (`WILLNEED`) → `0.0 → 1.0`
    - advice 4 (`DONTNEED`) → `0.0 → 0.0`
    - a 64 MiB file also warms `0.0 → 1.0`
  - remaining caveat: `warm_file_cache` still targets the dense `K256-*-metal.gguf`, not the `.experts` file the arena fill reads. the first Q2_0 fill of each phase is still the slow one: `q2_0_moce0` speed 1.1 s / 2.94 GB/s, `tf_ops_tools_q2_0_moce0` 1.3 s / 2.69 GB/s.
- unchanged from v3: the cap check is still log-only when `STRATA_METAL_ARENA_GB` is set (L1208). out of scope for this round.

## 5. tests / provenance

- **`route_parity`** (6 runs, all exit 0, deterministic):
  - default: `K=512 … mismatches 0, worst 2.16e-07` / `K=256 + MoCE=0.85 … mismatches 0, 3.06e-07 PASS`
  - fast: `2.15e-07` / `4.00e-07 PASS`
  - matches PLAN #30 exactly.
- **`--self-test`:** exit 0, output `scripts/run_phase0_phase2_m5_validation.py self-test: PASS`. the v3 garbled prefix is gone.
- **provenance:** all 7 prompt hashes and all 5 script/binary hashes match the disk.

## falsifier summary

| falsifier | result |
|---|---|
| `summary_comparison` vs raw | **0 mismatches** |
| tests | `route_parity` 6/6 PASS, `--self-test` PASS |
| v3 fix 1 (splits / prefill ms / fill ranges) | ✓ all recompute |
| v3 fix 2 (llama caveats) | ✓ |
| v3 fix 3 (5,600-B prompt archived) | ✓ sha `2549439506dd…` |
| v3 fix 4 (evict dropped, mincore test) | ✓, and the test discriminates |
| v3 fix 5 (`need` check, `p_tm.TM`) | ✓ |
| v3 fix 6 (TF turns) | ✓ complete turns, real JSON task, `<think>` unscored. n = 1/domain is still open. |
| unsourced PLAN numbers | **8 "cool in v3" values not in v3 or any file** (L579–582, L282). plus `17.0 GiB` / L1914, "96 graph splits", "93..163 pages", and "63 generated tokens" (unpatched; ≈64 visible, not printed) |
| unsourced claim | "sustained fanless thermal soak": no thermal data, and the runner comment says the speed runs were thermally unthrottled |

## what to fix

1. archive the v3-era cool numbers (e.g. `results/phase0_phase2_m5_validation_v3_cool_summary.json` from parent transcript row 176426) or drop the 8 values. better: make the runner write timestamped outputs so a re-run can't destroy the previous raw.
2. #29: mark `17.0 GiB` and "96 graph splits/token" as [unverified], or archive a `--repack` load log and a `GGML_SCHED` split count. fix "63 generated tokens" → "64 generated tokens (63 decode steps)" for llama-cli.
3. #30: source "93..163 pages" (print the class page sizes in the strata log) or drop it.
4. relabel "thermal soak" as a hypothesis, or log `pmset -g therm` / `powermetrics` around the runs. fix the runner's Phase A comment.
5. PLAN L583: say that ops_tools' 64 scored tokens are the visible JSON only, with no reasoning and no `</think>`. optionally fix the ops_tools `<think>` / `\n` template drift.