opus audit v4: verification of the v3 fixes (audits/opus_audit_phase0_2_m5_v3.md) #

scope: read-only. the JSON is a fresh run (generated_at_utc 2026-10-15T13:00:39Z, file mtime 16:00:39 local), so every strata number changed again since v3. i re-parsed all 22 output_tails with my own regexes (STRATA_RESULT, STRATA_KLD, STRATA_KLD_REF2, STRATA_PPL, expert stream, window T1, expert arena, locked, prompt scratch … unmapped, llama common_perf_print / [ Prompt: … ]). from that i recomputed every perf dict, every summary_comparison field, and every number on PLAN L274–282 (#29, #30) and L571–586 (§7.1 #5).

what i ran:

• vendor/strata/build-metal/route_parity: 3× default math, 3× STRATA_METAL_MATH=fast
• python3 -B scripts/run_phase0_phase2_m5_validation.py --self-test
• a mutation check of the new self-test: _madvise_file with advice 0 / 3 / 4 on F_NOCACHE-written files (256 KiB and 64 MiB), measured with the script's own file_cache_residency
• a GGUF-vocab detokenizer, plus a byte-level BPE using the GGUF merges, on the three eval_tf_* files and the unpatched llama output. the BPE round-trips the chat_json ids exactly.
• SHA-256 of the 7 prompt files and 5 scripts/binaries against provenance

no repo files were edited except this report.

verdict: FALSIFIER HIT (PLAN provenance only).

• all 4 requested fix items pass.
• summary_comparison: 0 mismatches (80 scalar fields + 3 KLD blocks).
• both tests pass.
• but PLAN still carries 8 numbers attributed to v3 that appear in no file on disk, plus 4 older #29/#30 numbers with no archived source (§1c).

1. PLAN / summary_comparison vs raw #

1a. perf + summary_comparison: PASS (0 mismatches) #

• all 14 strata perf dicts equal my parse field for field, including ref2 = STRATA_KLD_REF2 and pure_gpu = T1 GPU − host.

• strata_metal_iq3s_moce0_43tok / strata_metal_q2_0_moce0_43tok: 8 + 8 fields exact.

• strata_metal_optimized_k256_q2_0_moce085: all 50 fields exact.

  • miss reductions recompute: 1 − 40.9/84.3 = 51.48% → 51.5 ✓, and 1 − 40.9/108.9 = 62.44% → 62.4 ✓
  • footprint / RSS before→after recompute from the 1024 run: 4.73→4.68, 7.41→7.36 ✓
  • swapouts_mib = the max over all 22 runs = 0.0 ✓

• quality block: all per-domain KLD / PPL / top-1 / perf values exact. macros:

  block	macro mean KLD	macro top-1
  q2_0_moce0 vs iq3s	0.203236	90.63 (90.627)
  moce085 vs q2_0_moce0	0.058768	96.88 (96.877)
  moce085 vs iq3s	0.238592	89.58 (89.583)

  all ✓.

• llama blocks: the 6 + 8 fields equal the archived tails:

  • unpatched [ Prompt: 14.3 | Generation: 1.4 ], 21.7 at ub2048, disk0 105,851.73 / 85,725.73
  • patched 38.11 / 63.27 / 3.22 / 0.28, disk0 33,428.34 / 46,808.58 / 28,272.94

• no 65.0 / 65tok string anywhere in the JSON or the runner. the per-token divisor is gone ✓.

• nit: the quality block's per-domain eval_tokens: 64 sits next to eval_ms / eval_tok_per_s, which are over the 63 decode steps (10746.76 / 63 = 170.58). perf.eval_tokens is overwritten by the STRATA_KLD count. label clash only, nothing is wrong numerically.

1b. PLAN numbers that recompute (✓) #

• #29: 1.40 (= 1.4), 14.30 / 21.70 (= 14.3 / 21.7; the trailing zeros are false precision, cosmetic), 105,851.73, 85,725.73, 3.22, 22-token, 33,428.34, 63 decode (the patched run's "63 runs"), 5,600-byte, 1,264-token, 38.11, 63.27, 0 decode tokens (eval time = 0.00 ms / 1 runs + [end of text]), 46,808.58, 28,272.94, 0.28.
• #30:
  • route_parity: 200 windows, 2.16e-07 / 2.15e-07, 3.06e-07 / 4.00e-07
  • memory: 2.90 / 0.34 / 1.28 / 4.53; 3.24 = 2.90 + 0.34 (derived)
  • prompt scratch: 1.36, 4.73→4.68, 7.41→7.36, 0.052 (labelled hypothesis ✓)
  • expert files: 23.42 / 15.82 GiB (.experts = 23.419 / 15.820 GiB ✓). 85 pages: 15.82 GiB / 12288 = 84.4 × 16 KiB → 85 ✓
  • arena: 1,550 / 1,440 / 2,441 / 3.40
  • misses: 51.5%, 84.3→40.9, 304.5→127.2, 82.44→91.48, 62.4%, 108.9
  • soaked speeds: 229.99 / 268.23, 8.04–9.33 (the soaked MoCE decodes are 8.04 / 9.33 / 8.37)
  • KLD: 0.0588 / 96.88 / 0.2032 / 90.63 / 0.2386, per-domain top-1, 192 = 3 × 64, 244 / 159 / 456
  • cool: 320.58 / 353.49 are in v3 ✓
• §7.1 #5:
  • run count and watchdog: 22 = 5 + 9 + 4 + 4; watchdog 0.25 s / 2.2 GiB / 64.0 MiB = watchdog_policy; 0.0 MiB swap ✓
  • init: 424.45–504.58 (MoCE speed: 469.70 / 424.45 / 504.58) ✓, 424.45–506.90 (all 14) ✓
  • fill: 0.6–1.3 s ✓; 2.69–5.92 GB/s over the 10 Q2_0 runs ✓; 3.98–5.04 over the 4 IQ3_S runs ✓
  • prefill: 4,452.38 / 7,635.20 ✓
  • every soaked tok/s, ms/tok, wait, GPU, hit split, misses/tok and MB/tok on L579–582 ✓. the splits now sum correctly: 52.62 + 24.70 = 77.32, 63.86 + 18.58 = 82.44, 82.34 + 9.14 = 91.48, 84.38 + 8.25 = 92.63, 86.70 + 7.14 = 93.84.
  • KLD per domain is rounded to 4 dp, all ✓. 248,320 = GGUF tokenizer.ggml.tokens length ✓.
  • the v3-cited cool values that are in v3: 339.50–356.06, 3,194.19, 5,793.65, 9.51, 13.52, 16.11 + 56.29 ✓.
• the 7 v3 §1c offenders (69.43/7.89, 71.09/11.35, 3,194.23, 5,793.62, 6.39–6.44, 6.82–7.07, 0.5 s) are all gone ✓.

1c. unsourced PLAN numbers (falsifier) #

where the 8 cool numbers came from:

• the parent's transcript row 176426 holds the v3-era summary_comparison (decode_ms_per_tok 137.56 / 105.12 / 73.94 / 73.37 / 75.62).
• they were real values of the 05:03Z run that v3 recomputed, and each pair is internally consistent (1000/137.56 = 7.27 etc.).
• but that JSON was overwritten by this run: the runner writes the same --output path, and the file isn't in git.
• the cited audit file only quotes 6 of the cool values, so the citation can't be checked from the repo.

fix, either of:

1. re-archive the v3-era summary as a separate file (raw tails are lost), or
2. drop the 8 values / quote only what v3 contains, or
3. re-run cool and soaked and archive both to timestamped outputs.

1d. "thermal soak" attribution: unmeasured, and the runner contradicts it #

• PLAN labels all soaked numbers "after sustained fanless thermal soak".
• but the runner itself says (L506): "Phase A: Run all 5 speed benchmark runs first while the fanless M5 Air is thermally unthrottled". in this run the speed runs came first. in the v3-era run they were interleaved with the TF refs.
• no temperature, pmset -g therm or powermetrics data is archived.
• both sides of the decode time grew, and pure GPU grew too, so this isn't just I/O:
  • host wait doubled at near-identical misses/tok and MB/tok (43-tok MoCE: 16.11 → 34.15 ms at 126.2 vs 127.2 MB/tok). the implied SSD rate fell from ~7.8 to ~3.7 GB/s.
  • pure GPU rose 56.29 → 88.35.
• that is consistent with thermal throttling, but also with the binary rebuild in between (window.cpp / route_parity / CLI rebuilt 15:56–15:57) or with background load. label it a hypothesis or log thermals.

2. llama prompt archive + #29 / §7.1 caveats: PASS #

• data/m5_val_prompts/llama_prompt_1264tok.txt: 5,600 bytes, sha256 2549439506dda855058b64b48167d9b87aeb0fe5ab4fdfb1e191782218a9ac79. that equals provenance.prompt_files_sha256 and is byte-identical to /tmp/m5_prompt_1600tok.txt, which still exists.

• the patched echo tail "…You are solving a Termina [end of text]" equals the file's last bytes ✓. the unpatched echo begins with the file's first task ✓.

• #29 (L276) explicitly states:

  • unpatched = llama-cli chat mode
  • patched = llama-completion -no-cnv -t 8 raw
  • the patched ub1024/2048 runs hit EOS after prefill with 0 decode tokens, so their disk0 is load + prefill only
  • "the two runs also differ in llama-cli chat vs llama-completion -t 8 raw mode"

  ✓

• §7.1 #5 (L573–574): names both modes and says "hitting EOS after prefill with 0 decode tokens" / "prefill only" ✓. it has no explicit "not like-for-like" sentence the way #29 does, but the modes and the prefill-only status are on the lines. acceptable.

• summary_comparison key whole_run_disk0_mb_1264tok_ub*_prefill_only carries the caveat in the name. no / 65.0 ✓.

• data caveats (not falsifiers):

  • the 5,600-byte file is cut mid-word ("Termina"). that's the likely reason the raw-mode run emits EOS immediately.
  • the unpatched keys say 1264tok, but llama-cli wraps the text in the chat template, so its prompt is 1,264 + template tokens (never printed).

3. eval_tf_{ops_tools,chat_json,ja}.txt: PASS (with content caveats) #

• all three contain exactly 2 <|im_start|>: one complete user turn (ending in <|im_end|>\n) plus the assistant turn. 244 + 64, 159 + 64 and 456 + 64 match the cmds, and all 9 runs report eval_tokens=64. no <think> (248068) in any scored span ✓.
• chat_json is now a genuine schema-extraction task: title / date / start / alphabetical attendees. the scored JSON is correct ✓.
• in all 9 runs the per-run GEN_IDS (argmax) align position-for-position with the forced tokens (ops 49/64, chat 61/64, ja 54/64 on the ref). so teacher forcing is aligned and off-by-one is ruled out ✓.
• caveats:
  1. ops_tools' scored span has no reasoning and no </think>. the </think> is in the prompt, and the 64 tokens are visible JSON only. so PLAN L583's "scoring final <think> reasoning + </think> + complete visible JSON/Japanese output" holds for chat_json and ja, not ops_tools.
  2. ops_tools still has <think> with no following \n (v3 caveat 4), and </think>\n\n\n{ has a triple newline. this doesn't match the chat template.
  3. ops_tools' user turn is complete but is a mid-trajectory New Terminal Output observation. the task statement from earlier turns is absent.
  4. ja's reasoning is a keyword-extraction / question-writing trace (「質問文を作成する」), a data-synthesis artifact that doesn't match the user's QA request. its tail is in the scored span.
  5. still 1 turn × 64 tokens per domain, no CI. v3 fix 6's "n ≥ 3 docs/domain" is not done.

4. code fixes: PASS #

• prompt_gemm_need_bytes() (window.cpp L2148–2195):
  • declares PromptGemm p_tm; and uses p_tm.TM for both mr = C*TOPK + NEXP*p_tm.TM (L2161) and the tiles buffer 8*(mr/p_tm.TM + 1) (L2180). no hard-coded 16 remains ✓.
• prompt_gemm_init() L2137:
  • if (need != prompt_gemm_need_bytes()) { err = "metal window: prompt_gemm_need_bytes() != need"; return false; }, after make() in measure mode.
  • this is a hard runtime check, stronger than assert (which NDEBUG would compile out).
  • the string is in the strata_window_cli binary (sha matches provenance, built 15:57:05 after window.cpp 15:56:59). all 14 runs completed, so the check passed at C = 1024, PLE on, flash.
  • nit: on mismatch it returns after st_->region(need) has already mapped the region, and leaves it mapped.
• evict_file_cache removed: no occurrence in scripts/, results/ or PLAN ✓.
• file_cache_residency: ctypes mmap(PROT_READ, MAP_SHARED) + mincore, counting vec[i] & 1 (L62–90).
• self-test (L419–434): writes 16 × 16 KiB with F_NOCACHE, asserts residency == 0.0, calls warm_file_cache, asserts == 1.0 ✓.
  • the mutation check shows it discriminates:
    • advice 0 (no-op) → 0.0 → 0.0, which the test would fail
    • advice 3 (WILLNEED) → 0.0 → 1.0
    • advice 4 (DONTNEED) → 0.0 → 0.0
    • a 64 MiB file also warms 0.0 → 1.0
  • remaining caveat: warm_file_cache still targets the dense K256-*-metal.gguf, not the .experts file the arena fill reads. the first Q2_0 fill of each phase is still the slow one: q2_0_moce0 speed 1.1 s / 2.94 GB/s, tf_ops_tools_q2_0_moce0 1.3 s / 2.69 GB/s.
• unchanged from v3: the cap check is still log-only when STRATA_METAL_ARENA_GB is set (L1208). out of scope for this round.

5. tests / provenance #

• route_parity (6 runs, all exit 0, deterministic):
  • default: K=512 … mismatches 0, worst 2.16e-07 / K=256 + MoCE=0.85 … mismatches 0, 3.06e-07 PASS
  • fast: 2.15e-07 / 4.00e-07 PASS
  • matches PLAN #30 exactly.
• --self-test: exit 0, output scripts/run_phase0_phase2_m5_validation.py self-test: PASS. the v3 garbled prefix is gone.
• provenance: all 7 prompt hashes and all 5 script/binary hashes match the disk.

falsifier summary #

what to fix #

1. archive the v3-era cool numbers (e.g. results/phase0_phase2_m5_validation_v3_cool_summary.json from parent transcript row 176426) or drop the 8 values. better: make the runner write timestamped outputs so a re-run can't destroy the previous raw.
2. #29: mark 17.0 GiB and "96 graph splits/token" as [unverified], or archive a --repack load log and a GGML_SCHED split count. fix "63 generated tokens" → "64 generated tokens (63 decode steps)" for llama-cli.
3. #30: source "93..163 pages" (print the class page sizes in the strata log) or drop it.
4. relabel "thermal soak" as a hypothesis, or log pmset -g therm / powermetrics around the runs. fix the runner's Phase A comment.
5. PLAN L583: say that ops_tools' 64 scored tokens are the visible JSON only, with no reasoning and no </think>. optionally fix the ops_tools <think> / \n template drift.