opus audit v3: verification of the v2 fixes (audits/opus_audit_phase0_2_m5_v2.md) #

scope: read-only. i re-parsed every output_tail in results/phase0_phase2_m5_validation.json (generated 2026-10-15T05:03:32Z, a fresh run, so every strata number changed since v2) with my own regexes, and recomputed perf, summary_comparison and every number in PLAN.md §2.7.A #29–#30 and §7.1 #5 from that.

what i ran:

• route_parity 3× default math and 3× STRATA_METAL_MATH=fast
• python3 -B scripts/run_phase0_phase2_m5_validation.py --self-test
• a mincore probe of evict_file_cache / warm_file_cache on throwaway 256 MiB files in /tmp
• a GGUF-vocab detokenization of the three eval_tf_96tok_* files
• a Python mirror of prompt_gemm_need_bytes()

no repo files were edited except this report.

verdict: FALSIFIER HIT.

• summary_comparison: 0 mismatches against the raw runs.
• code fixes (#2): done.
• PLAN still quotes 9 numbers that are in no archived run: two hit-rate splits, two prefill ms values, and the fill-rate/fill-time ranges.
• evict_file_cache returns True but evicts nothing (mincore shows 100% still resident), and the self-test only checks the return value.
• the archived llama prompt is not the prompt the llama runs used. it's the first 1,600 chars of the 5,600-char /tmp file.
• the patched-vs-unpatched llama disk0 comparison in #29 isn't like for like.

---

1. PLAN / summary_comparison vs raw #

1a. summary_comparison: PASS (0 mismatches) #

• perf dicts equal my independent parse for all 14 strata runs. the only differences are float rounding of pure_gpu = T1 GPU − host to 2 dp.
• all 44 derived fields of strata_metal_optimized_k256_q2_0_moce085 equal the raw. the other 4 (locked 2.90, unpinned 0.34, hc 1.28, scratch 1.36) equal the log line.
• both 43-tok MoCE=0 blocks are exact.
• KLD macros, recomputed:
  • 0.327365 / 75.52
  • 0.064461 / 90.11 (90.107)
  • 0.356695 / 75.00
• miss reductions: 1−40.6/84.3 = 51.84% → 51.8 ✓ and 1−40.6/108.9 = 62.72% → 62.7 ✓.
• llama blocks are now computed from prev_llama_* (runner L665–692), and the values equal the archived llama tails:
  • unpatched: 14.3 / 21.7 / 1.4 / 105,851.73 / 1,628.5
  • patched: 38.11 / 63.27 / 3.22 / 0.28 / 33,428.34 / 514.3 / 46,808.58
• 0.0 MiB swapout in all 22 runs ✓. min_avail 2.85–3.36 GiB ✓.

1b. the 12 v2 mismatches (re-checked against the new raw) #

v2 item	PLAN now	raw now	status
43-tok arena + RA	82.17 + 9.38	82.17 + 9.38	✓
43-tok wait + GPU	16.11 + 56.29	16.11 + 56.29	✓
1024 arena + RA	84.23 + 8.50	84.23 + 8.50	✓
1024 wait + GPU	14.10 + 57.61	14.10 + 57.612	✓
2048 arena + RA	86.50 + 7.36	86.50 + 7.36	✓
fill rate range	6.39–6.44 Q2_0, 6.82–7.07 IQ3_S	Q2_0 3.66, 6.39–6.44. IQ3_S 6.10, 7.03–7.07	✗ (6.82 in no run; both ranges drop the first/cold run)
IQ3_S slots	1,550 / 1,440 filled	1550 / 1440	✓
run count	22	5 + 9 + 4 + 4 = 22	✓
route_parity	2.16e-07 / 3.06e-07 default, 2.15e-07 / 4.00e-07 fast	identical, deterministic ×3 each	✓
upstream 644–976 slots	removed	n/a	✓
init range scope	339.50–356.06 (MoCE speed), 335.05–418.56 (all 14)	exact	✓
ja PPL / user-turn caveat	TF switched to assistant tokens. ja PPL is now 25.57 → 29.52, so the blowup is gone	exact	✓ (see §3 for new caveats)

1c. NEW PLAN numbers with no raw source (falsifier) #

PLAN claim	line	raw
IQ3_S 43-tok split 69.43% arena + 7.89% RA	§7.1 L580	52.60% + 24.72% (sum 77.32 ✓)	✗
Q2_0 MoCE=0 split 71.09% arena + 11.35% RA	§7.1 L581	63.86% + 18.58% (sum 82.44 ✓)	✗
1024-tok prefill 3,194.23 ms	§7.1 L578	3,194.19	✗
2048-tok prefill 5,793.62 ms	§7.1 L578	5,793.65	✗
6.39–6.44 GB/s Q2_0 fill	§7.1 L577	3.66–6.44 (strata_q2_0_moce0_short_43tok)	✗
6.82–7.07 GB/s IQ3_S fill	§7.1 L577	6.10–7.07 (6.82 appears nowhere)	✗
0.5s profile fill	§7.1 L577	0.9 s on q2_0_moce0 speed, 0.4 s on IQ3_S TF runs	✗ (range is 0.4–0.9 s)

this is the same failure mode as v2. the sums match, which suggests the splits were written to make the sums right, or were carried over from a non-archived run. none of the 7 strings occurs in any tail.

everything else in #29 / #30 / §7.1 #5 recomputes:

• every tok/s, ms/tok, combined hit rate, misses/tok, MB/tok, host wait and pure GPU value
• prefill 320.58 / 353.49
• 2.90 / 0.34 / 1.28 / 4.53 (dense GGUF = 4.220 GiB = 4.53 GB ✓)
• phys_footprint 4.73 → 4.67 and RSS 7.65 → 7.60 (identical in the 1024 and 2048 runs)
• the 2441 / 1550 / 1440 slot counts and 3.40 GB in all 14 runs
• 23.42 GiB / 15.82 GiB expert files (.experts sizes 23.419 / 15.820 GiB ✓). 85 pages per Q2_0 slot: 15.82 GiB / 12288 = 1.38 MB = 84.4 × 16 KiB ✓, and 3.40 GB / (85 × 16 KiB) = 2441 ✓
• all KLD / PPL / top-1 values

1d. #29 llama comparison: numbers recompute, but the comparison is invalid #

• patched step2_ngl99_cmoe_ub1024 decoded nothing. it shows eval time = 0.00 ms / 1 runs, and the echo ends …You are solving a Termina [end of text]. unpatched step2_ub1024 (llama-cli chat mode) generated ~64 thinking tokens at 1.4 tok/s.
  • so "whole-run disk0 drops from 105,851.73 MB to 46,808.58 MB on the 1,264-token -ub 1024 run" compares prefill-only against prefill plus about 64 decode steps.
  • the same goes for 85,725.73 vs 28,272.94 at ub2048.
• different binaries and modes.
  • unpatched is llama-cli (chat template applied, -st, no -t).
  • patched is llama-completion -no-cnv -t 8 (raw text).
  • so the unpatched prompt isn't 1,264 tokens (template tokens are added, and llama-cli never prints a count), and the 14.3 → 38.11 prefill ratio mixes binary, template and threads.
• "1.40 → 3.22 tok/s decode" compares an unpatched 1,264-token-context chat run against a patched 22-token raw run.
• the / 65.0 divisor in whole_run_disk0_mb_per_65tok* (runner L680, L689) is a literal, and the runs' disk0 includes load + prefill, so "MB / 65 tok" isn't a per-token rate.

2. code fixes: PASS, with two notes #

• ✓ window.cpp L1207: pg_scratch_need = prompt_gemm_need_bytes(). no 1.36 * gb literal remains anywhere in vendor/strata/metal, patch_strata_metal.py or the runner (the only 1.36 is in the self-test sample).

  • my Python mirror of the function gives 1.3586 GB for C = 1024, 4224 cells, PLE on, flash. that matches the logged 1.36, and both the plan line and the prompt_gemm_init line print 1.36.

• ✓ the stale string is fixed. prompt_gemm_init (L2140–2142) prints "in a separate transient mmap while a prompt runs" when SEPARATE_PROMPT_BUF != 0. the "in the expert arena" text survives only in the =0 branch, and it appears in no tail.

• ✓ "0 arena slots stolen" / "0.00 GB anon": no match in source, patch or any tail.

• ✓ prompt_region(false) (L2203–2215):

  1. proc_mem() (phys_footprint + resident_size via TASK_VM_INFO)
  2. cx_->free
  3. delete pg_
  4. release_region() → munmap
  5. purge_device_allocations
  6. proc_mem()
  7. the log line with both pairs

  all 14 tails carry it.

• ⚠ prompt_gemm_need_bytes() is a hand-maintained duplicate of make() (L2101–2129 vs L2166–2191).

  • it re-derives maxrows with a hard-coded 16 for p.TM (L2159, L2178), where make() uses p.TM, which defaults to 16.
  • nothing asserts that it equals the measured need. an edit to make() would silently desync the plan.
  • suggest: assert(need == prompt_gemm_need_bytes()) in prompt_gemm_init, or run make() in measure mode from both places.

• ⚠ the cap check is still log-only when STRATA_METAL_ARENA_GB is set, which it is in all 14 runs. the budget is reduced by need only in auto mode (L1208–1210). with an explicit arena, the scratch only adds a "peak prefill above cap" suffix. that's fine for these runs (6.5 < 18.8 GB cap), but not enforced.

• ⚠ PLAN #30 says "XNU bills only the ~0.06 GB CPU-faulted p.R+p.emb slice". p.R + p.emb = 1024 × (4·2560 + 2560) × 4 B = 0.052 GB, consistent with the 2-dp delta. RSS also only drops 0.05 GB, though (7.65 → 7.60), so the 1.36 GB scratch was never resident by either metric. the mechanism is a hypothesis, not measured. label it so, or measure with vmmap/footprint.

3. runner / data #

evict_file_cache / warm_file_cache: FAIL (evict) / PASS (warm) #

• the implementation is now ctypes libc.mmap(PROT_READ, MAP_SHARED) + madvise + munmap (L33–64). no more from_buffer TypeError swallow. ✓
• mincore probe (separate process, 16 KiB pages, 256 MiB file):
  • evict_file_cache (MADV_DONTNEED=4) on a fully cached file → returns True, residency 1.00 → 1.00. on Darwin, MADV_DONTNEED deactivates pages; it does not drop clean UBC pages. evict is a no-op.
  • warm_file_cache (MADV_WILLNEED=3) on a file written with F_NOCACHE (residency 0.00) → 1.00. warm works.
• the self-test is tautological. assert evict_file_cache(__file__) is True checks only the madvise return code. it can't detect the no-op.
• it targets the wrong file anyway.
  • evict/warm touch K256-*-metal.gguf, the dense 4.53 GB file. that one is mlocked per run, so init is warm in every run (335–419 ms).
  • the arena fill reads K256-*-metal.gguf.experts with F_NOCACHE preads, and neither function touches it.
  • the first run per expert file is still the slow one: q2_0_moce0 speed run 0.9 s / 3.66 GB/s, and iq3s speed run 6.10 GB/s vs 7.03–7.07 later. so the cold/warm asymmetry v2 flagged on the MoCE=0 vs MoCE=0.85 pair remains, now on the fill rather than on init.
  • decode tok/s is measured after the fill, so the 9.51 → 13.52 pair is less affected than in v2, but n = 1.
• fix suggestions:
  • drop evict, or use F_NOCACHE + fcntl(F_PURGE…) / purge (needs root). or accept warm-for-all and warm the .experts file region the profile fills.
  • make the self-test assert residency via mincore.

uniform arena: PASS #

all 14 strata runs have env.STRATA_METAL_ARENA_GB = "3.4" and log expert arena 3.40 GB. Q2_0 runs have 2441 slots / 1 class, IQ3_S runs 1550 slots (1440 filled) / 3 classes.

eval_tf_96tok_*: structure PASS, content caveats #

• all three are 96 ids:
  • ids[0..28] = <|im_start|>user\n … <|im_end|>\n (29 tokens)
  • ids[29..31] = <|im_start|>assistant\n
  • ids[32..95] = the 64 scored assistant tokens
• --n-prompt 32 --n-gen 64 matches. all 9 KLD runs report eval_tokens=64. hashes match provenance.
• caveats, from decoding with the GGUF vocab:

  1. the user turns are left-truncated fragments.

     • ops_tools: );\n df.erase();\n\nroot@f71c07bf8043:/app#
     • chat_json: {19}~?$$\n\nSolve this step by step…
     • ja: カイブ\n36\n\n質問: 5G医療救急車…

     the responses refer to context that isn't there. ja says 「文章の内容を理解する」 about a document that's absent, and ops_tools says "The fix looks correct… testValidate()". so this is off-context teacher forcing.

  2. all 64 scored tokens are <think> reasoning. the first scored token is the <think> template token (248068), which is near-deterministic: 1/64 per domain is free agreement.

  3. chat_json is a modular-arithmetic math problem, not chat/JSON. the domain label is wrong.

  4. ops_tools has <think> with no following \n, unlike the other two and the chat template.

  5. still n = 64 tokens × 1 doc/domain, and no CI.

llama_prompt_1264tok.txt: FAIL (wrong file archived) #

• the archived file is 1,600 bytes (sha 334a7331…, matching provenance).
• every llama run's cmd uses -f /tmp/m5_prompt_1600tok.txt. that file still exists: 5,600 bytes, mtime 05:24:16, sha 2549439506dd….
• the archived file is exactly its first 1,600 chars.
• the patched run's prompt echo ends with "…Stop only when you believe the task is complete…\n\n\nYou are solving a Termina". that text is in the /tmp file's last 200 chars and absent from the archived file.
• so the runs used the 5,600-char file. the name "1600tok" was the char count of an earlier version, and "1264tok" now labels a ~350-token prefix.
• fix: copy /tmp/m5_prompt_1600tok.txt into data/m5_val_prompts/ before /tmp is cleared, and re-hash it.

llama_cpp_* from prev_llama_*: PASS #

• runner L665–692 indexes prev_llama_unpatched / prev_llama_patched by step. no literals remain except the / 65.0 divisor (§1d).
• the llama runs are still carried over (env: null), not re-run.

4. provenance / tests #

• all 5 script/binary SHA-256s and all 7 prompt hashes match the disk.

• mtimes:

  • patch 07:54:46
  • moe.metal 07:54:46.45
  • route_parity 07:54:47.65
  • strata_window_cli 07:54:49.78
  • runner 08:01:11.43
  • JSON 08:03:32.52 (generated_at_utc 05:03:32.517Z)

  runs sum to 138.55 s, leaving ~2.5 s for vm_stat/iostat and launch. consistent with build → edit runner → run → write.

• route_parity (6 runs, all exit 0, deterministic):

  • default math: K=512 … 2.16e-07, K=256 + MoCE=0.85 … 3.06e-07 PASS
  • STRATA_METAL_MATH=fast: 2.15e-07, 4.00e-07 PASS
  • matches PLAN #30 exactly.

• --self-test: exit 0, but the PASS line has a garbled prefix: self-test: 0 scripts/run_phase0_phase2_m5_validation.py self-test: PASS. the evict assertion is tautological (§3).

falsifier summary #

falsifier	result
PLAN vs raw	7 cells / 9 numbers unsourced (§1c): IQ3_S and Q2_0-m0 arena/RA splits, 2 prefill ms, the fill-rate ranges, the fill time
summary_comparison vs raw	0 mismatches
hard-coded literal	1.36 * gb gone ✓. remaining: the / 65.0 llama divisor, and the hard-coded 16 (TM) in the prompt_gemm_need_bytes() mirror
archived inputs	the llama prompt archive is the wrong file (1,600-char prefix of the 5,600-char prompt used)
cache control	evict_file_cache is a verified no-op, and it targets the dense file, not .experts

what to fix #

1. PLAN §7.1 L577–581: put in the raw splits (52.60 + 24.72, 63.86 + 18.58), the prefill ms (3,194.19, 5,793.65) and the fill ranges (3.66–6.44 Q2_0 incl. the cold first run, 6.10–7.07 IQ3_S, 0.4–0.9 s).
2. PLAN #29: state that the patched ub1024/ub2048 runs generated 0 tokens and used llama-completion raw vs llama-cli chat, so the disk0/prefill/decode ratios aren't like for like. or re-run both with the same binary, mode and -n.
3. archive the real /tmp/m5_prompt_1600tok.txt (5,600 B, sha 2549439506dd…).
4. replace or drop evict_file_cache, warm the .experts file if cold/warm parity matters, and test residency with mincore in --self-test.
5. assert prompt_gemm_need_bytes() == need in prompt_gemm_init.
6. TF eval: use untruncated user turns (or a document-free prompt), relabel chat_json, and exclude the template <think> token from scoring. n ≥ 3 docs/domain.