opus audit: phase 0 + phase 2 m5 air validation & strata metal patches #

scope: scripts/patch_strata_metal.py, scripts/prepare_strata_k256.py, vendor/strata/metal/src/convert_k256_q2_0.cpp, the patched vendor/strata/metal tree (git -C vendor/strata diff), scripts/patch_m5_lazy_moe.py, scripts/run_phase0_phase2_m5_validation.py, results/phase0_phase2_m5_validation.json, PLAN.md §2.7.A #29–#30 and §7.1 item 5. read-only, except one run of the existing vendor/strata/build-metal/route_parity binary.

verdict: ISSUES. the per-run raw numbers for the 1024/2048-token strata runs are real and recompute exactly. the patches work for K=256, and the Q2_0 layout is right. but:

1. the short-prompt headline row in PLAN §7.1 item 5 and §2.7.A #30 has no raw source. 13.88 tok/s, 72.04 ms/tok, 92.85% (87.80% + 5.05%), 34.3 misses/tok, 84.0 MB/tok, 13.60 ms wait, 57.4 ms GPU and the 83.8 -> 34.3 / "−59%" claim appear nowhere in the archived raw runs. the archived short run gives 13.80 tok/s, 91.24%, 42.1, 107.8 MB/tok, 16.05 ms, 55.79 ms. summary_comparison in the same JSON was edited by hand to the unarchived numbers.
2. the llama.cpp 3.81 tok/s decode and the -7.8x NVMe-transfer claim have no raw source either. the "MB per decode tok" figures (1,628 / 515) are whole-run disk0 deltas (load + prefill + decode) divided by 65, measured on runs with different prompts.
3. the code that produced the results is not the code in the tree. the results JSON is from 06:52:33. patch_strata_metal.py, every patched vendor file and strata_window_cli all have mtimes from 06:54:06–06:54:09. the JSON's schema is also not the one run_phase0_phase2_m5_validation.py writes, and no strata runner, cmdline or env (STRATA_METAL_MOCE_BONUS!) is archived.
4. code-level issues: a GPU/IO-thread race on smap inside the MoCE router can mis-scale w. ahead() does not mirror the top-3 protection. MADV_DONTNEED does not reclaim the prompt scratch on darwin, and hold_region(true) memsets all 1.36 GB on every prompt. the "dropped 1.60 GB" line is a hard-coded string, and PLAN's "+1.60 GB freed vs upstream" is wrong: upstream already skipped the BF16 HC originals, so the real saving is ≈0.34 GB. the IQ3_S/IQ4_NL→Q2_0 requantized model and MoCE bonus 0.85 have no quality measurement, so the phase 1c KLD numbers do not describe the model that was benchmarked.

---

1. metrics vs raw output (results/phase0_phase2_m5_validation.json) #

i recomputed every strata run from STRATA_RESULT and the expert stream line in output_tail. decode = 63 / eval_ms, prefill = prompt_tokens / prefill_ms, and combined = arena + read-ahead. the parsed perf dicts match the tails exactly.

claim (PLAN §7.1 #5 / §2.7.A / task)	raw source	status
320.80 tok/s 1K prefill (3.19 s)	step2: 1024 tok / 3191.98 ms = 320.80	✓
363.58 tok/s 2K prefill (5.63 s)	step3: 2048 / 5632.86 ms = 363.58	✓
13.53 tok/s (73.91 ms) after 1K	step2: 63 / 4656.22 ms = 13.53	✓
12.96 tok/s (77.17 ms) after 2K	step3: 63 / 4861.89 ms = 12.96	✓
91.32% / 99.9 MB/tok after 1K	0.8566 + 0.0566 = 0.9132; 99.9	✓
90.73% / 114.5 MB/tok after 2K	0.8343 + 0.0730 = 0.9073; 114.5	✓
13.88 tok/s (72.04 ms/tok) short prompt	step1: 13.80 tok/s, 72.49 ms/tok	✗ not in raw
92.85% = 87.80% arena + 5.05% RA	step1: 91.24% = 84.33% + 6.91%	✗ not in raw
34.3 misses/tok	step1/2/3: 42.1 / 41.7 / 44.5	✗ not in raw
84.0 MB/tok (lower end of "84.0–114.5")	step1/2/3: 107.8 / 99.9 / 114.5	✗ not in raw; true range 99.9–114.5
13.60 ms SSD wait + 57.4 ms GPU	step1: 16.05 ms wait, 55.79 ms pure GPU	✗ not in raw
83.8 -> 34.3 misses/tok (−59%)	no MoCE-off run archived at all	✗ unsupported (the arithmetic 49.5/83.8 = 59.1% is right)
"12.96–13.88 tok/s" decode range	raw range 12.96–13.80	✗ upper end
"90.73%–92.85%" hit-rate range	raw range 90.73%–91.32%	✗ upper end
0.34 s (341 ms) init	384.31 / 340.83 / 340.94 ms	✓ (the best 2 of 3; short run 384)
0.5 s (6.94 GB/s) arena fill	0.5 s 6.94 / 0.6 s 6.84 / 0.5 s 6.94	✓
2,728–3,015 slots, 3.8–4.2 GB arena	every run: 2728 slots, 3.80 GB	✗ 3,015 / 4.2 GB not in raw
2.90 GB mlocked	log locked 2.90 GB (measured locked_bytes)	✓, and matches the header: live dense 2.897 GB, page-rounded 2.904 GB
1.60 GB dead HC + embd dropped	log text is a hard-coded literal ("dropped 1.60 GB ..." in the patch)	⚠ size ≈ right by header (1.279 GB BF16 hc_* + 0.338 GB IQ4_XS token_embd = 1.617 GB), "dropped" unmeasured, see §2.3
1.36 GB scratch, 0 arena slots stolen	log line is printed by region() itself; "0 slots stolen" is a literal	⚠ true by construction (region_slots_.clear()), not a measurement
0.0 MiB swapouts	max_swapout_delta_mib = 0.0 in all 11 runs	✓ (0.25 s poll, min_avail 2.376–2.398 GiB on strata runs, 0.18–0.2 GiB above the 2.2 GiB fuse)
llama unpatched 1.40 tok/s, 14.30 tok/s "1K prefill"	unpatched step2_ub1024 bracket line: 14.3 / 1.4	✓ value, ✗ label: 1264-token prompt at -ub 1024, not a 1K prompt
llama patched 38.11 "1K" / 63.27 "2K prefill"	patched step2 ub1024 / ub2048	✓ values, ✗ labels: both are the same 1264-token prompt, -ub 1024 vs 2048. neither is a 2K prompt, so it is not comparable with strata's 2048-token 363.58
llama patched 3.22–3.81 tok/s decode, "1.4 -> 3.81"	only decode run: step1 3.22. step2 runs hit EOS at once (eval 0.00 ms / 1 runs). step3 --moe-cache-mib 1536 = 0.28 tok/s	✗ 3.81 not in raw. the 0.28 tok/s moe-cache regression isn't reported
1,628 MB/tok unpatched	unpatched step2_ub1024 disk0 105,851.73 MB / 65 = 1628.5	⚠ whole-run I/O (load + 1264-tok prefill + decode) / 65, not per decode token
515 MB/tok patched	patched step1 disk0 33,428.34 / 65 = 514.3	⚠ same whole-run metric, rounded up, and on a 22-token prompt, so the comparison with 1,628 mixes prompt lengths
-7.8x NVMe transfers	no pair of runs gives 7.8 (pairwise xfrs ratios 0.57–26.2; nearest 8.10 = unpatched step3 / patched step1)	✗ unsupported
"96 graph splits/token"	not in any archived output	unverified

the JSON's own summary_comparison.strata_metal_optimized_k256_q2_0 carries the unarchived numbers (13.88, 0.9285, 34.3, 84.0) next to raw runs that contradict them. summary_comparison.llama_cpp_patched_m5_lazy_moe.decode_tok_per_s = 3.81 contradicts the only raw decode (3.22). fix: either archive the runs those numbers came from (with env + cmdline), or replace them with the archived 13.80 / 91.24% / 42.1 / 107.8 / 3.22 and drop the −59% and −7.8x claims.

other measurement caveats:

• every strata number is n = 1 run of 63 decode tokens: no variance, no repeat.
• prefill_ms in strata_window_cli includes the lazy arena fill (~0.5–0.6 s, logged after ready in …ms), the 1.36 GB memset in hold_region(true), and the first decode step. so real GEMM prefill is faster than 320.8/363.6, but these are cold-start numbers, not prefill throughput.
• the 1024/2048-token prompt token files aren't archived. decoded GEN_IDS show step2 continues raw text without a chat template, and step3 ends a truncated chat turn. all three outputs are coherent: step1 begins "The user is asking about SQLite's WAL (Write-Ahead Logging) mode…".
• watchdog_policy.max_swapout_delta_mib = 128 in the JSON, but run_guarded_llama uses its 64 MiB default (main() never passes it), and the module docstring still says min_avail 3.0 GiB.

2. code review #

2.1 dynamic n_expert (K=256): PASS, with notes #

• router.metal route_body, moe.metal moe_route and prompt.metal route_topk mask idx >= n_expert to -INFINITY before the max, set exp to 0, and set masked v to -INFINITY after normalisation. the logits row stride stays 512, consistently with GemvArgs{…, n_exp_, …, 512} and logits_ = A(4*MAXT*512). ✓
• the fused router_route grid is (n_exp_+15)/16. the "last threadgroup routes" test uses ngx (threadgroups_per_grid), so it follows the grid size. plan_body loops e < a.n_expert. ✓
• route_topk packs n_expert << 16 | n_tok. chunks are ≤1024 tokens, so no overflow. PgPlan/PcMmPlanArgs.n_expert = n_exp_, and mm_scan launches n_exp_ threads. moe_mm.metal already handles n_expert < 512 ("simdgroups past the launched ones must count 0"). ✓
• stream.cpp: ne_ comes from blk.0.ffn_gate_exps.weight shape[2], lay_ slice sizes are bytes / ne_, the STRX header check is ne == ne_, and the STRP check is h[2] == ne_. where_/smap_ keep the 512 stride, which matches the l*512*4 smap offsets in window.cpp. ✓
• window.cpp decode gemv(…, logits_, 512, ntok) and the predL_ gemvs pass 512 as y_stride, and gemv() takes n_out from t.shape[1] = 256. no OOB. ✓
• waste, not a bug: PromptGemm and prompt_chain.inc still run gemm(…, Wr, logits, T, NEXP, N) with NEXP = 512. the F16 Wr copy has rows 256..511 that are never written, so prefill computes a router GEMM twice as large as needed. the extra rows are masked in route_topk. pass n_exp_ here.
• untested at K=256: i ran route_parity and it passes (0 mismatches, worst weight rel err 2.16e-07). but it sets a.n_expert = 512 and pad = 0. it exercises neither the K=256 masking nor MoCE, so PLAN's "(route_parity: 0 mismatches across 200 windows)" doesn't back the n_expert claim it's attached to. tests/token_chain.cpp only got the extra buffer(10) argument and still hard-codes 512 (GemvArgs{N, 512, …}, grid 512/16). the patch_strata_metal.py docstring wrongly lists it as a dynamic-n_expert patch target.
• the MTP path (window.cpp ~L1806) still has a.n_expert = NEXP and no smap. it's fine as long as the MTP head keeps 512 experts, but nobody has checked that.

2.2 MoCE cache-prior reranking: the logic is right, with one race and one mismatch #

the GPU logic does what PLAN says:

• ranks 0..2 are picked from unperturbed v. at rank == 3, every resident (smap[idx] != 0xFFFFFFFF), still-unpicked, v > 0 candidate is scaled by moce_scale = exp(bonus). moce_pad_ is exp(atof(STRATA_METAL_MOCE_BONUS)), bit-cast into the previously unused MoeArgs.pad. ✓
• for picks at ranks ≥3 whose smap is resident, true_prob = best / scale. ssum and selected use true_prob, so w is the unperturbed probability renormalised over the selected 10. ✓ multiplying the prob by e^b is the same as adding b to the logit, which is consistent with eff_lg in ahead().
• with stream_ == false, a.pad = 0, so the rcounter_ buffer bound as smap is never read. ✓

problems:

1. race on smap (medium). smap is read twice per candidate: once when scaling at rank 3 and once when restoring. I/O threads write smap_[l*NE+e] = code when a read completes (io_main, L368), and victim()/ahead() evictions write 0xFFFFFFFF. both run while the GPU executes the single 48-segment command buffer, and read-aheads for layer l+1/l+2 complete during it. if an expert's entry flips between the two reads, w for that expert is off by ×e^±0.85 (2.34×) with no error. fix: record a per-lane 16-bit "scaled" mask at rank 3 and test that mask instead of re-reading smap[ex] (a simd_shuffle of the owning lane's bit).
2. ahead() is only an approximate match for the GPU rule. eff_lg adds the bonus to every resident candidate, top-3 included, and uses where_ >= 0, which counts in-flight reads as resident. the GPU uses smap, which counts completed reads only, and protects the top 3. so a non-resident expert in the true top-3, which the GPU always picks, can be pushed out of the top K = 8 read-ahead by ≥8 boosted residents. the read-ahead under-predicts exactly the misses MoCE can't avoid. fix: rank by raw lg for the first 3, then by eff_lg.
3. exactness and quality are unmeasured. MoCE with bonus 0.85 (×2.34 prob) changes which experts run for ranks 3–9. phase 1b's closed-loop λ = 0.09 is a different scale, and PLAN itself says MoCE quality is deferred to phase 5. nothing in the archived runs records whether the env var was set, so we can't even tell whether the 91% hit rates are with or without MoCE.
4. the hot-slot protection (slot.hot, cap n - (ne_+32) = 2,440 of 2,728) is set only by the arena fill. hold_region(false) clears all hot flags after the first prompt, so from the second prompt of a multi-turn session onward, prompt streaming evicts decode-hot experts again. the single-prompt CLI benchmark can't show this. PLAN #30's "protected … during prompt streaming chunks" holds for the first prompt only.

2.3 zero-copy dense + mlock + DONTNEED: works, but the savings are overstated #

• the header of K256-q2_0-metal.gguf gives 1079 dense tensors, max_dense_end = min_exp_off = file size = 4.5317 GB. the expert tensor infos point past EOF, so this is a dense-only file and the wrap guard passes. newBufferWithBytesNoCopy is called on a PROT_READ, MAP_SHARED map with a page-aligned length. ✓ skip tensors keep their real file offsets, and since W_ wraps from f.map + 0, every non-skip offset stays valid. ✓
• live dense = 2.897 GB, 2.904 GB page-rounded, which matches the measured locked 2.90 GB. ✓ BF16 hc_* (non-norm) = 290 tensors / 1.279 GB, which matches HC read in Q8_0: 290 matrices, 680 MB (Q8 = 679.6 MB). token_embd = IQ4_XS 0.338 GB.
• PLAN #30's premise is wrong. upstream load_dense (git show HEAD:…/stream.cpp) already skipped the hc_* originals (offset = 0; continue). it copied only the other 3.24 GB (4.514 − 1.279) into anon memory, not "4.53 GB", via F_NOCACHE preads. the HC Q8 conversion also reads the BF16 originals through st_->read (F_NOCACHE pread) when streaming, so those mmap pages are never faulted in, and MADV_DONTNEED on them is a no-op. compared with upstream, the change trades 3.24 GB of anon memory for 2.90 GB of mlocked file pages, a real saving of ≈0.34 GB (token_embd), not "+1.60 GB freed for the expert arena". both kinds are equally unreclaimable once wired.
• the log line "(dropped 1.60 GB dead BF16 HC + CPU embd pages)" and "0.00 GB anon" are string literals in the patch. they don't count anything. compute and print the actual skip/embd byte totals.
• mlock failures are silent (locked_bytes just comes out smaller). fine, since the run printed 2.90.

2.4 separate PromptGemm scratch: the "reclaim" doesn't work as described #

• region() with STRATA_METAL_SEPARATE_PROMPT_BUF=1 (the default) mmaps a 1.36 GB MAP_ANON|MAP_PRIVATE buffer, and window.cpp wraps it with count_allocated=false. ✓ no arena slots are taken.
• hold_region(true) still does memset(region_, 0, 1.36 GB) on every prompt. that made sense when the region aliased arena slots. for a fresh separate buffer it dirties all 1.36 GB on every prompt and is counted in prefill_ms.
• MADV_DONTNEED on darwin anonymous memory does not discard contents or reduce phys_footprint. it's advisory and the contents are preserved. under pressure the dirty pages get compressed or swapped, not dropped. the darwin idiom is MADV_FREE_REUSABLE, with MADV_FREE_REUSE before reuse, or munmap + re-mmap (sources: Mozilla bug 1567366, Chromium's switch to MADV_FREE_REUSABLE, arcbox's measurement that DONTNEED leaves "footprint unchanged, contents preserved"). the buffer is also wrapped as a Metal buffer, which can keep it resident. so "reclaimed via MADV_DONTNEED after prefill" is unverified and probably false. nothing measured footprint after prefill: the only footprint line is printed before the prompt.
• count_allocated=false also takes the 1.36 GB out of the arena planner's "planned resident footprint 5.5 GB". the real peak during prefill is ≈ 5.0 + 1.36 GB, which the cap logic never sees. on this machine min_avail already bottomed at 2.38 GiB against a 2.2 GiB fuse, so a smaller-RAM or busier mac has less margin than the planner thinks. fix: count the scratch in the plan, skip the memset for the separate buffer, use MADV_FREE_REUSABLE/munmap, and log task_info phys_footprint after prefill to prove the reclaim.

2.5 convert_k256_q2_0.cpp vs ref_quant.cpp: layout PASS, quality unmeasured #

• block = 64 elements / 18 bytes. d is fp16 at bytes 0–1, then qs[16]. element i = 4j+k goes in byte 2 + j at shift 2k, which is exactly ref_quant.cpp deq_block(Q2_0): b[2 + j/4] >> ((j%4)*2) & 3. code = q + 1 and value = (code − 1)·d. ✓ it matches the ggml codec recorded in PLAN #28.
• slice = 2560·640/64·18 = 460,800 B and expert = 1,382,400 B. 48·256 experts = 16.987 GB, which matches the file size of K256-q2_0-metal.gguf.experts (16,986,947,584 B = 15.82 GiB). 85 × 16 KiB slots × 2728 = 3.799 GB arena. ✓ rows (2560 for gate/up, 640 for down) are multiples of 64, so no block straddles a row. ✓ the input .experts is 23.42 GiB. ✓
• the header patch rewrites only the type of the 144 *_exps tensor infos. gguf_lite recomputes bytes from type and shape, so lay_ sizes match the STRX table, and src_sz (the input file size) equals the byte-copied output file. ✓
• the quantizer searches d ∈ {0.5…1.0}·amax (9 steps) for minimal SSE on the asymmetric {−1,0,1,2} grid. that's reasonable, and better than ggml's fixed d = amax. note the error is evaluated with the unrounded d.
• the quality gap is high-impact. this is a second lossy quantization (IQ3_S 3.44 bpw / IQ4_NL 4.5 bpw → Q2_0 2.25 bpw on already-quantized weights). phase 1c's K256_reap_l2_sum KLD 0.564 and top-1 79.8% were measured on the IQ3_S GGUF, not on this file. no KLD, top-1 or tier-2 result exists for K256-q2_0-metal, alone or with MoCE 0.85. every strata speed number in §7.1 #5 is for a model of unknown quality.
• per-expert size isn't a multiple of 4 KiB (1,382,400 = 337.5 pages), so odd experts start 2 KiB off alignment for F_NOCACHE preads. upstream's mixed format was unaligned too, so this isn't a regression, but padding each expert to 85 × 16 KiB in the file would make every read aligned.

2.6 prepare_strata_k256.py #

• STRP: <4s5I = magic, ver, NL, NE, n, n, and the reader checks h[1]==NL, h[2]==ne_ and iterates h[4]. ✓ new expert ids follow keep-map order, which is sorted, and the slicer also sorts (slice_qwen4exp_gguf.py L511). ✓
• the profile is ranked by in-sample REAP calibration counts. if the eval prompts (the SQLite prompt, data/eval_ops_tools.txt) overlap that calibration distribution, the warm hit rates are optimistic. not checked.
• the docstring says "6.8 GiB dense" and "17.0 GiB STRX". the real files are 4.22 GiB and 23.42 GiB. stale.

2.7 patch_m5_lazy_moe.py #

• the lazy-range, F_RDAHEAD 0 and POSIX_MADV_WILLNEED prefetch logic is sound. the decode fast path (ids->ne[1] <= 2) gives each thread whole experts (one_chunk(…, 0, ne01, 0, cne1, …)) and returns early. that's correct, but load-balanced only to whole-expert granularity (10 experts over 8 threads).
• the patched --moe-cache-mib 1536 path collapsed to 0.28 tok/s (step3). PLAN never mentions it, and it's the one result that argues against that path.

3. provenance #

• results/phase0_phase2_m5_validation.json mtime 06:52:33. scripts/patch_strata_metal.py and every patched vendor/strata/metal/** file 06:54:06, build-metal/strata_window_cli 06:54:09. patch_strata_metal.py starts with git checkout -- ., so the vendor mtimes would move even for a byte-identical re-run, but the script itself was rewritten after the results. the measured binary is unrecoverable.
• run_phase0_phase2_m5_validation.py writes {hardware, model_variant, shards, initial_vm_stat, runs}. the results file has {summary_comparison, strata_optimized_runs, llama_cpp_patched_runs, llama_cpp_unpatched_runs, watchdog_policy}, so it was assembled by something that isn't in the repo. the strata runs carry no cmd, vm_before/vm_after or env, so STRATA_METAL_MOCE_BONUS, --n-expert, --ple-gguf and the prompt token files can't be recovered.

4. what to fix (in order) #

1. PLAN §7.1 #5, §2.7.A #29–#30 and summary_comparison: replace the unarchived numbers with the raw ones (13.80 / 91.24% / 42.1 misses / 107.8 MB/tok / 16.05 ms / 55.79 ms. llama decode 3.22, ssd metric relabelled "whole-run MB / 65 tok". prefill labels "1264-tok prompt at ub=1024/2048"). or re-run and archive the MoCE-on vs MoCE-off pair behind −59%. drop −7.8x and "+1.60 GB freed" (≈0.34 GB vs upstream). add the 0.28 tok/s moe-cache result.
2. commit a strata runner that records cmdline, env, binary hash and vendor diff hash into the JSON, plus the prompt token files. re-run 3× per config.
3. measure the quality of K256-q2_0-metal (KLD/top-1 vs the IQ3_S K256 and vs unpruned) at bonus 0 and 0.85 before anyone quotes its tok/s as the finalist's speed.
4. MoCE: a register mask instead of the second smap read, top-3 protection in ahead(), and a parity test with n_expert = 256, pad = exp(0.85) and a random smap, against a CPU replica.
5. scratch: no memset for the separate buffer, MADV_FREE_REUSABLE or munmap, count it in the arena plan, and log phys_footprint after prefill. replace the hard-coded "1.60 GB" / "0.00 GB anon" / "0 slots stolen" strings with computed values.
6. prompt router GEMM over n_exp_ rows, not NEXP. keep the hot-slot flags (or re-mark them) across prompts.