opus audit v2: verification of the 6 fixes from audits/opus_audit_phase0_2_m5.md §4 #

scope: read-only re-audit of PLAN.md §2.7.A #29–#30 and §7.1 item 5, results/phase0_phase2_m5_validation.json, scripts/run_phase0_phase2_m5_validation.py, scripts/patch_strata_metal.py, data/m5_val_prompts/*.txt, the patched vendor/strata/metal tree and vendor/strata/build-metal/{route_parity,strata_window_cli}. the only things i ran: route_parity (3×), run_phase0_phase2_m5_validation.py --self-test, and a GGUF header parse of K256-q2_0-metal.gguf. no repo files were edited.

verdict: MOSTLY FIXED, falsifier hit (PLAN only).

• every number in summary_comparison recomputes exactly from the archived raw tails: 0 mismatches across 14 strata runs, 3×3 KLD blocks, and 2 llama blocks.
• the provenance chain is now real: the hashes match, the cmd and env are archived, and the prompt files are archived and hashed.
• the MoCE race is fixed and ahead() mirrors the top-3 protection.
• but PLAN §7.1 #5 / §2.7.A still quote 12 numbers that are in no archived run. they look like they came from an earlier run: arena/read-ahead hit splits, host-wait/GPU ms, a fill rate, a slot count, the run count, and the route_parity errors.
• two hard-coded or stale log strings remain in the prompt-scratch path (§5).

1. PLAN / summary_comparison vs raw runs #

method: i re-parsed every output_tail myself (STRATA_RESULT, expert stream, window T1, expert arena, process footprint, STRATA_KLD, STRATA_KLD_REF2, llama common_perf_print / [ Prompt | Generation ]). then i recomputed perf, summary_comparison and the PLAN strings from that.

1a. summary_comparison: PASS (0 mismatches) #

• all 5 speed runs and 9 KLD runs: the perf dicts equal the tails, including the derived fields (combined = hit + ra, pure_gpu = T1 GPU − host_ms, misses/tok = miss_rate × 480).
• strata_metal_iq3s_moce0_43tok, strata_metal_q2_0_moce0_43tok and all 30 fields of strata_metal_optimized_k256_q2_0_moce085 equal the source runs.
• miss reductions: 1 − 40.1/83.2 = 51.80% ✓ and 1 − 40.1/108.9 = 63.18% ✓.
• llama: 105,851.73/65 = 1628.49 ✓, 33,428.34/65 = 514.28 ✓, 46,808.58 ✓, 0.28 tok/s (--moe-cache-mib 1536, 221,796.57 ms / 63) ✓, 3.22 ✓, 14.3/21.7/1.4 ✓, 38.11/63.27 on 1264 tokens ✓. the relabelling to "1,264-tok" and "whole-run disk0 / 65 tok" is correct.
• 0.0 MiB swapout in all 22 runs ✓. min_avail is 2.85–5.50 GiB, against the 2.2 GiB fuse.

1b. PLAN text: FAIL, 12 numbers have no raw source #

everything else in #29, #30 and §7.1 #5 matches:

• decode 7.22/9.75/12.97/13.46/12.44 tok/s, the ms/tok, combined hit rates, misses/tok and MB/tok.
• prefill 319.39 (3.21 s) / 342.87 (5.97 s).
• 2.90/0.34/1.28/4.53 GB. i rechecked these against the GGUF header: dense non-hc = 2.904 GB, token_embd = 0.338 GB, 290 BF16 hc_* = 1.279 GB, all dense = 4.521 GB.
• footprint 4.83 -> 4.77.
• all KLD/top-1 macros and per-domain values (§3).

fix: replace the 12 cells with the raw values above, or archive the runs they came from.

2. provenance: PASS, with notes #

• all 5 SHA-256s in provenance match the files on disk: runner, patch_strata_metal.py, patch_m5_lazy_moe.py, prepare_strata_k256.py, strata_window_cli. all 6 prompt-file hashes match too. token counts: 43, 1024 and 2048 for the speed prompts, 96 for each of the three eval_tf_96tok_* files.

• mtime order:

  • prompts 07:25:22
  • patch_strata_metal.py 07:34:49.890
  • patched vendor files 07:34:49.98
  • libstrata_metal_kernels.a 07:34:53.54
  • strata_window_cli 07:34:53.60
  • runner 07:36:34.34
  • JSON 07:38:52.72 (generated_at_utc 04:38:52.716Z = same instant, UTC+3)

  that is consistent with build → run → write. tight: the runs sum to 137.89 s, plus about 0.6 s of vm_stat/iostat overhead (measured at 19 ms per pair), which puts launch at about 07:36:34.1 ± 0.3 s. so the runner was saved roughly at the moment it was launched. this can't be resolved further. the schema, env keys and summary keys exactly match the code that is in the tree.

• the metal kernels are compiled at run time from vendor/strata/metal/kernels (STRATA_METAL_KERNELS_DEFAULT), so the benchmark used the 07:34:49 shaders.

• the cmd/env per strata run match the runner's construction. note that IQ3_S uses ARENA_GB=3.4 but the Q2_0 speed runs use 3.5 (1550 vs 2513 slots), so the "−63.2% vs IQ3_S" figure mixes format, slot count and MoCE.

• the llama runs were not re-run. the runner copies them from the previous JSON. they have env: null, and patch_m5_lazy_moe.py (05:22) predates them only as far as anyone can tell. the llama blocks of summary_comparison are hard-coded literals in the runner (lines 647–664), not computed. they happen to equal the raw. /tmp/m5_prompt_1600tok.txt, the llama prompt, is not archived.

• route_parity binary 07:23:30 vs route_parity.cpp 07:34:49: the source mtime moved because of the patch's git checkout -- . rewrite. the binary's format strings equal the current source, and the kernel under test is compiled at run time from the current moe.metal.

• evict_file_cache() is a silent no-op. (c_char*1).from_buffer() on a PROT_READ mmap raises TypeError: underlying buffer is not writable (reproduced on a /tmp probe), and except Exception: pass swallows it. the effect: strata_q2_0_moce0_short_43tok ran with a cold Q2_0 dense file (init 1038 ms, fill 3.06 GB/s), while the MOCE=0.85 runs ran warm. misses/tok is a count, so the −51.8% holds, but the tok/s pair isn't like for like. n = 1 everywhere.

• --self-test: self-test: PASS (exit 0).

3. quality measurement: PASS on the arithmetic, the caveats matter #

• strata_window_cli.cpp teacher-forcing logic is correct.
  • positions: logits after the last prompt token predict all_ids[n_prompt]. each decode step feeds all_ids[n_prompt+i] and scores all_ids[n_prompt+i+1]. that's 1 + 63 = 64 positions = n_gen per domain, 192 in total.
  • the FP64 log-sum-exp and Σ p·(log p − log q) run over the full n_vocab, which the header gives as token_embd [2560, 248320] ✓.
  • --logits-ref2 is scored against the same candidate logits ✓.
  • PPL = exp(mean NLL) ✓.
  • the moce085 run is TF too, so its arena state follows the same tokens as the reference.
• every per-domain value and macro recomputes exactly from the tails:
  • Q2_0 m0 vs IQ3_S m0 0.56976 / 72.40%
  • Q2_0 m0.85 vs Q2_0 m0 0.056163 / 92.71%
  • Q2_0 m0.85 vs IQ3_S m0 0.589351 / 71.36%
• caveats that PLAN doesn't state:
  1. the TF text is a user turn. every eval file starts 248045 846 198 (<|im_start|>user\n). it is not assistant output, so this measures prompt-text modelling, not the response distribution.
  2. the sample is n = 64 tokens × 1 document per domain. no CI.
  3. the ja PPL goes 53.87 -> 406.29 (7.5×) under Q2_0. PLAN quotes only KLD 0.96 and top-1 59%. the PPL blowup is the louder signal, and it belongs in §7.1.
  4. chat_json PPL improves (52.0 → 30.6) at KLD 0.18. with 64 tokens, PPL is noise-dominated.
  5. the reference is IQ3_S K256 (itself pruned and quantized), so these are increments, not distance to unpruned.
  6. the KLD runs use a 3.4 GB arena (2441 slots), the speed runs 3.5 GB (2513), so MoCE's routing-change rate differs slightly between the quality and speed numbers.
  7. p90 takes index floor(0.9·63) = p≈89, and the median is the upper median. both are minor.

4. MoCE smap race, ahead(), route_parity: PASS #

• moe.metal moe_route and router.metal route_body:

  • smap[idx] is read once, at rank == 3, and the result goes into a per-lane ushort scaled_mask.
  • the restoration for picks at rank ≥ 3 uses simd_broadcast(scaled_mask, ex & 31) and bit ex >> 5, with no second smap read. so an I/O-thread flip can no longer desync scale and unscale. the race is fixed ✓.
  • ranks 0..2 are picked from unscaled v ✓. lanes ≥ n_expert are -INF ✓.

• the decode path actually used is moe_route: routef_ needs STRATA_METAL_ROUTE=1, and ffn_gate_inp is F32. that's the kernel route_parity covers. router_route carries the identical body but has no parity test.

• stream.cpp ahead(): the first min(3,K) come from raw lg (unperturbed), and the rest are ranked by eff_lg = lg + bonus·[smap resident]. it now reads smap (completed reads), not where_. top-3 get prio +1e3 so they're submitted first ✓.

  • nit: eff_lg re-reads sm[e] racily during the selection, which is harmless for a prefetch heuristic.

• route_parity, 3 runs, identical output each time:

  • K=512 (pad=0, 200 windows): mismatches 0, worst weight rel err 2.16e-07
  • K=256 + MoCE=0.85 (random smap + 256..511 traps, 200 windows): mismatches 0, worst weight rel err 3.06e-07 PASS

  limits: smap is static per dispatch, so the test can't exercise the race, and the race fix rests on code review. route_topk (prompt, K=256) and router_route aren't covered.

5. separate PromptGemm scratch: logic PASS; hard-coded/stale strings FAIL #

• ✓ region() (SEPARATE_PROMPT_BUF=1): fresh MAP_ANON with region_bytes_ = 0, so hold_region(true)'s memset is skipped. there's no memset on fresh pages.

• ✓ prompt_region(false) order:

  1. proc_mem() (task_vm_info.phys_footprint)
  2. cx_->free on every all_bufs wrap
  3. delete pg_
  4. release_region() → munmap
  5. purge_device_allocations
  6. proc_mem() and the log

• ✓ load_dense:

  • locked_bytes sums the mlock successes (page-rounded), and unpinned_embd_bytes / skipped_hc_bytes sum t.bytes. all three are computed, and they match the header.
  • the "dropped 1.60 GB" literal is gone.
  • the ≈0.34 GB saving vs upstream is confirmed. git show HEAD: load_dense skips hc_* and anon-copies the remaining 3.241 GB. patched locks 2.904 GB, so 3.241 − 2.904 = 0.338 GB = token_embd.

• ✗ hard-coded plan value: window.cpp const double pg_scratch_est = (pg_on && sep_pg) ? 1.36 * gb : 0.0; is a literal (patch_strata_metal.py L888). the real need depends on PROMPT_CHUNK, max_cells (scores = MT·MB) and ple_on_. also, it's only printed ("6.6 GB peak prefill …") plus a "above cap" suffix. it doesn't enter budget, footprint_plan or the cap check, so the scratch still isn't counted in the plan. use the measured need, and feed it into the cap check.

• ✗ stale log string: prompt_gemm_init still prints "…1.36 GB of buffers, in the expert arena while a prompt runs" (patch L984 replaces it with itself). with the separate buffer that's false, and it appears in every archived tail.

• ⚠ tautological "measurements":

  • "0 arena slots stolen" = region_slots_.size() right after region_slots_.clear().
  • "0.00 GB anon" = the cx.allocated() delta across a loop that only wraps with count_allocated=false.

  both are computed, but they can't be anything except 0.

• ⚠ the logged reclaim doesn't show 1.36 GB. the 1024- and 2048-tok runs show 4.83 -> 4.77 GB (0.06 GB). the 43-tok and 32-tok runs show 4.55 -> 4.55 / 4.67 -> 4.67 (0.00). footprint after the fill was 4.7 GB, so prefill added only about 0.13 GB. either most of the 1.36 GB scratch is never resident, which would make "6.6 GB peak prefill" an overestimate, or phys_footprint misses GPU-touched no-copy pages. PLAN's "munmapped immediately after prefill (4.83 -> 4.77)" is literally true. it just isn't evidence of a 1.36 GB peak or reclaim.

• ⚠ mtp_src_ = nullptr on release. that's fine with MTP off (these runs), but an MTP decode after a prompt would lose its source. the "tight" fallback in region() mmaps without setting sep_region_bytes_, so it's never unmapped (SEPARATE_PROMPT_BUF=0 small-arena path only). the comments in stream.cpp L799 and stream.h L67 ("inside the arena", "places them (once)") are stale.

6. router GEMM over n_exp_ + multi-turn slot.hot: PASS #

• prompt_chain.inc L275 and window.cpp L2359 run gemm(…, Wr, logits, T, n_exp_, N). route_topk gets T | n_exp_<<16, and prompt.metal indexes logits + ulong(t) * ulong(n_exp), masking idx >= n_exp. that's consistent with the GEMM's Nn = n_exp_ row stride. PgPlan/PcMmPlanArgs and mm_scan use n_exp_ ✓. the buffers are still sized NEXP (over-allocation, harmless).
• hold_region(true) now re-marks hot on the first n − (ne_+32) slots of each class's LRU list (MRU first) on every prompt. hold_region(false) no longer clears the flags, and victim() honours hot only while in_prompt_. on the separate-buffer path, pg_ is deleted after each prompt, so the next prompt re-runs prompt_gemm_init → region() → hold_region(true), and the refresh happens. by code review that's correct. it's untested, since the CLI is single-prompt.

falsifier summary #

what to fix #

1. PLAN §7.1 #5 / §2.7.A #30: replace the 12 numbers in §1b with the raw values. add the ja PPL 53.9 → 406 and the "user-turn TF, 64 tok/domain" caveat.
2. window.cpp: log and plan the measured prompt need instead of 1.36 * gb, include it in the cap check, and fix the "in the expert arena" string. drop or label the tautological "0 slots stolen" / "0.00 GB anon".
3. runner: make evict_file_cache actually work, or remove it. compute the llama summary from prev_llama_* instead of literals. archive /tmp/m5_prompt_1600tok.txt. repeat runs (n ≥ 3), and use the same arena size for the IQ3_S and Q2_0 comparison.
4. explain the 0.06 GB footprint delta: either the scratch is mostly non-resident, so lower the peak estimate, or phys_footprint misses it, so measure differently.