strata metal opts v1 (opus partner) — hit rate, read-ahead, handshake, GPU floor #

all numbers below come from probes in /tmp/opus_metal_probes/ over the real cal_iq3s_1024 dump (1,022 tokens × 48 layers) and the real K256-q2_0-*metal.gguf routers / Q2_0 experts, plus one tiny Metal microbench (≈2 s of GPU, run while no strata process was up). the cache replays are closed-loop: routing re-runs MoCE against the simulated residency every layer, like router_route does. the patch is /tmp/opus_metal_probes/opus_v1.diff (208 changed lines, git apply --check clean on the current vendor tree, all three files pass clang++ -fsyntax-only; not built or run on the GPU: that's yours).

0. tl;dr (ranked by expected ms/token at the 7.5 GB ops_1024 point, 59.1 ms/tok = 16.9 tok/s) #

1+2+3+4+5 ≈ −11…−15 ms → ~45 ms/tok ≈ 22 tok/s, before speculation. 99% hit rate is not reachable by policy at 7.5 GB (needs ~13 GB, §1.2), and it's the wrong target anyway: stall time is.

1. MoE cache hit rate #

1.1 where the misses are (replay, stock = J3, λ=0.85, prefetch K=10 MRU, cap = slots, tokens 128..1021) #

your measured 7.5 GB ops_1024 run had misses 16.3/tok + ra 3.24% (15.6/tok); the replay's 9.82 + 9.92 is the same shape on a different (prefill-text) workload, so use it for relative deltas.

• yes, ranks 0–2 carry most of the misses once MoCE is on: 6.3 of 9.8 (64%) with prefetch, 14.0 of 19.65 (71%) without.
• but un-protecting them does not buy much: the bonus (0.85 logits) is big next to the router's spread (median z₁−z₃ = 0.35, z₂−z₃ = 0.12; only 31% / 6% of rank-1/2 picks lead rank 3 by > 0.5), so most protected misses are experts whose lead is bigger than the bonus. replay at 7.5 GB, prefetch on, quality = routed-output relative MSE vs λ=0 over 256 sampled tokens × 48 layers, exact Q2_0 SwiGLU experts (healed .experts):

• anchor: Q2_0 (healed) vs IQ3_S experts on the same routing is rel-L2 0.469 (relMSE 0.22), so stock MoCE adds ~+15% routed MSE on top of the quant error, J2 ~+18%, J1 ~+19%.
• J1/J2 at λ0.85 dominate raising λ: J1 gets fewer misses than J3 λ1.5 at lower error. if anything moves, move J, not λ.
• margin gating = J1 in practice (J3 with margin 0.5 or 1.0 reproduces J1 to 0.01 misses/tok): the margin almost never binds because the bonus already exceeds most gaps. skip it.
• adaptive top-k (drop rank 8–9 misses with small weight) is pointless under MoCE: rank 8–9 misses are ~0.0/tok because the bonus already pulls residents into the tail. 0.06–0.09 drops/tok.
• capacity is the real lever. stock policy vs slots: 5385 → 9.82, 6500 → 7.18, 7500 → 5.13, 8500 → 3.52, 9500 → 2.28 (99.05% hit), 10500 → 1.23. 99% needs ~13.2 GB of arena. ds4's 99% is a much larger resident fraction; not comparable.

1.2 eviction — tested, negative (don't spend time here) #

• Belady MIN on the stock routing trace at 5385: 10.71/tok vs LRU 19.65 (no prefetch): 45% headroom exists, but online policies don't reach it (same as ds4's −4%).
• popularity EMA + near-pick credit (top-16 unbiased candidates at 0.5 credit): 19.65 → 19.65.
• probationary insertion (misses enter below the MRU end, SLRU-style) by 10k/25k/50k/100k accesses: 19.66 / 20.03 / 25.49 / 48.47 — worse.

1.3 the read-ahead (predL_) is not broken #

• the guess W(l+1)·XF(l) is good: recall of layer l+1's real top-10 = 0.678 @10, 0.821 @16, 0.932 @32; the real top-1 is in the guessed top-10 93.2% of the time (weight-recall 0.762 @10). two layers ahead (W(l+2)·XF(l), PREFETCH2) is worse: 0.588 @10, top-1 86%.
• "3.2% read-ahead hits" is the expected number: 96% of lookups are hits already, and the read-ahead converts ~half of the would-be misses (replay at 7.5 GB: 19.65 → 9.82 urgent misses, 9.92 read-ahead hits). it is doing its job.
• what is wrong is around it:
  1. ahead() calls lru_front() on every resident guessed expert, so guesses that are never used push real recency down. no-touch: 9.82 → 9.40 (−4%).
  2. K=10 is too shallow for the budget: K=16 no-touch → 6.99 urgent misses/tok (−29% vs stock) for 45 reads/tok (vs 30); only ~0.8 new reads per layer, ~1.1 MB/layer, well inside one layer's GPU time at 3.9–6.8 GB/s. confidence gating (only guesses with p ≥ 0.02/0.04) loses recall faster than it saves reads; don't.
  3. with MoCE on, at_layer waits for every unused in-flight read-ahead of the layer to finish before it even looks up the layer's experts (done_cv_.wait in the moce_bonus_ > 0 block), inside the handshake, and that wait is counted in host but not in waiting. then it throws the slot away. the patch moves all of that to after the GPU is let go (§2.4).
  4. keeping unused read-aheads resident at MRU instead of dropping them (pf5 in the replay): 10.00 vs 9.82, slightly worse, and the dropped-slot design is fine once it's off the critical path.

2. the decode floor #

2.1 what "gpu" is #

win_gpu_ is GPUEndTime − GPUStartTime of the one segmented command buffer, so it includes every host turn and SSD wait ("GPU incl. the host's turns"). wall 59.1 = CB 50.87 + ~8.2 ms outside the CB (embeddings, PLE gather_batch, encode patching, finish). the CB = SSD wait 7.09 + 48 handshakes (~2.8–3.3 ms, §2.3) + host non-wait turn time + compute (~38–40 ms). run once with STRATA_METAL_PROFILE=1 to get the pre / rg / fin split of the 8.2 ms; the PLE row gather could move into layer 0's host turn (its GEMVs sit before layer 0 but are only needed after it).

2.2 bytes per token (why ~40 ms is close to the floor) #

dense reads per decoded token on this GGUF: attn_qkv 711 MB (Q6_K) + ssm_out 444 + attn_gate 379 + HC 4×315 BF16 → Q8_0 copies ≈ 670 + attn_q 262 + attn_output 151 + ffn_gate_inp F32 126 + the read-ahead GEMV re-reading the next layer's F32 router 123 + shexp 146 + indexer/k/v/ssm ≈ 70 + output.weight Q6_K 521 ≈ 3.6 GB, + experts 10 × 48 × 1.38 MB = 0.66 GB. 4.3 GB / ~40 ms ≈ 108 GB/s on a 153 GB/s part. kernel tuning can find maybe 10%; bytes are the lever: the Q8_0 read-ahead router (−92 MB) is free; the head (521 MB = ~4.5 ms) is the next biggest single item if you ever want a two-stage head.

2.3 handshakes (measured, hs.mm, 48 segments, M5) #

• encoder breaks are free; the cost is the event round trip, 57–73 µs of GPU idle each, ~2.8–3.5 ms/token over 48 layers.
• work encoded between encodeSignalEvent and encodeWaitForEvent runs while the host takes its turn: one extra 27 µs dispatch per segment made the whole thing faster (5.735 → 5.175). so the patch puts the shared expert (gate/up/gate_inp GEMVs, swiglu_prep, down; ~2.7 MB ≈ 25–45 µs a layer) there.
• "skip the wait when the guess says everything is resident" — tested in the replay and rejected: at 7.5 GB, 29 of 48 layers/token would qualify (MoCE guess top-16 all resident), but 3.8 of those still miss, dropping 4.4 experts/token holding 14% of their layer's routed weight. too costly for ~1.7 ms.

2.4 the patch (/tmp/opus_metal_probes/opus_v1.diff) #

• mtl_rt.{h,mm}: run_graph_seg(ds, cuts, sigs, at_cut, after_cut, user): signal at sigs[s], encode [sigs[s], cuts[s]) as its own encoder, wait at cuts[s]; after setting signaledValue, call after_cut(s). the old overload forwards with sigs = cuts, after_cut = nullptr (the prompt chain uses it unchanged).
• window.cpp:
  • layer(): when streaming, the shared expert's dispatches go to a side list and are spliced in right after router_route/moe_route; sigs_ records the position. barriers: the side list starts its own encoder; moe_route (non-fused path) keeps its barrier on the router GEMV; moe_down keeps its barrier on moe_gu (the old nb_ = true before the down-shexp GEMV is skipped).
  • after_cut(s) → st_->after_layer(s, …, predL_ + s); the prompt chain calls after_layer inline in at_cut.
  • STRATA_METAL_PRED_Q8 (default on): Q8_0 copy of the 48 routers (33 MB) for the read-ahead GEMV only (gemv_8_32_1_4_1 exists; Q8_0 reads the float x plane).
  • am.dense_file = st_->dense_unwired_bytes() for the arena rule.
• stream.{h,cpp}:
  • at_layer (decode, split_turn_): only the layer's own lookups, misses and the wait. the queued-read-ahead cancel, the MoCE wait/evict and ahead() move to after_layer.
  • after_layer(l): release_limbo() (last turn's retired read-aheads: wait for any still reading — they had a whole layer — then free in slot order, lru_back), retire_unused_pre(l, l, true) (this layer's unclaimed read-aheads: queued pieces dropped, slots parked in limbo_), then ahead(l+1).
  • determinism: victim() never picks a limbo slot, nor (in decode) a pre slot, so which slots are free at every decision point depends only on the order of turns, never on SSD timing. smap is untouched for read-aheads until claimed, as before. ahead() still pins with the current stamp, so it can't evict what the GPU is reading.
  • pre_touch_ (STRATA_METAL_PREFETCH_TOUCH, default 0): ahead() no longer lru_fronts resident guesses.
  • hold_region() releases limbo on both edges (prompt start/end), so prompt stream-ins can't leak.
  • report: (+ X ms after the GPU went on).
• arena_budget.h: ArenaMem::dense_file subtracted in arena_budget()'s RAM term and in arena_grow(); 0 keeps every old number (the arena_budget_test still passes).
• A/B switches: STRATA_METAL_SHEXP_OVERLAP=0, STRATA_METAL_SPLIT_TURN=0, STRATA_METAL_PREFETCH_TOUCH=1, STRATA_METAL_PRED_Q8=0 each restore the old behavior. STRATA_METAL_PREFETCH=16 is not a default in the patch; set it in the bench.

2.5 why 8.2 GB got slower, and the arena rule #

• compute (gpu − wait) climbs with the arena on both prompts: ops_1024 35.9 / 41.4 / 43.8 / 53.7 ms and wal_43 42.2 / 52.4 / 50.7 / 90.0 ms at 3.4 / 6.0 / 7.5 / 8.2 GB, while min_avail falls 5.4 → 3.8 → 3.4 → 2.9 GiB with 0 swap. that is paging, not the GPU page table: TLB reach depends on bytes touched per token (same at every arena size), not on arena size.
• mechanism: with MLOCK_DENSE=0 the 4.53 GB dense map is clean file cache, and GPU reads never set the CPU's referenced bits, so the pager sees those pages as idle and reclaims them first. on small Macs the arena is unwired too (WIRE_ARENA defaults on only at ≥ 64 GB), so it can be compressed between command buffers. every command buffer then pays to make it all resident again. arena_avail() makes it worse: it counts inactive file pages, the model's own dense pages included, as free.
• confirm in one run: vm_stat before/after a 64-token decode, Pageins and Decompressions deltas per token. tens of MB per token = this.
• what to run: MLOCK_DENSE=1 WIRE_ARENA=1, arena 6.0 / 6.5 / 7.0 GB; expect compute back near 36 ms. 24 GB: 4.5 dense + 7 arena = 11.5 GB wired, under the 19.6 GB wire limit, leaving ~10 GB for macOS + buffers.
• default rule on a 24 GB Mac: arena = available − keep − ple_reserve − dense_unwired, keep = 7 GB as now. your data says min_avail ≥ ~3.3 GiB is safe and ≤ 2.9 GiB is not. with dense wired, dense_unwired = 0 and the old rule applies.

3. what i'd like back from the M5 runs #

1. the full stream report line for 7.5 GB stock, especially host X ms per window, of it waiting Y (X−Y = what the split turn saves) and the late count.
2. STRATA_METAL_PROFILE=1 pre / rg / fin for one decode run.
3. vm_stat Pageins/Decompressions deltas across a decode at 7.5 GB, MLOCK_DENSE=0 vs 1.
4. A/B order: (a) MLOCK+WIRE alone at 6.5/7.0; (b) + patch defaults; (c) + PREFETCH=16; (d) + J2 with a KLD check (STRATA_METAL_LOGPOS); (e) --suffix-draft 1/2/3.

4. reproduce #

cd /tmp/opus_metal_probes
./logits models/K256_reap_l2_sum/K256-q2_0-metal.gguf <cal_iq3s_1024 prefix> cal   # router logits + guesses, checks 99.99% set agreement with .ids
./sim cal <K256-expert-profile.bin> main|caps|pf|ins|opt|dump                       # closed-loop replays
./recall cal                                                                        # read-ahead recall
./belady sel_ref.bin 5385 ; ./belady sel_j3b085.bin 5385
NTH=4 ./eval <healed .experts> <cal prefix> 4 sel_ref.bin sel_*.bin                  # routed-output error (validated: 0.086 rel-L2 vs the heal tool's own Q2 teacher)
./hs 4 2 ; ./hs 4 4                                                                 # handshake microbench (GPU, ~2 s)