# strata metal opts v1 (opus partner) — hit rate, read-ahead, handshake, GPU floor

all numbers below come from probes in `/tmp/opus_metal_probes/` over the real `cal_iq3s_1024` dump (1,022 tokens × 48 layers) and the
real `K256-q2_0-*metal.gguf` routers / Q2_0 experts, plus one tiny Metal microbench (≈2 s of GPU, run while no strata process was up).
the cache replays are closed-loop: routing re-runs MoCE against the simulated residency every layer, like `router_route` does.
the patch is `/tmp/opus_metal_probes/opus_v1.diff` (208 changed lines, `git apply --check` clean on the current vendor tree,
all three files pass `clang++ -fsyntax-only`; not built or run on the GPU: that's yours).

## 0. tl;dr (ranked by expected ms/token at the 7.5 GB ops_1024 point, 59.1 ms/tok = 16.9 tok/s)

| # | change | how | expected | risk |
|---|---|---|---|---|
| 1 | **wire dense + arena** | `STRATA_METAL_MLOCK_DENSE=1 STRATA_METAL_WIRE_ARENA=1`, arena 6.0–7.0 GB | **−5…−8 ms** (compute = gpu−wait grows 35.9 → 41.4 → 43.8 → 53.7 ms as the arena grows 3.4 → 6.0 → 7.5 → 8.2 GB) | needs ~11.5 GB wired; `vm.user_wire_limit` = 19.6 GB |
| 2 | prefetch K=16, no LRU touch of resident guesses | `STRATA_METAL_PREFETCH=16` + patch (`STRATA_METAL_PREFETCH_TOUCH=0` default) | urgent misses −29% (9.82 → 6.99/tok in replay) → wait 7.1 → ~5 ms: **−2 ms** | +15 reads/tok = +21 MB/tok (0.35 GB/s), SSD has 6.8 |
| 3 | shared expert between signal and wait | patch (`STRATA_METAL_SHEXP_OVERLAP=1` default) | **−1.5…−2.2 ms** (measured: a handshake costs 57–73 µs of GPU idle, and work placed between signal and wait is hidden) | none (same math, same order) |
| 4 | host turn split: read-ahead + pre-slot retirement after the GPU is released | patch (`STRATA_METAL_SPLIT_TURN=1` default) | removes `ahead()` and the MoCE "wait for unused in-flight read-aheads" stall from all 48 handshakes: **−0.5…−2 ms** (exact = `host − waiting` in the stream report, which the new report line splits out) | deterministic by construction (limbo list, see §2.4) |
| 5 | read-ahead GEMV on a Q8_0 copy of `ffn_gate_inp` | patch (`STRATA_METAL_PRED_Q8=1` default) | −92 MB/tok of F32 reads: **−0.7 ms**; +33 MB memory | zero quality risk (guess only picks what is read ahead) |
| 6 | MoCE protect top-2 instead of top-3 | `router.metal` `rank == 3` → `2` and `ahead()` `k_unpert` 3 → 2 | misses −14% (−1.4/tok at 7.5 GB) | routed-output relMSE +14% (0.0343 → 0.0392) — measure KLD first |
| 7 | speculative T=2 windows (suffix drafter already in the CLI) | `--suffix-draft 1..3` | T=2 moves 16.3 experts/layer instead of 10 → **+9.5% bytes for 2 positions**; any acceptance > ~10% pays | MoCE + more misses per window; measure |

1+2+3+4+5 ≈ −11…−15 ms → **~45 ms/tok ≈ 22 tok/s**, before speculation. 99% hit rate is not reachable by policy at 7.5 GB (needs ~13 GB, §1.2), and it's the wrong target anyway: stall time is.

## 1. MoE cache hit rate

### 1.1 where the misses are (replay, stock = J3, λ=0.85, prefetch K=10 MRU, cap = slots, tokens 128..1021)

| cap (GB) | policy | urgent miss/tok | read-ahead hits/tok | SSD reads/tok | miss by selected rank 0..9 /tok |
|---|---|---|---|---|---|
| 2441 (3.4) | stock | 24.89 | 26.01 | 72.6 | 2.1 4.0 6.1 4.2 3.5 2.5 1.5 0.7 0.2 0.1 |
| 4308 (6.0) | stock | 13.16 | 13.53 | 40.1 | 1.3 2.5 4.1 2.1 1.5 1.0 0.5 0.2 0.1 0.0 |
| 5385 (7.5) | stock | **9.82** | 9.92 | 30.2 | 1.0 2.0 3.3 1.4 1.0 0.6 0.3 0.1 0.0 0.0 |
| 5385 (7.5) | no prefetch | 19.65 | 0 | 19.7 | 3.5 4.7 5.8 2.5 1.6 0.9 0.4 0.1 0.0 0.0 |
| 5385 (7.5) | λ=0, no prefetch | 55.97 | 0 | 56.0 | 3.0 3.5 4.1 4.6 5.1 5.6 6.5 7.1 7.9 8.6 |

your measured 7.5 GB ops_1024 run had misses 16.3/tok + ra 3.24% (15.6/tok); the replay's 9.82 + 9.92 is the same shape on a different (prefill-text) workload, so use it for relative deltas.

- yes, ranks 0–2 carry most of the misses once MoCE is on: 6.3 of 9.8 (64%) with prefetch, 14.0 of 19.65 (71%) without.
- **but un-protecting them does not buy much**: the bonus (0.85 logits) is big next to the router's spread (median z₁−z₃ = 0.35, z₂−z₃ = 0.12; only 31% / 6% of rank-1/2 picks lead rank 3 by > 0.5), so most protected misses are experts whose lead is *bigger* than the bonus. replay at 7.5 GB, prefetch on, quality = routed-output relative MSE vs λ=0 over 256 sampled tokens × 48 layers, exact Q2_0 SwiGLU experts (healed `.experts`):

| policy | urgent miss/tok | Δ vs stock | routed relMSE | rel-L2 |
|---|---|---|---|---|
| J3 λ0.5 | 13.41 | +37% | 0.0223 | 0.149 |
| **J3 λ0.85 (stock)** | **9.82** | — | **0.0343** | 0.185 |
| J2 λ0.85 | 8.45 | −14% | 0.0392 | 0.198 |
| J1 λ0.85 (= Qualcomm top-1) | 7.82 | −20% | 0.0418 | 0.204 |
| J3 λ1.5 | 8.15 | −17% | 0.0440 | 0.210 |
| J2 λ1.5 | 5.16 | −47% | 0.0580 | 0.241 |
| J3 λ0.85 + drop tail misses w<0.05 (rank ≥ 6) | 9.76 | −0.6% | 0.0343 | 0.185 |

  - anchor: Q2_0 (healed) vs IQ3_S experts on the same routing is rel-L2 0.469 (relMSE 0.22), so stock MoCE adds ~+15% routed MSE on top of the quant error, J2 ~+18%, J1 ~+19%.
  - **J1/J2 at λ0.85 dominate raising λ**: J1 gets fewer misses than J3 λ1.5 at lower error. if anything moves, move J, not λ.
  - **margin gating = J1** in practice (J3 with margin 0.5 or 1.0 reproduces J1 to 0.01 misses/tok): the margin almost never binds because the bonus already exceeds most gaps. skip it.
  - **adaptive top-k (drop rank 8–9 misses with small weight) is pointless under MoCE**: rank 8–9 misses are ~0.0/tok because the bonus already pulls residents into the tail. 0.06–0.09 drops/tok.
- capacity is the real lever. stock policy vs slots: 5385 → 9.82, 6500 → 7.18, 7500 → 5.13, 8500 → 3.52, **9500 → 2.28 (99.05% hit)**, 10500 → 1.23. 99% needs ~13.2 GB of arena. ds4's 99% is a much larger resident fraction; not comparable.

### 1.2 eviction — tested, negative (don't spend time here)
- Belady MIN on the stock routing trace at 5385: 10.71/tok vs LRU 19.65 (no prefetch): 45% headroom exists, but online policies don't reach it (same as ds4's −4%).
- popularity EMA + near-pick credit (top-16 unbiased candidates at 0.5 credit): 19.65 → 19.65.
- probationary insertion (misses enter below the MRU end, SLRU-style) by 10k/25k/50k/100k accesses: 19.66 / 20.03 / 25.49 / 48.47 — worse.

### 1.3 the read-ahead (`predL_`) is not broken
- the guess `W(l+1)·XF(l)` is good: recall of layer l+1's real top-10 = **0.678 @10, 0.821 @16, 0.932 @32; the real top-1 is in the guessed top-10 93.2%** of the time (weight-recall 0.762 @10). two layers ahead (`W(l+2)·XF(l)`, `PREFETCH2`) is worse: 0.588 @10, top-1 86%.
- "3.2% read-ahead hits" is the expected number: 96% of lookups are hits already, and the read-ahead converts ~half of the would-be misses (replay at 7.5 GB: 19.65 → 9.82 urgent misses, 9.92 read-ahead hits). it is doing its job.
- what is wrong is around it:
  1. `ahead()` calls `lru_front()` on every **resident** guessed expert, so guesses that are never used push real recency down. no-touch: 9.82 → 9.40 (−4%).
  2. K=10 is too shallow for the budget: K=16 no-touch → **6.99** urgent misses/tok (−29% vs stock) for 45 reads/tok (vs 30); only ~0.8 new reads per layer, ~1.1 MB/layer, well inside one layer's GPU time at 3.9–6.8 GB/s. confidence gating (only guesses with p ≥ 0.02/0.04) loses recall faster than it saves reads; don't.
  3. with MoCE on, `at_layer` **waits for every unused in-flight read-ahead of the layer to finish before it even looks up the layer's experts** (`done_cv_.wait` in the `moce_bonus_ > 0` block), inside the handshake, and that wait is counted in `host` but not in `waiting`. then it throws the slot away. the patch moves all of that to after the GPU is let go (§2.4).
  4. keeping unused read-aheads resident at MRU instead of dropping them (pf5 in the replay): 10.00 vs 9.82, slightly worse, and the dropped-slot design is fine once it's off the critical path.

## 2. the decode floor

### 2.1 what "gpu" is
`win_gpu_` is `GPUEndTime − GPUStartTime` of the one segmented command buffer, so it **includes** every host turn and SSD wait ("GPU incl. the host's turns"). wall 59.1 = CB 50.87 + ~8.2 ms outside the CB (embeddings, PLE `gather_batch`, encode patching, `finish`). the CB = SSD wait 7.09 + 48 handshakes (~2.8–3.3 ms, §2.3) + host non-wait turn time + compute (~38–40 ms). run once with `STRATA_METAL_PROFILE=1` to get the pre / rg / fin split of the 8.2 ms; the PLE row gather could move into layer 0's host turn (its GEMVs sit before layer 0 but are only needed after it).

### 2.2 bytes per token (why ~40 ms is close to the floor)
dense reads per decoded token on this GGUF: attn_qkv 711 MB (Q6_K) + ssm_out 444 + attn_gate 379 + HC 4×315 BF16 → Q8_0 copies ≈ 670 + attn_q 262 + attn_output 151 + `ffn_gate_inp` F32 126 **+ the read-ahead GEMV re-reading the next layer's F32 router 123** + shexp 146 + indexer/k/v/ssm ≈ 70 + `output.weight` Q6_K 521 ≈ **3.6 GB**, + experts 10 × 48 × 1.38 MB = 0.66 GB. 4.3 GB / ~40 ms ≈ 108 GB/s on a 153 GB/s part. kernel tuning can find maybe 10%; bytes are the lever: the Q8_0 read-ahead router (−92 MB) is free; the head (521 MB = ~4.5 ms) is the next biggest single item if you ever want a two-stage head.

### 2.3 handshakes (measured, `hs.mm`, 48 segments, M5)
| mode | 4 MB × 2/seg | 4 MB × 4/seg | 16 MB × 4/seg |
|---|---|---|---|
| one encoder | 2.638 ms | 5.265 | 21.48 |
| 48 encoders, no events | 2.630 | 5.255 | 21.49 |
| 48 encoders + signal/wait (host spins, answers at once) | 5.506–5.735 | 8.750 | 24.20 |
| + 1 more dispatch per segment **between signal and wait** | 5.175 | 8.061 | — |

- **encoder breaks are free**; the cost is the event round trip, 57–73 µs of GPU idle each, ~2.8–3.5 ms/token over 48 layers.
- work encoded between `encodeSignalEvent` and `encodeWaitForEvent` runs while the host takes its turn: one extra 27 µs dispatch per segment made the whole thing *faster* (5.735 → 5.175). so the patch puts the shared expert (gate/up/gate_inp GEMVs, swiglu_prep, down; ~2.7 MB ≈ 25–45 µs a layer) there.
- "skip the wait when the guess says everything is resident" — tested in the replay and **rejected**: at 7.5 GB, 29 of 48 layers/token would qualify (MoCE guess top-16 all resident), but 3.8 of those still miss, dropping 4.4 experts/token holding 14% of their layer's routed weight. too costly for ~1.7 ms.

### 2.4 the patch (`/tmp/opus_metal_probes/opus_v1.diff`)
- `mtl_rt.{h,mm}`: `run_graph_seg(ds, cuts, sigs, at_cut, after_cut, user)`: signal at `sigs[s]`, encode `[sigs[s], cuts[s])` as its own encoder, wait at `cuts[s]`; after setting `signaledValue`, call `after_cut(s)`. the old overload forwards with `sigs = cuts`, `after_cut = nullptr` (the prompt chain uses it unchanged).
- `window.cpp`:
  - `layer()`: when streaming, the shared expert's dispatches go to a side list and are spliced in right after `router_route`/`moe_route`; `sigs_` records the position. barriers: the side list starts its own encoder; `moe_route` (non-fused path) keeps its barrier on the router GEMV; `moe_down` keeps its barrier on `moe_gu` (the old `nb_ = true` before the down-shexp GEMV is skipped).
  - `after_cut(s)` → `st_->after_layer(s, …, predL_ + s)`; the prompt chain calls `after_layer` inline in `at_cut`.
  - `STRATA_METAL_PRED_Q8` (default on): Q8_0 copy of the 48 routers (33 MB) for the read-ahead GEMV only (`gemv_8_32_1_4_1` exists; Q8_0 reads the float x plane).
  - `am.dense_file = st_->dense_unwired_bytes()` for the arena rule.
- `stream.{h,cpp}`:
  - `at_layer` (decode, `split_turn_`): only the layer's own lookups, misses and the wait. the queued-read-ahead cancel, the MoCE wait/evict and `ahead()` move to `after_layer`.
  - `after_layer(l)`: `release_limbo()` (last turn's retired read-aheads: wait for any still reading — they had a whole layer — then free in slot order, `lru_back`), `retire_unused_pre(l, l, true)` (this layer's unclaimed read-aheads: queued pieces dropped, slots parked in `limbo_`), then `ahead(l+1)`.
  - determinism: `victim()` never picks a `limbo` slot, nor (in decode) a `pre` slot, so which slots are free at every decision point depends only on the order of turns, never on SSD timing. smap is untouched for read-aheads until claimed, as before. `ahead()` still pins with the current stamp, so it can't evict what the GPU is reading.
  - `pre_touch_` (`STRATA_METAL_PREFETCH_TOUCH`, default 0): `ahead()` no longer `lru_front`s resident guesses.
  - `hold_region()` releases limbo on both edges (prompt start/end), so prompt stream-ins can't leak.
  - report: `(+ X ms after the GPU went on)`.
- `arena_budget.h`: `ArenaMem::dense_file` subtracted in `arena_budget()`'s RAM term and in `arena_grow()`; 0 keeps every old number (the `arena_budget_test` still passes).
- A/B switches: `STRATA_METAL_SHEXP_OVERLAP=0`, `STRATA_METAL_SPLIT_TURN=0`, `STRATA_METAL_PREFETCH_TOUCH=1`, `STRATA_METAL_PRED_Q8=0` each restore the old behavior. **`STRATA_METAL_PREFETCH=16` is not a default in the patch**; set it in the bench.

### 2.5 why 8.2 GB got slower, and the arena rule
- compute (gpu − wait) climbs with the arena on both prompts: ops_1024 35.9 / 41.4 / 43.8 / 53.7 ms and wal_43 42.2 / 52.4 / 50.7 / 90.0 ms at 3.4 / 6.0 / 7.5 / 8.2 GB, while min_avail falls 5.4 → 3.8 → 3.4 → 2.9 GiB with 0 swap. that is paging, not the GPU page table: TLB reach depends on bytes touched per token (same at every arena size), not on arena size.
- mechanism: with `MLOCK_DENSE=0` the 4.53 GB dense map is clean file cache, and GPU reads never set the CPU's referenced bits, so the pager sees those pages as idle and reclaims them first. on small Macs the arena is unwired too (`WIRE_ARENA` defaults on only at ≥ 64 GB), so it can be compressed between command buffers. every command buffer then pays to make it all resident again. `arena_avail()` makes it worse: it counts inactive file pages, the model's own dense pages included, as free.
- confirm in one run: `vm_stat` before/after a 64-token decode, Pageins and Decompressions deltas per token. tens of MB per token = this.
- what to run: `MLOCK_DENSE=1 WIRE_ARENA=1`, arena 6.0 / 6.5 / 7.0 GB; expect compute back near 36 ms. 24 GB: 4.5 dense + 7 arena = 11.5 GB wired, under the 19.6 GB wire limit, leaving ~10 GB for macOS + buffers.
- default rule on a 24 GB Mac: `arena = available − keep − ple_reserve − dense_unwired`, keep = 7 GB as now. your data says min_avail ≥ ~3.3 GiB is safe and ≤ 2.9 GiB is not. with dense wired, `dense_unwired` = 0 and the old rule applies.

## 3. what i'd like back from the M5 runs
1. the full stream report line for 7.5 GB stock, especially `host X ms per window, of it waiting Y` (X−Y = what the split turn saves) and the `late` count.
2. `STRATA_METAL_PROFILE=1` pre / rg / fin for one decode run.
3. `vm_stat` Pageins/Decompressions deltas across a decode at 7.5 GB, MLOCK_DENSE=0 vs 1.
4. A/B order: (a) MLOCK+WIRE alone at 6.5/7.0; (b) + patch defaults; (c) + `PREFETCH=16`; (d) + J2 with a KLD check (`STRATA_METAL_LOGPOS`); (e) `--suffix-draft 1/2/3`.

## 4. reproduce
```
cd /tmp/opus_metal_probes
./logits models/K256_reap_l2_sum/K256-q2_0-metal.gguf <cal_iq3s_1024 prefix> cal   # router logits + guesses, checks 99.99% set agreement with .ids
./sim cal <K256-expert-profile.bin> main|caps|pf|ins|opt|dump                       # closed-loop replays
./recall cal                                                                        # read-ahead recall
./belady sel_ref.bin 5385 ; ./belady sel_j3b085.bin 5385
NTH=4 ./eval <healed .experts> <cal prefix> 4 sel_ref.bin sel_*.bin                  # routed-output error (validated: 0.086 rel-L2 vs the heal tool's own Q2 teacher)
./hs 4 2 ; ./hs 4 4                                                                 # handshake microbench (GPU, ~2 s)
```