# strata metal opts v2 (opus partner): resident-first experts, read-ahead precision, prewarm, and v1's attribution

patch: `/tmp/opus_metal_probes/opus_v2.diff`, against the current vendor tree (v1 already in), 255 changed lines, `git apply --check` clean.
`stream.cpp` and `window.cpp` pass `clang++ -fsyntax-only`. the Metal library compiles, and every new pipeline specialization builds
(`mcheck.mm`, CPU only). the new plan was checked on the GPU (`plantest.mm`, a few ms of GPU; 200 random windows, T = 1..8, 60% resident,
λ = 0.85; ids, weights and every (token, k) → entry → expert/weight mapping identical to the unsplit plan; resident groups strictly first;
0/200 bad). the decode itself is not built or run: that's yours.

## 0. first, what v1's numbers do and don't say

| arm (ops_1024, 7.0 GB unless noted) | ms/tok | CB "GPU" | host / waiting | pagein / decomp over the run |
|---|---|---|---|---|
| 0 stock, unwired | 58.72 | 58.32 | 7.94 / 7.26 | 1.7 / 7.4 MiB |
| a old code, wired, 6.5 / 7.0 | 67.52 / 71.66 | 66.3 / 71.1 | 7.75 / 7.09 | 7.3 / 12.8 MiB |
| b v1 defaults, wired, 6.5 / 7.0 | 49.83 / 49.92 | 48.72 / 48.76 | 7.14 / 7.04 | 1.5 / 4.4 MiB |
| c b + PREFETCH=16 | 48.46 | 47.23 | 5.37 / 5.29 | 0.6 / 3.6 MiB |
| unpert K=6 / 7 | 47.89 / 48.05 | 46.75 / 46.85 | 4.99 / 4.91, 4.57 / 4.49 | — |

- **i was wrong about paging.** 1.7 MiB of page-ins over a whole stock run can't cost ~8 ms/token, and wiring alone made the old code
  slower (+9…+13 ms), not faster. retracting it. MLOCK/WIRE stay a stability choice, not a speed lever.
- so the −8.9 ms between arm 0 and arm b is v1's code, or run-to-run state, and i can't account for most of it: KPROF puts the kernels at 38.3 ms;
  b's 48.7 ≈ 38.3 + 7.0 wait + 0.1 host + ~3.3 handshakes, which closes. stock's 58.3 ≈ 38.3 + 7.3 + 0.7 + 3.3 + ~1.7 (shexp not hidden, F32
  read-ahead router) ≈ 51, leaving **~7 ms in the stock arm that nothing i changed explains**. the arms ran once each, in sequence, on a
  fanless machine.
- needed before anything gets a number in PLAN.md: **2×2 interleaved, 3 reps** (v1 off/on × wired off/on: ABCD ABCD ABCD, same arena,
  same PREFETCH), the stream line + profile line per run. if v1-on/unwired lands at ~49 ms the patch is the win and wiring is noise; if
  v1-off lands at ~51 the stock arm was a bad run.

## 1. v2 patch

### 1.1 resident-first experts (the big one: handshakes + part of the SSD wait hidden)
the router already reads `smap` (for MoCE). now `plan_body` (function constant 26, decode only) also orders the plan's groups
**resident-at-routing-time first, missing after**, with two more pairs of indirect grids (`P_SPLIT`: count, then IGU/IDN for each part).
`moe_gu`/`moe_down` take function constant 27: phase 1 = the resident groups, phase 2 = the missing ones.

window, per layer: `[… router/route] → signal → [shared expert, moe_gu ph1, moe_down ph1] → wait → [moe_gu ph2, moe_down ph2, combine]`.

- what moves off the critical path: the resident experts' gate/up + down. KPROF has 3.06 + 2.28 = 5.34 ms for 48 layers × 10, so
  ~96% resident ≈ **105 µs a layer**, plus the shared expert (~30–45 µs). that's more than the 57–73 µs event round trip, so
  **every handshake should hide completely (~3 ms)**. on layers with misses, ~140 µs of their SSD wait now overlaps GPU work too
  (expected −0.5…−1.5 ms at ~10 misses/tok). estimate **−3…−4.5 ms/tok**.
- exactness: entries keep their (token, k) → entry mapping (`P_TOKENT`), so the combine sums the same values in the same order; every
  number is bitwise the old one. without fc 26 the plan is the old plan bit for bit (no-miss arithmetic path, tested).
- why phase 1 may read the arena during the host's turn:
  - a chosen resident expert is pinned (`at_layer` pass 1) before any `victim()` call in that turn.
  - claims and finished reads only ever publish `smap` (~0 → slot), never unpublish a chosen expert.
  - the one writer that could race is the previous turn's `after_layer` → `ahead(l)`, which runs beside the GPU. **its victims now skip
    layer `ln`** (`victim(cls, avoid_layer)`).
  - this also fixes a v1 determinism hole: an `ahead(l+1)` eviction could land just before or just after `route(l+1)` read `smap`, which
    changed MoCE routing. that was rare but real; it can't happen now.
- switch: `STRATA_METAL_EXPERT_SPLIT=0`.

### 1.2 read-ahead hygiene (lateness went 49 → 118 with v1, 397 with K=16)
- v1 moved the cancel of queued, unwanted read-aheads to `after_layer`, so during `at_layer`'s wait they could still start and hold
  I/O threads ahead of the urgent misses. that's my regression. now `at_layer` retires them before it looks up a single expert
  (`retire_unused_pre(l, l, &limbo_new_, used)`, no waiting), and `after_layer` releases the previous turn's limbo (two generations,
  still deterministic).
- `STRATA_METAL_IO_URGENT=2` (default): the first 2 I/O threads take urgent reads only while decoding, so a miss never queues behind
  in-flight read-aheads. prompts keep all threads.

### 1.3 why unpert K=6–7 wins, and where to stop
closed-loop replay (7.0 GB = 5027 slots, stock MoCE, no read-ahead), for **non-resident** experts by their rank in the raw (unperturbed)
guess `W(l+1)·XF(l)`: P(that expert is then actually selected):

| raw rank | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10–15 | 16+ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P(selected) | .58 | .46 | .34 | .23 | .17 | .12 | .08 | .06 | .05 | .03 | .01–.03 | ≤ .01 |

so the k-th read-ahead is useful 58% … 8% of the time down to rank 6, then below ~6%, and every useless read costs SSD time and an LRU-tail
expert. K=6–7 sits right at the knee, which is what you measured (20.88 tok/s). the MoCE-ranked version wastes its slots on residents the bonus
already favours. leave unpert as the default and don't go past K=8 without the urgent-thread reservation.

### 1.4 prewarm (the 282 ms T=3 stall)
`prewarm_pipelines()` builds, and throws away, the dispatch lists for T = 2..max_t once, right before the first window is built. that
compiles every `#t` specialization, the QSA selection's (built at pos 4096), the PLE GEMVs and the head. no side effects: `rec_`,
`cuts_`, `sigs_`, `cur_inj_` and `nb_` are restored, nothing is encoded. switch: `STRATA_METAL_PREWARM=0`.
- on `--suffix-draft`: ops_1024 accepted only 3 T=3 windows in 61, so the drafter barely fires on this prompt, and a T=3 window's CB was
  86.5 ms (≈ 1.8× T=1: ~21 distinct experts a layer, so more misses). with resident-first experts the extra misses partly hide.
  re-measure on a repetitive tool-output prompt before deciding anything.

### 1.5 kernel sweep hooks (no default changes)
`STRATA_METAL_MOE_GU_TILE` / `STRATA_METAL_MOE_DN_TILE` = `"LPR,NR,SG"` select any instantiated `moe_gu_`/`moe_down_` tile.
- `moe_down_142_8_1_4` moves 48 × 10 × 0.46 MB = 221 MB in 2.28 ms = **97 GB/s**, while `moe_gu` does 144 GB/s and the dense GEMVs do 122–135.
  that's the one kernel clearly below the bandwidth line: ~0.7 ms on the table.
- instantiated candidates: `8,2,4`, `8,4,4`, `8,1,8`, `8,2,8`, `4,1,4`, `4,2,4`, `16,1,4`, `16,2,4`, `2,2,4`. run KPROF per tile and keep the best.

## 2. what's left in the 38 ms of kernels
| item | ms | GB/s | lever |
|---|---|---|---|
| HC `gr_up_q8` + `gr_down_q8` (192 Q8_0 matrices, 668 MB) | 5.45 | 122 | bytes only: Q8_0 → a 5–6-bit format with a gr kernel for it (−1.9 ms at 0.69 B/w), KLD-gated |
| `output.weight` Q6_K (521 MB) | 3.93 | 133 | bytes only: two-stage head (Q2/Q3 copy over 248K rows → exact Q6_K on the top 64): −2.5 ms, +~180 MB of memory; needs a top-20 recall check |
| attn_qkv + ssm_out + attn_gate + attn_out (Q6_K/Q5_K) | ~10 | 125–135 | at bandwidth; requant is a quality decision |
| `moe_down` | 2.28 | 97 | tile sweep (1.5) |
| `ffn_gate_inp` F32 ×95 | 1.77 | 141 | that KPROF run still shows 95 F32 GEMVs: was it built with `PRED_Q8=1`? with v1 it should be 48 F32 + 47 Q8_0 (−0.6 ms). a BF16 router would take another −0.45 ms but moves routing slightly: KLD first |
| `gdn_step` ×36 | 2.01 | ~114 (F32 state r+w) | none cheap |

## 3. asks for the next M5 round
1. the 2×2 interleaved A/B from §0, so v1 gets an honest number.
2. v2 defaults + unpert K=6/7 (stream + profile lines; watch `late`, `waiting`, and CB vs 38.3 + waiting).
3. `STRATA_METAL_EXPERT_SPLIT=0` vs 1, interleaved, same config (isolates §1.1).
4. KPROF once with v2, to confirm the 47 Q8_0 read-ahead GEMVs and the phase split.
5. MOE_DN_TILE sweep under KPROF.