strata metal opts v2 (opus partner): resident-first experts, read-ahead precision, prewarm, and v1's attribution #

patch: /tmp/opus_metal_probes/opus_v2.diff, against the current vendor tree (v1 already in), 255 changed lines, git apply --check clean. stream.cpp and window.cpp pass clang++ -fsyntax-only. the Metal library compiles, and every new pipeline specialization builds (mcheck.mm, CPU only). the new plan was checked on the GPU (plantest.mm, a few ms of GPU; 200 random windows, T = 1..8, 60% resident, λ = 0.85; ids, weights and every (token, k) → entry → expert/weight mapping identical to the unsplit plan; resident groups strictly first; 0/200 bad). the decode itself is not built or run: that's yours.

0. first, what v1's numbers do and don't say #

• i was wrong about paging. 1.7 MiB of page-ins over a whole stock run can't cost ~8 ms/token, and wiring alone made the old code slower (+9…+13 ms), not faster. retracting it. MLOCK/WIRE stay a stability choice, not a speed lever.
• so the −8.9 ms between arm 0 and arm b is v1's code, or run-to-run state, and i can't account for most of it: KPROF puts the kernels at 38.3 ms; b's 48.7 ≈ 38.3 + 7.0 wait + 0.1 host + ~3.3 handshakes, which closes. stock's 58.3 ≈ 38.3 + 7.3 + 0.7 + 3.3 + ~1.7 (shexp not hidden, F32 read-ahead router) ≈ 51, leaving ~7 ms in the stock arm that nothing i changed explains. the arms ran once each, in sequence, on a fanless machine.
• needed before anything gets a number in PLAN.md: 2×2 interleaved, 3 reps (v1 off/on × wired off/on: ABCD ABCD ABCD, same arena, same PREFETCH), the stream line + profile line per run. if v1-on/unwired lands at ~49 ms the patch is the win and wiring is noise; if v1-off lands at ~51 the stock arm was a bad run.

1. v2 patch #

1.1 resident-first experts (the big one: handshakes + part of the SSD wait hidden) #

the router already reads smap (for MoCE). now plan_body (function constant 26, decode only) also orders the plan's groups resident-at-routing-time first, missing after, with two more pairs of indirect grids (P_SPLIT: count, then IGU/IDN for each part). moe_gu/moe_down take function constant 27: phase 1 = the resident groups, phase 2 = the missing ones.

window, per layer: [… router/route] → signal → [shared expert, moe_gu ph1, moe_down ph1] → wait → [moe_gu ph2, moe_down ph2, combine].

• what moves off the critical path: the resident experts' gate/up + down. KPROF has 3.06 + 2.28 = 5.34 ms for 48 layers × 10, so ~96% resident ≈ 105 µs a layer, plus the shared expert (~30–45 µs). that's more than the 57–73 µs event round trip, so every handshake should hide completely (~3 ms). on layers with misses, ~140 µs of their SSD wait now overlaps GPU work too (expected −0.5…−1.5 ms at ~10 misses/tok). estimate −3…−4.5 ms/tok.
• exactness: entries keep their (token, k) → entry mapping (P_TOKENT), so the combine sums the same values in the same order; every number is bitwise the old one. without fc 26 the plan is the old plan bit for bit (no-miss arithmetic path, tested).
• why phase 1 may read the arena during the host's turn:
  • a chosen resident expert is pinned (at_layer pass 1) before any victim() call in that turn.
  • claims and finished reads only ever publish smap (~0 → slot), never unpublish a chosen expert.
  • the one writer that could race is the previous turn's after_layer → ahead(l), which runs beside the GPU. its victims now skip layer ln (victim(cls, avoid_layer)).
  • this also fixes a v1 determinism hole: an ahead(l+1) eviction could land just before or just after route(l+1) read smap, which changed MoCE routing. that was rare but real; it can't happen now.
• switch: STRATA_METAL_EXPERT_SPLIT=0.

1.2 read-ahead hygiene (lateness went 49 → 118 with v1, 397 with K=16) #

• v1 moved the cancel of queued, unwanted read-aheads to after_layer, so during at_layer's wait they could still start and hold I/O threads ahead of the urgent misses. that's my regression. now at_layer retires them before it looks up a single expert (retire_unused_pre(l, l, &limbo_new_, used), no waiting), and after_layer releases the previous turn's limbo (two generations, still deterministic).
• STRATA_METAL_IO_URGENT=2 (default): the first 2 I/O threads take urgent reads only while decoding, so a miss never queues behind in-flight read-aheads. prompts keep all threads.

1.3 why unpert K=6–7 wins, and where to stop #

closed-loop replay (7.0 GB = 5027 slots, stock MoCE, no read-ahead), for non-resident experts by their rank in the raw (unperturbed) guess W(l+1)·XF(l): P(that expert is then actually selected):

so the k-th read-ahead is useful 58% … 8% of the time down to rank 6, then below ~6%, and every useless read costs SSD time and an LRU-tail expert. K=6–7 sits right at the knee, which is what you measured (20.88 tok/s). the MoCE-ranked version wastes its slots on residents the bonus already favours. leave unpert as the default and don't go past K=8 without the urgent-thread reservation.

1.4 prewarm (the 282 ms T=3 stall) #

prewarm_pipelines() builds, and throws away, the dispatch lists for T = 2..max_t once, right before the first window is built. that compiles every #t specialization, the QSA selection's (built at pos 4096), the PLE GEMVs and the head. no side effects: rec_, cuts_, sigs_, cur_inj_ and nb_ are restored, nothing is encoded. switch: STRATA_METAL_PREWARM=0.

• on --suffix-draft: ops_1024 accepted only 3 T=3 windows in 61, so the drafter barely fires on this prompt, and a T=3 window's CB was 86.5 ms (≈ 1.8× T=1: ~21 distinct experts a layer, so more misses). with resident-first experts the extra misses partly hide. re-measure on a repetitive tool-output prompt before deciding anything.

1.5 kernel sweep hooks (no default changes) #

STRATA_METAL_MOE_GU_TILE / STRATA_METAL_MOE_DN_TILE = "LPR,NR,SG" select any instantiated moe_gu_/moe_down_ tile.

• moe_down_142_8_1_4 moves 48 × 10 × 0.46 MB = 221 MB in 2.28 ms = 97 GB/s, while moe_gu does 144 GB/s and the dense GEMVs do 122–135. that's the one kernel clearly below the bandwidth line: ~0.7 ms on the table.
• instantiated candidates: 8,2,4, 8,4,4, 8,1,8, 8,2,8, 4,1,4, 4,2,4, 16,1,4, 16,2,4, 2,2,4. run KPROF per tile and keep the best.

2. what's left in the 38 ms of kernels #

3. asks for the next M5 round #

1. the 2×2 interleaved A/B from §0, so v1 gets an honest number.
2. v2 defaults + unpert K=6/7 (stream + profile lines; watch late, waiting, and CB vs 38.3 + waiting).
3. STRATA_METAL_EXPERT_SPLIT=0 vs 1, interleaved, same config (isolates §1.1).
4. KPROF once with v2, to confirm the 47 Q8_0 read-ahead GEMVs and the phase split.
5. MOE_DN_TILE sweep under KPROF.