moe decode/prefill cache and prefetch policy audit #

verdict #

first test the existing STRATA_METAL_PREFETCH_UNPERT=0 switch in isolation. it already makes ranks 3 onward moce-aware under the default protect=3, r1-bonus=0 configuration. no new router kernel is needed. the patch below generalizes that behavior to the configurable router and adds an independently disabled scan-resistance experiment. neither is a measured speedup or approved replacement.

there are two important corrections to the initial hypothesis:

1. ahead() already defines eff_lg = logit + resident_bonus (stream.cpp:662–664). however, STRATA_METAL_PREFETCH_UNPERT defaults to 1, making k_unpert=K, so the adjusted-selection loop is bypassed entirely. the reported raw-top-6 behavior is the default, not the only implemented policy.
2. speculative completion cannot make a prefetched expert moce-resident. io_main() publishes smap only when !x.pre (391); at_layer() clears pre and publishes only after the actual router claims an expert (797–798). an unused read-ahead therefore cannot award itself a residency bonus. its victim eviction can still change another layer's future residency, which is a genuine closed-loop effect.

source pin: vendor/strata HEAD 296b59d2f73c8f04a805df29c2fa97655adace78; the actual inspected working files are snapshotted in /tmp/sol_moe_cache/original/. this matters because the shared checkout may contain changes beyond HEAD. no tracked vendor file was edited, no subagent was spawned, and no inference, GPU, or model-weight SSD benchmark was run.

1. what the supplied trace does and does not establish #

these numbers are supplied by the parent from call_3037605, not independently remeasured here:

• 480 lookups/token, approximately 436–441 arena hits, 25–28 prefetch hits including about 5 late hits, and 14.2–16.6 synchronous misses.
• 76.4–82.6 speculative experts issued/token; 4,326 arena slots / 24,576 possible layer-expert pairs = 17.60% residency.
• midpoint accounting: 80 issued / 26 eventually used = 32.5% issue usefulness; timely usefulness is about (26−5)/80 = 26.25%. 54 issued experts/token are not credited as useful within the reported accounting window.
• with the supplied 131.8–144.4 MB/token speculative-read volume, 67.5% unnecessary traffic would be about 89–97 MB/token. the task's approximately 82 MB/token waste estimate is plausible with a different actual average expert size/byte attribution, but is not exactly derivable from those aggregate ranges. at 22.5 tok/s, 82 MB/token is 1.845 GB/s; 89–97 MB/token is 2.0–2.2 GB/s. do not turn either into a measured speedup ceiling.

pre_issued_ counts submissions, not completed SSD bytes. queued pieces can be cancelled; boundary in-flight jobs may be consumed later. rigorously distinguish issued, cancelled-before-read, partially read, completed-unused, useful-late, and useful-timely. aggregate counters alone do not prove every issued-but-uncredited expert was fully read and never used.

a late prefetch hit is not an arena hit and not necessarily latency-equivalent to a synchronous miss: partial service may already be done. eliminating a late job can increase a sync-miss counter yet improve wall time, or vice versa. the requested falsifier explicitly forbids increased sync misses, so report both rather than trading one away silently.

2. exact router and predictor math #

for target layer l, let predicted logits be z_e, and R_e=1 only if the router-visible smap entry is valid. with default protected width p=3 and bonus b=0.85, the ideal router ranking is

first p picks: argmax z_e among unselected experts
subsequent picks: argmax (z_e + b R_e) among unselected experts.

softmax preserves logit order and multiplying its probabilities by exp(b) is equivalent to adding b to logits in exact arithmetic. exp(0.85)=2.33965. routing weights are recovered from unboosted probabilities after selection; the bonus changes identities, not a replacement weighting rule (moe.metal:183–239).

an unselected nonresident c loses to resident r whenever

z_c − z_r < b.

example: z_e=−0.1e, resident experts e=6..19. raw top-6 is {0,1,2,3,4,5}; moce-consistent first six picks are {0,1,2,6,7,8}. the raw predictor requests six missing experts; the consistent predictor requests only {0,1,2}. the CPU harness calls actual ahead() and confirms that reduction on this constructed perfect-predictor case. it is not evidence that this trace loses 50% of prefetch traffic.

“the GPU will never pick raw ranks 3..5” is too strong. pred is an early, approximate, sometimes Q8_0 router projection from a different hidden state (window.cpp:954–969), not the eventual target logits. residual error can overcome the margin. and an expert displaced from predicted top-6 can still appear in actual ranks 6..9. moce-aware prediction is a better-aligned heuristic, not a proof of uselessness.

STRATA_METAL_MOCE_PROTECT changes p (window.cpp:1309), and STRATA_METAL_MOCE_BONUS_R1 adds an intermediate resident preference from rank 1 until p (1310–1321). the old UNPERT=0 branch hardcodes p=3 and does not represent r1 preference. the proposed helper honors both settings. the new route mode is deliberately bypassed when b<=0, preserving the existing zero-bonus path. arithmetic equivalence is not a promise of GPU bitwise identity near float softmax underflow/ties; this helper predicts, and never replaces actual routing.

3. state machine and cache audit #

ahead, at_layer, after_layer #

• ahead ranks predictions, merges token candidates, skips already-mapped experts, obtains a victim, submits speculative I/O and pins the slot. no confidence, probability, byte-budget or deadline gate is applied.
• default PREFETCH_TOUCH=0 is already important: a guessed resident expert does not receive an LRU promotion. comments cite an earlier 4% miss regression from speculative touches (169–171).
• at_layer builds the actual unique wanted set, pins all mapped wanted slots before allocating misses (778–779), promotes queued wanted prefetch pieces into the urgent queue, and waits only for wanted work on the ordinary split path.
• split-turn decode is the default. unused target-layer speculation is retired into limbo_new_. queued work is cancelled, running work finishes, and retired slots cannot be eviction candidates.
• after_layer runs after GPU release (window.cpp:585–587), releases the previous limbo generation, rotates the current generation, then schedules the next read-ahead. release_limbo waits for running I/O and frees slots in sorted slot order. do not replace this with “free completed jobs whenever convenient”: SSD timing would leak into residency and hence moce routing.
• non-split moce cleanup instead waits for unused speculative work then removes it synchronously (753–775). trimming speculation can reduce this cleanup cost too, but this is not the default execution path.

victim #

victim scans class-local LRU from the tail, rejecting active I/O, current-layer pins, limbo, and (for default split decode) all speculative slots, regardless of completion state. ahead additionally avoids evicting the target layer. prompt mode makes two passes, first protecting the prompt-hot set and then allowing fallback. there is no demand-frequency protection in decode.

so speculative jobs consume arena capacity and can evict useful experts from other layers even while their own bytes remain invisible to smap. excluding all prefetched slots from victims preserves timing determinism but makes over-issuing especially expensive: they cannot be cheaply recycled until deterministic retirement/release.

I/O queues #

io_main takes urgent jobs first (377–383), and by default two threads are reserved for urgent decode I/O (nio_urgent_, 158). therefore “all eight workers are equally blocked behind speculative jobs” is not the current default design. reserved workers still share the physical SSD, and a read already in progress is not preempted. promoting a queued read-ahead cannot cancel other workers' in-flight speculative transfers.

prefill #

T>8 takes a different policy: layer-ranked streaming of up to a configured cap (820–845), not ahead's predicted candidates. current defaults are 256 earlier layers / 128 for l>=40, with PROMPT_MAX_PRE override. prompt-hot protection is refreshed in hold_region, preserving top class entries while reserving approximately 2*ne+32 tail slots (961–973); initial fill uses a different ne+32 hot threshold (599–607). these are heuristics, not measured per-layer marginal optima. the proposed decode flags do not redesign prefill.

4. arxiv sweep: primary sources and what transfers #

downloaded and read relevant primary HTML sections; saved HTML plus extracted text under /tmp/sol_moe_cache/. reported research results below are authors' results on their own systems, not reproduced on ours.

paper	inspected mechanism	implication for us
seqmoe, 2609.12978v1, §§6.2–7.2, sep 2026	confidence-ranked feasible task selection under deadline/EMA bandwidth constraints; “probabilistic belady” uses discounted predicted activation probabilities, not true future knowledge	budget useful bytes before deadlines, not fixed candidate count. needs a trained multi-step predictor before copying its eviction score. its 45% residency result is not comparable to our 17.6%.
moe-core, 2610.01950, §§3.1, 3.3, 3.6, oct 2026	per-layer hit curves measured with fixed initialization/replacement; transfer capacity from low marginal loss donors to high marginal gain receivers; distinct prefill buffering	measure marginal miss reduction, not merely low hit rate. its CPU/NPU weight-transfer regime is not our SSD regime; approximate expert substitution is outside this exact-execution patch.
cache-aware joint router adaptation, 2609.04895v2, §§3, 5, appendix b.3, sep 2026	trained temporal retention plus bounded spatio-temporal pre-access refinement; jointly changes backbone and auxiliary routers while preserving the top-k rule	not a training-free drop-in or an unchanged-model guarantee. useful emphasis on demand-plus-proactive traffic; appendix explicitly says modeled overlap is not measured end-to-end latency.
edge0, 2609.18063v2, §§3.2–4, sep 2026	prediction becomes routing, with recovery lora trained on that student path	zero unused predictions by construction, but changes the model. not an exact cache fix. reported 35b/24gb behavior cannot be transplanted onto K512 without training/quality work.
st-moe, 2606.15453v1, predictor/table update, jun 2026	2-bit saturating confidence, prefetch threshold 2; confirmed predictions increment and rejected ones decrement	useful training-free admission template. confidence counters are not calibrated probabilities; must train/update using all eligible candidates, including withheld ones, or the gate becomes selection-biased.
apex, 2608.11688v1, §iv-c, aug 2026	ordinal logistic CDF estimates probability that an extra-prefetch budget covers the actual top-k; selects smallest budget reaching tau; exact and approximate execution modes are separate	directly relevant learned coverage gate. not equivalent to thresholding native router softmax values. training/calibration data needed.
finemoe, 2502.05370, §4.3, 2025 preprint with revised HTML	retrieved expert maps, cosine-similarity-derived cumulative-mass threshold delta=clip(1−score,0,1), at least routing width candidates	adaptive selection precedents; low confidence can increase rather than decrease prefetch breadth. wrong objective if copied without SSD bandwidth cost.

classic scan resistance is older, not a new 2025/26 invention: s3-fifo, sosp 2023 source filters one-hit objects through small/main/ghost queues and bounded counters; lirs, sigmetrics 2002 uses inter-reference recency and a retained nonresident history. targeted 2025–26 searches did not establish a direct MoE S3-FIFO/LIRS evaluation. do not label our two-hit heuristic either full algorithm. belady MIN is an offline upper bound, not an implementable online policy without a prediction assumption.

5. why a raw softmax threshold is not the right confidence variable #

router p_e=softmax(z)_e expresses relative mixture importance, not q_e=Pr(expert e is actually selected at the target layer | early state, current residency). actual top-10 selections sum to ten indicator outcomes, while router probabilities sum to one. an expert with tiny p_e can be selected very reliably, and a large p_e can be unstable under hidden-state drift. a fixed p_e cutoff therefore has no stated false-negative guarantee.

a useful candidate admission objective is

admit e when q_e * expected_exposed_stall_saved
              > SSD_byte_shadow_price * bytes_e + expected_victim_loss.

include deadline success probability in the saved-stall term; useful-late jobs have a smaller, not necessarily zero, payoff. target-layer score margin after resident bias is a feature for q_e, not a calibrated q_e itself. other features: raw/biased rank, gap to predicted tenth selection, layer id, token/window size, recent validated precision, and queue occupancy. learn/calibrate on actual claimed sets, stratified by layer and rank, and keep a held-out workload.

not implemented here: a guessed “probability>0.01” gate, cancellation/reallocation based on actual SSD completion timing, or an online policy requiring a new inference-time model. those would be unvalidated changes to the determinism/performance envelope.

6. patch, ownership, invariants and limits #

/tmp/sol_moe_cache/moe_cache_policy.patch is an unapplied experimental patch from the vendor/strata root. exact modified source is in patched/; no Metal change is necessary because GPU routing semantics remain unchanged.

• STRATA_METAL_PREFETCH_ROUTE=1: opt into rank-dependent router-consistent prediction using published residency, configurable protected rank and r1 bonus. default off, and gated off for main moce bonus<=0. existing UNPERT behavior remains untouched when disabled. ties across merged token candidates use expert-id order in the new mode. fixed candidate union storage is widened from 256 to the 512-expert universe, avoiding reliance on T<=8; negative K is clamped safely.
• STRATA_METAL_DECODE_2Q=1: independently opt into a bounded two-hit frequency-protected LRU approximation, not full 2Q/LIRS/S3-FIFO. fresh slots start reuse=0; first real decode claim sets reuse=1 and stays at the LRU tail; second claim sets reuse=2 and promotes. decode victim first seeks cold eligible slots, then falls back to ordinary eligible LRU. speculation never increments reuse and remains tail-inserted under this flag. prompt-hot logic is unchanged. first-touch warm/profile experts are not specially seeded protected.
• all existing active-I/O, stamp pin, avoid-layer, speculative, and limbo exclusions remain enforced on every eviction pass. no I/O completion-driven policy update is added; state remains bounded by arena slots, not session history.
• helper selection is O(TNEK^2) with K<=32 and NE<=512; existing repeated-top-k selection already has this shape. union dedup is bounded by 512. worst-case victim scan doubles but is still O(class slots); no unbounded history/ghost list. no extra SSD reads are requested by these checks themselves.

the cache-policy part cannot satisfy a universal no-miss-increase promise. a CPU abstract cache replay with capacity 3, warm sequence ABABAB followed by CD repeated ten times gives ordinary LRU 4 misses versus frequency-protected LRU 22: stale A/B protection leaves only one cold slot for the new working set. a quota, aging or ghost-assisted policy would mitigate this but still cannot dominate LRU on every possible trace. this is why the flag stays disabled and why adoption requires the user's actual workload falsifier. “scan-resistant” is a property on scans, not a universal superiority theorem.

moce-aware prediction can also lose coverage if early logits are wrong; no proof or real-trace replay here establishes nonincreasing synchronous misses. under the strict empirical falsifier, neither flag should become a default yet.

7. actual checks run #

from project root:

python3 /tmp/sol_moe_cache/check.py

completed exit 0. it runs:

1. clang++ C++20 -Wall -Wextra -fsyntax-only on the whole patched stream.cpp, with the real engine/runtime headers. no warnings/errors.
2. a CPU-only executable linked with actual patched stream.cpp using dead stripping, no initialized Metal context or weight files: 2,000 randomized finite-logit comparisons of the helper's ten picks against an independent double-precision softmax/multiply simulation, covering protect=0..10 and r1 off/on; negative-K and constructed displacement checks; three assertions against actual victim() for cold preference, pinned fallback and no victim; three calls to actual ahead() verifying raw-six versus biased-three requests, unpublished speculative residency, and identical requests with unrelated speculation complete versus still running.
3. separate original/patched CPU executables calling actual ahead() on 100 randomized zero-main-bonus cases, T=1..8, K=6/10: complete emitted queued-slot/expert sequences match byte-for-byte. this tests default-policy parity, not a full decode determinism theorem or the independently enabled 2Q flag.
4. git apply --check from live vendor/strata: exit 0. it does not apply the patch.

visual final review: temporary git diff reviewed using view_diff and the new header using view_code. no benchmark was run. syntax checking is not a linked engine build; CPU harness linking exercises only reachable test code. no float/quality or throughput result is being claimed.

8. next measured experiment — after the GPU owner finishes #

1. keep cache/arena, split-turn mode, I/O workers, prompt, seed, prediction dtype and moce settings fixed. first compare existing UNPERT=1 vs UNPERT=0 with default protect/r1. this answers the main suspected mismatch with no patch.
2. collect candidate ids/scores and actual selections for all candidates, layerwise issued/completed/cancelled/timely/late/unused bytes, demand misses, wait, limbo capacity and output hashes. record actual resident snapshots before routing; the total trace aggregates alone cannot train a gate or replay this policy faithfully.
3. replay candidate policies CPU-side, then closed-loop run finalists: with moce>0, changed evictions alter future routing and hidden states, so fixed-route replay is only a screening test. include cold and warm multi-turn chat plus a deliberately shifted-domain workload.
4. test the generalized route flag alone before the cache flag; do not bundle effects. reject if sync misses increase on the paired target workloads, if moce=0 output/queue determinism breaks, or if lower traffic merely converts timely hits into exposed waits.
5. optimize layer budgets by marginal saved exposed time per byte, not minimum layer hit rate. for cache slots use empirical hit-curve differences subject to size-class capacity; for prefetch slots use validated q/deadline benefit, not a uniform K. reserve an urgent bandwidth margin. prefill is a separate profile with its own reuse/scan behavior and counters.

bottom line: the cheap, immediately testable knob already exists; the self-boosting-prefetch story does not. after fixing the predictor/router objective mismatch, the next real win is calibrated admission under a byte/deadline budget, not blindly swapping LRU for a famous cache acronym.