phase 1a v3 keep maps + phase 1b sweep audit (read-only) #

scope: scripts/select_and_eval_prunes.py (v3), scripts/replay_moe_cache.py, scripts/replay_moe_cache_fast.cpp, results/keep_maps/ (9), results/oos_prune_retention.json, results/phase1b_cache_moce_sweep.json, results/phase1b_heldout_cache_moce_sweep.json, checked against results/reap_saliency_v2.json (131,072 calib tok) and results/eval_reap_heldout.json (8,192 held-out tok). nothing in the repo was edited. the scratch differential test lives in /tmp/rpl/.

verdict: PASS (with minor notes). #93 can close. #

• the v2 shrink dead-expert bug is fixed: 0 of the 8 zero-hit experts (L0: 181,193,244,271,413,424; L1: 116; L47: 371) are kept in any of the 9 maps.
• the L47 max-norm outlier is fixed: the hybrid is now a per-layer rank-percentile average, and nothing normalizes across experts by max.
• C++ and Python replayers: identical math. with Python ties broken by lowest id (which is what C++ does), 96/96 configs match to 5e-7. that covers 6 policies, λ ∈ {0, .05, .12}, unpruned/K224/sparse-48 keep maps, and 22,297 forced-miss slots.
• the sweep outputs are internally consistent, and the in-sample and OOS sweeps cover the same 120 configs.

1. keep maps vs saliency (direct recompute) #

• verify_invariants passes on both the calib JSON and the held-out JSON.
• all 9 maps are exactly reproduced by compute_layer_scores + the script's ranking (0 layer mismatches). every layer has K unique ids, and the maps nest 224 ⊂ 256 ⊂ 320 for every metric.
• the assert zero_hit_kept == 0 is in place. shrink scores c=0 as 0, and the LCB mean − SE times c/(c+64) damping kills rare noise. experts kept with <100 calib hits: l2_sum 0, hybrid 0–2, shrink 2–3 (whole model).
• L47: expert 143 (mean 5.56, 9,549 hits) is kept everywhere, which is legit since it has real support. L47 reap retention at K224: l2_sum .945, hybrid .935, shrink .877.
• oos_prune_retention.json: I recomputed mean/min reap retention, mean prob retention and slot replacement from the held-out JSON. all 36 numbers match to 4 dp. OOS ranking is l2_sum > hybrid > shrink on every metric at every K (K256 mean reap: .797 / .782 / .734; slot replacement: .239 / .271 / .338).
• note (not a falsifier): kept experts with 0 held-out hits number 7–69 per map. that's sampling noise at 8K tokens, since all of them have ≥74 calib hits.

2. C++ vs Python replayer #

matches, line for line:

• forced misses max(0, k_used − n_chosen) go into both misses and total_picks, after warmup only.
• MoCE bonus: λ·(z0 − z[k−1]) on top-j or cached experts. sort order is (score desc, rank asc), then eid.
• eviction never evicts the current chosen set.
• lazy EMA: mult *= 1/d before crediting, rescale at 1e100. credits for near_pick and inv_router are the same.
• Belady: next-use is built from ids[:k] and the pointer advances on chosen. it's only valid at λ=0, and the sweep has no Belady λ>0 rows ✓.
• the ntok==0 header uses the actual token count in both (v2 edge 2 fixed). the remap is keep-list order and the JSON is sorted, so they agree.

differences (minor):

• tie-breaking: Python min/max over a set resolves ties in hash-iteration order, while C++ picks the lowest id. with the production Python as-is, 5/44 configs differed by ≤0.04% misses/tok. that's below sweep rounding and has no impact on conclusions. if you want bit-exact parity, add c as the last key in Python.
• fp32 (C++) vs fp64 (Python) logits/bonus: no observable effect in the test.
• C++ hardening gaps, none triggered by the sweep: there's no cache_cap >= k_used check (otherwise victim_idx = -1 is UB), no fopen null check, and it hardcodes 48 layers in the keep loader, ≤512 experts, top_m ≤ 64, warmup 64, decay .99 and near_pick_n 48.

3. sweep outputs #

• 120 configs per file, with identical key sets: 10 variants. K512 and the l2_sum maps get all 6 policies plus λ ∈ {.05, .09, .12} × j ∈ {2, 3} for lru/near_pick/inv_router. hybrid and shrink get lru and near_pick at λ=0 only.
• derived fields check out on all 240 rows: ssd_mb = misses·1.3824, the tok/s formulas for 3.9 and 6.8 GB/s + 12 ms, and misses = 480·(1−hit).
• Belady ≤ every λ=0 online policy per variant ✓. MoCE misses are monotone non-increasing in λ for every policy and variant ✓. j=2 vs j=3 differ by ≤0.01, so j is plumbed through but irrelevant.
• in-sample → OOS shift (misses/tok, 48L): K512 LRU 84.61 → 84.84, Belady 42.81 → 46.13, LRU λ.12 63.21 → 66.06. near_pick_ema λ.12 degrades most OOS (70.0 → 79.3). LRU is the best online policy everywhere, and LFU is the worst (155).
• caveat for interpretation: for pruned variants, misses only count cache traffic. substituted slots (24–41% at K224) are a quality cost that the miss numbers don't price. shrink/hybrid's slightly lower miss counts come from replacing more slots, so don't read them as better caching. both traces are also teacher-forced prefill, not decode.

recommendation #

the conclusion stands: reap_l2_sum is the prune metric (best OOS retention at every K), hybrid is the hedge ablation, and shrink can be dropped. use LRU + MoCE λ≈.12 as the online baseline.