# phase 1a v3 keep maps + phase 1b sweep audit (read-only)

scope: `scripts/select_and_eval_prunes.py` (v3), `scripts/replay_moe_cache.py`, `scripts/replay_moe_cache_fast.cpp`, `results/keep_maps/` (9), `results/oos_prune_retention.json`, `results/phase1b_cache_moce_sweep.json`, `results/phase1b_heldout_cache_moce_sweep.json`, checked against `results/reap_saliency_v2.json` (131,072 calib tok) and `results/eval_reap_heldout.json` (8,192 held-out tok). nothing in the repo was edited. the scratch differential test lives in `/tmp/rpl/`.

## verdict: PASS (with minor notes). #93 can close.

- the v2 shrink dead-expert bug is **fixed**: 0 of the 8 zero-hit experts (L0: 181,193,244,271,413,424; L1: 116; L47: 371) are kept in any of the 9 maps.
- the L47 max-norm outlier is **fixed**: the hybrid is now a per-layer rank-percentile average, and nothing normalizes across experts by max.
- C++ and Python replayers: **identical math**. with Python ties broken by lowest id (which is what C++ does), 96/96 configs match to 5e-7. that covers 6 policies, λ ∈ {0, .05, .12}, unpruned/K224/sparse-48 keep maps, and 22,297 forced-miss slots.
- the sweep outputs are internally consistent, and the in-sample and OOS sweeps cover the same 120 configs.

## 1. keep maps vs saliency (direct recompute)

- `verify_invariants` passes on both the calib JSON and the held-out JSON.
- all 9 maps are exactly reproduced by `compute_layer_scores` + the script's ranking (0 layer mismatches). every layer has K unique ids, and the maps nest 224 ⊂ 256 ⊂ 320 for every metric.
- the `assert zero_hit_kept == 0` is in place. shrink scores c=0 as 0, and the LCB `mean − SE` times `c/(c+64)` damping kills rare noise. experts kept with <100 calib hits: l2_sum 0, hybrid 0–2, shrink 2–3 (whole model).
- L47: expert 143 (mean 5.56, 9,549 hits) is kept everywhere, which is legit since it has real support. L47 reap retention at K224: l2_sum .945, hybrid .935, shrink .877.
- `oos_prune_retention.json`: I recomputed mean/min reap retention, mean prob retention and slot replacement from the held-out JSON. **all 36 numbers match to 4 dp.** OOS ranking is l2_sum > hybrid > shrink on every metric at every K (K256 mean reap: .797 / .782 / .734; slot replacement: .239 / .271 / .338).
- note (not a falsifier): kept experts with 0 *held-out* hits number 7–69 per map. that's sampling noise at 8K tokens, since all of them have ≥74 calib hits.

## 2. C++ vs Python replayer

matches, line for line:
- forced misses `max(0, k_used − n_chosen)` go into both misses and total_picks, after warmup only.
- MoCE bonus: `λ·(z0 − z[k−1])` on top-j or cached experts. sort order is (score desc, rank asc), then eid.
- eviction never evicts the current chosen set.
- lazy EMA: `mult *= 1/d` before crediting, rescale at 1e100. credits for near_pick and inv_router are the same.
- Belady: next-use is built from `ids[:k]` and the pointer advances on chosen. it's only valid at λ=0, and the sweep has no Belady λ>0 rows ✓.
- the `ntok==0` header uses the actual token count in both (v2 edge 2 fixed). the remap is keep-list order and the JSON is sorted, so they agree.

differences (minor):
- **tie-breaking**: Python `min/max` over a `set` resolves ties in hash-iteration order, while C++ picks the lowest id. with the production Python as-is, 5/44 configs differed by ≤0.04% misses/tok. that's below sweep rounding and has no impact on conclusions. if you want bit-exact parity, add `c` as the last key in Python.
- fp32 (C++) vs fp64 (Python) logits/bonus: no observable effect in the test.
- C++ hardening gaps, none triggered by the sweep: there's no `cache_cap >= k_used` check (otherwise `victim_idx = -1` is UB), no `fopen` null check, and it hardcodes 48 layers in the keep loader, ≤512 experts, `top_m ≤ 64`, warmup 64, decay .99 and near_pick_n 48.

## 3. sweep outputs

- 120 configs per file, with identical key sets: 10 variants. K512 and the l2_sum maps get all 6 policies plus λ ∈ {.05, .09, .12} × j ∈ {2, 3} for lru/near_pick/inv_router. hybrid and shrink get lru and near_pick at λ=0 only.
- derived fields check out on all 240 rows: `ssd_mb = misses·1.3824`, the tok/s formulas for 3.9 and 6.8 GB/s + 12 ms, and `misses = 480·(1−hit)`.
- Belady ≤ every λ=0 online policy per variant ✓. MoCE misses are monotone non-increasing in λ for every policy and variant ✓. j=2 vs j=3 differ by ≤0.01, so j is plumbed through but irrelevant.
- in-sample → OOS shift (misses/tok, 48L): K512 LRU 84.61 → 84.84, Belady 42.81 → 46.13, LRU λ.12 63.21 → 66.06. near_pick_ema λ.12 degrades most OOS (70.0 → 79.3). LRU is the best online policy everywhere, and LFU is the worst (155).
- caveat for interpretation: for pruned variants, misses only count cache traffic. substituted slots (24–41% at K224) are a quality cost that the miss numbers don't price. shrink/hybrid's slightly lower miss counts come *from* replacing more slots, so don't read them as better caching. both traces are also teacher-forced prefill, not decode.

## recommendation

the conclusion stands: `reap_l2_sum` is the prune metric (best OOS retention at every K), hybrid is the hedge ablation, and shrink can be dropped. use LRU + MoCE λ≈.12 as the online baseline.