phase 1a v2 / 1b replayer audit (read-only) #

scope: scripts/assemble_corpus.py, scripts/patch_imatrix.py, scripts/select_and_eval_prunes.py, scripts/replay_moe_cache.py, results/reap_saliency_v2.json, results/keep_maps/ (9 files), against audits/opus_audit_phase1_methodology.md. run audited (parent transcript seq 173661): llama-imatrix ... -f calibration_mix_v2.txt --parse-special -c 2048 -b 512 -ngl 99 --chunks 64 --output-frequency 5 --no-ppl.

verdict #

• JSON invariants: all pass on all 48 layers.
• v1 findings: the corpus/leak ones (A2, A4–A8) are fixed. A1/A3 are only partly fixed, B2/B4 partly, B3 half, C3 still open.
• new bug: reap_shrink's prior keeps every dead expert. all 8 zero-hit experts in the model are kept at every K, and the method replaces 40% of top-10 slots at K=224.
• replayer math: correct. lazy EMA matches an eager reference exactly. there are 3 edge cases (below).
• for 1c, prioritize reap_l2_sum and reap_hybrid. don't slice reap_shrink until its prior is fixed.

1. v1 findings vs v2 #

2. reap_saliency_v2.json invariants (direct check, all 48 layers) #

• Σcount == 1,310,720 on every layer ✓. Σprob_sum == 131,072.000 on every layer ✓. trace_tokens_written = 16384 ✓, nonfinite_drops = 0 ✓.
• count==0 ⇔ reap_l2_sum==0 on every layer ✓. all values finite ✓. Cauchy–Schwarz sq_sum ≥ sum²/count holds everywhere, apart from one rounding tie at L30 e87 (count 1, %.6f).
• zero-hit experts: 8 total (L0: 181, 193, 244, 271, 413, 424; L1: 116; L47: 371).
• the 9 keep maps match a re-run of compute_layer_scores exactly. they are sorted, unique, size K and nested across K (224 ⊂ 256 ⊂ 320).

per-layer view at K=224 (share of count kept / share of reap_l2_sum kept / smallest kept count):

whole-model top-10 slot replacement (exactly 1 − kept count share, over all 131K tokens):

3. shrink prior bug (select_and_eval_prunes.py L47–54) #

mu_layer = Σreap/Σcount is the count-weighted mean. heavy hitters push it above most experts' own means: 63% (L0) to 85% (L47) of experts have a mean below μ. a zero-hit expert scores exactly μ, so it lands in the top ~15–37%.

result: reap_shrink keeps all 8 dead experts at K=224, 256 and 320, plus 26–40 experts with fewer than 100 hits in L47. reap_hybrid inherits this at K=320 (8 dead) and in L47 at every K.

fixes, pick one:

• use the median of the per-expert means as the prior;
• use a lower confidence bound, mean − z·sd/√n (sd from reap_l2_sq_sum, which is already in the JSON);
• or set count ≥ c_min.

the hybrid's max-normalization (L66–69) is also fragile: one L47 outlier compresses the shrink term to p50 = 0.06. rank-averaging would be more robust.

4. replay_moe_cache.py #

• lazy decay (L137–211): stored S = Σ credit·d^−s, true score = S·d^t. eviction compares within one step, so the shared factor cancels. the rescale at 1e100 happens before the current credit is added, which is correct. checked against an eager decay-every-expert reference on a 3,000-token synthetic trace with decay 0.99 and 0.9 (0.9 triggers the rescale): miss counts match exactly for ema, near_pick_ema and inverse_router_ema. self-test passes.
• Belady pointer advance (L147–151) is correct when λ=0, and λ>0 is skipped (L288).
• edge 1, keep-map filtering (L58–68): after filtering, fewer than k_used survivors may remain in the top-64. select_moce_top_k then returns fewer than 10 experts, and the real pruned model would fetch rank >64 experts that aren't in the trace, so misses are undercounted. count these tokens and treat the missing slots as forced misses.
• edge 2: if actual_tokens < n_tokens (L33) means a header with ntok=0 parses 0 tokens. use actual_tokens when the header is 0. this is moot for v2 traces, since the header is written on every flush.
• edge 3: ema_decay and near_pick_n have no CLI flags, so 0.99 and 48 are hard-coded.
• validity: the v2 trace is the first 16,384 calibration tokens (≈ first 5 rounds). replaying keep maps on it is in-sample, because the maps were fit on those tokens. it's also teacher-forced prefill, not decode.
• perf: tuple parsing is ~100M Python objects for 16K × 48 × 128 (several GB). numpy would fix it.

5. what to slice in 1c #

1. reap_l2_sum at K=320 and 256. it has the lowest slot replacement by a wide margin, and kept experts all have ≥ 856 hits.
2. reap_hybrid at K=320 and 256, as the rare-specialist hedge. it shares 263–302 of 320 experts with l2_sum, so the KL gap is an informative ablation.
3. K=224: only reap_l2_sum (24% slot replacement on average, 41% at L0).
4. reap_shrink: don't slice as-is. regenerate with a median or LCB prior first.

the 8 dead experts may be vision or rare-path experts (C3). that's an argument for adding image calibration, not for keeping them by accident.