# phase 1a v2 / 1b replayer audit (read-only)

scope: `scripts/assemble_corpus.py`, `scripts/patch_imatrix.py`, `scripts/select_and_eval_prunes.py`, `scripts/replay_moe_cache.py`, `results/reap_saliency_v2.json`, `results/keep_maps/` (9 files), against `audits/opus_audit_phase1_methodology.md`.
run audited (parent transcript seq 173661): `llama-imatrix ... -f calibration_mix_v2.txt --parse-special -c 2048 -b 512 -ngl 99 --chunks 64 --output-frequency 5 --no-ppl`.

## verdict

- JSON invariants: **all pass** on all 48 layers.
- v1 findings: the corpus/leak ones (A2, A4–A8) are fixed. A1/A3 are only partly fixed, B2/B4 partly, B3 half, C3 still open.
- **new bug: `reap_shrink`'s prior keeps every dead expert.** all 8 zero-hit experts in the model are kept at every K, and the method replaces 40% of top-10 slots at K=224.
- replayer math: correct. lazy EMA matches an eager reference exactly. there are 3 edge cases (below).
- for 1c, prioritize `reap_l2_sum` and `reap_hybrid`. don't slice `reap_shrink` until its prior is fixed.

## 1. v1 findings vs v2

| id | status | evidence |
|---|---|---|
| A1 prefix only | **partial** | `--chunks 64 -c 2048` = 131,072 tok of a 3.78 MB file (~1M tok), so ~12%, about the first ~35–40 of ~330 rounds. the round-robin (assemble_corpus.py L219–229) makes the prefix mixed, but only ~40/300 ukisai, ~10/24 TB, ~40/260 whittle and ~40 JA docs are seen. the tail of the file is JA-only rounds (ja list ≈ 330 > 300). |
| A2 parse-special | fixed | flag is in the command. ukisai and hotchpotch are still hand-wrapped (L64, L184), not `apply_chat_template`. minor. |
| A3 char vs token | partial | the per-round budget is a char heuristic (comment at L208–213). it was never tokenized, so the shares aren't verified. |
| A4 TB leak | fixed | xhigh only, one prompt per task, task-disjoint assert (L80–91) |
| A5 ChiTako leak | fixed | eval JA now comes only from hotchpotch. ChiTako is shingle-deduped (L156–163), and hp is url-unique plus shingle-filtered against everything before it (L168–192). |
| A6 ukisai leak/preamble | fixed | last episode per trial, one trial per base task, disjoint assert, preamble kept in 2 docs only (L29–71). nit: the regex at L26 strips only one suffix, so `X_run2__dsv4…` → `X_run2`. |
| A7 whittle | fixed | `prompt+response`, kind quotas, left-truncation (L115–142). only shard 0 is used. |
| A8 hotchpotch split | fixed | explicit `test-00000-of-00001.parquet` (L167) |
| B2 hook ids | partial | ids are still read from `down_t->src[2]` (patch L224–230), not `ffn_moe_topk`. the non-MUL_MAT_ID case now aborts (L225, good), but `--moe-cache-mib` would still silently mis-attribute experts. not triggered by this run. |
| B3 trace | half | header rewritten on every flush (L107–111) ✓. no `(chunk,pos,token_id)` per row ✗ |
| B4 router imatrix | open | `return true` at L220. minor. |
| nonfinite / empty-JSON guard | fixed | L149–155, L163. JSON `nonfinite_drops = 0` |
| C1 metric | partial | shrink and hybrid were added, but the shrink prior is wrong (§3). no per-domain REAP. |
| C2 substitution screen | added | `score_trace_substitution` (select L73–107) |
| C3 vision | open | known. keep maps aren't exportable. |

## 2. `reap_saliency_v2.json` invariants (direct check, all 48 layers)

- `Σcount == 1,310,720` on every layer ✓. `Σprob_sum == 131,072.000` on every layer ✓. `trace_tokens_written = 16384` ✓, `nonfinite_drops = 0` ✓.
- `count==0 ⇔ reap_l2_sum==0` on every layer ✓. all values finite ✓. Cauchy–Schwarz `sq_sum ≥ sum²/count` holds everywhere, apart from one rounding tie at L30 e87 (count 1, %.6f).
- zero-hit experts: 8 total (L0: 181, 193, 244, 271, 413, 424; L1: 116; L47: 371).
- the 9 keep maps match a re-run of `compute_layer_scores` exactly. they are sorted, unique, size K and nested across K (224 ⊂ 256 ⊂ 320).

per-layer view at K=224 (share of `count` kept / share of `reap_l2_sum` kept / smallest kept count):

| L | l2_sum | hybrid | shrink | overlap sum∩hyb, sum∩shr, hyb∩shr | μ_layer vs median expert mean |
|---|---|---|---|---|---|
| 0 | .59/.78/927 | .56/.77/927 | .53/.74/**0** (6 dead kept) | 204,190,210 | .159 vs .126 |
| 10 | .73/.76/1552 | .67/.72/393 | .54/.61/177 | 186,140,177 | .118 vs .110 |
| 13 | .71/.75/1597 | .64/.70/333 | .55/.62/227 | 177,143,187 | .105 vs .097 |
| 22 | .67/.74/1599 | .61/.71/287 | .55/.66/287 | 184,162,201 | .169 vs .151 |
| 47 | .88/.95/856 | .74/.89/**0** | .59/.83/**0** | 165,142,201 | .616 vs .308 (expert 143: mean 5.56, expert 445: E[x²]=726) |

whole-model top-10 slot replacement (exactly `1 − kept count share`, over all 131K tokens):

| K | l2_sum mean/worst | hybrid | shrink |
|---|---|---|---|
| 320 | .116 / .226 (L0) | .169 / .258 | .231 / .295 |
| 256 | .194 / .343 | .258 / .372 | .342 / .408 |
| 224 | .243 / .409 | .308 / .440 | .402 / .470 |

## 3. shrink prior bug (select_and_eval_prunes.py L47–54)

`mu_layer = Σreap/Σcount` is the **count-weighted** mean. heavy hitters push it above most experts' own means: 63% (L0) to 85% (L47) of experts have a mean below μ. a zero-hit expert scores exactly μ, so it lands in the top ~15–37%.

result: `reap_shrink` keeps **all 8 dead experts at K=224, 256 and 320**, plus 26–40 experts with fewer than 100 hits in L47. `reap_hybrid` inherits this at K=320 (8 dead) and in L47 at every K.

fixes, pick one:
- use the median of the per-expert means as the prior;
- use a lower confidence bound, `mean − z·sd/√n` (sd from `reap_l2_sq_sum`, which is already in the JSON);
- or set `count ≥ c_min`.

the hybrid's max-normalization (L66–69) is also fragile: one L47 outlier compresses the shrink term to p50 = 0.06. rank-averaging would be more robust.

## 4. `replay_moe_cache.py`

- lazy decay (L137–211): stored S = Σ credit·d^−s, true score = S·d^t. eviction compares within one step, so the shared factor cancels. the rescale at 1e100 happens before the current credit is added, which is correct. checked against an eager decay-every-expert reference on a 3,000-token synthetic trace with decay 0.99 and 0.9 (0.9 triggers the rescale): miss counts match exactly for ema, near_pick_ema and inverse_router_ema. self-test passes.
- Belady pointer advance (L147–151) is correct when λ=0, and λ>0 is skipped (L288).
- edge 1, keep-map filtering (L58–68): after filtering, fewer than `k_used` survivors may remain in the top-64. `select_moce_top_k` then returns fewer than 10 experts, and the real pruned model would fetch rank >64 experts that aren't in the trace, so misses are **undercounted**. count these tokens and treat the missing slots as forced misses.
- edge 2: `if actual_tokens < n_tokens` (L33) means a header with ntok=0 parses **0 tokens**. use `actual_tokens` when the header is 0. this is moot for v2 traces, since the header is written on every flush.
- edge 3: `ema_decay` and `near_pick_n` have no CLI flags, so 0.99 and 48 are hard-coded.
- validity: the v2 trace is the first 16,384 calibration tokens (≈ first 5 rounds). replaying keep maps on it is **in-sample**, because the maps were fit on those tokens. it's also teacher-forced prefill, not decode.
- perf: tuple parsing is ~100M Python objects for 16K × 48 × 128 (several GB). numpy would fix it.

## 5. what to slice in 1c

1. `reap_l2_sum` at K=320 and 256. it has the lowest slot replacement by a wide margin, and kept experts all have ≥ 856 hits.
2. `reap_hybrid` at K=320 and 256, as the rare-specialist hedge. it shares 263–302 of 320 experts with l2_sum, so the KL gap is an informative ablation.
3. K=224: only `reap_l2_sum` (24% slot replacement on average, 41% at L0).
4. `reap_shrink`: don't slice as-is. regenerate with a median or LCB prior first.

the 8 dead experts may be vision or rare-path experts (C3). that's an argument for adding image calibration, not for keeping them by accident.