# phase 1 methodology audit: calibration corpus, REAP/trace hook, expert selection

read-only audit. scope: `scripts/assemble_corpus.py`, `scripts/patch_imatrix.py`, `scripts/select_and_eval_prunes.py`, PLAN.md §2.3 / §2.5 / §6.1 / §6.4.
the C++ was checked against current llama.cpp master (`tools/imatrix/imatrix.cpp`, `src/llama-graph.cpp::build_moe_ffn`, `src/models/qwen4exp.cpp`, `common/common.h`, `common/arg.cpp`), fetched for this audit. the Colab clone may be a slightly different commit; the version-sensitive points are marked.
the datasets were checked via the HF API / datasets-server (file listings, first 100–300 rows) and the real `calibration_ja.txt` was downloaded.
the run audited is the one launched in the parent session:
`llama-imatrix -m Swift-...-GSQ-RCO-IQ3_S-00001-of-00002.gguf -f calibration_mix_400k.txt -c 512 -b 512 -ngl 99 --chunks 128 --output-frequency 10 --no-ppl` (no `--parse-special`, no `--moe-cache-mib`).

---

## verdict

| # | finding | severity | affects the run already done? |
|---|---|---|---|
| A1 | `--chunks 128 -c 512` = **65,536 tokens**, read sequentially from the file start. the corpus is ~0.6–0.7M tokens. **ukisai and hotchpotch contribute 0 tokens**; ops = ~6–8 Terminal-Bench "low-effort failure rerun" transcripts; JA = only the first ~150–200 ChiTako paragraphs | **blocker** | yes |
| A2 | no `--parse-special`: `parse_special` defaults to `false` in imatrix, so every `<\|im_start\|>`, `<\|im_end\|>`, `<think>`, `</think>`, `<tool_call>` is tokenized as **literal text pieces**, never as the special tokens the model routes on in real use | **major** | yes |
| A3 | domains are balanced by **characters**, not tokens. JA is ~2–3× more tokens per char, so the 65K-token window is ~50–60% JA, ~20–25% ops, ~20% chat (PLAN: ~30/25/25+) | major | yes |
| A4 | Terminal-Bench split is by sorted path: calib = `low/traces/*` [0:20], eval = [20:25]. **2 of the 5 eval transcripts (`medium/.../adaptive-rejection-sampler`, `medium/.../chess-best-move`) are the same tasks as calib transcripts #1 and #2**, and both of those are inside the 65K tokens actually used | major (leak) | yes |
| A5 | ChiTako is split on `"\n\n"` paragraphs, not on its 916 chunks. 196 / 1,039 kept blocks (18.9%) are exact duplicates; Monte-Carlo over 5 random splits: **3–8 of the 40 eval blocks (~15%) share >50% of their 20-char shingles with the 250 calib blocks** | major (leak) | yes (the duplicated ones sit in the window) |
| A6 | ukisai split is by row, but `task` carries `_run2…_run5` and `__dsv4-trial-2` suffixes: several rollouts of the same base task can land on both sides. also every row's first user message starts with the same **2,834-char Terminus-2 preamble**, so after `[:6000]` ≈47% of every ukisai doc (calib and eval) is identical boilerplate | major (leak + waste) | no for calib (0 tokens used), **yes for `eval_ops_tools.txt`** |
| A7 | Whittle `prompt` is **already chat-templated** (`<\|im_start\|>system ... <\|im_start\|>assistant\n<think>\n`); the script wraps it again → nested `<\|im_start\|>user\n<\|im_start\|>system…`, two `<think>` openers. `prompt[:4000]` cuts the 38K-char agent prompts inside the identical tool schema; `response`/`tool_calls` are ignored, so 10% of rows have an empty answer and "sandbox-verified tool use" is mostly a repeated schema header. no `kind` filter despite the comment (first 100 rows: 45% math/gsm) | major | yes |
| A8 | hotchpotch file picked with `next(rglob("*.parquet"))`: filesystem order decides **train or test** split, non-deterministic across machines. some source `id`s repeat (1/100 in test) → same passage, different question, possible on both sides | minor | no (0 calib tokens) |
| B1 | `on_logits`, `on_weighted_out` stride/shape math, trace record layout, header offset, `save_all()` call sites: **correct** for this build and this command (details §2) | ok | — |
| B2 | version hazard: with `--moe-cache-mib > 0`, the down `MUL_MAT_ID`'s `src[2]` is **cache slot ids, not expert ids** → REAP stats silently attributed to the wrong experts. with `*_exps_s` scale tensors or a down bias, `ffn_moe_weighted->src[0]` is not `MUL_MAT_ID` → REAP stats **silently all zero** | major latent (not triggered by this command) | no |
| B3 | trace header `ntok` is only written at clean exit; trace has no token ids / positions / chunk ids, so §6.1.2's "token ids, turn boundaries, session ids" are not met; the 8,192-token trace is ~2 interleave rounds of teacher-forced prefill, not agent sessions | minor/scope | yes |
| B4 | intercepting `ffn_moe_logits-*` with an early `return true` drops the normal imatrix stats for `blk.*.ffn_gate_inp.weight` from `full_imatrix.gguf` | minor (only if that imatrix is reused to quantize) | yes |
| C1 | ranking uses `reap_l2_sum` = Σ g·‖f‖ = count × REAP. **published REAP is the mean over routed tokens.** the sum is frequency-weighted, so it inherits A3's domain-share skew and is biased against rare-token specialists — the exact "reliability knee" risk §2.3 names | major (method) | yes |
| C2 | "retained REAP mass" in the report is maximized by construction when ranking by the same metric; it ignores substitution (norm_topk renormalization) and is the kind of router statistic §2.3 already says can predict the opposite of behavior | minor (reporting) | — |
| C3 | calibration has no image tokens → vision experts will look dead and get pruned (PLAN's own ISTA lesson); current keep maps are not exportable as-is | major (known, re-flagged) | yes |

falsifier status: **hit.** train/eval leakage confirmed (A4, A5, A6), and the calibration actually used is not the corpus that was designed (A1–A3, A7). no arithmetic bug in the C++ hook for this build/command; two latent silent-failure paths (B2).

---

## 1. calibration corpus and split hygiene (`assemble_corpus.py`)

### 1.1 what the imatrix run actually sees (A1, A3)

`compute_imatrix` takes `n_chunk = min(--chunks, tokens/n_ctx)` and walks the token stream **from the start, in order** (imatrix.cpp ~L1464–1525). nothing is shuffled. with `-c 512 --chunks 128` that is the first **65,536 tokens**.

the file is built as `ops / chat / ja` 4,000-char blocks interleaved round-robin, where each stream is a concatenation in source order:

- ops = `tb_calib_texts` (20 × 8,000 chars = 160K chars) **then** `uki_calib_texts`
- chat = whittle docs (each ≤ ~10K chars)
- ja = `ja_calib_blocks[:250]` (avg 330 chars → ~83K chars) **then** `hp_calib_texts`

per round (12K chars) at plausible Qwen ratios (TB markdown ~3.2 ch/tok, English ~3.5–4, JA ~1.3–2): ~1,250 + ~1,100 + ~2,000–3,000 ≈ 4.4–5.4K tokens. 65,536 tokens ≈ **12–15 rounds ≈ 48–60K chars per stream.** the conclusion is robust to the exact ratios:

- ukisai only starts after 160K ops chars (~40 rounds, >180K tokens). **0 ukisai tokens.**
- hotchpotch only starts after ~83K JA chars (~20 rounds, >90K tokens). **0 hotchpotch tokens.**
- ops = TB transcripts #1–#7ish, sorted by path: `low/traces/adaptive-rejection-sampler, chess-best-move, dna-assembly, dna-insert, extract-moves-from-video, filter-js-from-html, gcode-to-text…`. the `low/` directory is the dataset's **failure reruns at low effort** (session paths read `tb21-low-failures-…`); the 89-task `xhigh` set is never touched.
- TB `transcript.md` is a markdown rendering (`## Tool activity / Tool: bash / Outcome: …`), not the Qwen chat template. it is off-distribution for routing compared to real agent turns.
- token shares in the window are roughly JA 50–60%, ops 20–25%, chat ~20%, versus PLAN §2.3's ~30% ops / ~25% chat+JSON / ~25% JA. the comment "~1.5 MB (~420K tokens)" assumes ~3.6 chars/token across the board, which JA breaks.

the trace (`max_trace_tokens = 8192`) is the first 16 chunks ≈ 2 rounds: about one TB transcript, 1–2 whittle docs and a handful of ChiTako paragraphs.

**fix:** run with `--chunks -1` (or ≥ corpus/512), and build the file so any prefix is representative: interleave at document granularity across *all* sources (round-robin per source within each domain), budget each domain by **tokens** (tokenize with the model's vocab when assembling), and shuffle document order with a fixed seed. at ~3 min per 128 chunks on the A100 (parent's estimate), the full ~1,200 chunks is ~30 min.

### 1.2 special tokens are not special (A2)

`common_params::parse_special = false` (common.h) and `compute_imatrix` tokenizes with `common_tokenize(ctx, params.prompt, true, params.parse_special)`. the run has no `--parse-special`. so the ChiTako text, the whittle templates and the hand-built `<|im_start|>{role}` wrappers all become runs of ordinary BPE pieces (`<`, `|`, `im`, `_start`, …). the router never sees the real role / think / tool-call tokens in this calibration, and those are exactly the positions where routing for "clean termination" and tool syntax happens. ChiTako's own README tells you to pass `--parse-special`.

the KL eval should tokenize the same way it will be served (special tokens parsed), otherwise it under-measures damage on template tokens.

### 1.3 leakage, per source

**Terminal-Bench (A4), confirmed from the repo listing.** 160 `transcript.md` files: `xhigh` 89, `medium` 27, `low` 22, `off` 20, 2 reruns. `sorted(rglob)` → [0:20] are 20 of the 22 `low/` tasks; [20:25] are `low/tune-mjcf`, `low/winning-avg-corewars`, `medium/adaptive-rejection-sampler`, `medium/build-pov-ray`, `medium/chess-best-move`. two eval transcripts are the same task (same container image, same files, same opening tool outputs) as calib transcripts #1 and #2, which are inside the 65K-token window. difflib ratio of the first 8,000 chars of `low/` vs `medium/` chess-best-move = 0.27: task-level leakage, not verbatim copies.
fix: split by task name (the path component after `traces/`), and prefer `xhigh/` for both sides.

**ChiTako (A5), confirmed on the real file.** `calibration_ja.txt` is 916 chunks joined by `"\n\n"`, but chunks contain internal `"\n\n"` paragraph breaks (wikipedia title/body, code blocks, multi-turn chats): splitting on `"\n\n"` gives 2,244 pieces, 1,039 of them >100 chars. 196 of those 1,039 are exact duplicates of another block (the builder reuses templates: e.g. one system prompt appears 21×, several Python helper snippets 7–8×). shuffling paragraphs then cutting 88/12 puts paragraphs of the same chunk, and literal duplicates, on both sides. over 5 random seeds, 3–8 of the 40 eval blocks had >50% shingle overlap with the 250 calib blocks (mean overlap 0.08–0.21). the exact rng state of the script couldn't be reproduced offline (it depends on the ukisai/whittle table sizes), so this is an estimate of the expected rate, not the exact count.
fix: dedupe first (exact + near-dup), and split on the builder's chunk boundaries or, simpler, take the eval JA from a disjoint source (e.g. hotchpotch **test** split, which is already sampled from fineweb2-edu-ja's test portion).

**ukisai (A6), confirmed from rows 0–100, 5000–5100, 10000–10100.** columns are `conversations, agent, model, model_provider, date, task, episode, run_id, trial_name`. `task` values include `nl2bash-1908-0000`, `nl2bash-1908-0000_run2 … _run5`, `inferredbugs-0001` and `inferredbugs-0001__dsv4-trial-2`. each row is one episode (user message = full history so far + assistant reply); the first user message is 4.5K–133K chars and **always begins with the same 2,834-char Terminus-2 instruction** before `Task Description`. consequences:
- a row-level shuffle can put `_run2` of a task in calib and `_run4` in eval: same task text, same initial terminal state.
- with `[:6000]`, ~47% of every ukisai document is that identical preamble, and the rest is mostly the task statement + first terminal output. 220 copies of the preamble is wasted calibration budget, and in `eval_ops_tools.txt` it gives the pruned model an easy, memorizable stretch that flatters KL.
- `uki_other = [i for i in uki_indices if i not in set(uki_nl2bash)]` rebuilds the set every iteration (O(N²), ~15K × 15K); slow, not wrong. `"term" in task` also matches any task id containing "term" — check what it actually selects.
fix: group-split on the base task id (`re.sub(r'(_run\d+|__dsv4-trial-\d+)$', '', task)`), keep one episode per trial (the last, which contains the whole trajectory), strip the shared preamble after its first occurrence, and render with the real chat template (tokenizer.apply_chat_template) instead of a hand-rolled one.

**whittle (A7), confirmed from rows 0–100 of shard 0.** no duplicate prompts in the sample, so no prompt-level leak seen. but the formatting is broken: `prompt` already contains the full template up to `<|im_start|>assistant\n<think>\n` (100/100 rows), and the script wraps it in another user turn. agent1 prompts are 38K chars, so `[:4000]` keeps only the start of the identical `# Tools` schema. the dataset's `response` (which carries `<tool_call>` blocks) and `tool_calls` columns are unused; 10/100 rows have an empty `answer`. no `kind` filter is applied, contrary to the comment (sample: math 31, longctx 20, agent1 15, json 15, gsm 14, chat 5).
fix: use `prompt + response` as-is (it is already the exact serving format), filter/stratify by `kind`, truncate from the left of long prompts (keep the end, where the task and the latest tool output are), and parse special tokens.

**hotchpotch (A8).** `next(rglob("*.parquet"))` returns train or test depending on directory order. test has 1 repeated `id` per 100 rows (several questions per passage); with 240 sampled rows the chance of one passage on both sides is small but nonzero. fix: pick the file explicitly, group by `id`/`url`.

### 1.4 context-length caveat

`-c 512` means every chunk starts cold (no prior context, recurrent GDN state cleared per chunk by `llama_memory_clear`), and documents are cut every ~4,000 chars with a domain switch in the middle of many windows. routing at positions <512 in a spliced window isn't the routing of a 50K-token agent session. it's what imatrix normally does, so it's not a bug, but for a prune that has to survive long tool loops, `-c 2048–4096` with the same token budget is closer to the workload.

---

## 2. C++ hook (`patch_imatrix.py`)

### 2.1 patch mechanics

- `replace("bool IMatrixCollector::collect_imatrix(", hook + ..., 1)`: inserts the struct before the definition (the in-class declaration has no `IMatrixCollector::` prefix). ok.
- `target_ask` matches master verbatim. ok.
- `target_lock`, replaced once: the first `std::lock_guard<std::mutex> lock(m_mutex);` in the file is inside `collect_imatrix` (char 20,833 > definition at 19,589). ok.
- `"g_collector.save_imatrix();"` is replaced **without a count**: it occurs twice in master, at L1899 (combine-only mode, no prompt) and L2041 (end of main). the periodic `--output-frequency 10` saves call `save_imatrix()` from inside `collect_imatrix` with no `g_collector.` prefix, so they are **not** patched. so in a normal run `save_all()` runs exactly once, at the end. good.
  - footgun: running `llama-imatrix --in-file a.gguf --in-file b.gguf` (combine mode) with `REAP_OUT` unset overwrites `/content/reap_saliency.json` with all zeros. low risk because the run used `REAP_OUT=/content/reap_saliency_full.json`, but worth a guard (`if (!any_data) return;`).
  - if the run dies (timeout / OOM), `save_all()` never runs: no JSON at all, and the trace header keeps `ntok = 0`. readers should derive `ntok = (filesize - 16) / 12288` and ignore the header when it's 0.

### 2.2 `on_logits` (router softmax mass + top-64 trace)

- graph facts (qwen4exp.cpp L1142–1159, build_moe_ffn): every one of the 48 trunk layers is MoE (`GGML_ASSERT(ffn_gate_inp != nullptr)`); gating is **softmax**, `exp_probs_b = nullptr`, `norm_w = true`, `w_scale = expert_weights_scale`; `ffn_moe_logits = mul_mat(gate_inp, cur)` → `[n_expert, n_tokens]`, freshly allocated, contiguous F32. so `data + row * n_expert` is the right stride, `prob_sum` is the correct router probability, and the top-10 of the trace's top-64 logits is exactly the real selection (no selection bias, no group masking).
- MTP block has `il = 48` → rejected by `il >= N_LAYERS`. ok.
- `il == 0` sizes the per-ubatch buffer and `il == 47` flushes it; `cur_ubatch_tokens` persists across layers 1..46 and is reset after the flush. with `-b 512 -c 512` there is one sequence per batch and one ubatch per chunk, so trace rows are in text order and every layer sees the same `n_tokens`. `trace_tokens_written` is only advanced at layer 47, after all 48 layers were filled. the clamp `min(n_tokens, max − written)` handles the last partial ubatch. correct.
  - caveat: with `-b > -c` imatrix packs `n_seq = n_batch / n_ctx` chunks into one batch, and the hybrid (GDN) memory splits ubatches per sequence, so trace rows would no longer be in text order and nothing in the record says which chunk/position a row is. the command used `-b 512`, so not triggered; keep it that way or log `(chunk, pos)` per row.
- record layout: per token, per layer, 64 × u16 ids then 64 × f16 logits = 256 B; ×48 = **12,288 B/token**. header = `Q4RT`(4) + ver(2) + n_layer(2) + n_expert(2) + top_m(2) + ntok(4) = 16 B, and `fseek(fp, 12)` hits the ntok field exactly. `fseek` + `fwrite` on a `"wb"` stream is legal. expected file size for the full run: 16 + 8,192 × 12,288 = **100,663,312 B**.
- nits: the softmax `exp` is computed twice per element; `inv_exp_safe(inv_sum)` can't be non-finite since `sum_exp ≥ 1`; a NaN logit would break `partial_sort`'s strict weak ordering (UB). none of these change results on finite input.

### 2.3 `on_weighted_out` (REAP numerator)

- `ffn_moe_weighted = ggml_mul(experts, weights)` with `experts = mul_mat_id(down_exps, …, selected_experts)` → `[n_embd, n_expert_used, n_tokens]`, and `weights` → `[1, n_expert_used, n_tokens]`, broadcast on dim 0. the `ggml_mul` result is a new contiguous tensor; `idx * nb[1] + row * nb[2]` addresses slot `idx` of token `row`. correct.
- `ids = down_t->src[2]` = `selected_experts`, an `[n_expert_used, n_tokens]` I32 that may be a non-contiguous view of the argsort; `ggml_backend_tensor_get(ids, buf, 0, ggml_nbytes(ids))` copies the view's strided extent from its data pointer, and `row * nb[1] + idx * nb[0]` is exactly upstream imatrix's own indexing. slot `idx` of the weighted tensor and `ids[idx, row]` refer to the same expert, because `mul_mat_id` writes output slot `idx` for `ids[idx, row]`. correct.
- `‖w·f‖₂ = w·‖f‖₂` because the softmax-derived weight is > 0, so `reap_l2_sum[e] / count[e]` is **exactly REAP's S_j** with the renormalized top-k gate (which is also Cerebras' current default, `renormalize_router_weights`). `w_scale` is a per-layer constant and doesn't change within-layer ranking.
- **latent silent failures (B2), not triggered by this command but present in master:**
  1. `build_lora_mm_id(..., slots)`: when the MoE cache is on (`--moe-cache-mib N`), the down `MUL_MAT_ID` gets `src[2] = slots` (cache slot indices). the hook would read slots as expert ids; they pass the `ex < 512` check and silently credit the wrong experts.
  2. if the GGUF has `ffn_down_exps_s` (per-expert scales) or a down bias, `ffn_moe_weighted->src[0]` is a `MUL` / `ADD_ID`, not `MUL_MAT_ID`; the `if` falls through and returns `true` with **no stats** → all REAP sums 0, and the selector silently degrades to ranking by `prob_sum` (the tie-break).
  the parent's smoke run reported non-zero REAP sums, so (2) didn't fire in that build, but nothing guards against it. fix: capture `ffn_moe_topk-{il}` in the callback and use it as the id source, and `GGML_ABORT` if `down_t` isn't a `MUL_MAT_ID` over the real expert ids.
- `if (std::isfinite(sq))` silently drops non-finite rows *and* their count; upstream imatrix aborts on non-finite instead. it should at least count drops.

### 2.4 side effect on the imatrix itself (B4)

returning early for `ffn_moe_logits-*` means the `MUL_MAT` on `blk.N.ffn_gate_inp.weight` never reaches the dense-collection branch, so `full_imatrix.gguf` has no entries for the router weights. harmless for pruning; it matters only if that imatrix gets reused for quantization (routers are usually kept at high precision anyway). fix: fall through to the normal path after collecting instead of `return true`.

### 2.5 invariants to check on the existing artifacts (cheap, do these first)

on `reap_saliency_full.json` (65,536 tokens, top-10):
- per layer: `Σ_e count == 655,360` for all 48 layers (catches B2-type skips and NaN drops)
- per layer: `Σ_e prob_sum ≈ 65,536` (±1e-3 relative)
- `trace_tokens_written == 8192`, trace file = 100,663,312 B, and for each trace row the top-10 ids appear in `count` (spot check)
- recompute `count` from the trace's top-10 for the first 8,192 tokens and compare against a separate run of the hook limited to 16 chunks: they must match exactly

---

## 3. saliency metric and selection (`select_and_eval_prunes.py`)

### 3.1 what each metric is

| field | formula | what it measures |
|---|---|---|
| `count` | \|X_j\| | routing frequency ("frequency" baseline in REAP; weakest) |
| `prob_sum` | Σ_x softmax(z(x))_j over **all** tokens | soft frequency; ignores output magnitude; includes mass on tokens where j wasn't selected |
| `reap_l2_sum` (default) | Σ_{x∈X_j} g_j(x)·‖f_j(x)‖ | **count × REAP**: expected gated output magnitude per calibration token |
| `reap_l2_sq_sum` | Σ_{x∈X_j} (g_j‖f_j‖)² | gated output energy, outlier-heavy; first-order "delete without replacement" error |
| *(missing)* REAP | `reap_l2_sum / count` | the published criterion (Lasby et al., arXiv 2510.13999): mean over routed tokens |

### 3.2 problems with the default (C1)

1. **it isn't REAP.** the paper deliberately averages over X_j so a rarely-routed expert that matters a lot when it fires isn't pruned just for being rare. the sum multiplies that back by frequency. that's a legitimate criterion ("expected contribution per token"), but it's a different method and should be named and ablated as such.
2. **it inherits the corpus mix linearly.** with A3 (JA over-weighted by tokens) and A1 (no ukisai/hotchpotch), the sum protects JA/wiki experts and under-protects terminal/tool-loop experts. the mean is less sensitive to mix (still not immune).
3. **rare-token specialists are what the sum prunes first.** §2.3's known risk ("reliability knee ~K=288–320: rare tokens, clean termination") is a rare-event failure. at K=224 (56% of experts removed) a frequency-weighted score is the wrong default. and with A2, the "clean termination" tokens (`<|im_end|>`, `</think>`) never even appeared as special tokens in calibration.
4. **the mean has its own trap:** with \|X_j\| small, `sum/count` is high-variance (one large-norm hit → top rank). use a shrinkage estimator, e.g. `(Σ + m·μ_layer) / (count + m)` with m≈20–50, or require `count ≥ c_min` before trusting the mean.

recommended ranking: compute per-domain REAP (separate imatrix passes per domain file, or log a domain id), normalize within each domain/layer, and keep an expert if it is in the top set for **any** domain (max over domains) or under a weighted mixture with explicit weights (e.g. ops 0.35 / chat+json 0.25 / JA 0.25 / reasoning 0.15). that makes the domain budget an explicit knob instead of an accident of chars-per-token. ablate {REAP mean, REAP-sum, per-domain-max REAP} at each K on the tier-1 KL screen; PLAN §2.3 already says to pick by behavior, not router stats.

### 3.3 pruning without router retraining (C2)

- with `norm_w = true` and softmax gating, the gate weights are `exp(z_i) / Σ_{i∈top10} exp(z_i)`: the global softmax denominator cancels. dropping router rows therefore doesn't inflate probabilities; it only changes **which** 10 experts are picked. a token whose top-10 contained a pruned expert gets the next survivor (rank 11, 12, …) and all 10 weights renormalize. so the per-token error is a **substitution** error, `Σ w'_i f_i − Σ w_i f_i`, not the deleted expert's mass. neither `reap_l2_sum` nor "retained REAP mass" measures that.
- you can measure it offline from the Q4RT trace (top-64 logits are enough: at K=224 the expected number of survivors in the top-64 is ~28): for each keep map, compute per token/layer the fraction of top-10 slots replaced, the old-weight mass that moved, and the rank of the substitutes. that's a better router-side screen than retained mass and costs nothing. (needs a trace drawn from the full, representative corpus; see A1.)
- "retained REAP mass" when selecting by the same metric is maximal by construction; report it, but don't use it to compare K or metrics.
- per-layer constant K is forced by GGUF; saliency concentration differs per layer, so the min-layer numbers the script already reports are the right thing to watch. stats come from the **unpruned** model: pruning layer l shifts layer l+1's inputs, which a one-shot score ignores. RCO (§2.3) handles this properly; one-shot REAP doesn't.
- the shared expert (`ffn_*_shexp`, sigmoid-gated) is untouched by slicing and buffers some substitution error. good, but it also means layer-level KL can look fine while specific rare-token behaviors break, which is why §6.4 tier 2 has to gate the decision.

### 3.4 smaller issues in the selector

- `kept = sorted(ranked[:k])`: ascending order, which is what GGUF slicing needs (router rows and expert slabs stay aligned). ok.
- tie-break `(score, prob_sum, count)`: fine; zero-hit experts fall back to `prob_sum`.
- no validation: `k ≤ n_exp`, every layer present, `Σcount` invariant (§2.5). add asserts so a silently-zero REAP file (B2) can't produce a keep map.
- the expert ids in the existing trace are pre-prune ids; any cache replay on a pruned model needs the keep-map remap plus re-selection of the top-10 among survivors from the stored top-64.
- vision (C3): no image tokens in calibration, so vision-path experts get ~0 score and go first. PLAN §2.3 says images are added before final export; the current keep maps must not be exported or used to judge vision.

---

## 4. recommended order of fixes

1. rebuild the corpus: document-level round-robin across all five sources, token-budgeted per domain, group-split by task/chunk/id, dedupe, whittle `prompt+response` as-is, ukisai last-episode-per-trial with the preamble kept once, TB split by task using `xhigh/`, hotchpotch eval from its `test` split, ChiTako split on chunks (or eval JA from a different source).
2. rerun imatrix with `--parse-special`, `--chunks -1` (all ~0.6–0.7M tokens), `-b 512`, `-c 2048` if time allows; keep `--moe-cache-mib` off.
3. harden the hook: ids from `ffn_moe_topk-{il}`, abort on unexpected `down_t`, count non-finite drops, fall through to normal imatrix collection for the router, write the trace header on every flush (so a killed run is readable), and log `(chunk, pos, token_id)` per trace row.
4. emit REAP mean (with shrinkage) and per-domain stats; build keep maps for {mean, sum, per-domain-max} × {320, 256, 224}.
5. screen with tier-1 KL on **leak-free** eval files (tokenized with special tokens), plus the trace-based substitution metrics from §3.3; finalists to §6.4 tier 2.
6. check the §2.5 invariants on the existing JSON/trace before deleting them: they tell you whether the hook itself was sound even though the corpus wasn't.