phase 1 methodology audit: calibration corpus, REAP/trace hook, expert selection #

read-only audit. scope: scripts/assemble_corpus.py, scripts/patch_imatrix.py, scripts/select_and_eval_prunes.py, PLAN.md §2.3 / §2.5 / §6.1 / §6.4. the C++ was checked against current llama.cpp master (tools/imatrix/imatrix.cpp, src/llama-graph.cpp::build_moe_ffn, src/models/qwen4exp.cpp, common/common.h, common/arg.cpp), fetched for this audit. the Colab clone may be a slightly different commit; the version-sensitive points are marked. the datasets were checked via the HF API / datasets-server (file listings, first 100–300 rows) and the real calibration_ja.txt was downloaded. the run audited is the one launched in the parent session: llama-imatrix -m Swift-...-GSQ-RCO-IQ3_S-00001-of-00002.gguf -f calibration_mix_400k.txt -c 512 -b 512 -ngl 99 --chunks 128 --output-frequency 10 --no-ppl (no --parse-special, no --moe-cache-mib).

---

verdict #

#	finding	severity	affects the run already done?
A1	--chunks 128 -c 512 = 65,536 tokens, read sequentially from the file start. the corpus is ~0.6–0.7M tokens. ukisai and hotchpotch contribute 0 tokens; ops = ~6–8 Terminal-Bench "low-effort failure rerun" transcripts; JA = only the first ~150–200 ChiTako paragraphs	blocker	yes
A2	no --parse-special: parse_special defaults to false in imatrix, so every <|im_start|>, <|im_end|>, <think>, </think>, <tool_call> is tokenized as literal text pieces, never as the special tokens the model routes on in real use	major	yes
A3	domains are balanced by characters, not tokens. JA is ~2–3× more tokens per char, so the 65K-token window is ~50–60% JA, ~20–25% ops, ~20% chat (PLAN: ~30/25/25+)	major	yes
A4	Terminal-Bench split is by sorted path: calib = low/traces/* [0:20], eval = [20:25]. 2 of the 5 eval transcripts (medium/.../adaptive-rejection-sampler, medium/.../chess-best-move) are the same tasks as calib transcripts #1 and #2, and both of those are inside the 65K tokens actually used	major (leak)	yes
A5	ChiTako is split on "\n\n" paragraphs, not on its 916 chunks. 196 / 1,039 kept blocks (18.9%) are exact duplicates; Monte-Carlo over 5 random splits: 3–8 of the 40 eval blocks (~15%) share >50% of their 20-char shingles with the 250 calib blocks	major (leak)	yes (the duplicated ones sit in the window)
A6	ukisai split is by row, but task carries _run2…_run5 and __dsv4-trial-2 suffixes: several rollouts of the same base task can land on both sides. also every row's first user message starts with the same 2,834-char Terminus-2 preamble, so after [:6000] ≈47% of every ukisai doc (calib and eval) is identical boilerplate	major (leak + waste)	no for calib (0 tokens used), yes for eval_ops_tools.txt
A7	Whittle prompt is already chat-templated (<|im_start|>system ... <|im_start|>assistant\n<think>\n); the script wraps it again → nested <|im_start|>user\n<|im_start|>system…, two <think> openers. prompt[:4000] cuts the 38K-char agent prompts inside the identical tool schema; response/tool_calls are ignored, so 10% of rows have an empty answer and "sandbox-verified tool use" is mostly a repeated schema header. no kind filter despite the comment (first 100 rows: 45% math/gsm)	major	yes
A8	hotchpotch file picked with next(rglob("*.parquet")): filesystem order decides train or test split, non-deterministic across machines. some source ids repeat (1/100 in test) → same passage, different question, possible on both sides	minor	no (0 calib tokens)
B1	on_logits, on_weighted_out stride/shape math, trace record layout, header offset, save_all() call sites: correct for this build and this command (details §2)	ok	—
B2	version hazard: with --moe-cache-mib > 0, the down MUL_MAT_ID's src[2] is cache slot ids, not expert ids → REAP stats silently attributed to the wrong experts. with *_exps_s scale tensors or a down bias, ffn_moe_weighted->src[0] is not MUL_MAT_ID → REAP stats silently all zero	major latent (not triggered by this command)	no
B3	trace header ntok is only written at clean exit; trace has no token ids / positions / chunk ids, so §6.1.2's "token ids, turn boundaries, session ids" are not met; the 8,192-token trace is ~2 interleave rounds of teacher-forced prefill, not agent sessions	minor/scope	yes
B4	intercepting ffn_moe_logits-* with an early return true drops the normal imatrix stats for blk.*.ffn_gate_inp.weight from full_imatrix.gguf	minor (only if that imatrix is reused to quantize)	yes
C1	ranking uses reap_l2_sum = Σ g·‖f‖ = count × REAP. published REAP is the mean over routed tokens. the sum is frequency-weighted, so it inherits A3's domain-share skew and is biased against rare-token specialists — the exact "reliability knee" risk §2.3 names	major (method)	yes
C2	"retained REAP mass" in the report is maximized by construction when ranking by the same metric; it ignores substitution (norm_topk renormalization) and is the kind of router statistic §2.3 already says can predict the opposite of behavior	minor (reporting)	—
C3	calibration has no image tokens → vision experts will look dead and get pruned (PLAN's own ISTA lesson); current keep maps are not exportable as-is	major (known, re-flagged)	yes

falsifier status: hit. train/eval leakage confirmed (A4, A5, A6), and the calibration actually used is not the corpus that was designed (A1–A3, A7). no arithmetic bug in the C++ hook for this build/command; two latent silent-failure paths (B2).

---

1. calibration corpus and split hygiene (assemble_corpus.py) #

1.1 what the imatrix run actually sees (A1, A3) #

compute_imatrix takes n_chunk = min(--chunks, tokens/n_ctx) and walks the token stream from the start, in order (imatrix.cpp ~L1464–1525). nothing is shuffled. with -c 512 --chunks 128 that is the first 65,536 tokens.

the file is built as ops / chat / ja 4,000-char blocks interleaved round-robin, where each stream is a concatenation in source order:

• ops = tb_calib_texts (20 × 8,000 chars = 160K chars) then uki_calib_texts
• chat = whittle docs (each ≤ ~10K chars)
• ja = ja_calib_blocks[:250] (avg 330 chars → ~83K chars) then hp_calib_texts

per round (12K chars) at plausible Qwen ratios (TB markdown ~3.2 ch/tok, English ~3.5–4, JA ~1.3–2): ~1,250 + ~1,100 + ~2,000–3,000 ≈ 4.4–5.4K tokens. 65,536 tokens ≈ 12–15 rounds ≈ 48–60K chars per stream. the conclusion is robust to the exact ratios:

• ukisai only starts after 160K ops chars (~40 rounds, >180K tokens). 0 ukisai tokens.
• hotchpotch only starts after ~83K JA chars (~20 rounds, >90K tokens). 0 hotchpotch tokens.
• ops = TB transcripts #1–#7ish, sorted by path: low/traces/adaptive-rejection-sampler, chess-best-move, dna-assembly, dna-insert, extract-moves-from-video, filter-js-from-html, gcode-to-text…. the low/ directory is the dataset's failure reruns at low effort (session paths read tb21-low-failures-…); the 89-task xhigh set is never touched.
• TB transcript.md is a markdown rendering (## Tool activity / Tool: bash / Outcome: …), not the Qwen chat template. it is off-distribution for routing compared to real agent turns.
• token shares in the window are roughly JA 50–60%, ops 20–25%, chat ~20%, versus PLAN §2.3's ~30% ops / ~25% chat+JSON / ~25% JA. the comment "~1.5 MB (~420K tokens)" assumes ~3.6 chars/token across the board, which JA breaks.

the trace (max_trace_tokens = 8192) is the first 16 chunks ≈ 2 rounds: about one TB transcript, 1–2 whittle docs and a handful of ChiTako paragraphs.

fix: run with --chunks -1 (or ≥ corpus/512), and build the file so any prefix is representative: interleave at document granularity across all sources (round-robin per source within each domain), budget each domain by tokens (tokenize with the model's vocab when assembling), and shuffle document order with a fixed seed. at ~3 min per 128 chunks on the A100 (parent's estimate), the full ~1,200 chunks is ~30 min.

1.2 special tokens are not special (A2) #

common_params::parse_special = false (common.h) and compute_imatrix tokenizes with common_tokenize(ctx, params.prompt, true, params.parse_special). the run has no --parse-special. so the ChiTako text, the whittle templates and the hand-built <|im_start|>{role} wrappers all become runs of ordinary BPE pieces (<, |, im, _start, …). the router never sees the real role / think / tool-call tokens in this calibration, and those are exactly the positions where routing for "clean termination" and tool syntax happens. ChiTako's own README tells you to pass --parse-special.

the KL eval should tokenize the same way it will be served (special tokens parsed), otherwise it under-measures damage on template tokens.

1.3 leakage, per source #

Terminal-Bench (A4), confirmed from the repo listing. 160 transcript.md files: xhigh 89, medium 27, low 22, off 20, 2 reruns. sorted(rglob) → [0:20] are 20 of the 22 low/ tasks; [20:25] are low/tune-mjcf, low/winning-avg-corewars, medium/adaptive-rejection-sampler, medium/build-pov-ray, medium/chess-best-move. two eval transcripts are the same task (same container image, same files, same opening tool outputs) as calib transcripts #1 and #2, which are inside the 65K-token window. difflib ratio of the first 8,000 chars of low/ vs medium/ chess-best-move = 0.27: task-level leakage, not verbatim copies. fix: split by task name (the path component after traces/), and prefer xhigh/ for both sides.

ChiTako (A5), confirmed on the real file. calibration_ja.txt is 916 chunks joined by "\n\n", but chunks contain internal "\n\n" paragraph breaks (wikipedia title/body, code blocks, multi-turn chats): splitting on "\n\n" gives 2,244 pieces, 1,039 of them >100 chars. 196 of those 1,039 are exact duplicates of another block (the builder reuses templates: e.g. one system prompt appears 21×, several Python helper snippets 7–8×). shuffling paragraphs then cutting 88/12 puts paragraphs of the same chunk, and literal duplicates, on both sides. over 5 random seeds, 3–8 of the 40 eval blocks had >50% shingle overlap with the 250 calib blocks (mean overlap 0.08–0.21). the exact rng state of the script couldn't be reproduced offline (it depends on the ukisai/whittle table sizes), so this is an estimate of the expected rate, not the exact count. fix: dedupe first (exact + near-dup), and split on the builder's chunk boundaries or, simpler, take the eval JA from a disjoint source (e.g. hotchpotch test split, which is already sampled from fineweb2-edu-ja's test portion).

ukisai (A6), confirmed from rows 0–100, 5000–5100, 10000–10100. columns are conversations, agent, model, model_provider, date, task, episode, run_id, trial_name. task values include nl2bash-1908-0000, nl2bash-1908-0000_run2 … _run5, inferredbugs-0001 and inferredbugs-0001__dsv4-trial-2. each row is one episode (user message = full history so far + assistant reply); the first user message is 4.5K–133K chars and always begins with the same 2,834-char Terminus-2 instruction before Task Description. consequences:

• a row-level shuffle can put _run2 of a task in calib and _run4 in eval: same task text, same initial terminal state.
• with [:6000], ~47% of every ukisai document is that identical preamble, and the rest is mostly the task statement + first terminal output. 220 copies of the preamble is wasted calibration budget, and in eval_ops_tools.txt it gives the pruned model an easy, memorizable stretch that flatters KL.
• uki_other = [i for i in uki_indices if i not in set(uki_nl2bash)] rebuilds the set every iteration (O(N²), ~15K × 15K); slow, not wrong. "term" in task also matches any task id containing "term" — check what it actually selects. fix: group-split on the base task id (re.sub(r'(_run\d+|__dsv4-trial-\d+)$', '', task)), keep one episode per trial (the last, which contains the whole trajectory), strip the shared preamble after its first occurrence, and render with the real chat template (tokenizer.apply_chat_template) instead of a hand-rolled one.

whittle (A7), confirmed from rows 0–100 of shard 0. no duplicate prompts in the sample, so no prompt-level leak seen. but the formatting is broken: prompt already contains the full template up to <|im_start|>assistant\n<think>\n (100/100 rows), and the script wraps it in another user turn. agent1 prompts are 38K chars, so [:4000] keeps only the start of the identical # Tools schema. the dataset's response (which carries <tool_call> blocks) and tool_calls columns are unused; 10/100 rows have an empty answer. no kind filter is applied, contrary to the comment (sample: math 31, longctx 20, agent1 15, json 15, gsm 14, chat 5). fix: use prompt + response as-is (it is already the exact serving format), filter/stratify by kind, truncate from the left of long prompts (keep the end, where the task and the latest tool output are), and parse special tokens.

hotchpotch (A8). next(rglob("*.parquet")) returns train or test depending on directory order. test has 1 repeated id per 100 rows (several questions per passage); with 240 sampled rows the chance of one passage on both sides is small but nonzero. fix: pick the file explicitly, group by id/url.

1.4 context-length caveat #

-c 512 means every chunk starts cold (no prior context, recurrent GDN state cleared per chunk by llama_memory_clear), and documents are cut every ~4,000 chars with a domain switch in the middle of many windows. routing at positions <512 in a spliced window isn't the routing of a 50K-token agent session. it's what imatrix normally does, so it's not a bug, but for a prune that has to survive long tool loops, -c 2048–4096 with the same token budget is closer to the workload.

---

2. C++ hook (patch_imatrix.py) #

2.1 patch mechanics #

• replace("bool IMatrixCollector::collect_imatrix(", hook + ..., 1): inserts the struct before the definition (the in-class declaration has no IMatrixCollector:: prefix). ok.
• target_ask matches master verbatim. ok.
• target_lock, replaced once: the first std::lock_guard<std::mutex> lock(m_mutex); in the file is inside collect_imatrix (char 20,833 > definition at 19,589). ok.
• "g_collector.save_imatrix();" is replaced without a count: it occurs twice in master, at L1899 (combine-only mode, no prompt) and L2041 (end of main). the periodic --output-frequency 10 saves call save_imatrix() from inside collect_imatrix with no g_collector. prefix, so they are not patched. so in a normal run save_all() runs exactly once, at the end. good.
  • footgun: running llama-imatrix --in-file a.gguf --in-file b.gguf (combine mode) with REAP_OUT unset overwrites /content/reap_saliency.json with all zeros. low risk because the run used REAP_OUT=/content/reap_saliency_full.json, but worth a guard (if (!any_data) return;).
  • if the run dies (timeout / OOM), save_all() never runs: no JSON at all, and the trace header keeps ntok = 0. readers should derive ntok = (filesize - 16) / 12288 and ignore the header when it's 0.

2.2 on_logits (router softmax mass + top-64 trace) #

• graph facts (qwen4exp.cpp L1142–1159, build_moe_ffn): every one of the 48 trunk layers is MoE (GGML_ASSERT(ffn_gate_inp != nullptr)); gating is softmax, exp_probs_b = nullptr, norm_w = true, w_scale = expert_weights_scale; ffn_moe_logits = mul_mat(gate_inp, cur) → [n_expert, n_tokens], freshly allocated, contiguous F32. so data + row * n_expert is the right stride, prob_sum is the correct router probability, and the top-10 of the trace's top-64 logits is exactly the real selection (no selection bias, no group masking).
• MTP block has il = 48 → rejected by il >= N_LAYERS. ok.
• il == 0 sizes the per-ubatch buffer and il == 47 flushes it; cur_ubatch_tokens persists across layers 1..46 and is reset after the flush. with -b 512 -c 512 there is one sequence per batch and one ubatch per chunk, so trace rows are in text order and every layer sees the same n_tokens. trace_tokens_written is only advanced at layer 47, after all 48 layers were filled. the clamp min(n_tokens, max − written) handles the last partial ubatch. correct.
  • caveat: with -b > -c imatrix packs n_seq = n_batch / n_ctx chunks into one batch, and the hybrid (GDN) memory splits ubatches per sequence, so trace rows would no longer be in text order and nothing in the record says which chunk/position a row is. the command used -b 512, so not triggered; keep it that way or log (chunk, pos) per row.
• record layout: per token, per layer, 64 × u16 ids then 64 × f16 logits = 256 B; ×48 = 12,288 B/token. header = Q4RT(4) + ver(2) + n_layer(2) + n_expert(2) + top_m(2) + ntok(4) = 16 B, and fseek(fp, 12) hits the ntok field exactly. fseek + fwrite on a "wb" stream is legal. expected file size for the full run: 16 + 8,192 × 12,288 = 100,663,312 B.
• nits: the softmax exp is computed twice per element; inv_exp_safe(inv_sum) can't be non-finite since sum_exp ≥ 1; a NaN logit would break partial_sort's strict weak ordering (UB). none of these change results on finite input.

2.3 on_weighted_out (REAP numerator) #

• ffn_moe_weighted = ggml_mul(experts, weights) with experts = mul_mat_id(down_exps, …, selected_experts) → [n_embd, n_expert_used, n_tokens], and weights → [1, n_expert_used, n_tokens], broadcast on dim 0. the ggml_mul result is a new contiguous tensor; idx * nb[1] + row * nb[2] addresses slot idx of token row. correct.
• ids = down_t->src[2] = selected_experts, an [n_expert_used, n_tokens] I32 that may be a non-contiguous view of the argsort; ggml_backend_tensor_get(ids, buf, 0, ggml_nbytes(ids)) copies the view's strided extent from its data pointer, and row * nb[1] + idx * nb[0] is exactly upstream imatrix's own indexing. slot idx of the weighted tensor and ids[idx, row] refer to the same expert, because mul_mat_id writes output slot idx for ids[idx, row]. correct.
• ‖w·f‖₂ = w·‖f‖₂ because the softmax-derived weight is > 0, so reap_l2_sum[e] / count[e] is exactly REAP's S_j with the renormalized top-k gate (which is also Cerebras' current default, renormalize_router_weights). w_scale is a per-layer constant and doesn't change within-layer ranking.
• latent silent failures (B2), not triggered by this command but present in master:
  1. build_lora_mm_id(..., slots): when the MoE cache is on (--moe-cache-mib N), the down MUL_MAT_ID gets src[2] = slots (cache slot indices). the hook would read slots as expert ids; they pass the ex < 512 check and silently credit the wrong experts.
  2. if the GGUF has ffn_down_exps_s (per-expert scales) or a down bias, ffn_moe_weighted->src[0] is a MUL / ADD_ID, not MUL_MAT_ID; the if falls through and returns true with no stats → all REAP sums 0, and the selector silently degrades to ranking by prob_sum (the tie-break). the parent's smoke run reported non-zero REAP sums, so (2) didn't fire in that build, but nothing guards against it. fix: capture ffn_moe_topk-{il} in the callback and use it as the id source, and GGML_ABORT if down_t isn't a MUL_MAT_ID over the real expert ids.
• if (std::isfinite(sq)) silently drops non-finite rows and their count; upstream imatrix aborts on non-finite instead. it should at least count drops.

2.4 side effect on the imatrix itself (B4) #

returning early for ffn_moe_logits-* means the MUL_MAT on blk.N.ffn_gate_inp.weight never reaches the dense-collection branch, so full_imatrix.gguf has no entries for the router weights. harmless for pruning; it matters only if that imatrix gets reused for quantization (routers are usually kept at high precision anyway). fix: fall through to the normal path after collecting instead of return true.

2.5 invariants to check on the existing artifacts (cheap, do these first) #

on reap_saliency_full.json (65,536 tokens, top-10):

• per layer: Σ_e count == 655,360 for all 48 layers (catches B2-type skips and NaN drops)
• per layer: Σ_e prob_sum ≈ 65,536 (±1e-3 relative)
• trace_tokens_written == 8192, trace file = 100,663,312 B, and for each trace row the top-10 ids appear in count (spot check)
• recompute count from the trace's top-10 for the first 8,192 tokens and compare against a separate run of the hook limited to 16 chunks: they must match exactly

---

3. saliency metric and selection (select_and_eval_prunes.py) #

3.1 what each metric is #

field	formula	what it measures
count	|X_j|	routing frequency ("frequency" baseline in REAP; weakest)
prob_sum	Σ_x softmax(z(x))_j over all tokens	soft frequency; ignores output magnitude; includes mass on tokens where j wasn't selected
reap_l2_sum (default)	Σ_{x∈X_j} g_j(x)·‖f_j(x)‖	count × REAP: expected gated output magnitude per calibration token
reap_l2_sq_sum	Σ_{x∈X_j} (g_j‖f_j‖)²	gated output energy, outlier-heavy; first-order "delete without replacement" error
(missing) REAP	reap_l2_sum / count	the published criterion (Lasby et al., arXiv 2510.13999): mean over routed tokens

3.2 problems with the default (C1) #

1. it isn't REAP. the paper deliberately averages over X_j so a rarely-routed expert that matters a lot when it fires isn't pruned just for being rare. the sum multiplies that back by frequency. that's a legitimate criterion ("expected contribution per token"), but it's a different method and should be named and ablated as such.
2. it inherits the corpus mix linearly. with A3 (JA over-weighted by tokens) and A1 (no ukisai/hotchpotch), the sum protects JA/wiki experts and under-protects terminal/tool-loop experts. the mean is less sensitive to mix (still not immune).
3. rare-token specialists are what the sum prunes first. §2.3's known risk ("reliability knee ~K=288–320: rare tokens, clean termination") is a rare-event failure. at K=224 (56% of experts removed) a frequency-weighted score is the wrong default. and with A2, the "clean termination" tokens (<|im_end|>, </think>) never even appeared as special tokens in calibration.
4. the mean has its own trap: with |X_j| small, sum/count is high-variance (one large-norm hit → top rank). use a shrinkage estimator, e.g. (Σ + m·μ_layer) / (count + m) with m≈20–50, or require count ≥ c_min before trusting the mean.

recommended ranking: compute per-domain REAP (separate imatrix passes per domain file, or log a domain id), normalize within each domain/layer, and keep an expert if it is in the top set for any domain (max over domains) or under a weighted mixture with explicit weights (e.g. ops 0.35 / chat+json 0.25 / JA 0.25 / reasoning 0.15). that makes the domain budget an explicit knob instead of an accident of chars-per-token. ablate {REAP mean, REAP-sum, per-domain-max REAP} at each K on the tier-1 KL screen; PLAN §2.3 already says to pick by behavior, not router stats.

3.3 pruning without router retraining (C2) #

• with norm_w = true and softmax gating, the gate weights are exp(z_i) / Σ_{i∈top10} exp(z_i): the global softmax denominator cancels. dropping router rows therefore doesn't inflate probabilities; it only changes which 10 experts are picked. a token whose top-10 contained a pruned expert gets the next survivor (rank 11, 12, …) and all 10 weights renormalize. so the per-token error is a substitution error, Σ w'_i f_i − Σ w_i f_i, not the deleted expert's mass. neither reap_l2_sum nor "retained REAP mass" measures that.
• you can measure it offline from the Q4RT trace (top-64 logits are enough: at K=224 the expected number of survivors in the top-64 is ~28): for each keep map, compute per token/layer the fraction of top-10 slots replaced, the old-weight mass that moved, and the rank of the substitutes. that's a better router-side screen than retained mass and costs nothing. (needs a trace drawn from the full, representative corpus; see A1.)
• "retained REAP mass" when selecting by the same metric is maximal by construction; report it, but don't use it to compare K or metrics.
• per-layer constant K is forced by GGUF; saliency concentration differs per layer, so the min-layer numbers the script already reports are the right thing to watch. stats come from the unpruned model: pruning layer l shifts layer l+1's inputs, which a one-shot score ignores. RCO (§2.3) handles this properly; one-shot REAP doesn't.
• the shared expert (ffn_*_shexp, sigmoid-gated) is untouched by slicing and buffers some substitution error. good, but it also means layer-level KL can look fine while specific rare-token behaviors break, which is why §6.4 tier 2 has to gate the decision.

3.4 smaller issues in the selector #

• kept = sorted(ranked[:k]): ascending order, which is what GGUF slicing needs (router rows and expert slabs stay aligned). ok.
• tie-break (score, prob_sum, count): fine; zero-hit experts fall back to prob_sum.
• no validation: k ≤ n_exp, every layer present, Σcount invariant (§2.5). add asserts so a silently-zero REAP file (B2) can't produce a keep map.
• the expert ids in the existing trace are pre-prune ids; any cache replay on a pruned model needs the keep-map remap plus re-selection of the top-10 among survivors from the stored top-64.
• vision (C3): no image tokens in calibration, so vision-path experts get ~0 score and go first. PLAN §2.3 says images are added before final export; the current keep maps must not be exported or used to judge vision.

---

4. recommended order of fixes #

1. rebuild the corpus: document-level round-robin across all five sources, token-budgeted per domain, group-split by task/chunk/id, dedupe, whittle prompt+response as-is, ukisai last-episode-per-trial with the preamble kept once, TB split by task using xhigh/, hotchpotch eval from its test split, ChiTako split on chunks (or eval JA from a different source).
2. rerun imatrix with --parse-special, --chunks -1 (all ~0.6–0.7M tokens), -b 512, -c 2048 if time allows; keep --moe-cache-mib off.
3. harden the hook: ids from ffn_moe_topk-{il}, abort on unexpected down_t, count non-finite drops, fall through to normal imatrix collection for the router, write the trace header on every flush (so a killed run is readable), and log (chunk, pos, token_id) per trace row.
4. emit REAP mean (with shrinkage) and per-domain stats; build keep maps for {mean, sum, per-domain-max} × {320, 256, 224}.
5. screen with tier-1 KL on leak-free eval files (tokenized with special tokens), plus the trace-based substitution metrics from §3.3; finalists to §6.4 tier 2.
6. check the §2.5 invariants on the existing JSON/trace before deleting them: they tell you whether the hook itself was sound even though the corpus wasn't.