replay_moe_cache.py (read-only) #

both self-tests pass (--self-test, exit 0). nothing edited.

slice_qwen4exp_gguf.py #

verified clean:

header: magic/version/u64 counts, KV raw round-trip (incl. nested arrays and string arrays), tensor info layout (name, u32 n_dims, u64 dims, u32 type, u64 offset). ok.
alignment: general.alignment honored; data section = align(header end); every new rel_offset aligned (l.330); tail zero-padding (l.373-377). ok. offsets are recomputed in directory order, so input data order doesn't matter (the l.159 comment claiming ascending order and padded_span l.158-168 are dead code, harmless).
GGML_TYPE_TRAITS: all match ggml (Q2_K 84, Q3_K 110, IQ2_XXS 66, IQ2_XS 74, IQ3_XXS (256,98), IQ1_S 50, IQ4_NL (32,18), IQ3_S (256,110), IQ2_S 82, IQ4_XS (256,136), IQ1_M 56, TQ1_0 54, TQ2_0 66, MXFP4 (32,17)). Q2_0=42 (64,18) = 2.25 bpw and Q1_0=41 (128,18), NVFP4=40 (64,36) match secondary quotes of upstream ggml-common.h (couldn't read the header itself).
dim order: ggml ne[] is innermost-first, so dims[-1] is the outermost axis: experts for ffn_*_exps [n_ff|n_embd, ., n_expert] and router rows for ffn_gate_inp [n_embd, n_expert]; per-expert bytes are contiguous, so nbytes // 512 slicing is right. router rows and expert slabs use the same keep order, so the permutation stays consistent.
split.count/split.no/split.tensors.count: tensor count doesn't change, so leaving them alone is right. shard 2+ have no expert_count KV (gguf-split puts it in shard 1 only), which is fine.

bugs / edge cases (most important first):

per-expert tensors the regex misses -> llama.cpp shape-mismatch load failure (l.50-52). only _exps.weight and ffn_gate_inp.{weight,bias} match. any blk.N.ffn_{gate,up,down,gate_up}_exps.bias, blk.N.ffn_*_exps.scale (per-expert scale tensors), or blk.N.exp_probs_b.bias ([512] expert-bias) gets copied with 512 rows while expert_count=K. fix: match ^blk\.(\d+)\.(ffn_\w*_exps\.(weight|bias|scale)|ffn_gate_inp\.(weight|bias)|exp_probs_b\.bias)$, and after the loop warn about (or fail on) any blk.* tensor still having dims[-1]==old_n_experts that wasn't sliced.
MTP layer blk.48. if the shard has nextn/MTP expert tensors at blk.48, l.307 raises KeyError because --keep-json is documented as '0'..'47' (l.477). that fails loud rather than corrupting anything, but the keep-JSON generator has to emit layer 48 (from MTP routing stats). fix the help text and the selection pipeline.
no expert_used_count <= K check (l.284-290). if K < 10 llama.cpp asserts at load. fix: parse *.expert_used_count too and raise if > new_k. also cross-check the header expert_count against old_n_experts (it's parsed at l.130 and never used).
in == out path destroys the input (l.353, O_TRUNC before the data is read). there's also no atomic write, so a crash leaves a truncated shard with a valid-looking header. fix: refuse os.path.samefile, write to out+'.partial', then os.replace.
>2 GiB tensors (l.362): os.pread(whole tensor) caps at 0x7ffff000 bytes on Linux (that's a loud EOFError there) and gives EINVAL on macOS. Q2_0/IQ3_S expert tensors are about 240-720 MB, so they're fine, but BF16/Q8 teacher-style ffn_gate_up_exps go over. pwrite results aren't checked either (l.355/371/377: short writes are silent). fix: per kept expert, pread(per_exp_bytes, abs_start + idx*per_exp_bytes), and loop pwrite until the whole buffer is written.
minor: the self-test comment at l.418 is stale ("32 elems = 10B"; Q2_0 is 64 elems = 18B). len(set(idx_list)) != new_k (l.281) does catch duplicates. fine.

replay_moe_cache.py #

verified clean:

select_moce_top_k: z' = z + λ·Δ·1[cached ∨ rank<J], stable tie-break on rank, top-J always survive (they get the bonus too). λ=0 gives the unbiased top-k. Δ is the per-token top1-topk range, not a corpus mean as the docstring says (fine, but say so).
belady_min: next-use lists are built on unbiased top-k, and pointers advance only on chosen experts. at λ=0 every use is a choice, so non-chosen candidates' pointers are correct; it evicts max-next-use among non-chosen. main() skips it for λ>0 (l.288), which is right. it matches the trace semantics.
warmup accounting and miss counting are right.

bugs:

EMA policies only decay resident experts (l.190-191, 195-196, 206-207). near_pick_ema and inverse_router_ema credit every top-N/top-m expert, resident or not, but only decay cache_set. so non-resident experts pile up an undecayed sum, and evicted experts keep a frozen score. demo: an expert sitting at a near-pick rank for 300 tokens builds up 104.9 (a true EMA would be ~33.7) against ~26 for the hot residents. once it's chosen once, it stays pinned for the next 99 tokens without ever being used again. plain ema has the frozen-on-eviction half of this. this biases exactly the near-pick vs inverse-router comparison that PLAN §2.5 #6 wants. fix: decay all n_experts each step (for c in range(n_experts), 512 mults is cheap), or do lazy decay with score*decay**(t-last_t).
cache_cap < k_used (l.165-166): the fallback can evict an expert chosen in this same step, including one admitted a moment earlier, and the miss count turns into garbage. fix: assert cache_cap >= k_used.
near_pick credits unbiased ranks only (l.197-202). under λ>0 the experts MoCE actually picked from outside the unbiased top-k get only near-pick credit (zero at rank ≥ near_pick_n). decide whether that's intended. if not, add credit = max(credit, 1.0) for eid in chosen_set.
trace shorter than warmup (l.217): misses=0, eval_tokens=1, so 0 misses/token goes silently into the 48-layer sum. fix: skip/flag it.
scope gaps vs PLAN §6.1 (these aren't math bugs, but they'll give wrong numbers if someone trusts the output):
- the docstring promises non-evicting next-layer prefetch eval, and none of it exists.
- the trace format has no session/turn/prefill-vs-decode markers, so the cache is never reset and there are no per-turn numbers.
- prefill is replayed token by token instead of as a per-chunk expert union (that misstates prefill misses).
- read_bytes() + Python tuples won't scale to the planned 12 GB / 1M-token trace. use a numpy memmap.
- λ>0 here is open-loop. it only models same-layer selection feedback, not downstream logit drift (§6.1 #6 already says closed loop is needed).
MB/expert inconsistency: l.305-306 uses 1.3824 MB (= 4.915M × 2.25/8, which is correct for Q2_0 (64,18)), but PLAN §5.2 l.373 says ~1.536 MB and the §3 Q2_0 column (37.7/23.6/18.9/16.5 GB) is computed at 2.5 bpw. so PLAN overstates Q2_0 bytes by ~11% (224 experts: 14.9 GB, not 16.5).