opus_r12_moemm — Round 12 Track A: mm_gu3 TG-memory shrink + Q6_K hoisted path + IQ2_XS sub-block hoist #

Scratch: /tmp/opus_r12_moemm/{base,vU,vQ6,vJ,vP,vP0,vQ6J,vNoH,pad1k..pad4k}/ (kernel dirs, runtime-compiled). Harness: prebuilt vendor/strata/build-metal/moe_mm_parity --v3 --gu F --dn Q2_0 --tile 16,32,32,4 --dtile 64,32,2 --gu-bench --T T --ab DIR with STRATA_METAL_KERNELS=<A dir>; 512 experts, M5. No change to moe_mm_parity.cpp was needed. Each run took ~1.6–5 s, mostly compile. GPU check: only the idle :8080 server was running. During my runs I gave sol-qsa-topk-and-flash its exclusive window on request.

Patch: /tmp/opus_r12_moemm/opus_r12_moemm.patch (1 line, git -C vendor/strata apply --check OK) #

Drops F != T_Q6_K from mm_gu3's pointer-hoisted fast-path guard, so Q6_K now takes the hoisted loop. W8<T_Q6_K> is unchanged (the ushort loads stay). With QK=256: boff=(k0>>8)*210, j_in=(k0+kl)&255.

Merge recommendation: conditional, see the T≈1k regression below.

Bitwise parity #

• --dump (HS + y, T=256, 64 experts) for base vs patched kernels: memcmp == 0 on all 11 mm_gu3 formats (Q4_K Q5_K Q6_K Q8_0 IQ4_XS Q2_0 IQ3_XXS IQ2_XXS IQ2_XS IQ2_S IQ3_S).
• Full reference parity on the patched kernels: Q6_K PASS (rel L2 1.99e-05 vs the fp16-operand reference); IQ2_XS PASS (rel L2 1.79e-05, base and patched identical).

Q6_K gate/up timing, paired --ab (A = base, B = patched; median µs per dispatch) #

The patch is a solid -4% at the kprof point (T=1983; ≈ -10 ms per 2k-token prompt across the 12 Q6_K layers). But it reproducibly loses 10–23% for T≈960–1100 (≈2 tiles per expert at 512 experts). The base kernel is also bimodal and non-monotonic in that band (T=512 24 ms, T=1000 23.6 ms, T=768 28 ms), which looks like cache/scheduling sensitivity between the 2 tiles of an expert that read the same weights. I did not root-cause it. Unroll_count(2) on the hoisted loop is far worse (+73% / +18%), and the j_in mask hoist (vQ6J) doesn't change it.

• Merge if prompt chunks are mostly ≥1280 or ≤768 tokens.
• Otherwise gate it host-side by T (not my file), or leave it out.

Target 1 — TG-memory union/shrink: FALSIFIED (no occupancy limit at 12,416 B) #

Padding probe: extra live threadgroup memory added to the base IQ2_XS mm_gu3 (T=2048, paired):

• +4 KB → 16.6 KB: +0.03%
• +8 KB → 20.6 KB: +0.01%
• +12 KB → 24.6 KB: -0.01%
• +16 KB → 28.6 KB: +4.0%

If 32 KB per core bounded residency, 16.6 KB would already drop it from 2 to 1 TG per core and cost a lot. It costs nothing. So per-core TG memory is larger than 32 KB, and at 12.4 KB something else (registers) limits occupancy. Shrinking below 12.4 KB can't add TGs.

Variant U (tab ∪ ot union = 8,192 B, plus a threadgroup barrier before accg.store), paired:

• IQ2_XS: +0.16% (T=512), +0.26/+0.27% (T=2048). Slower because of the barrier.
• Q6_K: -0.06% / +0.16%. Neutral.
• IQ3_S: -0.38/-0.32/-0.10% (T=2048), -0.47% (T=1024). Tiny; only 2 layers (≈0.2 ms).
• Not shipped.

Variant H (half or 2-round float ot): not built. The padding probe already shows TG memory isn't the limiter, so neither H nor U has occupancy to gain.

Target 2b — W8<T_IQ2_XS> sub-block hoisting: no gain #

Measured on top of vQ6, paired at T=512/2048:

• vJ (j_in = (k0&255)|(kl&31), which makes l, hh and the in-group offset provably loop-invariant): -0.31%/-0.03%. Noise; the compiler already hoists them.
• popcount signs instead of the ksg table lookup, IQ2_XS (vP0, vP): +1.8…+2.6% slower. Rejected.
• Control: forcing IQ2_XS off the hoisted path costs +2.0…+2.2% at T=1000/1024/2048, so the existing IQ2_XS fast path is confirmed good and stable at T≈1k (unlike Q6_K).