opus r8 track A: mm_gu3 register epilogue + W8 decoder micro-opts #
patch: /tmp/opus_r8_moemm/opus_r8_moemm.patch (applies with git -C vendor/strata apply, --check clean). touches only
metal/kernels/moe_mm.metal and metal/tests/moe_mm_parity.cpp.
method #
- scratch copy
/tmp/opus_r8_moemm/strata, own build-metal; kernel variants as separate kernel dirs (/tmp/opus_r8_moemm/k_*) selected withSTRATA_METAL_KERNELS. moe_mm_paritygains--dump <file>(HS + y bytes, skips the slow CPU reference -> bitwise A/B of two kernel versions in <0.1 s) and--gu-bench(times only the gate/up GEMM, ~2 s an invocation).- all runs:
--v3 --tile 16,32,32,4(the production mm_gu3 tile),--dn Q2_0, M5. gate/up time a layer, us, min of 3 interleaved runs (other agents were on the GPU at times; medians noisy).
Target 1: drop threadgroup float ot[SG][TM*TN] (8 KiB) from mm_gu3 -> REJECTED #
- direct register stores via
accg.get_multidimensional_index(i)(scalar half) and a coop_k4 half4 variant: both bitwise identical to baseline for IQ2_XS/IQ2_S/IQ2_XXS/IQ3_S/IQ3_XXS (T=256, HS+y). the half4 one confirms the M5 dst layout: elements i..i+3 = 4 consecutive f at one m, 8-byte aligned. - but slower: IQ2_XS T=2048 base 23973 / scalar 24131 / half4 24136 us (+0.7%, clean GPU, 3 reps, spread <0.1%).
- other tiles with half4: 16,64,32,4 +10%; 16,64,32,2 -7%; 32,32,32,2 -3%; 16,32,32,5 / 16,16,32,8 / 16,16,32,4 within ±0.4%. none beats 16,32,32,4 baseline.
- reading: mm_gu3 is decode-bound and threadgroup memory is not the binding occupancy limit (registers likely are); extra TGs/core where it does change do not help (16,64,32,4 gets worse, probably cache contention). not included in the patch.
Target 2: W8 micro-opts #
| change | result | kept |
|---|---|---|
| W8<IQ2_XS>: scales folded (l<2: sc0 = scb&15, sc1 = scb>>4) | T=2048 23963 -> 23343 (-2.6%) | yes |
| W8<IQ2_S>: same fold | 29208 -> 29281 (+0.25%, noise) | no |
| W8<IQ2_XS>: popcount signs instead of ksg tab | erases the fold gain (23947) | no |
| W8<IQ2_XXS>: popcount signs instead of ksg tab | 24704 -> 24489 (-0.9%), sweep below -1.8..-2.3% | yes |
| W8<IQ3_XXS>: existing MM_PROBE=4 popcount | inconclusive under GPU contention (min -2.3%, medians ±5%) | no change |
all kept changes bitwise identical (all 5 IQ formats, HS + y dumps).
final sweep (gate/up us, min of 3, base -> patched) #
| fmt | T=512 | T=1024 | T=2048 |
|---|---|---|---|
| IQ2_XS | 8375 -> 8383 (+0.1%) | 14526 -> 14058 (-3.2%) | 24166 -> 23629 (-2.2%) |
| IQ2_XXS | 8954 -> 8935 (-0.2%) | 15134 -> 14786 (-2.3%) | 26058 -> 25598 (-1.8%) |
| IQ2_S (unchanged code, control) | -0.1% | +0.0% | +0.1% |
| IQ3_S (unchanged code, control) | +0.1% | +0.3% | +2.3% (contended) |
expected prefill effect: ~2% of mm_gu3 IQ2_XS (806.7 ms) + ~2% of IQ2_XXS (67.9 ms) ~= 17-19 ms at T=1983 (~0.4% of total prefill). small but free and bitwise.