opus r23 quant: ksg parity signs, dq_f16 / DW / unroll (Round 23 Track A) #

Patch: /tmp/opus_r23_quant/opus_r23_quant.patch (git -C vendor/strata apply --check OK). Files touched: metal/kernels/quant.metal (shipped kernel change), metal/tests/dense_mm_parity.cpp (new A/B mode). Device: Apple M5 (24 GB). Scratch: /tmp/opus_r23_quant/{base,strata} (base = vendor at 296b59d + working tree).

TL;DR #

• ksigns_iq2xs[i] == i | (popcount(i) & 1) << 7 holds for all 128 entries (checked vs ggml-common.h). But computing it is not a uniform win. On M5 it depends on format and on where the lookup would otherwise go (constant vs threadgroup), so ksg now computes the sign byte only where it was measured faster:
  • IQ2_XS with QUANT_CTAB (production decode moe_gu, since window.cpp sets ctab for the routed gate/up): -2.5..-5.5% gate/up time across NE=1..4, ntok 1..8 (ratios 0.945..0.995, 2 runs)
  • IQ3_XXS with the threadgroup table (production prompt mm_gu3_18_16_32_32_4, which runs with ctab off): -1.6..-3.9% (5 paired runs, quiet GPU; A/A control within +-0.1%). Its 128-byte sign copy is gone, and fmt<T_IQ3_XXS>::TAB drops from 288 to 256 words.
  • everywhere else it stays a lookup, because computing it measured slower: IQ2_XXS moe_gu ctab +2..+13%, IQ3_XXS moe_gu ctab +0..+6%, IQ2_XS mm_gu3 +1.5..2% (matches opus_r8/opus_r12, which rejected popcount for IQ2_XS mm).
  • known trade-off: IQ3_XXS moe_gu with ctab off (only reachable with STRATA_METAL_CTAB=0) is +5..7% slower. The default decode path (ctab on) for IQ3_XXS is unchanged (0.998..1.006).
• dq_f16 is at the memory roofline. IQ4_XS 2560x10240 moves 67 MB (52 MB of fp16 out + 15 MB in) in ~509 us = ~132 GB/s on M5. None of the decode-side ideas helped, so none ship:
  • IQ4 DW without the table/barrier, using a 4-register packed codebook: slower, 1% on big shapes and 10-22% on small ones
  • IQ4 DW reading kvalues_iq4nl straight from constant memory: ~1% slower / neutral
  • a.perm guard (no integer division when perm == 0): 1.00x
  • 16-byte uint4 stores in place of the half4 pairs: 1.00x
• #pragma clang loop unroll(full) on the fixed-trip loops (load16, X::load, every Dec<F>, unit_combine, every deq<F> loop, put loops): net loss on gemv. IQ4_XS 2560x10240 was +8..9% (best tile, 2 sweeps), Q6_K 2560x10240 +2.8%, Q4_K/Q3_K/Q5_K/Q4_0/BF16 within +-1..2%. Not shipped. The compiler already unrolls these where it pays; forcing it costs registers.

Bitwise verification #

Every output buffer bitwise: I added an instrumentation hook to the scratch base mtl_rt.mm (not in the patch): with STRATA_ABHASH=file, after every Context::run/run_graph it hashes every buffer bound to the dispatches. Each test binary runs twice, with STRATA_METAL_KERNELS=base/kernels and then =candidate/kernels, and the hash logs are compared. A/A runs first confirmed the tests are deterministic. With the final patch, all 31 cases gave the same hashes (OK) and the same exit codes (0):

• dense_mm_parity (360 hashed buffer states), gemv_parity (26,316)
• expert_parity --gu {Q2_0,IQ3_XXS,IQ2_XXS,IQ2_XS,IQ2_S,IQ3_S,IQ4_XS,Q4_K,Q5_K,Q6_K,Q8_0} --ntok 3, --dn {Q2_0,IQ4_NL,Q5_1,Q8_0} --ntok 8
• moe_mm_parity --v3 --tile 16,32,32,4 --dtile 64,32,2 --T 48 --gu {Q2_0,IQ3_XXS,IQ2_XXS,IQ2_XS,IQ2_S,IQ3_S,IQ4_XS,Q4_K,Q8_0}, --dn {IQ4_NL,Q5_1,Q8_0}, --gu IQ2_XS --tile 16,64,64,2 (mm_gu), --gu IQ3_XXS --dn IQ4_NL
• for the ctab=on decode path, each expert_parity --ab run (both ctab on and ctab off, IQ2_XS/IQ3_XXS/IQ2_XXS/IQ2_S) passed its in-process memcmp of plan/h/ye/y. Trick: quant.metal and units.metal are empty stubs in the kernel dir, and moe.metal / the --ab file are each version's quant+units+moe concatenation, so A and B carry different quant.metal.
• moe_mm_parity --v3 --gu {IQ2_XS,IQ3_XXS} with QUANT_CTAB=1 (mm_gu3 ctab on): same hashes too
• dense_mm_parity --dq-ab with DQ_AB_KERNELS: memcmp=0 for every shape tried (IQ4_XS, IQ4_NL, IQ3_S, Q8_0)

Measurements (paired, M5) #

mm_gu3 (prompt, moe_mm_parity --v3 --tile 16,32,32,4 --dtile 64,32,2 --gu-bench --ab KB, kernel dir = base, T=2048) #

diff = (cand - base) / base, med / min, %

IQ2_S: removing its dead sign copy / shrinking TAB came out +1..3% but within that run's noise (A/A spread up to 11%). Left unchanged.

moe_gu (decode, expert_parity --bench --ab, cand/base gate/up ratio over NE=1,2,3,4 same + ntok=1,2,4,8 sh=0.3) #

• IQ2_XS ctab: 0.953 0.995 0.965 0.975 0.945 0.954 0.971 0.968 (earlier run 0.947..0.993)
• IQ2_XS tab (unchanged path): 0.992..1.007
• IQ3_XXS ctab (unchanged path): 0.998..1.006, plus one 1.064 at ntok=8
• IQ3_XXS tab: 1.047..1.076 (non-default path, see trade-off above)
• IQ2_XXS ctab / tab (unchanged path): 0.993..1.006 / 0.996..1.018

dq_f16 (DQ_AB_KERNELS=base dense_mm_parity --dq-ab TYPE, CHUNK totals) #

• A/A: IQ4_XS 60.20 -> 60.39 ms, IQ3_S 55.14 -> 55.09
• IQ4 4-register LUT: IQ4_XS 60.06 -> 60.61 (2560x128: 0.78x, 2560x48: 0.85x), IQ4_NL 65.44 -> 65.45 (320x10240p4: 0.82x)
• IQ4 constant kvalues_iq4nl: 56.53 -> 57.15
• perm guard: 56.55 -> 56.51
• 16-byte stores: IQ4_XS 56.75 -> 56.82, IQ3_S 55.18 -> 55.12, Q8_0 80.27 -> 76.45 (one 2560x10240 outlier; the other 12 shapes 1.00x)

gemv unroll pragmas (gemv_parity --one, 3 alternating runs per shape, min per tile) #

best tile base -> cand: IQ4_XS 2560x10240 104.6 -> 114.1 (+9.2%; first sweep +24%); Q6_K 2560x10240 160.0 -> 164.5 (+2.8%); IQ4_NL 640x2560 +3.3%; Q2_0 2560x640 +0.6%; Q3_K -2.2%; Q4_K -0.9%; Q5_K -0.7%; IQ3_S -0.3%; Q4_0 -0.1%; BF16 -0.1%. Rejected.

New test mode (in patch) #

DQ_AB_KERNELS=<dir> dense_mm_parity --dq-ab [type] [shape] compiles <dir>'s quant/units/mma/dense_mm next to this build's. It runs both at dq_tg's threadgroup size, memcmps the fp16 output and times them in paired rounds. The warm-up now covers both sides (before, the first shape was biased against A by ~14%). Without the env var the mode is unchanged (64 threads vs dq_tg's).

Notes / caveats #

• ksg picks by at (the grid's word count: 256 IQ3_XXS, 512 IQ2_XXS, 1024 IQ2_XS). It is a compile-time constant at every call site (quant.metal and moe_mm.metal), so the branch folds away. moe_mm.metal's call sites didn't need changing.
• The moe_mm.metal:691 ks pointer (tab + 256) is unused, so the IQ3_XXS shrink is safe. The only other readers of tab + G go through ksg.
• About the heartbeat's note: my moe_mm runs used kernel dir = base and --ab = candidate, so A = base and B = candidate. Negative means the candidate is faster, as reported here (the A/A controls confirm the direction).
• Not tested end to end (no full-model runs, per the rules). Expected effect: decode routed gate/up for the IQ2_XS K512 pack about -4%; prompt mm_gu3 for IQ3_XXS packs about -3%.