opus r26 track A: IQ4_XS / IQ4_NL packed-code GEMV + Q2_0R multi-token #
patch: /tmp/opus_r26_gemv/opus_r26_gemv.patch (git -C vendor/strata apply --check passes against the current working tree)
scratch: /tmp/opus_r26_gemv/strata (+ build/gemv_parity, variant kernel dirs v_*, base snapshot base_kernels)
what ships #
gemv.metal: an IQ4_XS path ingemvthat keeps each row's unit as its 16 code bytes (packed_uint4) + scale (5 regs/row instead offloat q[32]+float4 sc= 36), loadsxs[32]once per token for all NR rows, and decodes the half2 table lookups inside the combine in exactlyunit_combine<32>'s four FMA chains (s0 = even j, s1 = odd j, s2/s3 = the +16 half), thend*(s0+s1) + d*(s2+s3) - 0.0f. Bitwise identical toDec<T_IQ4_XS>+unit_combine<32>. Gated at compile time:F == T_IQ4_XS && (LPR <= 16 || NTOK >= 4).quant.metal:iq4xs_scale(b, ui)factored out ofdeq<T_IQ4_XS>(same expression; used by both).gemv_parity.cpp:GEMV_AB_COLD=1for--ab gemv: the timed dispatches walk up to 64 copies of W (≤512 MB), so the paired A/B measures weights coming from DRAM like a real decode, not hot in SLC.
what doesn't ship (measured, not a win) #
#pragma clang loop unroll(full)on the IQ4deqloops: no-op (compiler already unrolls). ratios 0.98-1.02 = noise.- IQ4_NL packed path (18-byte blocks are only 2-aligned → ushort/rd32 reads): 0.999-1.0016 everywhere. excluded.
- IQ4_XS packed path at LPR 32, NTOK < 4 (the production 32,2,4,1 decode tile): cold +0.4..+0.7% on 2560->6144/10240, hot -5% on 6144<->2560 (SLC-resident) — the hot win disappears once weights come from DRAM, so it stays on the old path.
- reading the codes as ushorts/rd32 instead of one
packed_uint4: +11..+16% slower. wide load is essential. - hoisting the lookups into
half2 h[NR][16]for NTOK ≥ 2: same as re-looking-up per token (±0.5%); kept the simpler one. - runtime-uniform gate on
a.n_in < 6144to dodge the 6144->2560 3-token loss: both paths in one kernel made it worse (6144 nt3 still +1.6%, 2560->640 nt3 gain gone). compile-time only. - Target 2 (T_Q2_0R NTOK ≥ 2 pk path): bitwise, but 5-rep paired: 8,1,4,1 +0.1/+0.3% (nt2/4), 32,2,4,1 -1.7/+1.0%,
16,1,4,1 -0.1/+0.3%. not a net win → dropped. also note:
window.cppnever dispatchesgemv_142_*(onlytoken_chain/pair_paritytests do; the routed down usesmoe_down_142), so there's no production consumer anyway. gemv_direct_xfor IQ4: superseded by the packed path (which reads xs once for both rows; direct-x re-reads x per row).
measurements (M5 Air, paired A/B, median of per-pair ratios, 3 runs each; live server on :8080 sharing the GPU) #
new/base ratio; cold = GEMV_AB_COLD=1 (realistic), hot = default --ab (same W every dispatch). production tiles from
Window::gemv_tile (m5). all rows: memcmp == 0.
| type | shape | ntok | tile | cold µs base -> new | cold | hot | parity |
|---|---|---|---|---|---|---|---|
| IQ4_XS | 2560->10240 | 1 | 32,2,4,1 | 110.36 -> 110.88 | -0.13% | +0.08% | bitwise |
| IQ4_XS | 2560->10240 | 2 | 16,2,4,1 | 112.33 -> 112.8 | -0.05% | -0.07% | bitwise |
| IQ4_XS | 2560->10240 | 4 | 16,2,4,1 | 138.3 -> 136.6 | -1.29% | -1.51% | bitwise |
| IQ4_XS | 6144->2560 | 1 | 32,2,4,1 | 70.86 -> 70.39 | +0.36% | +0.20% | bitwise |
| IQ4_XS | 6144->2560 | 2 | 16,2,4,1 | 71.8 -> 72.25 | +0.32% | -1.15% | bitwise |
| IQ4_XS | 6144->2560 | 4 | 32,2,4,1 | 106.22 -> 105.8 | -0.37% | +0.60% | bitwise |
| IQ4_XS | 2560->6144 | 1 | 32,2,4,1 | 67.9 -> 68.58 | -0.17% | +0.06% | bitwise |
| IQ4_XS | 2560->6144 | 2 | 16,2,4,1 | 69.08 -> 69.38 | +0.13% | -1.39% | bitwise |
| IQ4_XS | 2560->6144 | 4 | 16,2,4,1 | 85.73 -> 84.81 | -1.12% | -1.24% | bitwise |
| IQ4_XS | 2560->12288 | 1 | 32,2,4,1 | 133.17 -> 133.21 | -0.06% | -0.16% | bitwise |
| IQ4_XS | 2560->12288 | 2 | 16,2,4,1 | 135.73 -> 136.04 | -0.02% | +0.37% | bitwise |
| IQ4_XS | 2560->12288 | 4 | 16,2,4,1 | 165.81 -> 163.39 | -1.43% | -1.40% | bitwise |
| IQ4_XS | 2560->640 | 1 | 16,2,4,1 | 11.9 -> 11.88 | -0.42% | -3.61% | bitwise |
| IQ4_XS | 2560->640 | 2 | 16,2,4,1 | 26.12 -> 25.88 | -2.44% | -1.82% | bitwise |
| IQ4_XS | 2560->640 | 4 | 16,2,4,1 | 35.08 -> 34.15 | -2.86% | -2.16% | bitwise |
| IQ4_NL | 640->2560 | 1 | 8,1,4,1 | 14.81 -> 14.6 | +0.05% | +0.02% | bitwise |
| IQ4_NL | 640->2560 | 2 | 4,2,4,1 | 34.31 -> 34.33 | -0.04% | -0.10% | bitwise |
| IQ4_NL | 640->2560 | 4 | 4,2,4,1 | 45.91 -> 45.82 | -0.08% | +0.01% | bitwise |
extra 5-rep cold runs: 2560->640 nt1 -5.0% (ratios 0.946-0.952; at a faster clock state the same shape gave -1.5%, the gain is clock-state dependent but never negative), 2560->640 nt3 -2.8%, 2560->10240 nt3 -2.0%, 2560->6144 nt3 -1.6%, 6144->2560 nt3 (16,2,4,1) +1.4% (the one consistent regression; nt2 +0.3%, nt4 uses 32,2,4,1 → -0.4%).
rough per-step accounting with the census counts (cold, µs): ntok 1: only shexp gate+up (57 × 2560->640) changes, ≈ -0.4..-1.5 µs each → ≈ -25..-85 µs/token. ntok 2: ≈ -26 µs. ntok 3: ≈ -130 µs (qkv/gate/shexp wins outweigh ssm_out/attn_out +1.4%). ntok 4: everything 16,2,4,1/32,2,4,1 -1..-3%. small but net-positive at every ntok; the big DRAM-bound shapes at ntok 1 (qkv, q, LM head 2560->248320) are at ~125-135 GB/s and nothing here moves them.
verification #
gemv_parity --abparity-only: IQ4_XS 2560->1000 (ragged n_out) × all 19 valid kCfgs tiles × ntok 1,2,3,4,5,8: all bitwise.- controls (unchanged code paths) bitwise: Q8_0 (32,1,4,4 / 32,2,4,1), IQ3_S (32,2,4,1 / 8,1,4,1), Q6_K (16,1,4,1 / 32,2,4,1), Q2_0R (8,1,4,1 nt 1/3), IQ4_NL nt3, IQ4_XS nt8, LM head 2560->248320.
- full
gemv_parity(every format × n_in 2560/640/10240 × ntok 1/3/8 × 20 tiles, vs double + window == one-token bitwise): ALL PASS (exit 0, 49 s — longer than the 10 s guideline; it's the whole suite, run once).
side observation (not my file, for whoever owns window.cpp) #
timings swing 2-3× between runs (e.g. 2560->640 base 11.9 vs 30.9 µs, IQ4_NL 640->2560 14.8 vs 34 µs) — GPU clock state / the live server. paired ratios were stable (±0.5%) inside a run, so all conclusions above are from ratios, not absolutes.