# opus r26 track A: IQ4_XS / IQ4_NL packed-code GEMV + Q2_0R multi-token

patch: `/tmp/opus_r26_gemv/opus_r26_gemv.patch` (`git -C vendor/strata apply --check` passes against the current working tree)
scratch: `/tmp/opus_r26_gemv/strata` (+ `build/gemv_parity`, variant kernel dirs `v_*`, base snapshot `base_kernels`)

## what ships

1. `gemv.metal`: an IQ4_XS path in `gemv` that keeps each row's unit as its 16 code bytes (`packed_uint4`) + scale
   (5 regs/row instead of `float q[32]` + `float4 sc` = 36), loads `xs[32]` once per token for all NR rows, and decodes
   the half2 table lookups inside the combine in **exactly** `unit_combine<32>`'s four FMA chains (s0 = even j, s1 = odd j,
   s2/s3 = the +16 half), then `d*(s0+s1) + d*(s2+s3) - 0.0f`. Bitwise identical to `Dec<T_IQ4_XS>` + `unit_combine<32>`.
   Gated at compile time: `F == T_IQ4_XS && (LPR <= 16 || NTOK >= 4)`.
2. `quant.metal`: `iq4xs_scale(b, ui)` factored out of `deq<T_IQ4_XS>` (same expression; used by both).
3. `gemv_parity.cpp`: `GEMV_AB_COLD=1` for `--ab gemv`: the timed dispatches walk up to 64 copies of W (≤512 MB), so
   the paired A/B measures weights coming from DRAM like a real decode, not hot in SLC.

## what doesn't ship (measured, not a win)

- `#pragma clang loop unroll(full)` on the IQ4 `deq` loops: no-op (compiler already unrolls). ratios 0.98-1.02 = noise.
- IQ4_NL packed path (18-byte blocks are only 2-aligned → ushort/rd32 reads): 0.999-1.0016 everywhere. excluded.
- IQ4_XS packed path at LPR 32, NTOK < 4 (the production 32,2,4,1 decode tile): cold +0.4..+0.7% on 2560->6144/10240,
  hot -5% on 6144<->2560 (SLC-resident) — the hot win disappears once weights come from DRAM, so it stays on the old path.
- reading the codes as ushorts/rd32 instead of one `packed_uint4`: **+11..+16% slower**. wide load is essential.
- hoisting the lookups into `half2 h[NR][16]` for NTOK ≥ 2: same as re-looking-up per token (±0.5%); kept the simpler one.
- runtime-uniform gate on `a.n_in < 6144` to dodge the 6144->2560 3-token loss: both paths in one kernel made it worse
  (6144 nt3 still +1.6%, 2560->640 nt3 gain gone). compile-time only.
- **Target 2 (T_Q2_0R NTOK ≥ 2 pk path)**: bitwise, but 5-rep paired: 8,1,4,1 +0.1/+0.3% (nt2/4), 32,2,4,1 -1.7/+1.0%,
  16,1,4,1 -0.1/+0.3%. not a net win → dropped. also note: `window.cpp` never dispatches `gemv_142_*` (only
  `token_chain`/`pair_parity` tests do; the routed down uses `moe_down_142`), so there's no production consumer anyway.
- `gemv_direct_x` for IQ4: superseded by the packed path (which reads xs once for both rows; direct-x re-reads x per row).

## measurements (M5 Air, paired A/B, median of per-pair ratios, 3 runs each; live server on :8080 sharing the GPU)

new/base ratio; cold = `GEMV_AB_COLD=1` (realistic), hot = default `--ab` (same W every dispatch). production tiles from
`Window::gemv_tile` (m5). all rows: `memcmp == 0`.

| type | shape | ntok | tile | cold µs base -> new | cold | hot | parity |
|---|---|---|---|---|---|---|---|
| IQ4_XS | 2560->10240 | 1 | 32,2,4,1 | 110.36 -> 110.88 | -0.13% | +0.08% | bitwise |
| IQ4_XS | 2560->10240 | 2 | 16,2,4,1 | 112.33 -> 112.8 | -0.05% | -0.07% | bitwise |
| IQ4_XS | 2560->10240 | 4 | 16,2,4,1 | 138.3 -> 136.6 | -1.29% | -1.51% | bitwise |
| IQ4_XS | 6144->2560 | 1 | 32,2,4,1 | 70.86 -> 70.39 | +0.36% | +0.20% | bitwise |
| IQ4_XS | 6144->2560 | 2 | 16,2,4,1 | 71.8 -> 72.25 | +0.32% | -1.15% | bitwise |
| IQ4_XS | 6144->2560 | 4 | 32,2,4,1 | 106.22 -> 105.8 | -0.37% | +0.60% | bitwise |
| IQ4_XS | 2560->6144 | 1 | 32,2,4,1 | 67.9 -> 68.58 | -0.17% | +0.06% | bitwise |
| IQ4_XS | 2560->6144 | 2 | 16,2,4,1 | 69.08 -> 69.38 | +0.13% | -1.39% | bitwise |
| IQ4_XS | 2560->6144 | 4 | 16,2,4,1 | 85.73 -> 84.81 | -1.12% | -1.24% | bitwise |
| IQ4_XS | 2560->12288 | 1 | 32,2,4,1 | 133.17 -> 133.21 | -0.06% | -0.16% | bitwise |
| IQ4_XS | 2560->12288 | 2 | 16,2,4,1 | 135.73 -> 136.04 | -0.02% | +0.37% | bitwise |
| IQ4_XS | 2560->12288 | 4 | 16,2,4,1 | 165.81 -> 163.39 | -1.43% | -1.40% | bitwise |
| IQ4_XS | 2560->640 | 1 | 16,2,4,1 | 11.9 -> 11.88 | -0.42% | -3.61% | bitwise |
| IQ4_XS | 2560->640 | 2 | 16,2,4,1 | 26.12 -> 25.88 | -2.44% | -1.82% | bitwise |
| IQ4_XS | 2560->640 | 4 | 16,2,4,1 | 35.08 -> 34.15 | -2.86% | -2.16% | bitwise |
| IQ4_NL | 640->2560 | 1 | 8,1,4,1 | 14.81 -> 14.6 | +0.05% | +0.02% | bitwise |
| IQ4_NL | 640->2560 | 2 | 4,2,4,1 | 34.31 -> 34.33 | -0.04% | -0.10% | bitwise |
| IQ4_NL | 640->2560 | 4 | 4,2,4,1 | 45.91 -> 45.82 | -0.08% | +0.01% | bitwise |

extra 5-rep cold runs: 2560->640 nt1 -5.0% (ratios 0.946-0.952; at a faster clock state the same shape gave -1.5%, the
gain is clock-state dependent but never negative), 2560->640 nt3 -2.8%, 2560->10240 nt3 -2.0%, 2560->6144 nt3 -1.6%,
**6144->2560 nt3 (16,2,4,1) +1.4%** (the one consistent regression; nt2 +0.3%, nt4 uses 32,2,4,1 → -0.4%).

rough per-step accounting with the census counts (cold, µs): ntok 1: only shexp gate+up (57 × 2560->640) changes,
≈ -0.4..-1.5 µs each → ≈ -25..-85 µs/token. ntok 2: ≈ -26 µs. ntok 3: ≈ -130 µs (qkv/gate/shexp wins outweigh
ssm_out/attn_out +1.4%). ntok 4: everything 16,2,4,1/32,2,4,1 -1..-3%. small but net-positive at every ntok;
the big DRAM-bound shapes at ntok 1 (qkv, q, LM head 2560->248320) are at ~125-135 GB/s and nothing here moves them.

## verification

- `gemv_parity --ab` parity-only: IQ4_XS 2560->1000 (ragged n_out) × all 19 valid kCfgs tiles × ntok 1,2,3,4,5,8: all bitwise.
- controls (unchanged code paths) bitwise: Q8_0 (32,1,4,4 / 32,2,4,1), IQ3_S (32,2,4,1 / 8,1,4,1), Q6_K (16,1,4,1 / 32,2,4,1),
  Q2_0R (8,1,4,1 nt 1/3), IQ4_NL nt3, IQ4_XS nt8, LM head 2560->248320.
- full `gemv_parity` (every format × n_in 2560/640/10240 × ntok 1/3/8 × 20 tiles, vs double + window == one-token bitwise):
  **ALL PASS** (exit 0, 49 s — longer than the 10 s guideline; it's the whole suite, run once).

## side observation (not my file, for whoever owns window.cpp)

timings swing 2-3× between runs (e.g. 2560->640 base 11.9 vs 30.9 µs, IQ4_NL 640->2560 14.8 vs 34 µs) — GPU clock state /
the live server. paired ratios were stable (±0.5%) inside a run, so all conclusions above are from ratios, not absolutes.