24 kB 287 lines
5dc99052
Round 27 blitz: gdn_step_t 5->4 barriers + dt/ssm_a hoist + red2[IB-1] (-3.0..-7.9%), qsa_merge2/qsa_merge float2(m,l) + simd_max (-8..-20%), gemv_tile n_out=512 32,2,4,1 (-3.2..-25.5%) + Q2_0 640->2560 8,2,4,1
dawn
⋯
audits/opus_r27_gemv.md
sourcerendered
download raw history
at main Markdown

Round 27 Track A — M5 gemv_tile audit (2560→512, 640→2560) + gemv_cat for IQ4_XS / IQ3_S / Q6_K #

author: opus-r27-gemv-tiles-and-cat · device: Apple M5 (10-core GPU) · live :8080 server was running during all measurements (it shares the GPU), so every number is the min over 3 runs of a 15-round paired median; runs that hit interference were visibly off by 3-5x and the min filters them.

deliverables

  • /tmp/opus_r27_gemv/opus_r27_gemv.patch — the change to apply (git -C vendor/strata apply --check OK against the current dirty tree): gemv_tile tiles + gemv_parity tooling/invariant. gemv.metal is not touched by it.
  • /tmp/opus_r27_gemv/opus_r27_gemv_cat_experiment.patch — NOT recommended; the gemv_cat IQ4_XS packed path + GVC instances for IQ4_XS/IQ3_S/Q6_K, kept only so the cat A/B below can be reproduced (applies on top of the main patch).
  • scratch tree /tmp/opus_r27_gemv/strata, build /tmp/opus_r27_gemv/build (gemv_parity binary there).

TL;DR #

  1. Bug confirmed: every 2560 -> 512 QSA attn_k/attn_v (15 IQ4_XS, 8 Q6_K, 1 IQ3_S) fell through to {32,1,4,1} because the M5 branches tested n_out == 640. Fixed:
    • IQ4_XS 512 -> {16,2,4,1} all ntok (also picks up R26's packed-IQ4 path): cold -14% (nt1) / -25% / -31% / -36% (nt4)
    • Q6_K 512 -> {8,1,4,1} all ntok: cold 0% / -14% / -23% / -23%
    • IQ3_S 512 -> {32,2,4,1} all ntok: cold -1% / -14% / -21% / -20%
    • LPR is constant across ntok for all three, so a window stays bitwise the one-token answer for these tensors.
    • inside the real group (k + v beside a 2560→12288 IQ4_XS attn_q, as Window::layer encodes them with no barrier) the new tiles are still the best or within 1% of best: IQ4_XS group -3..-8%, Q6_K -1..-5%, IQ3_S -1..-5%.
  2. 640 -> 2560 (ffn_down_shexp): the current table is already right for IQ4_NL and Q8_0 (cold optimum matches exactly). Only change: Q2_0 at ntok>=4 {8,1,4,1} -> {8,2,4,1} (-4..-6% cold, same LPR => bitwise identical sums). The R26 claim "Q2_0R {8,1,4,1} 39.55 us at ntok 4 vs {16,1,4,1} 18.19" does not reproduce: {8,1,4,1} is the best Q2_0R tile at ntok 2..4 here (14.84 us cold at nt4; {16,1,4,1} 17.17, {4,2,4,1} 16.83). The 39.55 was almost certainly GPU interference (I saw identical 30-40 us outliers on unrelated shapes while the server was busy).
  3. 2560 -> 640 IQ3_S (gate/up shexp, 23 tensors) at ntok 2..4: {8,1,8,1} -> {32,2,4,1} (-1..-8% cold), which also makes its LPR constant across ntok (it was 32 at nt1, 8 at nt>=2).
  4. gemv_cat for IQ4_XS/IQ3_S/Q6_K: NOT a win -> not wired. It is bitwise (verified, nt 1/2/3/4/8, 512+512 and 640+640), but Window::layer already encodes k/v and gate/up as concurrent dispatches (nb_ = true), and these GEMVs are DRAM-bound, so fusing saves no bytes and only a dispatch: IQ4_XS -2..+8% (worse at nt>=3), Q6_K -4..+15% (worse at nt>=3 and on 640), IQ3_S -6..+6% (noise-level, median slightly negative). Same finding for the existing Q8_0 cat (hot 2560→512+512 nt2: +2.4%).
  5. The LPR insight is verified and is now a permanent check in gemv_parity: for every format, n_in, ntok, all KS=1 tiles with the same LPR give bitwise-identical outputs (across NR and SG). gemv_parity: ALL PASS (13 formats × 3 n_in × ntok 1/3/8, rel L2 < 1e-5, window == one token bitwise, same-LPR bitwise).

Estimated decode effect: per token (nt=1) ~18 µs saved over 15 IQ4_XS k/v (isolated cold), per 4-token verify window ~135 µs (IQ4_XS 95 + Q6_K 36 + IQ3_S 3) plus ~1 µs × 23 IQ3_S gate/up at nt 2..3. Against ~47 ms/tok that is 0.04-0.3% — real but small; it is NOT measured end-to-end (full-model runs were off-limits).

Shapes in k512-iq2_xs-metal.gguf (parsed from the GGUF header) #

tensor shape types
attn_k (12) 2560→512 IQ4_XS 8, Q6_K 3, IQ3_S 1
attn_v (12) 2560→512 IQ4_XS 7, Q6_K 5
ffn_gate_shexp (48) 2560→640 IQ4_XS 31, IQ3_S 9, Q6_K 8
ffn_up_shexp (48) 2560→640 IQ4_XS 26, IQ3_S 14, Q6_K 8
ffn_down_shexp (48) 640→2560 IQ4_NL 36, Q8_0 7, Q2_0 5 (prepared x, kernel gemv_42_*, not Q2_0R)

same-type pairs: k+v 7/12 (IQ4_XS 5, Q6_K 2); gate+up 23/48 (IQ4_XS 18, IQ3_S 3, Q6_K 2).

Target 1a: 2560 -> 512, isolated (µs per dispatch, behind a barrier) #

gemv_parity --tiles TYPE 2560 512 NT <tiles...> (GEMV_AB_COLD=1 walks 256 copies of W ≈ 180 MB, so W comes from DRAM)

type weights ntok 32,1,4,1 16,2,4,1 32,2,4,1 16,1,4,1 8,1,4,1 8,2,4,1 8,1,8,1 4,2,4,1 best
IQ4_XS cold 1 8.67 7.45 8.36 7.46 7.61 8.13 7.62 12.54 16,2,4,1
IQ4_XS cold 2 11.36 8.57 9.78 9.47 9.50 9.50 9.61 14.72 16,2,4,1
IQ4_XS cold 3 14.05 9.71 11.36 10.69 11.10 11.31 11.24 18.36 16,2,4,1
IQ4_XS cold 4 17.57 11.22 13.08 13.16 13.08 13.20 13.20 22.00 16,2,4,1
IQ4_XS hot 1 8.21 6.58 7.21 6.82 7.06 6.32 7.04 7.68 8,2,4,1
IQ4_XS hot 2 14.04 10.10 11.29 11.03 11.12 10.03 11.12 14.72 8,2,4,1
IQ4_XS hot 3 14.21 9.46 10.74 10.55 10.66 10.19 10.65 16.32 16,2,4,1
IQ4_XS hot 4 17.01 10.74 12.24 12.39 12.34 11.75 12.39 19.98 16,2,4,1
Q6_K cold 1 10.26 11.09 10.58 10.45 10.24 11.24 10.24 17.85 8,1,8,1
Q6_K cold 2 12.43 11.61 12.14 10.98 10.64 13.72 10.74 22.93 8,1,4,1
Q6_K cold 3 15.60 13.43 15.26 12.45 12.04 18.08 12.18 31.62 8,1,4,1
Q6_K cold 4 19.07 15.64 17.63 14.79 14.63 21.81 14.75 40.22 8,1,4,1
Q6_K hot 1 7.12 6.57 6.78 6.46 6.87 8.21 6.87 14.54 16,1,4,1
Q6_K hot 2 10.26 8.08 9.24 8.68 8.80 11.12 8.99 19.99 16,2,4,1
Q6_K hot 3 13.63 10.40 12.09 11.07 10.96 15.54 10.99 28.37 16,2,4,1
Q6_K hot 4 17.15 12.89 14.77 13.56 13.52 19.30 13.68 36.18 16,2,4,1
IQ3_S cold 1 7.98 8.54 7.90 8.03 9.70 11.10 9.62 16.18 32,2,4,1
IQ3_S cold 2 10.64 9.49 9.14 9.75 10.59 12.36 10.52 18.50 32,2,4,1
IQ3_S cold 3 13.45 11.06 10.57 11.79 11.86 14.47 11.78 22.20 32,2,4,1
IQ3_S cold 4 16.06 12.82 12.78 13.34 13.65 16.89 13.71 28.33 32,2,4,1
IQ3_S hot 1 7.48 7.30 6.81 7.58 7.93 8.22 7.83 12.92 32,2,4,1
IQ3_S hot 2 11.14 9.49 9.22 10.04 10.24 11.38 10.39 17.50 32,2,4,1
IQ3_S hot 3 13.12 10.22 9.92 10.79 11.01 11.40 10.96 19.34 32,2,4,1
IQ3_S hot 4 17.06 13.86 13.09 13.53 14.14 16.73 14.11 28.43 32,2,4,1

Target 1a': 2560 -> 512 inside the real group (k + v beside attn_q IQ4_XS 2560→12288, one barrier) #

GEMV_BESIDE=IQ4_XS,12288,<q tile> GEMV_AB_COLD=1 gemv_parity --tiles ... (µs per group; q tile = gemv_tile's: 32,2,4,1 nt1, 16,2,4,1 nt>=2)

type weights ntok 32,1,4,1 16,2,4,1 32,2,4,1 8,1,4,1 best
IQ4_XS cold 1 184.57 178.86 180.51 180.03 16,2,4,1
IQ4_XS cold 2 200.33 186.88 191.96 190.08 16,2,4,1
IQ4_XS cold 3 241.82 223.59 229.72 227.78 16,2,4,1
IQ4_XS cold 4 298.04 274.62 281.90 279.07 16,2,4,1
Q6_K cold 1 228.76 227.32 223.86 225.61 32,2,4,1
Q6_K cold 2 258.73 253.91 254.12 254.85 16,2,4,1
Q6_K cold 3 335.32 320.63 322.67 320.89 16,2,4,1
Q6_K cold 4 400.36 381.26 386.35 381.40 16,2,4,1
IQ3_S cold 1 259.87 259.62 256.99 257.80 32,2,4,1
IQ3_S cold 2 285.11 275.59 276.66 278.94 16,2,4,1
IQ3_S cold 3 335.79 319.72 320.34 325.83 16,2,4,1
IQ3_S cold 4 380.44 359.75 362.64 365.00 16,2,4,1

Target 1b: 640 -> 2560 (µs per dispatch) #

current table: IQ4_NL nt1 8,1,4,1 / nt>=2 4,2,4,1; Q8_0 nt1-2 32,1,4,1 / nt>=3 4,1,4,1; Q2_0 8,1,4,1.

type weights ntok 8,1,4,1 4,2,4,1 8,2,4,1 16,1,4,1 16,2,4,1 32,1,4,1 4,1,4,1 4,4,4,1 2,2,4,1 best
Q2_0 cold 1 6.42 6.56 6.24 5.93 6.07 7.42 6.42 7.58 8.90 16,1,4,1
Q2_0 cold 2 7.30 7.62 7.46 7.99 7.64 10.69 7.69 10.31 9.92 8,1,4,1
Q2_0 cold 3 8.79 9.31 9.17 10.36 11.38 14.42 9.52 12.79 11.74 8,1,4,1
Q2_0 cold 4 12.15 11.60 11.47 13.52 13.91 18.45 12.01 15.03 14.76 8,2,4,1
Q2_0 hot 1 4.60 4.50 4.36 5.44 4.91 7.15 5.08 5.06 5.84 8,2,4,1
Q2_0 hot 2 9.02 8.94 8.72 10.99 10.04 15.32 9.23 12.37 10.19 8,2,4,1
Q2_0 hot 3 9.10 9.17 9.05 11.19 10.18 16.33 9.43 12.54 10.25 8,2,4,1
Q2_0 hot 4 10.39 10.58 10.41 12.66 12.17 18.29 10.62 13.06 13.00 8,1,4,1
IQ4_NL cold 1 9.81 10.32 10.47 10.51 10.73 11.57 10.23 10.70 10.64 8,1,4,1
IQ4_NL cold 2 11.01 10.68 11.35 13.03 12.11 15.31 10.65 12.02 11.47 4,1,4,1
IQ4_NL cold 3 15.63 13.09 14.93 19.77 16.86 23.95 14.45 18.66 14.89 4,2,4,1
IQ4_NL cold 4 18.36 14.97 17.07 23.50 19.96 28.95 16.69 21.71 17.85 4,2,4,1
IQ4_NL hot 1 8.13 6.85 7.46 10.10 8.59 11.85 7.35 6.96 7.02 4,2,4,1
IQ4_NL hot 2 11.02 8.41 9.13 13.82 10.91 16.81 9.42 9.16 8.98 4,2,4,1
IQ4_NL hot 3 12.68 9.51 10.65 16.52 13.04 20.70 11.07 12.33 11.02 4,2,4,1
IQ4_NL hot 4 15.57 12.18 13.43 20.78 16.64 26.63 13.79 16.13 14.21 4,2,4,1
Q8_0 cold 1 16.71 17.18 16.74 15.56 16.43 14.68 16.57 17.94 18.76 32,1,4,1
Q8_0 cold 2 16.88 17.48 17.44 17.95 18.45 16.57 16.95 20.47 19.14 32,1,4,1
Q8_0 cold 3 18.86 19.86 20.33 23.39 23.45 21.17 18.03 30.34 22.56 4,1,4,1
Q8_0 cold 4 22.27 20.71 24.12 26.74 29.20 28.05 19.75 35.07 24.07 4,1,4,1
Q8_0 hot 1 7.66 6.80 7.35 8.89 8.62 9.92 7.14 8.37 7.28 4,2,4,1
Q8_0 hot 2 9.26 7.61 9.21 11.43 11.08 13.60 8.16 10.68 9.65 4,2,4,1
Q8_0 hot 3 12.80 10.74 12.58 16.86 16.01 19.23 11.44 17.02 12.83 4,2,4,1
Q8_0 hot 4 16.06 13.25 15.52 20.87 19.81 24.71 13.97 20.68 15.39 4,2,4,1
Q2_0R cold 1 8.55 9.03 8.67 8.40 8.87 10.15 8.69 10.33 11.53 16,1,4,1
Q2_0R cold 2 10.35 11.12 11.18 11.30 12.02 14.80 10.88 14.81 13.77 8,1,4,1
Q2_0R cold 3 12.52 13.74 12.92 13.51 16.88 18.73 12.52 17.48 16.19 8,1,4,1
Q2_0R cold 4 14.84 16.83 16.24 17.17 20.23 24.87 15.30 21.32 20.14 8,1,4,1
Q2_0R hot 1 5.77 6.33 5.86 6.49 6.14 7.68 6.10 6.81 6.80 8,1,4,1
Q2_0R hot 2 8.26 9.10 8.66 9.98 9.88 13.61 8.51 10.95 9.31 8,1,4,1
Q2_0R hot 3 10.88 11.69 11.35 13.68 13.32 18.94 11.57 14.91 12.59 8,1,4,1
Q2_0R hot 4 12.45 13.70 13.30 15.43 15.52 22.76 12.89 17.02 15.22 8,1,4,1

Q2_0 re-run with 5 repeats: nt1 8,1 6.33 / 8,2 6.15 / 16,1 5.95; nt4 8,1 14.21 / 8,2 13.68 / 16,1 15.97. nt1 16,1,4,1 is -6% cold but +18% hot and changes LPR across ntok; left alone.

2560 -> 640 check (cold, µs) — the shexp gate/up tiles (cat would use them) #

type weights ntok 32,1,4,1 16,2,4,1 32,2,4,1 16,1,4,1 8,1,4,1 8,1,8,1 best
IQ4_XS cold 1 10.40 8.49 9.65 8.75 8.43 8.41 8,1,8,1
IQ4_XS cold 2 15.32 10.34 12.12 12.36 11.30 10.90 16,2,4,1
IQ4_XS cold 3 19.39 11.44 14.82 14.02 13.09 13.09 16,2,4,1
IQ4_XS cold 4 24.94 13.64 17.42 17.63 15.97 15.85 16,2,4,1
Q6_K cold 1 12.43 12.89 12.68 12.47 12.63 12.60 32,1,4,1
Q6_K cold 2 17.32 15.06 15.95 14.47 13.81 13.80 8,1,8,1
Q6_K cold 3 22.77 18.00 22.91 16.89 15.54 15.52 8,1,8,1
Q6_K cold 4 28.06 21.12 26.50 20.43 19.12 19.15 8,1,4,1
IQ3_S cold 1 10.86 10.36 10.32 10.73 12.07 11.76 32,2,4,1
IQ3_S cold 2 15.59 12.38 12.51 14.04 13.65 13.52 16,2,4,1
IQ3_S cold 3 20.06 14.32 14.40 15.82 15.27 15.12 16,2,4,1
IQ3_S cold 4 24.20 16.99 17.47 18.29 17.74 17.54 16,2,4,1

IQ3_S re-run (5 repeats, cold): nt2 8,1,8,1 10.98 / 32,2,4,1 10.19 / 16,2,4,1 10.05; nt3 12.90 / 11.83 / 12.19; nt4 14.50 / 14.34 / 13.98. 16,2 and 32,2 tie; 32,2 chosen so LPR matches nt1.

Target 2: gemv_cat vs two concurrent gemvs (gemv_parity --cat, experiment patch applied) #

Each pair behind one barrier, the two gemvs with no barrier between (as Window::layer), 2 runs each:

    cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 cold: 2 gemv 13.737 us, gemv_cat 13.753 us (+0.1%)
    cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 cold: 2 gemv 13.819 us, gemv_cat 13.852 us (+0.2%)
    cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 cold: 2 gemv 14.479 us, gemv_cat 14.354 us (-0.9%)
    cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 cold: 2 gemv 14.475 us, gemv_cat 14.383 us (-0.6%)
    cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 cold: 2 gemv 17.481 us, gemv_cat 18.855 us (+7.9%)
    cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 cold: 2 gemv 18.378 us, gemv_cat 18.888 us (+2.8%)
    cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 cold: 2 gemv 21.094 us, gemv_cat 21.678 us (+2.8%)
    cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 cold: 2 gemv 20.690 us, gemv_cat 21.110 us (+2.0%)
    cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 hot: 2 gemv 10.519 us, gemv_cat 10.541 us (+0.2%)
    cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 hot: 2 gemv 10.540 us, gemv_cat 10.569 us (+0.3%)
    cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 hot: 2 gemv 12.803 us, gemv_cat 12.771 us (-0.3%)
    cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 hot: 2 gemv 12.800 us, gemv_cat 12.769 us (-0.2%)
    cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 hot: 2 gemv 14.901 us, gemv_cat 15.067 us (+1.1%)
    cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 hot: 2 gemv 14.943 us, gemv_cat 15.125 us (+1.2%)
    cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 hot: 2 gemv 17.509 us, gemv_cat 17.592 us (+0.5%)
    cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 hot: 2 gemv 17.450 us, gemv_cat 17.559 us (+0.6%)
    cat Q6_K 2560 512+512 nt 1 tile 8,1,4 cold: 2 gemv 20.273 us, gemv_cat 20.993 us (+3.6%)
    cat Q6_K 2560 512+512 nt 1 tile 8,1,4 cold: 2 gemv 20.458 us, gemv_cat 21.054 us (+2.9%)
    cat Q6_K 2560 512+512 nt 2 tile 8,1,4 cold: 2 gemv 21.199 us, gemv_cat 21.048 us (-0.7%)
    cat Q6_K 2560 512+512 nt 2 tile 8,1,4 cold: 2 gemv 21.735 us, gemv_cat 21.322 us (-1.9%)
    cat Q6_K 2560 512+512 nt 3 tile 8,1,4 cold: 2 gemv 22.902 us, gemv_cat 23.149 us (+1.1%)
    cat Q6_K 2560 512+512 nt 3 tile 8,1,4 cold: 2 gemv 22.877 us, gemv_cat 23.111 us (+1.0%)
    cat Q6_K 2560 512+512 nt 4 tile 8,1,4 cold: 2 gemv 26.476 us, gemv_cat 28.002 us (+5.8%)
    cat Q6_K 2560 512+512 nt 4 tile 8,1,4 cold: 2 gemv 28.318 us, gemv_cat 30.941 us (+9.3%)
    cat Q6_K 2560 512+512 nt 1 tile 8,1,4 hot: 2 gemv 13.059 us, gemv_cat 12.609 us (-3.4%)
    cat Q6_K 2560 512+512 nt 1 tile 8,1,4 hot: 2 gemv 12.212 us, gemv_cat 11.709 us (-4.1%)
    cat Q6_K 2560 512+512 nt 2 tile 8,1,4 hot: 2 gemv 16.915 us, gemv_cat 16.627 us (-1.7%)
    cat Q6_K 2560 512+512 nt 2 tile 8,1,4 hot: 2 gemv 16.135 us, gemv_cat 16.060 us (-0.5%)
    cat Q6_K 2560 512+512 nt 3 tile 8,1,4 hot: 2 gemv 20.912 us, gemv_cat 20.629 us (-1.4%)
    cat Q6_K 2560 512+512 nt 3 tile 8,1,4 hot: 2 gemv 20.641 us, gemv_cat 20.426 us (-1.0%)
    cat Q6_K 2560 512+512 nt 4 tile 8,1,4 hot: 2 gemv 25.688 us, gemv_cat 25.883 us (+0.8%)
    cat Q6_K 2560 512+512 nt 4 tile 8,1,4 hot: 2 gemv 25.375 us, gemv_cat 25.923 us (+2.2%)
    cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 cold: 2 gemv 16.619 us, gemv_cat 16.692 us (+0.4%)
    cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 cold: 2 gemv 16.687 us, gemv_cat 16.696 us (+0.1%)
    cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 cold: 2 gemv 16.941 us, gemv_cat 16.938 us (-0.0%)
    cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 cold: 2 gemv 17.346 us, gemv_cat 17.026 us (-1.8%)
    cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 cold: 2 gemv 22.748 us, gemv_cat 23.210 us (+2.0%)
    cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 cold: 2 gemv 22.205 us, gemv_cat 23.013 us (+3.6%)
    cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 cold: 2 gemv 25.746 us, gemv_cat 26.951 us (+4.7%)
    cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 cold: 2 gemv 25.801 us, gemv_cat 26.759 us (+3.7%)
    cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 hot: 2 gemv 12.635 us, gemv_cat 12.580 us (-0.4%)
    cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 hot: 2 gemv 11.794 us, gemv_cat 11.828 us (+0.3%)
    cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 hot: 2 gemv 14.355 us, gemv_cat 14.294 us (-0.4%)
    cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 hot: 2 gemv 14.347 us, gemv_cat 14.337 us (-0.1%)
    cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 hot: 2 gemv 16.780 us, gemv_cat 17.007 us (+1.4%)
    cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 hot: 2 gemv 16.950 us, gemv_cat 17.069 us (+0.7%)
    cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 hot: 2 gemv 20.724 us, gemv_cat 20.625 us (-0.5%)
    cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 hot: 2 gemv 20.848 us, gemv_cat 21.028 us (+0.9%)
    cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 15.959 us, gemv_cat 16.974 us (+6.4%)
    cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 17.358 us, gemv_cat 16.771 us (-3.4%)
    cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 cold: 2 gemv 21.662 us, gemv_cat 21.334 us (-1.5%)
    cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 cold: 2 gemv 22.133 us, gemv_cat 22.124 us (-0.0%)
    cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 cold: 2 gemv 27.146 us, gemv_cat 26.946 us (-0.7%)
    cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 cold: 2 gemv 27.741 us, gemv_cat 27.671 us (-0.3%)
    cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 cold: 2 gemv 32.387 us, gemv_cat 32.224 us (-0.5%)
    cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 cold: 2 gemv 31.193 us, gemv_cat 31.570 us (+1.2%)
    cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 15.602 us, gemv_cat 15.247 us (-2.3%)
    cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 15.583 us, gemv_cat 14.706 us (-5.6%)
    cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 hot: 2 gemv 19.240 us, gemv_cat 18.755 us (-2.5%)
    cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 hot: 2 gemv 19.042 us, gemv_cat 18.566 us (-2.5%)
    cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 hot: 2 gemv 23.257 us, gemv_cat 22.716 us (-2.3%)
    cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 hot: 2 gemv 23.515 us, gemv_cat 23.379 us (-0.6%)
    cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 hot: 2 gemv 29.370 us, gemv_cat 28.880 us (-1.7%)
    cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 hot: 2 gemv 28.697 us, gemv_cat 28.610 us (-0.3%)
    cat Q6_K 2560 640+640 nt 2 tile 8,1,4 cold: 2 gemv 25.246 us, gemv_cat 26.583 us (+5.3%)
    cat Q6_K 2560 640+640 nt 2 tile 8,1,4 cold: 2 gemv 27.044 us, gemv_cat 28.421 us (+5.1%)
    cat Q6_K 2560 640+640 nt 3 tile 8,1,4 cold: 2 gemv 33.306 us, gemv_cat 33.523 us (+0.7%)
    cat Q6_K 2560 640+640 nt 3 tile 8,1,4 cold: 2 gemv 32.500 us, gemv_cat 33.620 us (+3.4%)
    cat Q6_K 2560 640+640 nt 4 tile 8,1,4 cold: 2 gemv 37.989 us, gemv_cat 40.487 us (+6.6%)
    cat Q6_K 2560 640+640 nt 4 tile 8,1,4 cold: 2 gemv 37.104 us, gemv_cat 40.660 us (+9.6%)
    cat Q6_K 2560 640+640 nt 2 tile 8,1,4 hot: 2 gemv 21.201 us, gemv_cat 24.492 us (+15.5%)
    cat Q6_K 2560 640+640 nt 2 tile 8,1,4 hot: 2 gemv 21.022 us, gemv_cat 23.201 us (+10.4%)
    cat Q6_K 2560 640+640 nt 3 tile 8,1,4 hot: 2 gemv 26.668 us, gemv_cat 27.057 us (+1.5%)
    cat Q6_K 2560 640+640 nt 3 tile 8,1,4 hot: 2 gemv 26.145 us, gemv_cat 26.466 us (+1.2%)
    cat Q6_K 2560 640+640 nt 4 tile 8,1,4 hot: 2 gemv 32.347 us, gemv_cat 33.714 us (+4.2%)
    cat Q6_K 2560 640+640 nt 4 tile 8,1,4 hot: 2 gemv 33.220 us, gemv_cat 34.377 us (+3.5%)
    cat Q6_K 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 23.505 us, gemv_cat 23.975 us (+2.0%)
    cat Q6_K 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 23.916 us, gemv_cat 24.354 us (+1.8%)
    cat Q6_K 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 19.528 us, gemv_cat 20.770 us (+6.4%)
    cat Q6_K 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 19.303 us, gemv_cat 19.804 us (+2.6%)

Why it cannot win: the two GEMVs already overlap on the GPU, x is 10 KB (L1/L2), and the cost is DRAM traffic for W (2×0.7–1.2 MB), which fusing does not reduce. At nt>=3 the cat kernel is slightly worse (extra row-select per row pointer, worse register allocation). Not wired in Window::gemv_cat / Window::layer. Note for whoever owns MTP: the same applies to the existing Q8_0 gm_cat in the draft step — worth re-checking with --cat Q8_0 2560 512 1 32,1,4 cold before keeping it.

Bitwise / ntok analysis (the "LPR decides the sums" claim) #

  • In gemv (KS=1) a row's sum is lane_sum<LPR> over lanes each accumulating units u = li, li+LPR, ... in order; NR only says how many rows a lane carries, SG how many simdgroups share a threadgroup — neither changes a row's order of operations. Format paths that depend on more than LPR (gemv_iq4_packed<LPR> with NTOK, gemv_direct_x with NR/NTOK for Q8_0) are written to produce the same FMA chains, and the new parity check confirms it for all 13 formats: same LPR ⇒ bitwise, for any NR/SG. KS>1 changes the order (partial sums), as expected.
  • Engine tests do not compare a window against single-token runs through Window::gemv_tile (the parity tests call the kernels directly; prompt_parity uses dense_mm), and the existing table already switches LPR across ntok (e.g. old IQ3_S 640, Q8_0, IQ4_XS n_in>=6144). The new 512 tiles and the IQ3_S 640 change remove LPR switches rather than add them; the Q2_0 nt4 change keeps LPR 8.

Verification #

  • gemv_parity (default, all formats/shapes): ALL PASS (includes the new same-LPR bitwise check). Note this full suite takes ~34 s (it always did; the CPU double reference dominates) — the only invocation above 10 s.
  • gemv_parity --cat parity: bitwise for IQ4_XS 16,2,4 / Q6_K 8,1,4 & 32,2,4 / IQ3_S 32,2,4 at 512+512 and 640+640, nt 1/2/3/4/8 (experiment patch) and for the existing Q8_0 instance (main patch).
  • window.cpp compiles (strata_metal_kernels object, scratch build). New tiles are all existing GV_SET instances.
  • Not done (forbidden here): full-model strata_window_cli / server A/B. Recommended next: one decode A/B with --spec 3 where most windows are 2..4 tokens, which is where the 512 fix pays most.

gemv_parity additions (main patch) #

  • --tiles TYPE NIN NOUT NTOK L,R,S,K ... paired multi-tile timing (cold with GEMV_AB_COLD=1, Q2_0R accepted), rel L2 per tile; GEMV_BESIDE=TYPE,NOUT,L,R,S,K times each tile as k+v beside a big GEMV.
  • --cat TYPE NIN NOUT NTOK L,R,S gemv_cat vs its two concurrent gemvs: bitwise + paired timing.
  • default parity: "same LPR: bitwise" invariant.