# Round 27 Track A — M5 `gemv_tile` audit (2560→512, 640→2560) + `gemv_cat` for IQ4_XS / IQ3_S / Q6_K author: `opus-r27-gemv-tiles-and-cat` · device: Apple M5 (10-core GPU) · live `:8080` server was running during all measurements (it shares the GPU), so every number is the **min over 3 runs of a 15-round paired median**; runs that hit interference were visibly off by 3-5x and the min filters them. deliverables - `/tmp/opus_r27_gemv/opus_r27_gemv.patch` — the change to apply (`git -C vendor/strata apply --check` OK against the current dirty tree): `gemv_tile` tiles + `gemv_parity` tooling/invariant. `gemv.metal` is **not** touched by it. - `/tmp/opus_r27_gemv/opus_r27_gemv_cat_experiment.patch` — NOT recommended; the gemv_cat IQ4_XS packed path + GVC instances for IQ4_XS/IQ3_S/Q6_K, kept only so the cat A/B below can be reproduced (applies on top of the main patch). - scratch tree `/tmp/opus_r27_gemv/strata`, build `/tmp/opus_r27_gemv/build` (`gemv_parity` binary there). ## TL;DR 1. **Bug confirmed**: every `2560 -> 512` QSA `attn_k`/`attn_v` (15 IQ4_XS, 8 Q6_K, 1 IQ3_S) fell through to `{32,1,4,1}` because the M5 branches tested `n_out == 640`. Fixed: - IQ4_XS 512 -> `{16,2,4,1}` all ntok (also picks up R26's packed-IQ4 path): cold **-14% (nt1) / -25% / -31% / -36% (nt4)** - Q6_K 512 -> `{8,1,4,1}` all ntok: cold 0% / -14% / -23% / -23% - IQ3_S 512 -> `{32,2,4,1}` all ntok: cold -1% / -14% / -21% / -20% - LPR is constant across ntok for all three, so a window stays bitwise the one-token answer for these tensors. - inside the real group (k + v beside a 2560→12288 IQ4_XS `attn_q`, as `Window::layer` encodes them with no barrier) the new tiles are still the best or within 1% of best: IQ4_XS group -3..-8%, Q6_K -1..-5%, IQ3_S -1..-5%. 2. **640 -> 2560 (`ffn_down_shexp`)**: the current table is already right for IQ4_NL and Q8_0 (cold optimum matches exactly). Only change: Q2_0 at ntok>=4 `{8,1,4,1} -> {8,2,4,1}` (-4..-6% cold, same LPR => bitwise identical sums). **The R26 claim "Q2_0R {8,1,4,1} 39.55 us at ntok 4 vs {16,1,4,1} 18.19" does not reproduce**: {8,1,4,1} is the best Q2_0R tile at ntok 2..4 here (14.84 us cold at nt4; {16,1,4,1} 17.17, {4,2,4,1} 16.83). The 39.55 was almost certainly GPU interference (I saw identical 30-40 us outliers on unrelated shapes while the server was busy). 3. **2560 -> 640 IQ3_S** (gate/up shexp, 23 tensors) at ntok 2..4: `{8,1,8,1} -> {32,2,4,1}` (-1..-8% cold), which also makes its LPR constant across ntok (it was 32 at nt1, 8 at nt>=2). 4. **gemv_cat for IQ4_XS/IQ3_S/Q6_K: NOT a win -> not wired.** It is bitwise (verified, nt 1/2/3/4/8, 512+512 and 640+640), but `Window::layer` already encodes k/v and gate/up as concurrent dispatches (`nb_ = true`), and these GEMVs are DRAM-bound, so fusing saves no bytes and only a dispatch: IQ4_XS -2..+8% (worse at nt>=3), Q6_K -4..+15% (worse at nt>=3 and on 640), IQ3_S -6..+6% (noise-level, median slightly negative). Same finding for the existing Q8_0 cat (hot 2560→512+512 nt2: +2.4%). 5. **The LPR insight is verified** and is now a permanent check in `gemv_parity`: for every format, n_in, ntok, all KS=1 tiles with the same LPR give bitwise-identical outputs (across NR and SG). `gemv_parity`: **ALL PASS** (13 formats × 3 n_in × ntok 1/3/8, rel L2 < 1e-5, window == one token bitwise, same-LPR bitwise). Estimated decode effect: per token (nt=1) ~18 µs saved over 15 IQ4_XS k/v (isolated cold), per 4-token verify window ~135 µs (IQ4_XS 95 + Q6_K 36 + IQ3_S 3) plus ~1 µs × 23 IQ3_S gate/up at nt 2..3. Against ~47 ms/tok that is 0.04-0.3% — real but small; it is NOT measured end-to-end (full-model runs were off-limits). ## Shapes in `k512-iq2_xs-metal.gguf` (parsed from the GGUF header) | tensor | shape | types | |---|---|---| | attn_k (12) | 2560→512 | IQ4_XS 8, Q6_K 3, IQ3_S 1 | | attn_v (12) | 2560→512 | IQ4_XS 7, Q6_K 5 | | ffn_gate_shexp (48) | 2560→640 | IQ4_XS 31, IQ3_S 9, Q6_K 8 | | ffn_up_shexp (48) | 2560→640 | IQ4_XS 26, IQ3_S 14, Q6_K 8 | | ffn_down_shexp (48) | 640→2560 | IQ4_NL 36, Q8_0 7, Q2_0 5 (prepared x, kernel `gemv_42_*`, not Q2_0R) | same-type pairs: k+v 7/12 (IQ4_XS 5, Q6_K 2); gate+up 23/48 (IQ4_XS 18, IQ3_S 3, Q6_K 2). ## Target 1a: 2560 -> 512, isolated (µs per dispatch, behind a barrier) `gemv_parity --tiles TYPE 2560 512 NT ` (GEMV_AB_COLD=1 walks 256 copies of W ≈ 180 MB, so W comes from DRAM) | type | weights | ntok | 32,1,4,1 | 16,2,4,1 | 32,2,4,1 | 16,1,4,1 | 8,1,4,1 | 8,2,4,1 | 8,1,8,1 | 4,2,4,1 | best | |---|---|---|---|---|---|---|---|---|---|---|---| | IQ4_XS | cold | 1 | 8.67 | **7.45** | 8.36 | 7.46 | 7.61 | 8.13 | 7.62 | 12.54 | 16,2,4,1 | | IQ4_XS | cold | 2 | 11.36 | **8.57** | 9.78 | 9.47 | 9.50 | 9.50 | 9.61 | 14.72 | 16,2,4,1 | | IQ4_XS | cold | 3 | 14.05 | **9.71** | 11.36 | 10.69 | 11.10 | 11.31 | 11.24 | 18.36 | 16,2,4,1 | | IQ4_XS | cold | 4 | 17.57 | **11.22** | 13.08 | 13.16 | 13.08 | 13.20 | 13.20 | 22.00 | 16,2,4,1 | | IQ4_XS | hot | 1 | 8.21 | 6.58 | 7.21 | 6.82 | 7.06 | **6.32** | 7.04 | 7.68 | 8,2,4,1 | | IQ4_XS | hot | 2 | 14.04 | 10.10 | 11.29 | 11.03 | 11.12 | **10.03** | 11.12 | 14.72 | 8,2,4,1 | | IQ4_XS | hot | 3 | 14.21 | **9.46** | 10.74 | 10.55 | 10.66 | 10.19 | 10.65 | 16.32 | 16,2,4,1 | | IQ4_XS | hot | 4 | 17.01 | **10.74** | 12.24 | 12.39 | 12.34 | 11.75 | 12.39 | 19.98 | 16,2,4,1 | | Q6_K | cold | 1 | 10.26 | 11.09 | 10.58 | 10.45 | 10.24 | 11.24 | **10.24** | 17.85 | 8,1,8,1 | | Q6_K | cold | 2 | 12.43 | 11.61 | 12.14 | 10.98 | **10.64** | 13.72 | 10.74 | 22.93 | 8,1,4,1 | | Q6_K | cold | 3 | 15.60 | 13.43 | 15.26 | 12.45 | **12.04** | 18.08 | 12.18 | 31.62 | 8,1,4,1 | | Q6_K | cold | 4 | 19.07 | 15.64 | 17.63 | 14.79 | **14.63** | 21.81 | 14.75 | 40.22 | 8,1,4,1 | | Q6_K | hot | 1 | 7.12 | 6.57 | 6.78 | **6.46** | 6.87 | 8.21 | 6.87 | 14.54 | 16,1,4,1 | | Q6_K | hot | 2 | 10.26 | **8.08** | 9.24 | 8.68 | 8.80 | 11.12 | 8.99 | 19.99 | 16,2,4,1 | | Q6_K | hot | 3 | 13.63 | **10.40** | 12.09 | 11.07 | 10.96 | 15.54 | 10.99 | 28.37 | 16,2,4,1 | | Q6_K | hot | 4 | 17.15 | **12.89** | 14.77 | 13.56 | 13.52 | 19.30 | 13.68 | 36.18 | 16,2,4,1 | | IQ3_S | cold | 1 | 7.98 | 8.54 | **7.90** | 8.03 | 9.70 | 11.10 | 9.62 | 16.18 | 32,2,4,1 | | IQ3_S | cold | 2 | 10.64 | 9.49 | **9.14** | 9.75 | 10.59 | 12.36 | 10.52 | 18.50 | 32,2,4,1 | | IQ3_S | cold | 3 | 13.45 | 11.06 | **10.57** | 11.79 | 11.86 | 14.47 | 11.78 | 22.20 | 32,2,4,1 | | IQ3_S | cold | 4 | 16.06 | 12.82 | **12.78** | 13.34 | 13.65 | 16.89 | 13.71 | 28.33 | 32,2,4,1 | | IQ3_S | hot | 1 | 7.48 | 7.30 | **6.81** | 7.58 | 7.93 | 8.22 | 7.83 | 12.92 | 32,2,4,1 | | IQ3_S | hot | 2 | 11.14 | 9.49 | **9.22** | 10.04 | 10.24 | 11.38 | 10.39 | 17.50 | 32,2,4,1 | | IQ3_S | hot | 3 | 13.12 | 10.22 | **9.92** | 10.79 | 11.01 | 11.40 | 10.96 | 19.34 | 32,2,4,1 | | IQ3_S | hot | 4 | 17.06 | 13.86 | **13.09** | 13.53 | 14.14 | 16.73 | 14.11 | 28.43 | 32,2,4,1 | ## Target 1a': 2560 -> 512 inside the real group (k + v beside attn_q IQ4_XS 2560→12288, one barrier) `GEMV_BESIDE=IQ4_XS,12288, GEMV_AB_COLD=1 gemv_parity --tiles ...` (µs per group; q tile = gemv_tile's: 32,2,4,1 nt1, 16,2,4,1 nt>=2) | type | weights | ntok | 32,1,4,1 | 16,2,4,1 | 32,2,4,1 | 8,1,4,1 | best | |---|---|---|---|---|---|---|---| | IQ4_XS | cold | 1 | 184.57 | **178.86** | 180.51 | 180.03 | 16,2,4,1 | | IQ4_XS | cold | 2 | 200.33 | **186.88** | 191.96 | 190.08 | 16,2,4,1 | | IQ4_XS | cold | 3 | 241.82 | **223.59** | 229.72 | 227.78 | 16,2,4,1 | | IQ4_XS | cold | 4 | 298.04 | **274.62** | 281.90 | 279.07 | 16,2,4,1 | | Q6_K | cold | 1 | 228.76 | 227.32 | **223.86** | 225.61 | 32,2,4,1 | | Q6_K | cold | 2 | 258.73 | **253.91** | 254.12 | 254.85 | 16,2,4,1 | | Q6_K | cold | 3 | 335.32 | **320.63** | 322.67 | 320.89 | 16,2,4,1 | | Q6_K | cold | 4 | 400.36 | **381.26** | 386.35 | 381.40 | 16,2,4,1 | | IQ3_S | cold | 1 | 259.87 | 259.62 | **256.99** | 257.80 | 32,2,4,1 | | IQ3_S | cold | 2 | 285.11 | **275.59** | 276.66 | 278.94 | 16,2,4,1 | | IQ3_S | cold | 3 | 335.79 | **319.72** | 320.34 | 325.83 | 16,2,4,1 | | IQ3_S | cold | 4 | 380.44 | **359.75** | 362.64 | 365.00 | 16,2,4,1 | ## Target 1b: 640 -> 2560 (µs per dispatch) current table: IQ4_NL nt1 8,1,4,1 / nt>=2 4,2,4,1; Q8_0 nt1-2 32,1,4,1 / nt>=3 4,1,4,1; Q2_0 8,1,4,1. | type | weights | ntok | 8,1,4,1 | 4,2,4,1 | 8,2,4,1 | 16,1,4,1 | 16,2,4,1 | 32,1,4,1 | 4,1,4,1 | 4,4,4,1 | 2,2,4,1 | best | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Q2_0 | cold | 1 | 6.42 | 6.56 | 6.24 | **5.93** | 6.07 | 7.42 | 6.42 | 7.58 | 8.90 | 16,1,4,1 | | Q2_0 | cold | 2 | **7.30** | 7.62 | 7.46 | 7.99 | 7.64 | 10.69 | 7.69 | 10.31 | 9.92 | 8,1,4,1 | | Q2_0 | cold | 3 | **8.79** | 9.31 | 9.17 | 10.36 | 11.38 | 14.42 | 9.52 | 12.79 | 11.74 | 8,1,4,1 | | Q2_0 | cold | 4 | 12.15 | 11.60 | **11.47** | 13.52 | 13.91 | 18.45 | 12.01 | 15.03 | 14.76 | 8,2,4,1 | | Q2_0 | hot | 1 | 4.60 | 4.50 | **4.36** | 5.44 | 4.91 | 7.15 | 5.08 | 5.06 | 5.84 | 8,2,4,1 | | Q2_0 | hot | 2 | 9.02 | 8.94 | **8.72** | 10.99 | 10.04 | 15.32 | 9.23 | 12.37 | 10.19 | 8,2,4,1 | | Q2_0 | hot | 3 | 9.10 | 9.17 | **9.05** | 11.19 | 10.18 | 16.33 | 9.43 | 12.54 | 10.25 | 8,2,4,1 | | Q2_0 | hot | 4 | **10.39** | 10.58 | 10.41 | 12.66 | 12.17 | 18.29 | 10.62 | 13.06 | 13.00 | 8,1,4,1 | | IQ4_NL | cold | 1 | **9.81** | 10.32 | 10.47 | 10.51 | 10.73 | 11.57 | 10.23 | 10.70 | 10.64 | 8,1,4,1 | | IQ4_NL | cold | 2 | 11.01 | 10.68 | 11.35 | 13.03 | 12.11 | 15.31 | **10.65** | 12.02 | 11.47 | 4,1,4,1 | | IQ4_NL | cold | 3 | 15.63 | **13.09** | 14.93 | 19.77 | 16.86 | 23.95 | 14.45 | 18.66 | 14.89 | 4,2,4,1 | | IQ4_NL | cold | 4 | 18.36 | **14.97** | 17.07 | 23.50 | 19.96 | 28.95 | 16.69 | 21.71 | 17.85 | 4,2,4,1 | | IQ4_NL | hot | 1 | 8.13 | **6.85** | 7.46 | 10.10 | 8.59 | 11.85 | 7.35 | 6.96 | 7.02 | 4,2,4,1 | | IQ4_NL | hot | 2 | 11.02 | **8.41** | 9.13 | 13.82 | 10.91 | 16.81 | 9.42 | 9.16 | 8.98 | 4,2,4,1 | | IQ4_NL | hot | 3 | 12.68 | **9.51** | 10.65 | 16.52 | 13.04 | 20.70 | 11.07 | 12.33 | 11.02 | 4,2,4,1 | | IQ4_NL | hot | 4 | 15.57 | **12.18** | 13.43 | 20.78 | 16.64 | 26.63 | 13.79 | 16.13 | 14.21 | 4,2,4,1 | | Q8_0 | cold | 1 | 16.71 | 17.18 | 16.74 | 15.56 | 16.43 | **14.68** | 16.57 | 17.94 | 18.76 | 32,1,4,1 | | Q8_0 | cold | 2 | 16.88 | 17.48 | 17.44 | 17.95 | 18.45 | **16.57** | 16.95 | 20.47 | 19.14 | 32,1,4,1 | | Q8_0 | cold | 3 | 18.86 | 19.86 | 20.33 | 23.39 | 23.45 | 21.17 | **18.03** | 30.34 | 22.56 | 4,1,4,1 | | Q8_0 | cold | 4 | 22.27 | 20.71 | 24.12 | 26.74 | 29.20 | 28.05 | **19.75** | 35.07 | 24.07 | 4,1,4,1 | | Q8_0 | hot | 1 | 7.66 | **6.80** | 7.35 | 8.89 | 8.62 | 9.92 | 7.14 | 8.37 | 7.28 | 4,2,4,1 | | Q8_0 | hot | 2 | 9.26 | **7.61** | 9.21 | 11.43 | 11.08 | 13.60 | 8.16 | 10.68 | 9.65 | 4,2,4,1 | | Q8_0 | hot | 3 | 12.80 | **10.74** | 12.58 | 16.86 | 16.01 | 19.23 | 11.44 | 17.02 | 12.83 | 4,2,4,1 | | Q8_0 | hot | 4 | 16.06 | **13.25** | 15.52 | 20.87 | 19.81 | 24.71 | 13.97 | 20.68 | 15.39 | 4,2,4,1 | | Q2_0R | cold | 1 | 8.55 | 9.03 | 8.67 | **8.40** | 8.87 | 10.15 | 8.69 | 10.33 | 11.53 | 16,1,4,1 | | Q2_0R | cold | 2 | **10.35** | 11.12 | 11.18 | 11.30 | 12.02 | 14.80 | 10.88 | 14.81 | 13.77 | 8,1,4,1 | | Q2_0R | cold | 3 | **12.52** | 13.74 | 12.92 | 13.51 | 16.88 | 18.73 | 12.52 | 17.48 | 16.19 | 8,1,4,1 | | Q2_0R | cold | 4 | **14.84** | 16.83 | 16.24 | 17.17 | 20.23 | 24.87 | 15.30 | 21.32 | 20.14 | 8,1,4,1 | | Q2_0R | hot | 1 | **5.77** | 6.33 | 5.86 | 6.49 | 6.14 | 7.68 | 6.10 | 6.81 | 6.80 | 8,1,4,1 | | Q2_0R | hot | 2 | **8.26** | 9.10 | 8.66 | 9.98 | 9.88 | 13.61 | 8.51 | 10.95 | 9.31 | 8,1,4,1 | | Q2_0R | hot | 3 | **10.88** | 11.69 | 11.35 | 13.68 | 13.32 | 18.94 | 11.57 | 14.91 | 12.59 | 8,1,4,1 | | Q2_0R | hot | 4 | **12.45** | 13.70 | 13.30 | 15.43 | 15.52 | 22.76 | 12.89 | 17.02 | 15.22 | 8,1,4,1 | Q2_0 re-run with 5 repeats: nt1 8,1 6.33 / 8,2 6.15 / 16,1 5.95; nt4 8,1 14.21 / **8,2 13.68** / 16,1 15.97. nt1 16,1,4,1 is -6% cold but +18% hot and changes LPR across ntok; left alone. ## 2560 -> 640 check (cold, µs) — the shexp gate/up tiles (cat would use them) | type | weights | ntok | 32,1,4,1 | 16,2,4,1 | 32,2,4,1 | 16,1,4,1 | 8,1,4,1 | 8,1,8,1 | best | |---|---|---|---|---|---|---|---|---|---| | IQ4_XS | cold | 1 | 10.40 | 8.49 | 9.65 | 8.75 | 8.43 | **8.41** | 8,1,8,1 | | IQ4_XS | cold | 2 | 15.32 | **10.34** | 12.12 | 12.36 | 11.30 | 10.90 | 16,2,4,1 | | IQ4_XS | cold | 3 | 19.39 | **11.44** | 14.82 | 14.02 | 13.09 | 13.09 | 16,2,4,1 | | IQ4_XS | cold | 4 | 24.94 | **13.64** | 17.42 | 17.63 | 15.97 | 15.85 | 16,2,4,1 | | Q6_K | cold | 1 | **12.43** | 12.89 | 12.68 | 12.47 | 12.63 | 12.60 | 32,1,4,1 | | Q6_K | cold | 2 | 17.32 | 15.06 | 15.95 | 14.47 | 13.81 | **13.80** | 8,1,8,1 | | Q6_K | cold | 3 | 22.77 | 18.00 | 22.91 | 16.89 | 15.54 | **15.52** | 8,1,8,1 | | Q6_K | cold | 4 | 28.06 | 21.12 | 26.50 | 20.43 | **19.12** | 19.15 | 8,1,4,1 | | IQ3_S | cold | 1 | 10.86 | 10.36 | **10.32** | 10.73 | 12.07 | 11.76 | 32,2,4,1 | | IQ3_S | cold | 2 | 15.59 | **12.38** | 12.51 | 14.04 | 13.65 | 13.52 | 16,2,4,1 | | IQ3_S | cold | 3 | 20.06 | **14.32** | 14.40 | 15.82 | 15.27 | 15.12 | 16,2,4,1 | | IQ3_S | cold | 4 | 24.20 | **16.99** | 17.47 | 18.29 | 17.74 | 17.54 | 16,2,4,1 | IQ3_S re-run (5 repeats, cold): nt2 8,1,8,1 10.98 / 32,2,4,1 10.19 / 16,2,4,1 10.05; nt3 12.90 / 11.83 / 12.19; nt4 14.50 / 14.34 / 13.98. 16,2 and 32,2 tie; 32,2 chosen so LPR matches nt1. ## Target 2: gemv_cat vs two concurrent gemvs (`gemv_parity --cat`, experiment patch applied) Each pair behind one barrier, the two gemvs with no barrier between (as `Window::layer`), 2 runs each: ``` cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 cold: 2 gemv 13.737 us, gemv_cat 13.753 us (+0.1%) cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 cold: 2 gemv 13.819 us, gemv_cat 13.852 us (+0.2%) cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 cold: 2 gemv 14.479 us, gemv_cat 14.354 us (-0.9%) cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 cold: 2 gemv 14.475 us, gemv_cat 14.383 us (-0.6%) cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 cold: 2 gemv 17.481 us, gemv_cat 18.855 us (+7.9%) cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 cold: 2 gemv 18.378 us, gemv_cat 18.888 us (+2.8%) cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 cold: 2 gemv 21.094 us, gemv_cat 21.678 us (+2.8%) cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 cold: 2 gemv 20.690 us, gemv_cat 21.110 us (+2.0%) cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 hot: 2 gemv 10.519 us, gemv_cat 10.541 us (+0.2%) cat IQ4_XS 2560 512+512 nt 1 tile 16,2,4 hot: 2 gemv 10.540 us, gemv_cat 10.569 us (+0.3%) cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 hot: 2 gemv 12.803 us, gemv_cat 12.771 us (-0.3%) cat IQ4_XS 2560 512+512 nt 2 tile 16,2,4 hot: 2 gemv 12.800 us, gemv_cat 12.769 us (-0.2%) cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 hot: 2 gemv 14.901 us, gemv_cat 15.067 us (+1.1%) cat IQ4_XS 2560 512+512 nt 3 tile 16,2,4 hot: 2 gemv 14.943 us, gemv_cat 15.125 us (+1.2%) cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 hot: 2 gemv 17.509 us, gemv_cat 17.592 us (+0.5%) cat IQ4_XS 2560 512+512 nt 4 tile 16,2,4 hot: 2 gemv 17.450 us, gemv_cat 17.559 us (+0.6%) cat Q6_K 2560 512+512 nt 1 tile 8,1,4 cold: 2 gemv 20.273 us, gemv_cat 20.993 us (+3.6%) cat Q6_K 2560 512+512 nt 1 tile 8,1,4 cold: 2 gemv 20.458 us, gemv_cat 21.054 us (+2.9%) cat Q6_K 2560 512+512 nt 2 tile 8,1,4 cold: 2 gemv 21.199 us, gemv_cat 21.048 us (-0.7%) cat Q6_K 2560 512+512 nt 2 tile 8,1,4 cold: 2 gemv 21.735 us, gemv_cat 21.322 us (-1.9%) cat Q6_K 2560 512+512 nt 3 tile 8,1,4 cold: 2 gemv 22.902 us, gemv_cat 23.149 us (+1.1%) cat Q6_K 2560 512+512 nt 3 tile 8,1,4 cold: 2 gemv 22.877 us, gemv_cat 23.111 us (+1.0%) cat Q6_K 2560 512+512 nt 4 tile 8,1,4 cold: 2 gemv 26.476 us, gemv_cat 28.002 us (+5.8%) cat Q6_K 2560 512+512 nt 4 tile 8,1,4 cold: 2 gemv 28.318 us, gemv_cat 30.941 us (+9.3%) cat Q6_K 2560 512+512 nt 1 tile 8,1,4 hot: 2 gemv 13.059 us, gemv_cat 12.609 us (-3.4%) cat Q6_K 2560 512+512 nt 1 tile 8,1,4 hot: 2 gemv 12.212 us, gemv_cat 11.709 us (-4.1%) cat Q6_K 2560 512+512 nt 2 tile 8,1,4 hot: 2 gemv 16.915 us, gemv_cat 16.627 us (-1.7%) cat Q6_K 2560 512+512 nt 2 tile 8,1,4 hot: 2 gemv 16.135 us, gemv_cat 16.060 us (-0.5%) cat Q6_K 2560 512+512 nt 3 tile 8,1,4 hot: 2 gemv 20.912 us, gemv_cat 20.629 us (-1.4%) cat Q6_K 2560 512+512 nt 3 tile 8,1,4 hot: 2 gemv 20.641 us, gemv_cat 20.426 us (-1.0%) cat Q6_K 2560 512+512 nt 4 tile 8,1,4 hot: 2 gemv 25.688 us, gemv_cat 25.883 us (+0.8%) cat Q6_K 2560 512+512 nt 4 tile 8,1,4 hot: 2 gemv 25.375 us, gemv_cat 25.923 us (+2.2%) cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 cold: 2 gemv 16.619 us, gemv_cat 16.692 us (+0.4%) cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 cold: 2 gemv 16.687 us, gemv_cat 16.696 us (+0.1%) cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 cold: 2 gemv 16.941 us, gemv_cat 16.938 us (-0.0%) cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 cold: 2 gemv 17.346 us, gemv_cat 17.026 us (-1.8%) cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 cold: 2 gemv 22.748 us, gemv_cat 23.210 us (+2.0%) cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 cold: 2 gemv 22.205 us, gemv_cat 23.013 us (+3.6%) cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 cold: 2 gemv 25.746 us, gemv_cat 26.951 us (+4.7%) cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 cold: 2 gemv 25.801 us, gemv_cat 26.759 us (+3.7%) cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 hot: 2 gemv 12.635 us, gemv_cat 12.580 us (-0.4%) cat IQ4_XS 2560 640+640 nt 1 tile 16,2,4 hot: 2 gemv 11.794 us, gemv_cat 11.828 us (+0.3%) cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 hot: 2 gemv 14.355 us, gemv_cat 14.294 us (-0.4%) cat IQ4_XS 2560 640+640 nt 2 tile 16,2,4 hot: 2 gemv 14.347 us, gemv_cat 14.337 us (-0.1%) cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 hot: 2 gemv 16.780 us, gemv_cat 17.007 us (+1.4%) cat IQ4_XS 2560 640+640 nt 3 tile 16,2,4 hot: 2 gemv 16.950 us, gemv_cat 17.069 us (+0.7%) cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 hot: 2 gemv 20.724 us, gemv_cat 20.625 us (-0.5%) cat IQ4_XS 2560 640+640 nt 4 tile 16,2,4 hot: 2 gemv 20.848 us, gemv_cat 21.028 us (+0.9%) cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 15.959 us, gemv_cat 16.974 us (+6.4%) cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 17.358 us, gemv_cat 16.771 us (-3.4%) cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 cold: 2 gemv 21.662 us, gemv_cat 21.334 us (-1.5%) cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 cold: 2 gemv 22.133 us, gemv_cat 22.124 us (-0.0%) cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 cold: 2 gemv 27.146 us, gemv_cat 26.946 us (-0.7%) cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 cold: 2 gemv 27.741 us, gemv_cat 27.671 us (-0.3%) cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 cold: 2 gemv 32.387 us, gemv_cat 32.224 us (-0.5%) cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 cold: 2 gemv 31.193 us, gemv_cat 31.570 us (+1.2%) cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 15.602 us, gemv_cat 15.247 us (-2.3%) cat IQ3_S 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 15.583 us, gemv_cat 14.706 us (-5.6%) cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 hot: 2 gemv 19.240 us, gemv_cat 18.755 us (-2.5%) cat IQ3_S 2560 640+640 nt 2 tile 32,2,4 hot: 2 gemv 19.042 us, gemv_cat 18.566 us (-2.5%) cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 hot: 2 gemv 23.257 us, gemv_cat 22.716 us (-2.3%) cat IQ3_S 2560 640+640 nt 3 tile 32,2,4 hot: 2 gemv 23.515 us, gemv_cat 23.379 us (-0.6%) cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 hot: 2 gemv 29.370 us, gemv_cat 28.880 us (-1.7%) cat IQ3_S 2560 640+640 nt 4 tile 32,2,4 hot: 2 gemv 28.697 us, gemv_cat 28.610 us (-0.3%) cat Q6_K 2560 640+640 nt 2 tile 8,1,4 cold: 2 gemv 25.246 us, gemv_cat 26.583 us (+5.3%) cat Q6_K 2560 640+640 nt 2 tile 8,1,4 cold: 2 gemv 27.044 us, gemv_cat 28.421 us (+5.1%) cat Q6_K 2560 640+640 nt 3 tile 8,1,4 cold: 2 gemv 33.306 us, gemv_cat 33.523 us (+0.7%) cat Q6_K 2560 640+640 nt 3 tile 8,1,4 cold: 2 gemv 32.500 us, gemv_cat 33.620 us (+3.4%) cat Q6_K 2560 640+640 nt 4 tile 8,1,4 cold: 2 gemv 37.989 us, gemv_cat 40.487 us (+6.6%) cat Q6_K 2560 640+640 nt 4 tile 8,1,4 cold: 2 gemv 37.104 us, gemv_cat 40.660 us (+9.6%) cat Q6_K 2560 640+640 nt 2 tile 8,1,4 hot: 2 gemv 21.201 us, gemv_cat 24.492 us (+15.5%) cat Q6_K 2560 640+640 nt 2 tile 8,1,4 hot: 2 gemv 21.022 us, gemv_cat 23.201 us (+10.4%) cat Q6_K 2560 640+640 nt 3 tile 8,1,4 hot: 2 gemv 26.668 us, gemv_cat 27.057 us (+1.5%) cat Q6_K 2560 640+640 nt 3 tile 8,1,4 hot: 2 gemv 26.145 us, gemv_cat 26.466 us (+1.2%) cat Q6_K 2560 640+640 nt 4 tile 8,1,4 hot: 2 gemv 32.347 us, gemv_cat 33.714 us (+4.2%) cat Q6_K 2560 640+640 nt 4 tile 8,1,4 hot: 2 gemv 33.220 us, gemv_cat 34.377 us (+3.5%) cat Q6_K 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 23.505 us, gemv_cat 23.975 us (+2.0%) cat Q6_K 2560 640+640 nt 1 tile 32,2,4 cold: 2 gemv 23.916 us, gemv_cat 24.354 us (+1.8%) cat Q6_K 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 19.528 us, gemv_cat 20.770 us (+6.4%) cat Q6_K 2560 640+640 nt 1 tile 32,2,4 hot: 2 gemv 19.303 us, gemv_cat 19.804 us (+2.6%) ``` Why it cannot win: the two GEMVs already overlap on the GPU, x is 10 KB (L1/L2), and the cost is DRAM traffic for W (2×0.7–1.2 MB), which fusing does not reduce. At nt>=3 the cat kernel is slightly worse (extra row-select per row pointer, worse register allocation). Not wired in `Window::gemv_cat` / `Window::layer`. Note for whoever owns MTP: the same applies to the existing Q8_0 `gm_cat` in the draft step — worth re-checking with `--cat Q8_0 2560 512 1 32,1,4` cold before keeping it. ## Bitwise / ntok analysis (the "LPR decides the sums" claim) - In `gemv` (KS=1) a row's sum is `lane_sum` over lanes each accumulating units `u = li, li+LPR, ...` in order; NR only says how many rows a lane carries, SG how many simdgroups share a threadgroup — neither changes a row's order of operations. Format paths that depend on more than LPR (`gemv_iq4_packed` with NTOK, `gemv_direct_x` with NR/NTOK for Q8_0) are written to produce the same FMA chains, and the new parity check confirms it for all 13 formats: **same LPR ⇒ bitwise, for any NR/SG**. KS>1 changes the order (partial sums), as expected. - Engine tests do not compare a window against single-token runs through `Window::gemv_tile` (the parity tests call the kernels directly; `prompt_parity` uses `dense_mm`), and the existing table already switches LPR across ntok (e.g. old IQ3_S 640, Q8_0, IQ4_XS n_in>=6144). The new 512 tiles and the IQ3_S 640 change *remove* LPR switches rather than add them; the Q2_0 nt4 change keeps LPR 8. ## Verification - `gemv_parity` (default, all formats/shapes): **ALL PASS** (includes the new same-LPR bitwise check). Note this full suite takes ~34 s (it always did; the CPU double reference dominates) — the only invocation above 10 s. - `gemv_parity --cat` parity: bitwise for IQ4_XS 16,2,4 / Q6_K 8,1,4 & 32,2,4 / IQ3_S 32,2,4 at 512+512 and 640+640, nt 1/2/3/4/8 (experiment patch) and for the existing Q8_0 instance (main patch). - `window.cpp` compiles (`strata_metal_kernels` object, scratch build). New tiles are all existing `GV_SET` instances. - Not done (forbidden here): full-model `strata_window_cli` / server A/B. Recommended next: one decode A/B with `--spec 3` where most windows are 2..4 tokens, which is where the 512 fix pays most. ## gemv_parity additions (main patch) - `--tiles TYPE NIN NOUT NTOK L,R,S,K ...` paired multi-tile timing (cold with `GEMV_AB_COLD=1`, `Q2_0R` accepted), rel L2 per tile; `GEMV_BESIDE=TYPE,NOUT,L,R,S,K` times each tile as k+v beside a big GEMV. - `--cat TYPE NIN NOUT NTOK L,R,S` gemv_cat vs its two concurrent gemvs: bitwise + paired timing. - default parity: "same LPR: bitwise" invariant.