round 23 track b: measured gates, not blanket raw-dot/q2 replacement #

patch: /tmp/sol_r23_gemv/sol_r23_gemv.patch (clean git -C vendor/strata apply --check). scratch: /tmp/sol_r23_gemv/strata; only metal/kernels/gemv.metal and metal/tests/gemv_parity.cpp changed. no tracked vendor files edited, no server or full-model runs.

experiment contract and provenance #

question: can raw float4 activations / packed q2 weights reduce register pressure without changing outputs or slowing production tiles? baseline: working-tree source copied before edits, vendor HEAD 296b59d2f73c8f04a805df29c2fa97655adace78; local baseline snapshot commit 7e412b5492d6c7886826a01aa388dfc6403a6724. baseline files preserved in /tmp/sol_r23_gemv/base_kernels and baseline_gemv_parity.cpp.

metric: same-tile GPU µs, deterministic random weight seeds 99/100 and activation seed 5; safe math. paired alternating baseline/candidate order. initial exploration: 7 pairs × 32 dispatches, 20ms warmup. confirmation: 15 pairs × 64 dispatches, 150ms warmup. predeclared meaningful threshold: about 2%; reject repeatable regressions. machine: apple m5, macos 27.0 build 26A428. timings are repeated hot-weight standalone projections, not rotating 1GB weights or full-model latency. live server untouched; exclusive machine idleness was not established.

limits: each standalone invocation limited to 9s, no :8080 requests. initial source compile ~5s; cached later runs usually <1s. two final untrimmed fuse attempts hit the 9s deadline (preserved logs); successful final trimmed-instantiation fuse run took 6.74s. no invocation was allowed to keep running beyond its deadline.

final source sha256:

• gemv.metal: c426737360f3e015f8c1ff1a25cab681ad700a9bd34bbb81750b156d836ca5ff
• gemv_parity.cpp: 75b8dce9508ec9aa5186f659877b7980e0ca29be996d21e767e843d5cb864f47

retained #

• gemv_raw_combine: same four FMA chains and final scale/subtract order as unit_combine; float4s read once across NR rows, no materialised xs array. enabled only for q8_0 ntok=1, and q8_0 ntok=2 / LPR16 / NR1, in gemv/cat/catk. all wider windows and other formats keep the original path.
• compressed weights only for q2_0r ntok=1 in gemv: ushort2 + scale per row, codes unpacked inside the FMA chain. no q[NR][16] or sc[NR] array. multi-token q2r unchanged.
• standalone gemv_parity --ab BASE_DIR TYPE NIN NOUT NTOK MODE [L R S K], modes gemv/pair/cat/catk, performs memcmp then paired timings; GEMV_AB_PARITY_ONLY=1 skips timings. unsupported shapes/configurations return 2.

q8 nt1 small projections: -3.41%, -5.41%, -3.62%; large projections essentially neutral. q8 2560→512 nt2: -2.56% (independent repeated confirmations -2.30/-1.89/-2.13%). q2r nt1: -2.16..-4.06% across all six shapes. nt>=2 q2r and q8 nt4 unchanged.

rejected #

• blanket raw-dot replacement: bitwise correct, but q8 NR2 nt4 +2..14%, iq4_xs generally neutral/slower; no blanket register-count claim translates directly into speed. dropping forced j unroll did not establish a broader winner. initial/rolled sources retained as candidate_initial.metal and candidate_rolled.metal.
• prepared q2_0 packed combine: NR1 production tiles neutral/slower; NR2 large synthetic tiles sometimes -8%, but the actual 2560→640 NR2 case confirmed only -1.55/-1.66/-1.65%, below the predeclared meaningful threshold. not shipped.
• gemv_pair q2 combine: no consistent win (-0..+2% in representative cells), unchanged. its token loops already had full-unroll pragmas, and it has no inner r loop to annotate; redundant pragmas were not added.
• the M5 tile lookup used is the actual working-tree window.cpp:955–1018: q2 2560→640 nt1 uses 32,2,4,1 and nt>=2 uses 16,1,4,1. also checked the requested 32,1,4,1 and 8,1,8,1 alternative tiles (q2confirm.json). no window.cpp changes.

parity and non-vacuity #

• 324/324 final A/B cases bitwise: q8/q2/q2r all 20 tiles at ntok 1/2/4/8 (Q8 32,1,8,8 uses n_in10240 to satisfy existing cfg_ok); 11 other formats at ntok1/2/4/8; cat/catk/pair ntok1..8. n_out101 exercises tail rows. final source logs: final_parity.json.
• 84/84 final six-shape/five-format benchmark cases bitwise, plus six final cat/catk cases. six requested IQ4_XS/IQ3_S n_in640 cells are invalid: their 256-weight blocks do not divide 640. rejected rather than truncated or fabricated.
• fuse_parity: ALL PASS initially (7.83s), and final source ALL PASS with unused GEMV_TYPES instantiations trimmed (6.74s). /tmp/sol_r23_gemv/fuse_kernels/gemv.metal is exactly final gemv plus #define GEMV_TYPES GV_SET(T_BF16) GV_SET(T_F32) GV_SET(T_Q8_0); no executed entry point changed. the trimming is only to fit the compilation under the 10s cap. full final untrimmed attempts timed out before results, not claimed passing.
• non-vacuity: perturbing one raw FMA gives memcmp FAIL; perturbing one packed q2r FMA gives memcmp FAIL. restoring final source passes. logs mutation.json, q2_mutation.json.
• invalid token count, invalid IQ4_XS input width, unsupported q8 pair each return 2 before GPU work.
• reviewed the complete patch with view_diff and changed context with view_code; existing gr/moe helpers were inspected. they cannot be reused directly because standalone gemv compilation does not include gr/moe.

final paired measurements #

µs: median baseline→candidate; parentheses: median per-pair change, not ratio of independent medians. tile is recorded per cell in final_matrix.json. near-zero entries are noise / unchanged code, not claimed wins.

final cat 2560→512+512, tile32,2,4,1: nt1 -2.49%; catk tile32,1,4,4 nt1 -2.98%. inactive cat/catk windows retain old code; isolated nonzero timing ratios there are noise, not optimization claims.

reproduction and artifacts #

cmake -S /tmp/sol_r23_gemv/strata/metal -B /tmp/sol_r23_gemv/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build /tmp/sol_r23_gemv/build --target gemv_parity fuse_parity -j4
/tmp/sol_r23_gemv/build/gemv_parity --ab /tmp/sol_r23_gemv/base_kernels Q2_0R 640 2560 1 gemv 8 1 4 1
GEMV_AB_PARITY_ONLY=1 /tmp/sol_r23_gemv/build/gemv_parity --ab /tmp/sol_r23_gemv/base_kernels Q8_0 2560 101 8 gemv 32 2 4 1
STRATA_METAL_KERNELS=/tmp/sol_r23_gemv/fuse_kernels /tmp/sol_r23_gemv/build/fuse_parity
git -C vendor/strata apply --check /tmp/sol_r23_gemv/sol_r23_gemv.patch

all raw paired logs: /tmp/sol_r23_gemv/logs/. machine-readable stages: initial_matrix.json, production_matrix.json, rolled_matrix.json, q2confirm.json, selected.json, repeats_fixed.json, final_matrix.json, final_extra.json, final_parity.json. successful final fusion log final_fuse_trimmed.txt; deadline logs final_fuse_parity.txt, final_fuse_retry.txt. no end-to-end tok/s claim.