opus_r33_gr — Round 33 Track A: hyper-connection read (gr_norm, gr_norm_comb, gr_down_f, gr_up_f) #
Patch: /tmp/opus_r33_gr/opus_r33_gr.patch (git -C vendor/strata apply --check OK; touches only metal/kernels/gr.metal
and metal/tests/fuse_parity.cpp). Scratch tree: /tmp/opus_r33_gr/strata; baseline kernels: /tmp/opus_r33_gr/base_kernels;
raw final logs: /tmp/opus_r33_gr/final_bench.txt. Device: Apple M5 (10-core GPU), GPU quiet for the final run.
Shipped (all bitwise PASS, ntok 1..8, every gr.metal/gr2.metal entry point incl. BF16/Q4/Q5/F32-inject; default fuse_parity ALL PASS) #
gr_norm:applybranch hoisted out of the 10-step unrolled loop, pointers pre-offset (rp = r + tid,bp = bo + t*2560 + tid,o += tid; w += tid) so every access is an immediate[i * 256]. -6..-11% at ntok 1..4, -2..-3% at 7..8 (~0.2 µs per dispatch; gr_norm is ~2.3-4.3 µs, mostly launch-bound).grd_accQ8_0, NR = 2 (gr_down_q8_*_r2,gr_down_q8f_*_r2): aGrdUnit<F, NR>holder with aGrdUnit<T_Q8_0, 2>specialization that keeps the codes packed aschar4[8]per row (8 regs instead of 32 floats) and widens withfloat4(char4)per token. Same fmas in the same order, so bitwise identical. The generic path is unchanged codegen (all other formats/NR = 1: ±0.5%, noise). ntok 7: -13..-16%, ntok 8: -10..-15%; ntok 5-6 neutral; ntok ≤ 4 never uses_r2in the engine.fuse_parity --gr-unroll ... bench: 16 pairs × 128 iters, reports the median pair ratio (A/A noise went from ±15% to ±0.3% for gr_down/gr_up, ~±2% for the tiny gr_norm_comb); gr_norm/gr_norm_comb now timed; added same-entry A/B benches for the engine's real entry points (gr_down_q8_4_4,_4_4_r2,q8f_4_4(_r2),q8_8_8(_r2)).
Final paired timings (median Δ, quiet GPU) #
| kernel | ntok | base µs | new µs | median Δ |
|---|---|---|---|---|
| gr_down_q8_4_4 | 1 | 23.213 | 23.210 | +0.0% |
| gr_down_q8_4_4 | 2 | 23.836 | 23.831 | +0.0% |
| gr_down_q8_4_4 | 3 | 31.574 | 31.606 | +0.1% |
| gr_down_q8_4_4 | 4 | 40.602 | 40.610 | +0.0% |
| gr_down_q8_4_4 | 5 | 55.545 | 55.580 | +0.0% |
| gr_down_q8_4_4 | 6 | 75.455 | 75.441 | +0.0% |
| gr_down_q8_4_4 | 7 | 97.495 | 97.445 | -0.0% |
| gr_down_q8_4_4 | 8 | 119.752 | 119.762 | -0.0% |
| gr_down_q8_4_4_r2 | 1 | 23.348 | 23.443 | +0.4% |
| gr_down_q8_4_4_r2 | 2 | 23.814 | 23.786 | -0.2% |
| gr_down_q8_4_4_r2 | 3 | 32.917 | 32.977 | +0.2% |
| gr_down_q8_4_4_r2 | 4 | 41.266 | 41.331 | +0.1% |
| gr_down_q8_4_4_r2 | 5 | 49.493 | 49.428 | -0.1% |
| gr_down_q8_4_4_r2 | 6 | 58.338 | 58.370 | +0.1% |
| gr_down_q8_4_4_r2 | 7 | 79.517 | 68.066 | -14.4% |
| gr_down_q8_4_4_r2 | 8 | 90.285 | 79.752 | -11.7% |
| gr_down_q8_8_8 | 1 | 23.584 | 23.729 | +0.6% |
| gr_down_q8_8_8 | 2 | 24.839 | 24.843 | -0.0% |
| gr_down_q8_8_8 | 3 | 34.813 | 34.802 | -0.1% |
| gr_down_q8_8_8 | 4 | 45.466 | 45.464 | -0.0% |
| gr_down_q8_8_8 | 5 | 62.634 | 62.615 | -0.0% |
| gr_down_q8_8_8 | 6 | 83.515 | 83.568 | +0.1% |
| gr_down_q8_8_8 | 7 | 115.168 | 115.243 | +0.0% |
| gr_down_q8_8_8 | 8 | 140.286 | 140.303 | +0.1% |
| gr_down_q8_8_8_r2 | 1 | 23.953 | 24.036 | +0.3% |
| gr_down_q8_8_8_r2 | 2 | 25.738 | 25.852 | +0.5% |
| gr_down_q8_8_8_r2 | 3 | 34.776 | 34.844 | +0.2% |
| gr_down_q8_8_8_r2 | 4 | 46.604 | 46.623 | +0.0% |
| gr_down_q8_8_8_r2 | 5 | 58.045 | 58.184 | +0.3% |
| gr_down_q8_8_8_r2 | 6 | 70.499 | 70.514 | -0.1% |
| gr_down_q8_8_8_r2 | 7 | 100.167 | 83.844 | -16.3% |
| gr_down_q8_8_8_r2 | 8 | 114.260 | 97.570 | -14.6% |
| gr_down_q8f_4_4_r2 | 1 | 24.422 | 24.454 | +0.1% |
| gr_down_q8f_4_4_r2 | 2 | 24.889 | 24.863 | -0.4% |
| gr_down_q8f_4_4_r2 | 3 | 33.414 | 33.478 | +0.2% |
| gr_down_q8f_4_4_r2 | 4 | 41.238 | 41.304 | +0.2% |
| gr_down_q8f_4_4_r2 | 5 | 49.481 | 49.448 | -0.1% |
| gr_down_q8f_4_4_r2 | 6 | 58.534 | 58.637 | +0.2% |
| gr_down_q8f_4_4_r2 | 7 | 80.516 | 70.128 | -12.9% |
| gr_down_q8f_4_4_r2 | 8 | 91.064 | 81.766 | -10.2% |
| gr_norm apply 0 | 1 | 3.241 | 2.882 | -11.1% |
| gr_norm apply 0 | 2 | 2.478 | 2.227 | -10.4% |
| gr_norm apply 0 | 3 | 2.506 | 2.269 | -9.2% |
| gr_norm apply 0 | 4 | 2.578 | 2.400 | -7.1% |
| gr_norm apply 0 | 5 | 2.668 | 2.523 | -5.5% |
| gr_norm apply 0 | 6 | 3.039 | 2.849 | -6.4% |
| gr_norm apply 0 | 7 | 3.044 | 2.958 | -2.8% |
| gr_norm apply 0 | 8 | 3.185 | 3.084 | -3.3% |
| gr_norm apply 1 | 1 | 3.293 | 3.088 | -6.1% |
| gr_norm apply 1 | 2 | 3.347 | 3.096 | -7.7% |
| gr_norm apply 1 | 3 | 3.361 | 3.160 | -6.2% |
| gr_norm apply 1 | 4 | 3.468 | 3.256 | -5.9% |
| gr_norm apply 1 | 5 | 3.620 | 3.428 | -5.4% |
| gr_norm apply 1 | 6 | 3.834 | 3.763 | -1.8% |
| gr_norm apply 1 | 7 | 4.069 | 3.980 | -2.2% |
| gr_norm apply 1 | 8 | 4.346 | 4.255 | -2.2% |
| gr_norm_comb | 1 | 5.135 | 5.047 | -1.4% |
| gr_norm_comb | 2 | 5.335 | 5.363 | +0.7% |
| gr_norm_comb | 3 | 6.176 | 6.248 | +1.3% |
| gr_norm_comb | 4 | 6.754 | 6.678 | -1.0% |
| gr_norm_comb | 5 | 6.985 | 6.874 | -1.5% |
| gr_norm_comb | 6 | 8.941 | 8.889 | -0.7% |
| gr_norm_comb | 7 | 9.091 | 9.133 | +0.5% |
| gr_norm_comb | 8 | 10.624 | 10.563 | -0.7% |
| gr_up_q8 | 1 | 23.770 | 23.762 | -0.0% |
| gr_up_q8 | 2 | 24.547 | 24.535 | -0.1% |
| gr_up_q8 | 3 | 26.172 | 26.186 | +0.1% |
| gr_up_q8 | 4 | 29.213 | 29.184 | -0.1% |
| gr_up_q8 | 5 | 32.855 | 32.889 | +0.1% |
| gr_up_q8 | 6 | 37.063 | 37.025 | -0.1% |
| gr_up_q8 | 7 | 42.019 | 41.957 | -0.1% |
| gr_up_q8 | 8 | 47.111 | 47.046 | -0.1% |
Tried and rejected (measured, bitwise-safe but not wins) #
deq<T_Q8_0>#pragma unroll(full): 0.0% (the compiler already unrolls it). Not shipped, quant.metal untouched.- Packed Q8 for NR = 1 gr_down: +1..3% at ntok 2 (
q8_8_8+3.2%), neutral elsewhere → restricted to NR = 2. - Packed Q8 + always-unrolled token loop at ntok 7-8: +12..45% (registers). Rolled-from-ntok-5 for NR = 2:
4_4_r2-3% at ntok 5 but8_8_r2+0.4..1%; kept the existing ≤ 6 threshold. unroll_count(3)on the gr_downuloop: +2..42% everywhere. Software-pipelined prefetch of the next Q8 block: +3..53%.- Aligned 9×
uintQ8 block load + funnel shift instead of 16ushortloads: +25..66% (much worse). gr_up_q8: staginglointhreadgroup float[8*320]: +1..6% (lo reads are already L1 broadcast hits); packed Q8 dot: +2..8% at ntok 4-8;unroll_count(3): +4..30%.gr_norm_combapply-hoist +gsguard: -0.3..-0.8% vs repeated A/A (within the ~1-2% order bias of that bench) → not shipped.
Why the rest is stuck (diagnosis) #
- ntok 1: gr_down_q8 (~23.2 µs for 3.48 MB) and gr_up_q8 (~23.8 µs, 3.48 MB) are both at ~150 GB/s = M5 DRAM roofline.
- ntok 4 gr_down_q8_4_4 (40.6 µs): non-bitwise probes — removing the xn loads → 25.8 µs, removing the weight loads →
~26-31 µs, a fully coalesced (wrong-math) xn pattern → no change. It is the sum of DRAM weights + L2 xn re-reads
(1 row/threadgroup × ntok × 40 KB), and
_r2(half the xn traffic) is no faster at ntok 4 → latency/occupancy bound. Real gains need a different row→threadgroup mapping / grid, i.e. host changes inwindow.cpp(out of my file scope).