5.2 kB 64 lines
9d9dc8ae
Round 35 blitz: Wr/Wr2 router dq reuse (-47 dispatches/chunk), gemm_f16_16_96_2 Walbe (-13.6%), M%128<=64 64_128_4 (1.14x @ M=306), hc_up_tile row-tail trim (-2.5% @ T=289), mtp_rms_xprep (-37.5%) & mtp_embed (-21.8%) & mtp_slop_pen (-96.5%) -> 629.05 tok/s GPU prefill, 25.26 tok/s @10.2K decode, mtp_ms 192.17/355.15ms
dawn
⋯
audits/opus_r35_dense.md
sourcerendered
download raw history
at main Markdown

opus r35 track A: router dq reuse + gemm_f16 tiles (TN=96, M%128 padding), SLC swizzle (rejected) #

patch: /tmp/opus_r35_dense/opus_r35_dense.patch (git -C vendor/strata apply --check = OK) scratch tree: /tmp/opus_r35_dense/strata; test binary /tmp/opus_r35_dense/bin/dense_mm_parity (linked against vendor/strata/build-metal/libmtl_rt.a, run with STRATA_METAL_KERNELS=/tmp/opus_r35_dense/strata/metal/kernels). Files touched: metal/kernels/dense_mm.metal (+1 instantiation), window.cpp (gemm selector 3458-3470, router 3668-3687 only), metal/tests/dense_mm_parity.cpp. No full-model runs, :8080 untouched.

1. router dequant reuse (window.cpp prompt_chunk) #

Under stream_ && pred_on && !p.ids3.ptr the router of layer l+1 that layer l dequantizes for the read-ahead is reused as layer l+1's own router: w_r / w_r2 ping-pong between p.Wr and p.Wr2 by layer parity, dq(tr, w_r) only at l == 0 (or when reuse is off). PRED2 (p.ids3) overwrites Wr2 with l+2, so it keeps the old path. Same tensor, same dq call, same perm → bitwise-identical weights. Hazards: the dq into w_r2 at layer l targets the buffer layer l-1's router GEMM read (several barriers back, in hc_read); w_r / w_r2 are always distinct within a layer. run_graph_seg runs the segments in order with no replay, so layer l+1 always sees layer l's dq. Saves 47 dq_f16 dispatches (512x2560) per streamed chunk; measured dq cost per call (--dq-ab, 2560x512): 17-20 us (quantized types), 30 us BF16, 57 us F32 → 0.8-2.7 ms of dq work per chunk (they overlap other dqs, so wall saving is lower). Compiles (clang++ -fsyntax-only with the build's flags); not measured end-to-end (no full-model runs allowed). Needs a prompt_parity / streamed-prompt logits check before merging (I couldn't run the model).

2. gemm_f16 tiles (dense_mm_parity --tiles / --ab, all memcmp == 0 vs 128_64_4) #

Methodology: the GPU was shared (other agents/server: Device Utilization swinging 0-84%), so medians of 5 rounds were useless (2-4x swings). --tiles now reports the min over interleaved rounds (DENSE_ROUNDS, default 5/9) and can be restricted with DENSE_TILES=a,b; repeated focused runs agreed within ~1-3%.

Walbe (N=96, K=2560, 36 calls): new gemm_f16_16_96_2, used at M >= 640 #

M 16_64_2 64_32_2 16_96_2 32_96_2 64_96_4
512 29.6 31.5 30.2 39.7 40.1
640 33.0 34.3 30.9 41.0 40.5
768 39.8 37.4 (cur) 33.0 41.4 46.4
1024 55.4 42.9 (cur) 41.5 47.5 48.0
1983 116.8 103.8 (cur) 91.6 92.0 94.1
2048 122.8 98.3 (cur) 94.3 96.0 95.2
64_96_2 and 128_96_4 are 2-3x slower (not shipped). -3..-12% on the call, but only ~0.3-0.4 ms/chunk total.

big-N GEMMs (Wqkv, Wz, Wout, Wq, Wpv, Wshd; the else branch), M < 1536 #

128_64_4 loses whenever M % 128 is 1..64 (a 128-row tail tile that is up to half padding); 64_128_4 ties it elsewhere. New rule: M < 1536 && (M % 128 == 0 || M % 128 > 64) ? 128_64_4 : 64_128_4. Focused runs (us/call, 128_64_4 → 64_128_4):

  • M=192: Wqkv 920 → 787, Wz 708 → 552, Wout 683 → 563, Wq 1135 → 942, Wshd 64 → 51
  • M=306: Wqkv 1686 → 1412, Wz 970 → 847, Wout 1013 → 888, Wq 2044 → 1710, Wshd 92 → 80
  • M=448: Wqkv 2250 → 1990, Wout 1353 → 1233, Wq 3240 → 2862; M=576/700: -7..-9%; M=1300/1450: -4..-6%
  • ties (rule keeps 128_64_4): M=256, 384, 640, 896, 1000, 1100 --ab chunk totals (sum of isolated GEMMs x calls, r34 → new): M=150 1.17x, 192 1.14x, 306 1.13-1.14x, 448 1.09x, 520 1.07x, 700/800 1.05x, 1050 1.03-1.05x, 1200 1.04x; 200/330/640/1000/1400/1600/1983/2048 1.00x (unchanged tiles bar Walbe). So the win is on the tail / short chunks (a 306-token prompt: ~22 ms of dense GEMM per chunk), not 2K chunks.

W512 (N=512, 95 calls), Wshgu (640), Wk (1152/1664): no change #

The current picks (32_64_2 <384, 32_128_2 above; Nn<=1664 rule) are within ~1-3% of the best everywhere, e.g. W512 M=1983: 32_128_2 402-404, 64_128_4 403-404, 64_64_2 405, 128_64_4 416; M=1024: 32_128_2 197, 64_64_2 197, 64_128_4 210. Only W512 M=768 (128_64_4 -4%) and Wshgu 64_64_2 at M~1024 (-2.5%) showed anything, inside run-to-run noise. Not shipped.

SLC swizzle (rejected, not in the patch) #

Tried gemm_f16_sw<TM,TN,SG,G>: threadgroup ids remapped so G column tiles of B run down all row tiles of A (same tiles, same bits, memcmp 0). 128_64_4_sw{2,4,8} and 64_128_4_sw{2,4,8} vs plain on Wqkv/Wz/Wout/Wq at M = 256..2048: never a reproducible win; 128_64_4_sw2 ≈ plain (±1%), sw4/sw8 up to +10-60% slower, 64_128_4_sw* often 2-3x slower (e.g. Wqkv M=1983: 7827 plain vs 28523 sw2 / 23062 sw4). The default dispatch order already keeps B cached well enough; the swizzle kills the row-tile reuse of A between concurrently running threadgroups.

verification #

  • dense_mm_parity (parity): ALL PASS
  • --tiles M (all 12 tiles incl. 16_96_2 x 13 shapes) at M = 100, 256, 306, 640, 1000, 1983, 2048: ALL BITWISE
  • --ab M at 150..2048: ALL BITWISE
  • window.cpp: compiles (-fsyntax-only, build flags). Router reuse not run end-to-end (forbidden here).