3.2 kB 39 lines
34041844
Round 36 blitz: predL_/predL2_ token-0 fast path when PREFETCH_DRAFT<=0 (-16.0% hot / -6.8% cold), TM>32 row-tail trim in gemm_hc_down (-4.7% @ T=1025) & gemm_swiglu_up (-1.4% @ T=1025, -0.8% @ T=1987) -> mtp_ms 190.62ms, 42.26 tok/s decode
dawn
⋯
audits/opus_r36_decode.md
sourcerendered
download raw history
at main Markdown

opus r36 track A: decode predL_/predL2_ token-0 fast path + gemv micro-opts #

patch: /tmp/opus_r36_decode/opus_r36_decode.patch (git -C vendor/strata apply --check clean against the current working tree) scratch: /tmp/opus_r36_decode/strata (build in build-metal, METAL_CPP_DIR set; window.cpp + strata_window_cli build clean, no new warnings)

what changed #

  • window.cpp Window::layer pred block: when STRATA_METAL_PREFETCH_DRAFT <= 0 (read the same way as stream.cpp env_int, default 1) the predL_ / predL2_ gemvs run at ntok = 1 on the window's tile (tile_ntok = ntok). otherwise unchanged. verified readers: ExpertStream::ahead is the only consumer of the float pred rows (via at_layer / after_layer), and with k_draft_max <= 0 it skips every t > 0. prompt chains pass pred = nullptr (cut_ids_), so they're unaffected. live server (pid 78503) runs with STRATA_METAL_PREFETCH_DRAFT=0, so this path is on in production. rows t >= 1 of predL_/predL2_ now hold stale data; nothing reads them in this mode.
  • Window::gemv: new trailing int tile_ntok = 0 (0 = use ntok). NOTE: this touches the declaration at window.cpp:320 (outside the listed line ranges, but the default parameter can't be added any other way).
  • prewarm: no change needed, prewarm_pipelines runs layer() for t = 1..max_t_, so it makes gemv_8_16_1_4_1#1.
  • gemv_parity --pred TYPE NIN NOUT NTOK L,R,S: new mode. token 0 of 2x gemv#NTOK vs 2x gemv#1 vs 1x gemvc#1, memcmp, then 3-way rotated paired timings (GEMV_AB_COLD=1 for DRAM weights).
  • gemv.metal: no change shipped (see below).

measurements (M5, live server running, so a few runs were noisy outliers; the 6x repeats below were stable) #

parity: token 0 bitwise for nt = 2, 3, 4 (all three variants). gemv_parity default: ALL PASS (7.9 s).

Q8_0 2560 -> 512+512, tile 16,1,4,1, median of 15 rotated rounds, 6 invocations each:

2 gemv #4 (before) 2 gemv #1 (shipped) gemvc #1
hot 21.64-21.66 us 18.09-18.14 us (-16.3%) 18.47-18.54 us (-14.5%)
cold 24.07-24.32 us 22.72-23.13 us (-5.6%) 22.82-23.15 us (-5.2%)

nt=3: hot -3.1%, cold ~0. nt=2: hot +1.3%, cold -0.8% (#2 already takes the raw-x direct path, so it's a wash). gemv_cat at #1 loses to the concurrent pair in every config, so it's NOT wired. expected saving per 4-token verify window: ~46 x (1.4 .. 3.5) us = ~65-160 us of GPU time, before overlap with the router GEMV they sit beside. small but free; not measured end-to-end (no full-model runs per rules).

gemv.metal experiments (rejected, all bitwise, --ab vs pristine kernels) #

  • gemv_direct_x widened to NTOK <= 4 at LPR 16 NR 1: nt=4 +1.7% hot / +4.3% cold (2560->512), +1.3..1.4% (2560->10240); nt=3 ~0. matches the existing "wider raw-dot windows did not improve" note. reverted.
  • [[max_total_threads_per_threadgroup(SG * 32)]] on gemv: 0.0 +- 0.9% across nt 1/4 hot/cold. reverted.
  • for reference, single gemv #1 2560->512 Q8_0: 16,1,4,1 5.9 us hot / 12.0 cold, 32,1,4,4 7.1 / 11.7, 32,1,4,1 6.4 / 11.6, so a non-bitwise tile swap for the pred path wouldn't buy anything either.