6.9 kB 80 lines
0958c6b1
Round 29 blitz: mm_gu3 T_IQ2_S scale caching (-86.2 ms / -10% in kprof) & T_IQ2_XXS byte loads (-6.5%), moe_route e512/res_mask (-20.6..-26.4%) + deq<T_IQ2_S/XXS> (-1.1..-6.9%)
dawn
⋯
audits/opus_r29_prompt.md
sourcerendered
download raw history
at main Markdown

opus_r29_prompt — Round 29 Track A: gemm_swiglu_gu shared-expert gate+up fusion + route_topk unrolls #

patch: /tmp/opus_r29_prompt/opus_r29_prompt.patch (git -C vendor/strata apply --check OK). It touches metal/kernels/prompt.metal (route_topk, +3/−1 lines) and metal/tests/comb_fold_parity.cpp (new gu + topk modes). window.cpp is unchanged. The fusion doesn't win robustly (details below), so I didn't wire it in. Scratch tree + build: /tmp/opus_r29_prompt/strata, /tmp/opus_r29_prompt/build (run with STRATA_METAL_KERNELS=/tmp/opus_r29_prompt/strata/metal/kernels).

TL;DR #

  • gemm_swiglu_gu (two accumulators, one A tile) is bitwise identical but not faster. On average it ties gate gemm_f16 + gemm_swiglu_up (±2–3%, which is the noise floor here). It wins −3..−5% at T = 192/256, loses +10..+14% at T = 224/288, and loses +1..+3% at 384..1536 in the most recent repeated runs. The kernel is compute-bound on the matrix units, and the gsh round trip (the "503 MB") overlaps with that compute. Two TM×TN accumulators halve occupancy, which eats the saving. This matches Round 11's finding (audits/opus_r11_dense.md: 2-acc fused 1016 µs vs 976 µs unfused). Not shipped.
  • Interleaved single-accumulator variant gemm_swiglu_il (one GEMM over B = gate/up rows interleaved, row 2j = gate j, 2j+1 = up j; each M5 lane's 4 columns = (g,u) of two outputs, half2 store) is also bitwise identical. It is the only shape with a repeatable win: −3.0..−4.4% at T = 1024 (≈ −1.0 ms/chunk, 6 of 6 runs, tiles il128_64_4 / il128_128_8 / il64_256_8). It ties at 2048 (−0.8..+1.2%) and is +1..+3% at 512. Shipping it needs dq(Wshg)/dq(Wshu) writing one interleaved [1280, 2560] buffer (dq_f16's perm = 2 over the pair; window.cpp lines 3674-3676 + dense_mm.metal). That is outside my boundaries, so it stays in the test as a measured follow-up candidate.
  • route_topk: the #pragma clang loop unroll(full) additions are no-ops. With only the pragmas (lines 35, 41, 104, 106, 125) the A/B ratio is 1.000–1.002, so the compiler already fully unrolls these constant-trip loops. The real cost was v[ex / 32] = -INFINITY, a runtime index into float v[16]. Replacing it with an unrolled select v[i] = i == (ex >> 5) ? -INFINITY : v[i] gives −0.7..−1.2% (ratio 0.988–0.993 in 3 of 3 quiet runs, every T and MoCE setting). It is bitwise identical (ids + w, 512 and 256 experts, MoCE 0 / 320335 / −16.3 / −0.05). The absolute gain is tiny: route_topk is ~2.3 ms per 48 at T=2048, so ≈ 20 µs/chunk. That is the only shipping change. I left the pragmas out of the diff because they measured nothing.

Parity (all memcmp == 0, outputs pre-filled 0xCD) #

  • comb_fold_parity 0 gu: T ∈ {1, 7, 161, 256, 512, 1024, 1983, 2048} × 7 gu tiles + window path → 65 PASS
  • comb_fold_parity 0 gu check il64_128_4,il128_64_4,64_64_4 → 33 PASS (interleaved)
  • During exploration, also bitwise: threadgroup-parked silu(g) (one accumulator, 8–32 KB TG) and K-stepped TK=32 multiply_accumulate (two accumulators) at 3 tiles each, at every T.
  • comb_fold_parity 0 topk base_prompt.metal → 25 PASS; route_parity PASS (K=512 and K=256+MoCE)
  • regressions: comb_fold_parity 1024 21 PASS, 0 epi 113 PASS, 256 hcup 16 PASS, 0 ab base_prompt.metal 22 PASS
  • window.cpp variant (gu in the mm_hist wave next to Wsg, swiglu_h only when !swu) compiled clean in the scratch engine build before I reverted it. The full model was never run (rules).

Timings: 48 shared experts, ms per chunk, median of 9 alternating; Δ vs gate + gemm_swiglu_up (the window at T > 160) #

GPU shared with the live :8080 server, so absolute numbers drift up to 2×. Runs whose baseline inflated are discarded or marked.

T gate+swiglu_up best 2-acc gu (Δ, repeated runs) best interleaved il (Δ)
192 4.6–5.0 64_64_4: −3.1, −3.3, −4.8, −4.8 il64_128_4: −8.7, −7.3 (one run each way)
224 — 32_64_2: +1.3 median; 64_64_4: +9.9 —
256 5.9–6.4 64_64_4: −3.3, −5.5, −2.9, −5.0 il128_64_4: −8.3 median (noisy: −18.7..−2.5)
288 — 64_64_4: +13.9 median —
320 — 64_64_4: +2.8 median il32_64_2: +2.2
384 8.7–10.0 64_128_4: −1.4, −1.0, then +3.0, +3.0, +1.5, +3.4 —
512 11.6–12.9 64_128_4: −2.1, −1.6, then +2.5, +4.4, +2.6, +3.7 il128_64_4: +1.2, +1.8, +2.2
1024 24.2–27 64_128_4: ±1.5 (8 runs, median ~0) il128_64_4: −3.8, −4.4, −3.7, −5.2, −3.0, −3.0
1536 37–41 128_128_8 / 64_128_4: −1..+2 —
2048 50–58 64_128_4: −2.1, −1.7, −0.4, −2.4, +1.4, +1.1 il128_64_4: +0.5, +0.3, −0.8

Other shapes swept (all slower or tied): 32_64_2 / 32_128_4 / 64_64_4 at T ≥ 512 (+14..+97%), 128_128_4 (+20..+120%), 64_128_8 (+19..+48% at ≥ 512); TG-parked silu(g) (64_128_4 +0.7..+9%, smaller tiles same as 2-acc); K-stepped TK=32 (+9..+13% at 1024/2048), TK=64 at 64_128_4 (+20..+26%). At T ≤ 160 the overlapped gate ∥ up + swiglu_h beats every fused form (+11..+48%), so swu's 160 threshold stands.

Why the "503 MB" doesn't show up #

gsh is T·640·4 B written and read once per layer. At T=1024 that is 5.2 MB ≈ 40 µs of DRAM per layer, but it runs concurrently with a ~500 µs compute-bound GEMM. Removing a round trip only pays when the extra accumulator costs nothing. The single-accumulator interleave keeps occupancy, which is why only il shows a (small) T=1024 win. The "mixh read twice from DRAM" claim doesn't hold either: each A row-block is already re-read by NFF/TN = 5–10 N-tiles, so A is cache-resident either way.

Test usage #

  • comb_fold_parity T gu [bench [tile,...]]: tiles TM_TN_SG (two-accumulator gemm_swiglu_gu) or ilTM_TN_SG (interleaved). T 0 = the parity sweep. bench rows: gate ∥ up + swiglu_h, gate + gemm_swiglu_up, then each tile. Without bench, pass any word in place of bench to give a tile list (e.g. 0 gu check il128_64_4).
  • comb_fold_parity T topk baseline-prompt.metal: route_topk bitwise vs a baseline copy (512 and 256 experts, 4 MoCE settings), plus paired ×48 times at 512 experts. T 0 = 512, 1024, 2048.
  • Both fused kernels live in the test (kSwigluFused), not in prompt.metal: zero runtime-library cost, no dead code in the engine's kernels.

Follow-up (needs files outside this track) #

Interleaved shared gate/up: dq both into one [2·NFF, N] f16 buffer with rows interleaved (dense_mm.metal dq_f16 needs a row offset, or the two GGUF tensors dequantised as one perm = 2 pair), then dispatch gemm_swiglu_il (il128_64_4) at T ≥ ~768 in the mm_hist wave. Expected ≈ −1 ms/chunk at T=1024, ~0 at 2048. Worth it only if the 1024-row streaming chunk is the common case.