7.3 kB 77 lines
69531aa5
Round 26 blitz: qsa_attn_mm 32-bit ko/vo + float4 loads (-23.3..-29.9%), gemv IQ4_XS packed_uint4 5-reg codes (-1.5..-5% cold DRAM), DW<IQ4_XS/IQ4_NL/Q6_K> unroll + MM_V3_ONLY fast compile
dawn
⋯
audits/sol_r26_qsa.md
sourcerendered
download raw history
at main Markdown

round 26 track b: qsa prompt mm #

contract (before measurement) #

owner: metal/kernels/qsa_prompt.metal; only attention's cooperative operand address preparation, softmax loop hints/output copy, and score loop hints change. no dispatch/layout/interface changes; fallback cooperative layout must remain valid. byte offsets are uint32, matching the existing cache addressing bound.

baseline: working-tree snapshot of the owned kernel at vendor commit 296b59d2f73c8f04a805df29c2fa97655adace78, saved at /tmp/sol_r26_qsa/baseline_qsa_prompt.metal (the working tree already contains prior-round changes).

question: do 32-bit operand offsets, fixed-loop unroll hints and vector threadgroup output loads reduce m5 qsa mm latency without changing any output byte?

evaluator: existing qsa_prompt_parity --mm-compare, deterministic rng seed 5, alternating paired dispatch order, full output buffer memcmp (including poisoned unwritten regions). primary matrix: t=96,1984 and pos0=0,2048,8192,30000. screen using 5 pairs, confirm using 9 pairs; reject bitwise changes or repeatable regressions. require roughly >=2% repeatable long-prompt improvement to ship a kernel change; otherwise preserve a negative/inconclusive result. ablate vector load and score hints if combined result loses. no full-model runs; no server requests; each test timeout 20s for cold compile, target <10s elapsed.

artifacts and immutable variant snapshots live in /tmp/sol_r26_qsa/; gpu tests run sequentially to avoid self-contention. budget: at most 60 short invocations plus 2 compile repairs.

decision #

ship attention offsets + fixed-loop hints + vector threadgroup loads. do not ship score unroll hints: bitwise pass, but repeated paired results did not establish a reliable gain. qsa_score_mm is unchanged in the patch. qsa_flash and all other kernels are unchanged.

  • patch: /tmp/sol_r26_qsa/sol_r26_qsa.patch (two owned files only).
  • git -C vendor/strata apply --check /tmp/sol_r26_qsa/sol_r26_qsa.patch: exit 0.
  • no tracked vendor files edited; no requests to :8080; no full-model runs.
  • existing --mm-compare reused; harness now checks post-timing partials and merged/prepared activation buffers bitwise, in addition to the pre-timing whole partial-buffer memcmp.

measured confirmation #

apple m5; macos 27.0 build 26a428; cooperative layout probe true; safe math (no reassociation). timings below are medians of 9 paired alternating-order single dispatches, milliseconds. full partials, post-timing partials, and all merged/prepared output bytes passed memcmp == 0 for every row. tg memory unchanged at 7424 bytes.

t pos0 baseline ms candidate ms reduction
96 0 0.362500 0.291625 19.55%
96 2048 6.023500 4.092000 32.07%
96 8192 5.318875 3.293667 38.08%
96 30000 1.918875 1.493208 22.18%
1984 0 14.569125 11.676958 19.85%
1984 2048 30.884125 23.683833 23.31%
1984 8192 31.790833 22.275375 29.93%
1984 30000 37.322500 28.089667 24.74%

final_attn_*.log contains all samples. a final review rerun at t1984,pos8192 gave 31.815083 → 22.381875 ms (-29.65%), all three byte checks passed (final_review_attn_1984_8192.log). at 12 qsa layers this isolated-kernel delta is roughly 113 ms/chunk; this is not an end-to-end server claim.

short-case absolute clocks varied considerably across processes (the live server was left alone). long-case paired gains were stable across screening, ablation and confirmation. neither clocks nor competing external activity were controlled; use the paired within-process deltas rather than comparing unrelated absolute samples.

ablations / negatives #

  • offsets + all attention loop hints but scalar copy: long t1984,pos2048/8192/30000 medians 30.879→24.543 / 33.366→25.781 / 36.744→28.541 ms. (ablate_offsets_softmax_*.log)
  • vector load incremental against that variant, measured directly paired: 24.394500→23.880250 ms at pos2048 (-2.11%), 23.688458→22.506375 at pos8192 (-4.99%), 28.195500→27.637125 at pos30000 (-1.98%). t96,pos8192 2.514417→2.353750 (-6.39%). all bitwise checks passed. (vector_increment_*.log)
  • score hints: screen t1984,pos30000 initially looked faster (10.396→9.670 ms), but 15-pair confirmation gave 5.177625→5.169417 (-0.16%). pos2048 0.481500→0.478792 (-0.56%), pos8192 1.450417→1.449083 (-0.09%). t96 pos8192 regressed 0.077625→0.081000 (+4.35%). no robust benefit: rejected. (confirm_score_*.log; candidate source combined.metal / score.metal retained)
  • the literal proposed #pragma clang loop unroll(4) failed compilation: metal clang accepts unroll_count(4), used here. one compile repair; failed attempt preserved in screen_combined_attn_1984_8192.log.

parity coverage / limits #

  • primary 2×4 t/pos matrix above: exact pre/post partials and merged/prepared equality, including causal tail chunks, crossing the 2051-cell budget at pos2048, invalid implicit ids poisoned to -1, and untouched buffer regions poisoned 0xa5.
  • force generic cooperative decode (COOP_K4=0) at t96,pos0/8192/30000: all partial and merge/prepared byte checks pass on m5. this is not physical m4 validation.
  • final score+topk pipeline checks at t96,pos2048 and t1984,pos8192/30000: scores, ids and post-timing equality pass. stage code is unchanged in final candidate.
  • all successful benchmark invocations completed under 5s, within requested <10s limit; timeout 20s reserved for cold compilation. gpu runs were sequential; no helper agents spawned.
  • rng seed 5; no exhaustive adversarial float corpus or physical non-m5 testing. safe-math compilation and unaltered arithmetic sequencing preserved the tested bit patterns.

reproduce / preserved sources #

scratch tree: /tmp/sol_r26_qsa/strata; immutable source snapshots: baseline_qsa_prompt.metal, baseline_qsa.metal, offsets_softmax.metal, vector.metal, score.metal, combined.metal in the experiment directory. final kernel in scratch differs from vector.metal only by the alignment comment. baseline kernel sha256: c768c25c3186e0715a77ad2d9a58277871b4aab50d51fca0d4dd3fa42962ac6a.

build command (existing runtime library is unmodified):

clang++ -std=c++20 -O2 -I/tmp/sol_r26_qsa/strata/metal/src -I<workspace>/vendor/strata/third_party/ggml /tmp/sol_r26_qsa/strata/metal/tests/qsa_prompt_parity.cpp <workspace>/vendor/strata/build-metal/libmtl_rt.a -framework Metal -framework Foundation -o /tmp/sol_r26_qsa/qsa_prompt_parity

run each matrix row sequentially:

STRATA_METAL_KERNELS=/tmp/sol_r26_qsa/strata/metal/kernels /tmp/sol_r26_qsa/qsa_prompt_parity --mm-compare attn 1984 8192 /tmp/sol_r26_qsa/baseline_qsa_prompt.metal 9

replace attn with score to isolate score; --score-compare t pos baseline_prompt baseline_qsa pairs checks score/topk pipeline. final source sha256 values:

  • /tmp/sol_r26_qsa/strata/metal/kernels/qsa_prompt.metal: 4602c462fa5096f3ee9e65b03460c5b441db3f4666c573fe0b30a5ffbf0f2971
  • /tmp/sol_r26_qsa/strata/metal/tests/qsa_prompt_parity.cpp: 5bc787fecab37b0d7f8bb6695cf4957a89694650084ed509ceb81678f6f37d23
  • /tmp/sol_r26_qsa/sol_r26_qsa.patch: c7e42096f883395eb76aa8022711f1a48514d077d17a34c78f9c38b212ea726a