round 27 track b — qsa prep and merge #
decision #
ship parallel aligned float2 (m,l) loads, threadgroup-cached l, and lane-strided simd_max in qsa_merge2 and qsa_merge. keep the serial L and output FMA chains, all three merge barriers, and buffer layouts unchanged. qsa_merge2 is the production decode path.
do not ship raw-row single-barrier norm_rope fusion or reduction unroll: bitwise-correct, but confirmation found no useful speedup. qsa_score already fully unrolls its four-value dot products; qsa_attn2 already uses char4 key loads and fully unrolls the eight-value and six-head inner loops. neither changed without evidence.
artifacts and provenance #
- patch:
/tmp/sol_r27_qsa/sol_r27_qsa.patch - isolated candidate:
/tmp/sol_r27_qsa/strata/ - immutable baseline kernels:
/tmp/sol_r27_qsa/baseline/ - rejected fusion source:
/tmp/sol_r27_qsa/qsa-with-prep-fusion.metal - exact source hashes:
/tmp/sol_r27_qsa/manifest.json - vendor head:
296b59d2f73c8f04a805df29c2fa97655adace78(baseline kernel hashes, not head alone, identify measured working-tree inputs) - all raw logs:
/tmp/sol_r27_qsa/*.log
only the three owned files occur in the patch. no tracked vendor file edited, no server calls or full-model benchmarks. git diff --check and git -C vendor/strata apply --check /tmp/sol_r27_qsa/sol_r27_qsa.patch both exit 0.
evaluation #
apple m5; runtime library vendor/strata/build-metal/libmtl_rt.a; clang++ c++20 -O2. kernel runtime defaults retained. alternating A/B timing order, median nine paired rounds, batches of 32 existing bitwise dispatches or 16 new full-window dispatches. large prep uses one dispatch per batch. gpu clock/load clearly varied across invocations; use within-pair ratios, not absolute times across runs. concurrent live-server load was not controlled.
build:
clang++ -std=c++20 -O2 -Ivendor/strata/metal/src /tmp/sol_r27_qsa/strata/metal/tests/qsa_parity.cpp vendor/strata/build-metal/libmtl_rt.a -framework Metal -framework Foundation -o /tmp/sol_r27_qsa/qsa_parity
set STRATA_METAL_KERNELS=/tmp/sol_r27_qsa/strata/metal/kernels for every invocation. existing --bitwise /tmp/sol_r27_qsa/qsa.metal.baseline tests v1; add QSA_ATTN=2 and use qsa2.metal.baseline for v2. new --paired /tmp/sol_r27_qsa/baseline P --tokens N compares every buffer byte and times prep/score/topk/attn/merge separately. --prep-only avoids allocating enormous attention partial buffers for prefill; full-window tokens capped at 128, prep-only at 2048.
correctness #
- final v1: 40/40 random-KV
(position, ntok)cases, fullpartandxpmemcmp equal; positions 0,1,63,64,512,2047,2048,2050,8192,30000; tokens 1,2,4,8. - final v2: identical 40/40 matrix, full
partandxpmemcmp equal. - final full-window seeded-input tests:
(P,N,attn,mrope)=(0,1,1,0),(3,3,1,0),(2047,8,1,0),(2050,8,2,0),(3000,8,2,1). all 13 complete buffers pass:qcur qidx kq vq ksc vsc tail dead pooled scores ids part xp. identical nonzero seeded history for tail/dead/pooled, other outputs initialized identically. prior KV history is zero in this mode; existing bitwise mode supplies random KV codes/scales/selection instead. - prep-only T=2048 full-buffer comparison passes; the untouched attention buffers are trivially equal in this mode.
- all final commands exit 0; final bitwise invocations 1.70s/1.63s, final full-window invocations 0.13–0.61s, final prefill 0.25s.
bitwise claims cover tested finite production-style inputs and these specialization modes, not all possible NaNs or all function constants. no summation reassociation: L remains serial fma(l[k], ek[k], L) in ascending k; output chain unchanged. (m,l) alignment follows even QD=256, head stride 258 and chunk stride 6192. maxima are parallelized, sums are not.
measured merge wins #
final existing random-KV evaluator, gpu microseconds baseline / candidate:
| kernel | P | tokens | baseline | candidate | latency reduction |
|---|---|---|---|---|---|
| merge2 | 512 | 1 | 18.69 | 15.94 | 14.7% |
| merge2 | 512 | 4 | 9.45 | 8.12 | 14.1% |
| merge2 | 2048 | 1 | 12.28 | 9.87 | 19.6% |
| merge2 | 2048 | 4 | 50.52 | 46.22 | 8.5% |
| merge2 | 8192 | 1 | 12.25 | 9.82 | 19.8% |
| merge2 | 8192 | 4 | 50.53 | 46.58 | 7.8% |
| merge | 512 | 1 | 13.45 | 11.66 | 13.3% |
| merge | 512 | 4 | 13.90 | 11.70 | 15.8% |
| merge | 2048 | 1 | 7.73 | 5.89 | 23.8% |
| merge | 2048 | 4 | 24.88 | 23.41 | 5.9% |
| merge | 8192 | 1 | 7.57 | 5.87 | 22.5% |
| merge | 8192 | 4 | 24.91 | 23.34 | 6.3% |
initial independent random-KV runs showed the same direction. three additional merge2 full-window confirmations at P=2050,N=8 gave 101.081/90.672, 101.065/90.281, 100.320/90.599 us (~10%). small-context v1 tradeoff: P=3,N=3 final 10.703/10.948 us (~2.3% slower), P=0,N=1 7.023/7.125 (~1.5% slower). these are below the predeclared 3% effect threshold, not a universal speedup claim. score and attention are unchanged; apparent differences there are load/clock noise.
rejected prep fusion #
initial full-window T=2048 timing looked ~33% faster, but this did not survive isolated confirmation. three prep-only confirmations plain: 2919.667/2918.042, 3041.292/3052.083, 3045.125/3048.708 us. mrope: 1591.625/1597.875, 3055.542/3099.292, 2983.917/2993.292. neutral or slightly worse.
parent-requested decode confirmation at P=3000, two independent runs per token count:
| tokens | run 1 baseline/candidate us | run 2 baseline/candidate us |
|---|---|---|
| 1 | 6.880/6.893 | 6.875/6.922 |
| 2 | 8.448/8.508 | 8.477/8.523 |
| 4 | 12.190/12.143 | 12.164/12.156 |
| 8 | 20.018/19.872 | 20.018/19.901 |
no >3% benefit; baseline helper restored. the compiler may already eliminate some barrier/reduction overhead, or added raw-row/gamma loads may offset it; no assembly claim made.
budget incident / limits #
one early full-window T=2048 invocation took 6.11s, exceeding the requested 5s. the old partial allocation scales to multiple gigabytes at that size. parent was notified immediately; added prep-only mode and a full-window token cap. all subsequent invocations were below 5s. no full-model or :8080 use. no full-model end-to-end throughput claim.