Round 36 blitz: predL_/predL2_ token-0 fast path when PREFETCH_DRAFT<=0 (-16.0% hot / -6.8% cold), TM>32 row-tail trim in gemm_hc_down (-4.7% @ T=1025) & gemm_swiglu_up (-1.4% @ T=1025, -0.8% @ T=1987) -> mtp_ms 190.62ms, 42.26 tok/s decode
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950# Round 36 Track C: `qsa_prep` Single-Barrier `norm_rope` & `w == 128` `tg_sum` + `qsa_score_mm` Loop Unrolling
## Summary & Decision
- **Owner:** `argon-r36-qsa-prep-score-and-gdn`- **Owned files:** `metal/kernels/qsa.metal`, `metal/kernels/qsa2.metal`, `metal/kernels/qsa_prompt.metal`, `metal/tests/qsa_parity.cpp`, `metal/tests/qsa_prompt_parity.cpp`- **Kernel decision:** **No-ship on kernel changes (`qsa.metal`, `qsa2.metal`, `qsa_prompt.metal` unchanged)**; **ship paired A/B harness improvements (`--ab` in `qsa_parity.cpp` and `qsa_prompt_parity.cpp`)**. - All tested `qsa_prep` and `qsa_score_mm` kernel variants achieved **100% IEEE-754 bitwise parity (`memcmp == 0`)** across every output buffer (`qcur`, `qidx`, `kq`, `vq`, `ksc`, `vsc`, `tail`, `dead`, `pooled`, `scores`, `ids`, `part`, `xp`), including `QSA_MROPE=1` and `QSA_Q_F16_FC=true`. - However, on quiet Apple M5 (10-core GPU) in warmed 15-pair alternating A/B benchmarks, the current working-tree `qsa_prep` and `qsa_score_mm` kernels are already at their M5 optimum: 1. **Single-barrier `norm_rope` (`if (rope && tid < QROT) row[tid] = x;` before `tg_sum256` + partner scale `(rs * row[tid ^ 32]) * g[tid ^ 32]`):** Bitwise identical (`memcmp == 0`), but slightly slower across decode and prompt shapes (`ratio 1.0006–1.0136`, `+0.5%` to `+1.4%` at `T = 1, 1984, 2048`) because holding `x` live across `tg_sum256` + issuing a second uncoalesced/divergent `device const float* g` load (`g[tid ^ 32]`) for `tid < 64` outweighs the threadgroup barrier savings on M5. 2. **`tg_sum256` `#pragma clang loop unroll(full)` + `nsg = (w <= 128 ? 4 : 8)`:** Bitwise identical (`memcmp == 0`), neutral within `±0.2%` at `T = 1984, 2048` (`961.05 -> 962.95 us`, `ratio 1.0010` at `P=0, T=1984`; `1000.15 -> 1005.97 us`, `ratio 0.9994` at `P=3000, T=1984`; `1039.76 -> 1044.30 us`, `ratio 1.0004` at `P=3000, T=2048`), because Metal Clang already unrolls `nsg = 8` when constant-folded and `qsa_prep` is memory-bandwidth bound on `qfull` / `kcur` / `vcur` / `kq` / `vq`. 3. **`qsa_score_mm` `#pragma clang loop unroll(full)` + `qp` pre-offsetting (`qsa_prompt.metal`):** Bitwise identical (`memcmp == 0`), neutral (`0.1788 -> 0.1770 ms` at `pos0=0`, `0.4654 -> 0.4699 ms` at `pos0=2051`, `1.3466 -> 1.3326 ms` at `pos0=8192`, `4.4671 -> 4.4764 ms` at `pos0=30000` for `T = 1984`), confirming Round 26 (`audits/sol_r26_qsa.md`) that `qsa_score_mm` is dominated by the `64x64x128` `matmul2d` cooperative tensor op and DRAM reads/writes.
## Measured Quiet-GPU A/B Results
### 1. `qsa_prep` Variants (`qsa_parity --paired` / `--ab`, 15 alternating pairs of 8x/32x dispatches)
| Shape (`P`, `nt`) | Baseline (`us`) | V1: Single-Barrier `norm_rope` + `nsg=4` (`us` / ratio) | V2: Two-Barrier + `nsg=4` + `unroll` (`us` / ratio) | V3: `tg_sum256` `unroll` only (`us` / ratio) | Bitwise `memcmp` ||---|---:|---:|---:|---:|---|| `P=3000, nt=1` | `7.047` | `7.090` (`1.0136`) | `6.939` (`0.9881`) | `7.039` (`0.9952`) | `PASS` (all 13 buffers) || `P=3000, nt=2` | `2.720` | `2.730` (`1.0029`) | `2.699` (`0.9957`) | `2.329` (`0.9967`) | `PASS` (all 13 buffers) || `P=3000, nt=4` | `2.879` | `2.859` (`1.0004`) | `2.879` (`0.9982`) | `2.854` (`0.9959`) | `PASS` (all 13 buffers) || `P=3000, nt=8` | `4.611` | `4.598` (`0.9972`) | `4.581` (`0.9835`) | `4.579` (`0.9693`) | `PASS` (all 13 buffers) || `P=0, nt=1984` | `969.125` | `969.359` (`1.0006`) | `962.948` (`1.0010`) | `963.656` (`0.9987`) | `PASS` (all 13 buffers) || `P=3000, nt=1984` | `1001.609` | `1013.505` (`1.0071`) | `1005.969` (`0.9994`) | `1006.901` (`1.0026`) | `PASS` (all 13 buffers) || `P=3000, nt=2048` | `1038.339` | `1042.750` (`1.0051`) | `1044.297` (`1.0004`) | `1036.656` (`1.0036`) | `PASS` (all 13 buffers) |
### 2. `qsa_score_mm` Loop Unrolling + `qp` Pre-offsetting (`qsa_prompt_parity --mm-compare score`, 15 alternating pairs)
| `T` | `pos0` | Baseline (`ms`) | Candidate (`ms`) | Delta | `static_tg_mem` | Pre/Post `memcmp` ||---:|---:|---:|---:|---:|---:|---|| `1984` | `0` | `0.178750` | `0.176958` | `-1.00%` | `16384 -> 16384` | `PASS / PASS` || `1984` | `2051` | `0.465417` | `0.469875` | `+0.96%` | `16384 -> 16384` | `PASS / PASS` || `1984` | `8192` | `1.346583` | `1.332583` | `-1.04%` | `16384 -> 16384` | `PASS / PASS` || `1984` | `30000` | `4.467125` | `4.476417` | `+0.21%` | `16384 -> 16384` | `PASS / PASS` |
## Harness Additions Shipped in Patch
- **`metal/tests/qsa_parity.cpp`:** - Added `--ab REF_DIR [P]` mode that runs full-buffer bitwise parity (`memcmp == 0`) and warmed alternating paired timings across decode `nt = 1, 2, 4, 8` and prompt `nt = 1984, 2048` in a single `< 6s` invocation. - Switched `--paired` timing loops to use `cx.run({ds[v][i]}, repeats)` with `cx.warm(...)` and removes per-turn CPU `memset` of GPU buffers inside the timing loop.- **`metal/tests/qsa_prompt_parity.cpp`:** - Added `--ab T pos0 baseline_qsa_prompt.metal baseline_qsa.metal pairs` mode testing `qsa_prep`, `qsa_score_mm`, and `combined` (`qsa_prep + qsa_score_mm`) with full 10-buffer bitwise `memcmp` verification (supporting `QSA_MROPE=1` and `QSA_QF16=1`) and 12-layer-per-command-buffer paired A/B timings. - Added `cx.warm(...)` before `--mm-compare` paired timing loops.
## Verification Artifacts
- Unified patch: `/tmp/argon_r36_qsa/argon_r36_qsa.patch`- `git -C vendor/strata apply --check /tmp/argon_r36_qsa/argon_r36_qsa.patch`: exit `0` (clean).- Baseline kernel snapshots preserved in `/tmp/argon_r36_qsa/base_kernels/`.123456789101112131415161718192021222324252627282930313233343536373839# opus r36 track A: decode predL_/predL2_ token-0 fast path + gemv micro-opts
patch: `/tmp/opus_r36_decode/opus_r36_decode.patch` (`git -C vendor/strata apply --check` clean against the current working tree)scratch: `/tmp/opus_r36_decode/strata` (build in `build-metal`, METAL_CPP_DIR set; `window.cpp` + `strata_window_cli` build clean, no new warnings)
## what changed- `window.cpp` `Window::layer` pred block: when `STRATA_METAL_PREFETCH_DRAFT <= 0` (read the same way as stream.cpp `env_int`, default 1) the `predL_` / `predL2_` gemvs run at `ntok = 1` on the window's tile (`tile_ntok = ntok`). otherwise unchanged. verified readers: `ExpertStream::ahead` is the only consumer of the float pred rows (via `at_layer` / `after_layer`), and with `k_draft_max <= 0` it skips every `t > 0`. prompt chains pass `pred = nullptr` (cut_ids_), so they're unaffected. live server (pid 78503) runs with `STRATA_METAL_PREFETCH_DRAFT=0`, so this path is on in production. rows t >= 1 of predL_/predL2_ now hold stale data; nothing reads them in this mode.- `Window::gemv`: new trailing `int tile_ntok = 0` (0 = use ntok). **NOTE: this touches the declaration at window.cpp:320** (outside the listed line ranges, but the default parameter can't be added any other way).- prewarm: no change needed, `prewarm_pipelines` runs `layer()` for t = 1..max_t_, so it makes `gemv_8_16_1_4_1#1`.- `gemv_parity --pred TYPE NIN NOUT NTOK L,R,S`: new mode. token 0 of 2x gemv#NTOK vs 2x gemv#1 vs 1x gemvc#1, memcmp, then 3-way rotated paired timings (GEMV_AB_COLD=1 for DRAM weights).- `gemv.metal`: **no change shipped** (see below).
## measurements (M5, live server running, so a few runs were noisy outliers; the 6x repeats below were stable)parity: token 0 bitwise for nt = 2, 3, 4 (all three variants). `gemv_parity` default: ALL PASS (7.9 s).
Q8_0 2560 -> 512+512, tile 16,1,4,1, median of 15 rotated rounds, 6 invocations each:| | 2 gemv #4 (before) | 2 gemv #1 (shipped) | gemvc #1 ||-|-|-|-|| hot | 21.64-21.66 us | 18.09-18.14 us (-16.3%) | 18.47-18.54 us (-14.5%) || cold | 24.07-24.32 us | 22.72-23.13 us (-5.6%) | 22.82-23.15 us (-5.2%) |
nt=3: hot -3.1%, cold ~0. nt=2: hot +1.3%, cold -0.8% (#2 already takes the raw-x direct path, so it's a wash).gemv_cat at #1 loses to the concurrent pair in every config, so it's NOT wired.expected saving per 4-token verify window: ~46 x (1.4 .. 3.5) us = ~65-160 us of GPU time, before overlap with therouter GEMV they sit beside. small but free; not measured end-to-end (no full-model runs per rules).
## gemv.metal experiments (rejected, all bitwise, `--ab` vs pristine kernels)- `gemv_direct_x` widened to NTOK <= 4 at LPR 16 NR 1: nt=4 +1.7% hot / +4.3% cold (2560->512), +1.3..1.4% (2560->10240); nt=3 ~0. matches the existing "wider raw-dot windows did not improve" note. reverted.- `[[max_total_threads_per_threadgroup(SG * 32)]]` on gemv: 0.0 +- 0.9% across nt 1/4 hot/cold. reverted.- for reference, single gemv #1 2560->512 Q8_0: 16,1,4,1 5.9 us hot / 12.0 cold, 32,1,4,4 7.1 / 11.7, 32,1,4,1 6.4 / 11.6, so a non-bitwise tile swap for the pred path wouldn't buy anything either.12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879# round 36 track b: bounded prompt row-tail trimming
## decision
ship `/tmp/sol_r36_prompt/sol_r36_prompt.patch`: only `metal/kernels/prompt.metal` and `metal/tests/comb_fold_parity.cpp` change. **dense_mm.metal and window.cpp remain byte-identical.** the patch passes `git -C vendor/strata apply --check`.
- trim 64-row parents to 16/32 rows in hc-down, both the ordinary column tile and its 96-column tail. factor the row selection into `hc_down_rows`.- factor swiglu's existing accumulator epilogue into `swiglu_up_tile`; trim 64-row parents to 16/32 rows, retaining original grid offsets and SG count.- deliberately leave 32-row parents untrimmed. the initial broad variant's small wins did not survive confirmation; hc-down SG2 showed a repeatable ~0.5–1.5% full-tile penalty. limiting the descriptors to parents larger than 32 avoids it. small-T 289/297 are therefore not claimed wins.- keep swiglu's current 64_128_4 selection. test-only 32_128_2 was slower in every requested gate+up chain sample.- reject dense row trimming: correctness passed, but the same-tile source comparison exposed a large Wout tail regression that the existing tile-policy `dense_mm_parity --ab` cannot detect.
## setup and contract
apple M5, local standalone Metal tests only. upstream git HEAD at snapshot: `296b59d2f73c8f04a805df29c2fa97655adace78`; working-tree sources, not merely HEAD, are snapshotted in `/tmp/sol_r36_prompt/baseline-{prompt,dense_mm}.metal`. final source snapshot: `/tmp/sol_r36_prompt/final-source/`; hashes: `sha256.json`. all implementation/build activity stayed under `/tmp/sol_r36_prompt/strata` and `/tmp/sol_r36_prompt/build`. no tracked strata file, live server, port 8080, or full model was touched; no subagents spawned.
initial question/decision rule: `/tmp/sol_r36_prompt/plan.md`. require exact memcmp parity, repeatable >1% improvement to select a new tile policy, and no >1% repeatable full-tile regression. existing source A/B uses 12 paired samples × 8 dispatches, alternating order and rotating output-buffer placement. brackets in logs are the sorted sample ratios at indices 2 and 9, not confidence intervals. GPU tests were serialized; every invocation completed under 10 seconds (largest observed requested tile-chain sweep: 6.60 s). builds run separately. clocks/temperature were not externally locked; noisy exploratory measurements are retained rather than presented as precise end-to-end predictions.
build:
```textcmake -S /tmp/sol_r36_prompt/strata/metal -B /tmp/sol_r36_prompt/build -G Ninja -DCMAKE_BUILD_TYPE=Releasecmake --build /tmp/sol_r36_prompt/build --target comb_fold_parity dense_mm_parity -j 4```
## final prompt measurements
ratio = candidate / baseline; lower is faster. `final-comb-*.log` is the final-source run; `bounded-{1,2}-*.log` confirms the same bounded-descriptor variant before the final cosmetic regrouping.
| T | hc-down ratio | swiglu ratio | interpretation ||---:|---:|---:|---|| 289 | 0.9994 | 1.0039 | TM32 unchanged; noise || 297 | 0.9963 | 1.0004 | TM32 unchanged; noise || 306 | 0.9986 | 1.0039 | no eligible tail || 512 | 1.0004 | 1.0002 | full tiles || 1024 | 1.0013 | 0.9989 | full tiles || 1025 | **0.9536** | **0.9861** | 1-row tail; confirmed || 1041 | **0.9669** | **0.9894** | 17-row tail; confirmed || 1983 | 0.9859 | 0.9995 | hc-down TM32 unchanged; do not attribute its fluctuation to trimming || 1987 | 0.9909 | 0.9937 | swiglu 3-row tail; hc-down TM32 unchanged || 2048 | 0.9932 | 1.0011 | full tiles |
at 1025/1041, the two earlier confirmation runs gave hc-down 0.9542/0.9539 and 0.9661/0.9651; swiglu 0.9887/0.9832 and 0.9875/0.9881. the smaller swiglu 1987 benefit was usually ~0.5–1.0%, but one late noisy confirmation was flat; do not sell that as a large speedup. no full-model latency estimate is claimed.
## swiglu tile policy experiment
`comb_fold_parity T epi bench 32_128_2,64_128_4`, requested T values; `tiles-T.log`. values are ms for 48 gate+fused-up chains, not the up kernel alone. both candidates use the same gate dispatch in each process.
| T | 32_128_2 | 64_128_4 | SG2 / SG4 ||---:|---:|---:|---:|| 384 | 8.784 | 8.721 | 1.0072 || 512 | 11.623 | 11.576 | 1.0041 || 1024 | 24.332 | 24.288 | 1.0018 || 1983 | 48.320 | 47.927 | 1.0082 || 1987 | 50.424 | 50.145 | 1.0056 || 2048 | 61.928 | 60.397 | 1.0253 |
no selection change is justified. these exploratory runs used the broad row-trimming candidate, which preserves the same TM64 behavior as final; their TM32 candidate had the extra tail descriptor. full-T 384/512/1024/2048 still reject SG2 without depending on a partial row tile.
## dense negative result, and why the new harness exists
`dense_mm_parity --ab` compares r34-vs-current **tile policies using one shader revision**. it is not source A/B. both policies can share a regression and still print “1.00x; ALL BITWISE”. the new `comb_fold_parity T denseab baseline-dense_mm.metal [shape]` compiles both source revisions and compares exactly the same tile, inputs, and rotating output buffers across revisions. it uses the corrected window tile policy, supports all 13 dense shapes, and fails for an unknown requested shape or T outside 1..2048.
broad dense candidate: factor gemm_f16_tile, retain SG, add TM16/TM32 parent-tail dispatch. 13 shapes × eight requested M values all passed memcmp; logs `dense-source-ab-M.log`. but Wout (N=2560, K=6144, tile64_128_4) at M1987 gave ratio 1.6845 in the exploratory sweep, then **1.7014 and 1.7785** on isolated confirmation, while candidate-self source A/B was **1.0003**. even TM32-only trimming regressed 1.6347. see `dense-Wout-confirm-1987.log`, `dense-Wout-confirm2.log`, `dense-Wout-self.log`, `dense-Wout-32only-1987.log`.
this is consistent with a descriptor/resource or tail-path performance cliff, but register spilling itself was not measured. a late reverse-direction check (original as current, rejected candidate as baseline) showed only 0.9209, so the precise magnitude depends on run conditions; it still did not rescue the candidate. artifact `candidate-dense-v1.metal` retains the rejected implementation. dense is entirely excluded from the deliverable patch, as parent requested.
## verification
final-source tests:
- `comb_fold_parity 0 epi`: **ALL PASS**, 18 token counts × seven tiles × both epilogues = 252 bitwise output checks. includes 16/17/32/33 boundary rows, 289/297/306, 1025/1041 and 1987. `final-comb-0.log`.- `comb_fold_parity T ab /tmp/sol_r36_prompt/baseline-prompt.metal gemm`, T=289,297,306,512,1024,1025,1041,1983,1987,2048: **ALL PASS**; includes hc-up as an unchanged control. `final-comb-T.log`.- `dense_mm_parity`: **ALL PASS** (float-reference numerical tolerance for GEMM, exact dequant/operand conversion). `final-dense-default.log`. do not mislabel its float-reference check as bitwise GEMM parity.- `DENSE_ROUNDS=3 dense_mm_parity --ab M`, M=289,297,306,512,1024,1983,1987,2048: **ALL BITWISE** across all 13 shapes. `final-dense-policy-ab-M.log`. these final runs use the unchanged dense shader.- source A/B harness rejected unknown shape with exit 1 (`final-dense-unknown-shape.log`); original/candidate reverse check remained bitwise correct (`final-dense-rejected-reverse.log`).- final visual diff/context review completed, `git diff --check` passes, and patch application check passes against the actual vendor checkout.
all raw evidence, baseline and rejected source snapshots, build products and the patch are in `/tmp/sol_r36_prompt/`. only this report is written into the main checkout.