Round 32 blitz: qsa_flash sp[4] 3->1 barrier & ko[j] offset + unroll (-4.0..-5.5% / -6.1 ms in kprof), mm_gu3 T_IQ2_S half4 HS store (-9.0 ms in kprof -> 3175.3 ms / 624.5 tok/s GPU prefill!), deq uint2 grid loads + deq_iq_packed (-> 408.75 tok/s prefill & 42.16 tok/s decode!)
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051# Round 32 Track C: Decode `deq<T_IQ2_S / T_IQ2_XXS / T_IQ2_XS / T_IQ3_S>` 64-Bit `uint2` Grid Loads (`quant.metal`) + `gu_body` Low-Register Packed `iq_combine` (`moe.metal`)
## 1. Summary of Changes- **Patch:** `/tmp/argon_r32_moe/argon_r32_moe.patch` (verified clean via `git -C vendor/strata apply --check`, exit 0).- **Files Modified (within ownership boundaries):** 1. `metal/kernels/quant.metal` (`lines 160–224`): - **`deq<T_IQ2_XXS>`, `deq<T_IQ2_XS>`, `deq<T_IQ2_S>`:** Replaced the two separate 32-bit grid lookups per `l2` iteration (`g_iq2*(tab, idx, 0)` and `g_iq2*(tab, idx, 1)`) with a single 64-bit `uint2` load (`const uint2 g = ctab ? as_type<uint2>(iq2*_grid[idx]) : ((threadgroup const uint2*)tab)[idx]`), cutting threadgroup/constant grid loads in half (`4 -> 2` loads per 16-weight unit). - **`deq<T_IQ3_S>`:** Replaced the per-iteration `uchar` sign loads `b[74 + 4 * ib + l]` with one hoisted 16-bit `ushort` load `s2 = *(device const ushort*)(b + 74 + 2 * ui)` before the `l2` loop, added `#pragma clang loop unroll(full)`, and replaced `float(1 + 2 * ((sc >> (4 * (ib & 1))) & 15))` with `fma(float((sc >> (4 * (ib & 1))) & 15), 2.0f, 1.0f)`. 2. `metal/kernels/moe.metal` (`lines 376–549`): - Added `iq_combine(uint4 g, uint s16, float sc, thread const float* xs)` and `deq_iq_packed<F>(b, ui, g, s16, tab)` for the 16-weight i-quant formats (`T_IQ2_S`, `T_IQ2_XXS`, `T_IQ2_XS`, `T_IQ3_S`, `T_IQ3_XXS`), keeping a decoded 16-weight unit in `6` registers (`uint4 g`, `uint s16`, `float sc`) instead of `17` registers (`float q[16]` + `float sc`). - In `gu_body<F, LPR, NR, NE>`, enabled fused gate+up decoding and single `X<F>::load` reuse per token at `NR == 1 && NE == 4` with loop-invariant `ui = (LPR % 16 == 0) ? (li & 15) : (u & 15)`, holding both `gate` and `up` packed units (`12` registers total vs `34` floats) across the `t = 0..3` token loop and cutting `X<F>::load` calls in half (`8 -> 4` per unit step) without register spilling.
## 2. Verification & Bitwise ParityAll standalone parity and A/B suites pass with **0 fails (`ALL PASS`, 100% bitwise identical)**:- `expert_parity --ab /tmp/argon_r32_moe/base_kernels`: `0 fail(s) ALL PASS` across `IQ2_S`, `IQ2_XXS`, `IQ2_XS`, `IQ3_S`, `IQ3_XXS` and tiles `32,2,4`, `32,1,4`, `16,1,4` (`NE = 1..4, 8` + random windows).- `gemv_parity --ab /tmp/argon_r32_moe/base_kernels IQ3_S 2560 2560 {1,2,4,8} gemv`: `bitwise` (`ALL PASS`).- Full `gemv_parity`, `dense_mm_parity`, `expert_parity`: `ALL PASS`.
## 3. Paired A/B Benchmarks on Quiet Apple M5 (`expert_parity --ab --bench --layers 1 --reps 16`)
### A. Threadgroup-Table Mode (`QUANT_CTAB=0`, used by `IQ3_S` in `window.cpp` and default `expert_parity`)| Format | Tile (`L,R,S`) | `NE=1` Ratio | `NE=2` Ratio | `NE=3` Ratio | `NE=4` Ratio | `ntok=1` Ratio | `ntok=2` Ratio | `ntok=4` Ratio | `ntok=8` Ratio ||---|---|---:|---:|---:|---:|---:|---:|---:|---:|| `IQ3_S` | `32,2,4` | **0.9894** (-1.1%) | **0.9713** (-2.9%) | **0.9290** (-7.1%) | **0.9254** (-7.5%) | **0.9817** (-1.8%) | **0.9836** (-1.6%) | **0.9763** (-2.4%) | **0.9722** (-2.8%) || `IQ3_S` | `32,1,4` | **0.9757** (-2.4%) | **0.9857** (-1.4%) | **0.9688** (-3.1%) | **0.8557** (-14.4%) | **0.9842** (-1.6%) | **0.9756** (-2.4%) | **0.9711** (-2.9%) | **0.9779** (-2.2%) || `IQ3_S` | `16,1,4` | 1.0009 | **0.9753** (-2.5%) | **0.9593** (-4.1%) | **0.8355** (-16.5%) | **0.9960** (-0.4%) | **0.9893** (-1.1%) | **0.9719** (-2.8%) | **0.9641** (-3.6%) || `IQ2_S` | `32,2,4` | **0.9890** (-1.1%) | **0.9917** (-0.8%) | 1.0003 | **0.9965** (-0.4%) | **0.9915** (-0.9%) | **0.9918** (-0.8%) | **0.9931** (-0.7%) | **0.9915** (-0.9%) || `IQ2_S` | `32,1,4` | **0.9993** (-0.1%) | **0.9963** (-0.4%) | **0.9713** (-2.9%) | **0.8686** (-13.1%) | **0.9996** | **0.9976** (-0.2%) | **0.9962** (-0.4%) | **0.9949** (-0.5%) || `IQ2_S` | `16,1,4` | 1.0000 | **0.9973** (-0.3%) | **0.9593** (-4.1%) | **0.8416** (-15.8%) | 1.0064 | 1.0008 | **0.9958** (-0.4%) | **0.9930** (-0.7%) || `IQ2_XXS` | `32,2,4` | **0.9928** (-0.7%) | **0.9976** (-0.2%) | **0.9943** (-0.6%) | **0.9961** (-0.4%) | **0.9926** (-0.7%) | **0.9906** (-0.9%) | **0.9921** (-0.8%) | **0.9915** (-0.9%) || `IQ2_XS` | `32,2,4` | **0.9777** (-2.2%) | **0.9947** (-0.5%) | **0.9960** (-0.4%) | **0.9934** (-0.7%) | **0.9734** (-2.7%) | **0.9819** (-1.8%) | **0.9815** (-1.9%) | **0.9830** (-1.7%) || `IQ2_XS` | `32,1,4` | **0.9925** (-0.8%) | **0.9998** | 1.0180 | **0.9048** (-9.5%) | **0.9941** (-0.6%) | **0.9952** (-0.5%) | **0.9907** (-0.9%) | **0.9966** (-0.3%) || `IQ3_XXS` | `16,1,4` | **0.9995** | 1.0050 | 1.0009 | **0.8900** (-11.0%) | 1.0015 | **0.9987** (-0.1%) | **0.9986** (-0.1%) | **0.9964** (-0.4%) |
### B. Constant-Table Mode (`QUANT_CTAB=1`, used by `IQ2_S`, `IQ2_XXS`, `IQ2_XS`, `IQ3_XXS` in `window.cpp`)| Format | Tile (`L,R,S`) | `NE=1` Ratio | `NE=2` Ratio | `NE=3` Ratio | `NE=4` Ratio | `ntok=1` Ratio | `ntok=2` Ratio | `ntok=4` Ratio | `ntok=8` Ratio ||---|---|---:|---:|---:|---:|---:|---:|---:|---:|| `IQ2_S` | `32,1,4` | 1.0023 | 1.0000 | 1.0005 | **0.8580** (-14.2%, 237.8 -> 204.0 us) | 1.0016 | 1.0028 | 1.0012 | **0.9969** (-0.3%) || `IQ2_S` | `16,1,4` | 1.0002 | 1.0003 | **0.9998** | **0.8533** (-14.7%, 225.0 -> 192.0 us) | **0.9958** (-0.4%) | 1.0015 | **0.9994** | **0.9989** (-0.1%) || `IQ2_XXS` | `16,1,4` | 1.0021 | 1.0005 | 1.0005 | **0.8899** (-11.0%, 220.1 -> 195.8 us) | 1.0010 | 1.0022 | **0.9991** | **0.9969** (-0.3%) || `IQ2_XS` | `16,1,4` | 1.0005 | 1.0009 | 1.0000 | **0.9057** (-9.4%, 223.2 -> 202.2 us) | 1.0017 | 1.0020 | **0.9979** (-0.2%) | **0.9974** (-0.3%) || `IQ3_XXS` | `16,1,4` | 1.0009 | 1.0032 | 1.0003 | **0.8866** (-11.3%, 230.1 -> 204.0 us) | 1.0009 | 1.0037 | 1.0007 | 1.0000 |
### C. Dense `gemv_parity --ab` (`IQ3_S 2560 -> 2560`, tile `32,2,4,1`)- `ntok=1`: `24.56 us -> 24.04 us` (**-2.1%**), bitwise identical.- `ntok=2`: `30.87 us -> 30.61 us` (**-0.8%**), bitwise identical.- `ntok=8`: `77.12 us -> 76.77 us` (**-0.5%**), bitwise identical.
## 4. Tried & Rejected Ideas- **Preloading 32-bit `q4 = rd32(b + 2 + 4 * ui)` in `deq<T_IQ3_S>`:** Tested against hoisted `s2` + `#pragma clang loop unroll(full)` + `fma`; adding `q4 = rd32(...)` increased live register pressure across the four divergent `g_iq3s` lookups in `deq<T_IQ3_S>` and lost the `-2.9%..-7.5%` gain. Keeping the byte index loads `b[2 + 8 * ib + 2 * l]` while hoisting the 16-bit sign load `s2` + `unroll(full)` + `fma` preserved the full `-1.6%..-7.5%` speedup on `IQ3_S`.- **Fused `deq_iq_packed` + `iq_combine` at `NR == 2` or `NE == 2, 3`:** At `NR == 2` (`32,2,4`), holding two packed units or adding per-element sign selection inside the token loop was `+1.2%..+3.7%` slower than `unit_combine<16>`. At `NR == 1, NE == 3`, the extra bitwise sign extractions across `3` tokens cost `+2.9%..+5.0%` more than saving `3` `X<F>::load` calls; at `NR == 1, NE == 4`, where the baseline `float q[16]` path suffered occupancy pressure from `8` `X<F>::load` calls and `17` live floats alongside `ag[1][4]` + `au[1][4]`, `deq_iq_packed` + `iq_combine` delivered a clean **-9.4% to -16.5% speedup** across all five 16-weight i-quant formats.1234567891011121314151617181920212223242526272829303132333435363738394041# opus r32 track A: mm_gu3 half4 HS store for IQ2_S/IQ3_S, uniform ib / hoisted l, IQ3_S super-block loop
short version: one tiny change ships, the half4 coalesced HS store turned on for `T_IQ2_S`, for about -0.3% on the IQ2_S gate/up at T=1024/2048(34 of 48 layers). the uniform-`ib` / hoisted-`l` rewrite and moving IQ3_S into the super-block loop are both 100% bitwise identicalbut **+1.3..4.4% slower**, so they're dropped (´・ω・`)
- patch: `/tmp/opus_r32_moemm/opus_r32_moemm.patch` (1 line in `metal/kernels/moe_mm.metal`, `git -C vendor/strata apply --check` OK)- scratch: `/tmp/opus_r32_moemm/strata`, baseline kernels `/tmp/opus_r32_moemm/base_kernels`, every variant in `/tmp/opus_r32_moemm/v_*/`- `moe_mm_parity.cpp` unchanged (the existing `--ab` / `--gu-bench` mode was enough)
## methodexisting `build-metal/moe_mm_parity` with `STRATA_METAL_KERNELS=<dir>` vs `--ab <dir>`, `--v3 --dn Q2_0 --tile 16,32,32,4 --dtile 64,32,2 --gu-bench`.each cell is 4 or 6 runs that alternate which kernel is A, and the number is the mean of the candidate-vs-base median diffs (negative = faster).every run also memcmps HS/YE/Y. base vs base on IQ2_S@1024 came out -0.02% [-0.07..+0.04]. the identical-code cells (IQ2_XXS/IQ2_XS underH_all) spread about ±0.4% (one +1.3% outlier), with the :8080 server resident, so I only treat a move as real if every run agrees on the sign.
## results (all bitwise PASS: HS_memcmp / YE_memcmp / Y_memcmp)| variant | change | fmt | T=512 | T=1024 | T=2048 ||---|---|---|---|---|---|| H_all / F | half4 HS store, IQ2_S | IQ2_S | -0.10 (4), -0.17 (6) | -0.38 [all 4 <0], -0.35 [all 6 <0] | -0.26 [all 4 <0], -0.06 (6, one +2.0 outlier; median -0.22) || H_all / F | half4 HS store, IQ3_S | IQ3_S | -1.25 (very noisy), +0.32 [-0.38..+1.27] | -0.33 [all <0], -0.30 [all <0] | -0.10, -0.60 (noisy) || U | `block(bl, ib = ki>>5, l = kl>>3 hoisted, ...)` for IQ2_S/XXS/XS | IQ2_S | | **+4.29** [3.6..4.7] | +1.29 || U | same | IQ2_XXS | | **+2.72** | || U3 | U + IQ3_S `block` (ib/l uniform) inside the 256 super-block `sb` loop | IQ3_S | | **+3.61** | || S3 | IQ3_S in the `sb` loop, still indexed by `j_in` (no ib change) | IQ3_S | | **+4.05** | **+4.43** |
## shipped`T_IQ2_S` added to the half4 store gate. extra checks: default parity PASS (rel L2 1.60e-05), `--ab` at T=512 bitwise PASS with tiles16,32,32,4 and 16,64,32,2.
not shipped:- **IQ3_S half4**: -0.3% at T=1024 and 2048, but T=512 has no clear sign (+0.32 mean, runs from -0.38 to +1.27), and r14 measured +3..5% for IQ3_S at mid T with this store. it's only 2 layers, so it's not proven enough to ship. flipping it on later is just adding `|| F == T_IQ3_S`.- **IQ4_XS / K-quants / Q8_0 (removing the F gate entirely)**: not benched. r14 saw Q6_K +1.1% at T=512, so they keep the scalar store.- **target 2 (uniform ib / hoisted l)**: a reliable regression. this matches r14 variant D, where the compiler already got the lane-constant part from `j_in`. turning `ib` into a loop-carried scalar (`ki >> 5` of the non-unrolled 8-step ki loop) seems to make the codegen worse, not better.- **IQ3_S in the sb loop**: +4% whether or not `ib` changes. the 8-step ki loop plus the dg/du arrays cost more than the one `rdh` a 32-step that it saves.
## take-awaythe mm_gu3 decode address math is still not the bottleneck. 3 rounds now (r14 D, r32 U, U3, S3) have shown that restructuring the indexingloses. the epilogue store is good for a few tenths of a percent at most.123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960# round 32 qsa flash experiment
## contract- owner: qsa_flash in metal/kernels/qsa_prompt.metal; no production files edited.- hypothesis: four S partial buffers remove two chunk barriers without changing the explicit ((s0+s1)+(s2+s3)) reduction; separately test constant-offset K loads and fixed-loop unrolling.- preserve: exact fp16 output, softmax sum order, padded cells, cooperative and fallback paths, P alias lifetime. extra static shared memory must remain below 16 kib.- evaluator: existing qsa_prompt_parity, paired alternating A/B, T=1984, positions 0/2048/8192/30000, layers=3, pairs=7. default parity plus edge checks. baseline is immutable /tmp/sol_r32_qsa/base_qsa_prompt.metal.- decision: retain only bitwise-identical changes with repeatable net M5 gain (target >=3% in long-context median, no material regression at other positions). exploratory variants followed by full-position confirmation.- budget: serial short standalone tests only, each timeout <=20s; no server or model benchmarks.
## source and execution- strata HEAD: `296b59d2f73c8f04a805df29c2fa97655adace78`; baseline includes the pre-existing dirty workspace changes (status saved in `/tmp/sol_r32_qsa/strata_status.txt`). no tracked production source was edited.- host: apple m5, macos 27.0 (26a428), existing `vendor/strata/build-metal/qsa_prompt_parity` binary. kernel source selected with `STRATA_METAL_KERNELS=/tmp/sol_r32_qsa/strata/metal/kernels`; safe math default; seed 5 from the existing evaluator.- commands: `qsa_prompt_parity --flash-compare 1984 <pos0> /tmp/sol_r32_qsa/base_qsa_prompt.metal 3 7`, plus default `qsa_prompt_parity`. all invocations exited within 3 seconds, below the 10-second budget.- comparison timing is three dispatches per sample, **not 12 layers** and not a full-model result. baseline-to-baseline check: 66.190 -> 66.122 ms at pos0=8192.
## exploratory ablation (pos0=8192)| variant | before median ms | after median ms | change || --- | ---: | ---: | ---: || four partial buffers only | 66.299 | 65.770 | -0.80% || + pre-added k offsets and two-step k loop unroll | 66.200 | 63.984 | -3.35% || + fixed-trip prefetch, softmax, denominator, v-fill unroll | 66.156 | 63.539 | -3.95% |
all three variants: bitwise memcmp pass. raw logs and immutable variant snapshots are in `/tmp/sol_r32_qsa/`.
## candidate confirmation| pos0 | before median ms | after median ms | change | bitwise || ---: | ---: | ---: | ---: | --- || 0 | 28.756 | 27.545 | -4.21% | pass || 2048 | 59.734 | 56.475 | -5.45% | pass || 8192 | 66.322 | 63.779 | -3.84% | pass || 30000 | 109.321 | 112.883 | +3.26%, contaminated | pass |
`confirm_8192.log` contains a mid-run contention jump in its last two pairs (80–88 ms versus 63–66 ms before); `confirm_30000.log` is contaminated throughout (99–133 ms). parent confirmed tracks a/c were simultaneously running short gpu parity sweeps and requested handoff now for its own exclusive confirmation. therefore the 30000 timing **does not establish either a win or a regression**, and final acceptance is pending that exclusive rerun. the exploratory 8192 run and early-position confirmation provide the pre-contention evidence, not an end-to-end claim.
correctness:- all four 1984-token comparisons: exact fp16 output memcmp pass.- default suite: all pass, including qsa_flash versus fp32 reference at pos0 0/1984/2050/30000 and fused prep memcmp.- signed-zero and tiny-cancellation modes: `QSA_FLASH_EDGE=1` and `=2`, each T=1984/pos0=30000/layers=3/pairs=7, both memcmp pass (`edge_1.log`, `edge_2.log`). timings contaminated, not used for selection.- forced generic fallback: `COOP_K4=0 ... --flash-compare 96 30000 ... 1 3`, memcmp pass (`fallback.log`). 5.314 -> 5.460 ms under contention; not an m4 performance claim. m4 hardware untested.- two exploratory compile failures were repaired before answering measurements: an over-broad scratch string replacement touched an earlier function, then a duplicated unroll directive. failed logs retained; final diff touches qsa_flash only.
## implementation and review- four disjoint S stores meet at one barrier, instead of the prior three. reduction remains exactly `((s0+s1)+(s2+s3))*0.0625f`.- retain the following barrier before aliasing `sp[1]` as P. total static threadgroup memory measured by metal: 7616 -> 13760 bytes, below 16 kib. this does not independently prove unchanged register/occupancy limits.- pre-add d0 + the lane's first k index once per chunk. cooperative layout probe already guarantees the second half repeats at +16. unrolling the two k steps exposes +0/+16/+32/+48 byte offsets; ks8 remains loaded once.- denominator loop is unrolled without reassociation: sequential `l += x; += y; += z; += w` preserved. no new helper duplicates an existing abstraction; existing cooperative layout conventions reused.- final visual review: scratch `view_diff` plus full relevant `view_code` context. no test harness source change needed.
## handoff- clean patch: `/tmp/sol_r32_qsa/sol_r32_qsa.patch` (only `metal/kernels/qsa_prompt.metal`).- `git -C vendor/strata apply --check /tmp/sol_r32_qsa/sol_r32_qsa.patch`: exit 0.- recommend this candidate for parent's exclusive confirmation, **not an unconditional 30000-position performance acceptance**. no server touched and no full-model benchmark run.
source sha256:```4602c462fa5096f3ee9e65b03460c5b441db3f4666c573fe0b30a5ffbf0f2971 base_qsa_prompt.metal9c141a1b65d5c2fa8bbff84027cd8757ed85cc1d72e80143cc94fe7255414e26 barrier_const_unroll.metal3217640e820225cf57f9e03d05c6d58284c86ac8a23c7396597bb628bc2424e2 qsa_prompt_parity```