Round 31 blitz: mm_gu3 IQ2_S/IQ2_XXS/IQ2_XS/IQ3_S tab+hh & shift (+6.8..+8.2% / -57.9 ms in kprof), moe_down/gemv Q2_0R 0x6400 half-mantissa decode (-7.6..-17.4%), dq_f16 IQ4 tab init & perm=4 inverse contiguous write
12345678910111213141516171819202122232425262728293031323334353637# Round 31 Track C Audit: `dq_f16` (`dense_mm.metal`) & `DW<F>` (`mma.metal`)
## 1. Summary & Scope- **Owned files**: - `metal/kernels/dense_mm.metal` (`dq_f16` lines 28–66) - `metal/kernels/mma.metal` (`DW<T_IQ4_NL>`, `DW<T_IQ4_XS>` lines 26–69) - `metal/tests/dense_mm_parity.cpp` (`--dq-ab <base_dir> [type] [shape]` CLI support & uncontended pair filter)- **Patch**: `/tmp/argon_r31_dq/argon_r31_dq.patch` (verified via `git -C vendor/strata apply --check`)- **Parity**: 100% bitwise identical (`memcmp == 0`) across all 14 formats and all 13 shapes in `dense_mm_parity --dq-ab`, default `dense_mm_parity`, and `mma_parity`.
## 2. Key Findings & Shipped Optimizations
1. **DRAM Bandwidth Ceiling Diagnosis on Large Shapes (`2560x12288`, `2560x10240`, `2560x6144`, `6144x2560`)**: - On M5 (153.6 GB/s theoretical peak unified memory bandwidth), `dq_f16` on the large matrices already achieves **125–142 GB/s** (`81%–92%` of theoretical DRAM peak), dominated by writing the 62.9 MB (`2560x12288`) / 52.4 MB (`2560x10240`) FP16 output buffer `Wh`.
2. **Eliminating Divergent `constant`-Memory Table Init in `dq_f16<T_IQ4_XS / T_IQ4_NL>` (`dense_mm.metal`)**: - `load_tab<T_IQ4_NL>` previously executed a dynamic `for (int i = tid; i < 256; i += nth)` loop in every threadgroup (10,240+ threadgroups per matrix) with divergent `constant`-memory reads `kvalues_iq4nl[i & 15]` and `kvalues_iq4nl[i >> 4]`, which serialize across SIMD lanes on Apple GPUs. - Because `nth.x >= 64` for every `dq_f16` dispatch, threads `tid < 64` now populate all 256 `half2` entries of `tab` in 4 unrolled bank-conflict-free stores (`tab[tid + {0, 64, 128, 192}]`) directly from immediate register constants (`0xbfad9881u`, `0xf6eaddcfu`, `0x26190d01u`, `0x71594535u`), sharing the invariant low nibble `h0` across all 4 writes with zero `constant`-memory loads and zero loop overhead.
3. **Guarded `a.perm` & Inverse Row Mapping (`ro = gid.y`) for Sub-8-Bit Formats (`dense_mm.metal`)**: - Guarded the `a.perm` division/modulo inside `if (a.perm)` so `a.perm == 0` (~430 of 530 dispatches) executes zero integer divisions/modulos, and folded `k0 = 0` into `DW<F>::run(..., u, 0, w, p0, p1, tab)`. - For `a.perm != 0` (`320x10240p4`, 96 dispatches/chunk) on sub-8-bit formats (`fmt<F>::BB * 16 < fmt<F>::QK * 17`), inverted the bijection so output rows `ro = gid.y` are written contiguously in `Wh` and input row `r` is computed via shift/mask `(ro & 3u) * (a.n_out >> 2) + (ro >> 2)` when `a.perm == 4` (`1.03x–1.05x` speedup on `320x10240p4` for `IQ4_NL`, `Q4_0`, `Q5_0`, `Q5_1`).
4. **Decoupled Scale Load in `DW<T_IQ4_XS>` and `DW<T_IQ4_NL>` (`mma.metal`)**: - Unpacking the 32 `tab` values into `w[0..31]` before loading/applying the block scale `d` hides the device memory latency of the block scale header (`b[0..7]`) behind the threadgroup table lookups, matching the two-phase pattern of `DW<F>`.
## 3. Ablations Tested & Rejected (Not Shipped)- **Dedicated `DW<T_Q8_0>` & `DW<T_IQ3_S>` fusing `* d` into the unpack loop**: Slightly regressed `6144x2560` (`0.98x`) compared to generic `DW<F>` + `deq<F>`, because `deq<F>` already unrolls fixed trip counts (`16` and `2`) while deferring `rdh(b)` until after the integer unpack loop.- **`DW<T_Q2_0>` `0x6400` FP16 mantissa-OR trick & `rd32`**: Bitwise identical (`memcmp == 0`) once byte-masked (`q[g]`), but `0.99x` (`57.84 -> 58.15 ms`) vs `ushort` `p[0], p[1]` in `DW<T_Q2_0>` because `DW<T_Q2_0>` outputs `float w[16]` (requiring an extra `half`-to-`float` conversion per weight after the `half4` FMA).- **`packed_uint4` / `uint* p` in `DW<T_IQ4_XS>`**: `0.99x` vs `ushort* p` because wider 32/128-bit loads delay the first `tab` lookup and increase live register pressure.- **32x `simd_shuffle` nibble LUT in `DW<T_IQ4_XS / T_IQ4_NL>`**: `0.66x` on `2560x640` / `2560x512` (32 shuffles per thread vs 16 `half2` threadgroup loads).
## 4. Quiet-GPU Paired Benchmark Results (`dense_mm_parity --dq-ab`)- `IQ4_XS`: `58.83 -> 58.56 ms` (`memcmp=0` on all shapes; `6144x2560` `1.01x`, `2560x640` `1.01x`, `2560x512` `1.01x`, `2560x48` `1.03x`, `10240x4` `1.07x`, `2560x1` `1.03x`)- `IQ4_NL`: `64.70 -> 64.50 ms` (`memcmp=0` on all shapes; `320x10240p4` `44.9 -> 42.6 us` `1.05x`, `6144x2560` `1.01x`, `2560x640` `1.01x`, `640x2560` `1.01x`, `2560x512` `1.01x`, `2560x48` `1.05x`, `10240x4` `1.04x`, `2560x1` `1.06x`)- `IQ3_S`: `57.02 -> 56.92 ms` (`memcmp=0` on all shapes; `2560x12288` `610.0 -> 606.7 us` `1.01x`, `2560x128` `1.07x`, `2560x48` `1.02x`)- `320x10240p4` (`perm=4`): `Q4_0` `43.8 -> 42.4 us` (`1.03x`), `Q5_0` `46.2 -> 45.2 us` (`1.02x`), `Q5_1` `47.1 -> 46.5 us` (`1.01x`), `IQ4_NL` `44.9 -> 42.6 us` (`1.05x`), all `memcmp=0`.123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051# opus R31 Track A: `mm_gu3` IQ2_S / IQ2_XXS / IQ2_XS / IQ3_S decode arithmetic
Patch: `/tmp/opus_r31_moemm/opus_r31_moemm.patch` (only `metal/kernels/moe_mm.metal`; `git -C vendor/strata apply --check` passes).Scratch tree: `/tmp/opus_r31_moemm/strata`, build `/tmp/opus_r31_moemm/build`, baseline kernels `/tmp/opus_r31_moemm/base_kernels`.Exhaustive checker: `/tmp/opus_r31_moemm/chk.{metal,cpp}` (binary `/tmp/opus_r31_moemm/chk`).
## What ships (all integer and pointer rewrites, no FP change)1. **`W8<T_IQ2_S>::block`**: the lane's `hh = (kl >> 2) & 1` comes from `mm_gu3` (fixed for the whole K loop). The grid word is read as `(tab + hh)[2 * i]`, and the sign byte as `s >> (hh << 2)` instead of `hh ? s >> 4 : s`. Measured separately: the shift alone gives about +4.7%, and `tab + hh` adds about +3.5% on top.2. **`W8<T_IQ2_XXS>`**: there's a new `block(bl, k, d, tab, hh)`, and `mm_gu3` runs IQ2_XXS with the 256-wide super-block loop (cached `dg/du`) the way IQ2_S does, with the lane `hh` and `tab + hh`. The `sign4_iq2xs` select stays. `run()` is a thin wrapper, so the other callers don't change.3. **`W8<T_IQ2_XS>`**: same change as IQ2_XXS (`block` with cached d, lane hh, `tab + hh`).4. **`W8<T_IQ3_S>::run`**: `s >> (4 * hh)` instead of the select.5. `mm_gu3`: `constexpr bool sb = IQ2_S || IQ2_XXS || IQ2_XS` picks the 256 super-block loop. The QUANT_CTAB path is kept inline (`ctab ? grid[i] >> 32*hh : th[2*i]`).
## Results: `moe_mm_parity --v3 --tile 16,32,32,4 --dtile 16,64,32,2 --dn Q2_0 --ab base --gu-bench`diff = (base - new) / new, so a positive number means new is faster. Medians are from paired, interleaved runs. The machine was shared with the live server, so I give the quiet-run ranges.
| gate/up | T=512 | T=1024 | T=2048 | bitwise ||---|---|---|---|---|| IQ2_S | +6.6..+11.7% | +7.9..+8.9% | **+7.9..+8.8%** (25.1 ms vs 27.1 ms) | PASS || IQ2_XXS | +3.0..+5.9% | +4.4..+5.7% | **+5.8..+7.6%** | PASS || IQ3_S | +4.6..+5.4% | +2.4..+2.6% | +2.0..+3.0% | PASS || IQ2_XS | +3.5..+3.8% | +3.5..+5.4% | +6.6..+6.9% | PASS |
All of them hit `HS_memcmp=PASS YE_memcmp=PASS Y_memcmp=PASS`. I also checked 72 extra A/B parity runs: 4 formats × tiles {16,32,32,4 / 16,64,32,2 / 32,64,32,2 / 16,16,32,8 / 16,32,16,4 / 16,64,64,2} × {T=256, T=1000, T=512 with QUANT_CTAB=1}. All passed bitwise.
Estimated full-model kprof savings, extrapolated from the T=2048 standalone numbers (I didn't run a full-model check, per the rules): IQ2_S 763.8 ms × ~7.4% ≈ **-56 ms**, IQ2_XXS 220.1 × ~6% ≈ **-13 ms**, IQ3_S 46.7 × ~2.5% ≈ -1 ms. That's roughly **-70 ms total (about 2% of prefill GPU time)**.
## Exhaustive GPU checks (chk): 65,536 fp16 d × 16 nibbles × v in 1..63 × all 16 sign masks (1.06 G evaluations)- `(d*0.125f) * float(2n+1)` vs `d * fma(n, 0.25f, 0.125f)`: bit-identical for every d. IQ3_XXS `(d*0.25f)*float(2n+1)` vs `d*fma(n,.5,.25)` is also identical. This is expected: scaling by a power of 2 is exact, the product `v*sc` (≤ 6+5+11 bits) is exact in fp32, so there's a single rounding to fp16 either way.- These `w4_signed` rewrites are bit-identical to the original for every d, including NaN/Inf: the fp32 sign-XOR (`as_type<uint4>(float4(uchar4)) ^ (s << {31..28})`), and the fp16 0x6400 mantissa trick with sign XOR on the fp16 codes.- Sign-XOR on the **fp16 output** is identical for every finite d but differs on NaN-d sign bits, so it is not 100% bit-exact.
## Tried and rejected (all bitwise PASS, all slower; measured on IQ2_S T=2048 unless noted)| variant | result ||---|---|| `w4_signed` fp32 sign-XOR | -11.3% (IQ2_XXS -16%) || `w4_signed` fp16 0x6400 codes + sign-XOR | -12% (IQ2_XXS -15.5%) || `w4_signed` 0x6400 codes, keep select | -11.2% || `w4_signed` select on the scale (`s&1 ? -sc : sc`) | -3.4% (IQ2_XXS ±0) || scale as `fma(float(n), d/4, d/8)`, d/4 and d/8 cached | -9.5% (d/8 computed at use: -2.6%; with tab+hh: -13%) || pre-scaled `dg = d*0.125`, `sc = dg * float(((scb<<1)&30)|1)` | -1.5% vs base (about 9% behind the shipped one) || `tab + hh` with the select kept for the sign | -2.5% || index pre-doubled `(p0<<1) | ((qh<<(9-2l))&0x600)` | ±0 vs shipped || `l` taken from the lane as well | ±0 vs shipped (dropped) || IQ2_XXS `sign4_iq2xs` as a shift `(j|parity<<7) >> 4*hh` | -10% (also -8% with cached d + tab+hh) || IQ3_S cached d via the 256 loop + lane hh | +1.5% (the shift alone is better: +2.8%; lane hh without caching: same as shift alone) |
Takeaway: on the M5 the decode is ALU-issue bound (each extra op per W4 costs about 1%). `d * fma(float(nibble), 0.25f, 0.125f)` with constant operands, and `float(uchar)` plus the select-negate in `w4_signed`, are already the cheapest forms. The wins came from removing per-lookup integer work (`+ hh`, the `hh ?` select on the sign byte) and per-call scale loads. Selects are not uniformly bad, though: `sign4_iq2xs`'s select beats the shift form.
Not done: IQ3_XXS (not in this model's hot list). The non-coop_k4 (M4) path still calls `W4<F>::run`, which is unchanged; I couldn't run it on the M5.12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364# round 31 track b experiment plan
owner: q2 decode combine helpers and dn_body in isolated /tmp/sol_r31_moe/strata. no interfaces, summation ordering, format layout, other kernels or live server changes.
hypotheses: half-mantissa and float-mantissa decode remove integer conversion overhead; packed loads/pointer stepping reduce address overhead. each is compared against immutable base kernel snapshot, with expert_parity --ab exact memcmp over NE=1..4,8/random and paired dn timings; gemv uses its built-in ab check. baseline self-ab first. accept only reproducible >=2% improvement without material regressions (>2%) across required cases; repeat apparent winners three times. all gpu invocations serial, reps=4/layers=1 initially to stay <10s (20s timeout for cold compile), no full-model work. reject failed parity, stop after bounded variants if gains not established. live server noise is uncontrolled; report rather than overclaim.
## result: ship half-mantissa combine only
patch: `/tmp/sol_r31_moe/sol_r31_moe.patch` (git apply --check against vendor/strata: exit 0). changes only `q2_combine<T_Q2_0R>` in moe.metal and `gemv_q2_raw_combine` in gemv.metal. no test code or dn loop changes survive. neither tracked vendor source nor :8080 touched.
machine: apple m5. baseline vendor head: `296b59d2f73c8f04a805df29c2fa97655adace78`, **dirty working tree**, so commit alone is not the measured source. exact baseline kernels preserved at `/tmp/sol_r31_moe/base/`; exact candidate at `/tmp/sol_r31_moe/half/` and `/tmp/sol_r31_moe/strata/metal/kernels/`. source hashes in `/tmp/sol_r31_moe/source_hashes.json`. prebuilt parity binaries from vendor/strata/build-metal; no evaluator edits. default safe math.
### correctness
all expert invocations exited 0 and printed `0 fail(s) ALL PASS`; the harness compares full plan, h, ye, y via memcmp for same-expert ne=1,2,3,4,8 and random routing at ntok=1,2,3,4,8. candidate was checked repeatedly against the immutable baseline. host exhaustive check of all 65,536 packed ushort words, all eight code positions, also exactly reproduces code-1 via half-bit decode. summation/fma order is unchanged.
gemv: bitwise parity at 640→2560, ntok=1,2,4 for **both Q2_0 and Q2_0R (142)**. Q2_0R is essential: Q2_0 uses the prepared-input path and does not exercise the changed helper. the raw helper is used only for ntok=1; raw ntok=2,4 retain the decoded path. logs: `/tmp/sol_r31_moe/gem_q2_nt{1,2,4}.log`, `/tmp/sol_r31_moe/gem_half_confirm*_nt{1,2,4}.log`.
### paired expert timings
command (same for baseline and all candidates):
```STRATA_METAL_KERNELS=/tmp/sol_r31_moe/strata/metal/kernels \ vendor/strata/build-metal/expert_parity --gu IQ2_S --dn Q2_0 \ --ab /tmp/sol_r31_moe/base --bench --layers 1 --reps 4```
baseline self-ab ratios 0.9868–1.0116 (baseline.log). first clean half run 0.8299–0.9218 (half.log). three confirmation runs, seven alternating pairs per case:
| case | confirm 1 | confirm 2 | confirm 3 | median ratio ||---|---:|---:|---:|---:|| ne=1 | 0.9145 | 0.9007 | 0.9086 | 0.9086 || ne=2 | 0.8600 | 0.8506 | 0.8539 | 0.8539 || ne=3 | 0.8417 | 0.8250 | 0.8261 | 0.8261 || ne=4 | 0.8221 | 0.8271 | 0.8258 | 0.8258 || ntok=1 | 0.9059 | 0.9036 | 0.9155 | 0.9059 || ntok=2 | 0.9246 | 0.9154 | 0.9173 | 0.9173 || ntok=4 | 0.9277 | 0.9194 | 0.9236 | 0.9236 || ntok=8 | 0.8900 | 0.8865 | 0.8936 | 0.8900 |
this is **7.6–17.4% lower dn latency** by median case ratio, not an end-to-end throughput claim. example confirmation 3: ne=1 45.4→40.8 us; ne=4 98.4→81.2 us; mixed ntok=8 266.7→238.2 us. unchanged gu is a useful control (~1.00). logs half_confirm1/2/3.log. all gpu invocations serial in this session; other tracks and live server may overlap, so parent will do exclusive confirmation. reps/layers were intentionally kept small to honor <10s invocations.
### gemv timings
```STRATA_METAL_KERNELS=/tmp/sol_r31_moe/strata/metal/kernels \ vendor/strata/build-metal/gemv_parity --ab /tmp/sol_r31_moe/base Q2_0R 640 2560 1 gemv```
repeat with ntok 2/4 and with Q2_0. raw ntok=1 median paired ratios on confirmation repeats 2/3: **0.8754 / 0.8930** (12.5% / 10.7% lower). raw ntok=2/4 ratios ~0.995–0.999, as expected for untouched paths; Q2_0 ratios 0.9961–0.9999. first raw runs showed frequency ramping, recorded rather than discarded, and were followed by stable repeats. no material regressions observed.
### rejected alternatives / evidence
- packed_ushort2 down loads: bitwise pass, ratio 0.9970–1.0173, no established gain (`packed.log`).- pointer stepping + packed load: bitwise pass, ne=3 +2.6% regression, mixed cases -2.1..-4.2% (`pointer.log`). combined with winning half against half-only: ratio 0.9824–1.0124, no >=2% incremental improvement (`half_pointer_vs_half.log`). excluded.- unroll_count(5): bitwise pass, ne=2 +7.9%, ne=3 +6.0%, mixed ntok=8 +8.0% (`unroll.log`). excluded.- fp32 mantissa: bitwise pass but severe external gpu interference (unchanged gu and dn times ballooned) makes its timing **inconclusive**, not a valid performance rejection (`float.log`). do not ship because half has reproducible wins while fp32 does not have clean confirmation.
all standalone processes finished below 10s (slowest float run 7.84s); timeout=20 allowed cold compilation. no full model benchmark. experiments stopped after confirmed half winner; quiet confirmation belongs to parent.
### review
reviewed complete scratch git diff visually, searched existing half-mantissa idioms (matches moe_mm), checked field ordering x/y/z/w and exact fma sequence. the two small helpers stay local to their kernel translation units; moving them to units.metal would violate ownership and broaden the patch. no unrelated edits, added debug output, or hard-coded shape assumptions.