round 29 track b: paired moe gate/up decode #

result #

• patch: /tmp/sol_r29_moemm/sol_r29_moemm.patch (sha256 0504a66525ab48e1a297ec9d4b7c39018de9f43c5fe4ccee566f1fa224c09645). git -C vendor/strata apply --check exits 0.
• isolated source: /tmp/sol_r29_moemm/strata; baseline kernels: /tmp/sol_r29_moemm/baseline. vendor head at snapshot: 296b59d2f73c8f04a805df29c2fa97655adace78; snapshot includes its then-current working files, not just head.
• all four requested formats at all three requested token counts passed HS_memcmp=PASS YE_memcmp=PASS Y_memcmp=PASS on final source, exit 0.
• all invocations were individually under 10 seconds (final maximum 9.768 seconds). no server calls or full-model benchmarks.
• iq2_s scale caching has a reproducible positive signal (~2–5% gate/up latency reduction). iq2_xxs direct byte loads have a ~2–3% preliminary signal. iq3_s fma and table trim alone did not establish a speed gain.
• later runs became heavily contended/noisy (unchanged iq2_xs also varied dramatically). do not interpret final noisy non-iq2_s medians as clean regressions or wins; confirm in a quiet gpu window before promoting.

implementation #

only owned lines/files changed:

1. iq2_s table: 2080 -> 2048 uints. loader writes exactly 2048, and index max is 2047. source-level static allocation drops from 16512 to 16384 bytes for the selected tile; this alone did not measurably change time.
2. iq2_s paired decoder now has an in-block helper accepting the already-loaded scale. mm_gu3 caches gate/up row scales once per 256-weight block and reuses them across eight k32 matmuls, preserving accumulation order. cache is TN * TK / 256 floats per operand/lane (four floats each for the target tile), no new threadgroup storage. existing wrapper retains full-row semantics for other callers. non-iq2_s keeps a single-iteration inner loop.
3. iq2_xxs: replace 32-bit grid-index read plus variable byte shifts with two byte reads. no changes to sign or scale math.
4. iq3_s: integer odd-scale construction -> equivalent fma.
5. standalone test: shared bitwise comparator used in both parity and isolated a/b bench paths; supports the requested -T alias and four-number down tile, retaining existing three-number form. mismatched down TM fails pre-gpu with exit 2 (tested). a/b timing remains the original alternating 19 paired samples and original filtering.

evaluator / command #

apple m5, existing libmtl_rt.a, clang++ c++20 -O2. standalone executable /tmp/sol_r29_moemm/moe_mm_parity built from scratch test source linked to vendor/strata/build-metal/libmtl_rt.a and metal/foundation frameworks. runtime uses the scratch kernels via STRATA_METAL_KERNELS:

STRATA_METAL_KERNELS=/tmp/sol_r29_moemm/strata/metal/kernels \
/tmp/sol_r29_moemm/moe_mm_parity --v3 --gu FORMAT --dn Q2_0 \
  --tile 16,32,32,4 --dtile 16,64,32,2 -T TOKENS \
  --ab /tmp/sol_r29_moemm/baseline --gu-bench

formats: IQ2_S / IQ2_XXS / IQ3_S / IQ2_XS; tokens: 512 / 1024 / 2048. seeded existing data generator; 512 experts. existing runtime fast compilation supplies MM_V3_ONLY. a = candidate, b = baseline. positive printed diff is (b-a)/a, not latency reduction (b-a)/b.

earlier low-noise evidence #

logs/v1_*: table trim + iq2_xxs byte loads + iq3_s fma, before scale caching:

logs/v2_IQ2_S_*: cached-scale source with a redundant min-bounded loop, subsequently simplified:

an independent scale-cache 1024 run (v2fix_IQ2_S_1024.txt) gave 15992.5 / 16870.9 us (5.20% reduction). the min-bounded loop unexpectedly regressed iq3_s (~8–25%); replacing it with a fixed compile-time trip count removed that in v3_IQ3_S_1024.txt: 16860.9 / 16775.4 us (-0.51%, noise). this correction is in the patch.

final exact-source matrix #

all final logs: /tmp/sol_r29_moemm/logs/final_FORMAT_T.txt. all 12 have 3/3 memcmp passes. timing below is raw, including severe contention later in the matrix:

final iq2_s relative reductions: 2.61%, 3.84%, 2.76%. later samples show multi-fold slowdowns, including the unchanged iq2_xs control. raw medians must not be cherry-picked into claims.

limitations / next check #

bitwise correctness is established for requested formats/tile/token counts against frozen baseline, not an exhaustive format/tile proof. no full-model throughput estimate is claimed. quiet confirmation is needed for the final non-iq2_s timing matrix. source reviews used view_diff and view_code; no tracked vendor files were modified.