round 20 track b — gather and plan #
decision #
per the parent’s follow-up, the final proposed patch includes the verified mm_hist / mm_place changes, width2560 float/half gather unrolling, and the short parameterized evaluator. window.cpp remains unchanged at tg256; mm_scan and meta[8] remain unchanged. the parent identified the late timing spikes as concurrent track a/c parity runs.
performance caveat: the copy did not meet this experiment’s original repeatable >5% threshold. tg256 remained as fast as or faster than 32/64/160 in the relatively stable sweeps. plan improvements were 3.5–7.8% in selected quiet measurements (including 12.224 -> 11.276 us at t2048), but other repetitions regressed. this is a correctness-verified candidate with small/noisy performance changes, not a robust large speedup claim.
- proposed patch:
/tmp/sol_r20_gather/sol_r20_gather.patch - kernel-only experimental patch (now also included in the main patch):
/tmp/sol_r20_gather/experimental-not-recommended.patch - isolated source/build:
/tmp/sol_r20_gather/strata/ - measured candidate:
/tmp/sol_r20_gather/variants/all-fastpaths.metal - baseline snapshot:
/tmp/sol_r20_gather/baseline/
both patches passed git -C vendor/strata apply --check. no tracked vendor files were edited. no :8080 requests, full-model runs, or child agents.
contract and setup #
owner: the existing plan kernels own deterministic sorting/offsets; gather owns activation conversion/copy; the engine only selects threadgroup size. preserve meta[8] = all_rows, padding, buffer layouts, entry order, and generic-width/expert-count fallbacks. plan: /tmp/sol_r20_gather/plan.md.
apple m5. baseline was the dirty vendor working-tree snapshot, not pristine upstream HEAD; vendor HEAD was 296b59d2f73c8f04a805df29c2fa97655adace78. source hashes and artifacts are in /tmp/sol_r20_gather/manifest.json. the live server was left alone. other concurrent GPU activity was not controlled; later measurements became severely unstable. the parent subsequently confirmed track a/c standalone benchmarks overlapped those runs, so those samples cannot support a speedup claim.
candidate tested all requested specializations together:
- histogram single conditional initialization/readback for <=1024 experts;
- place two explicit initialization stores at 512 experts, full unroll of the 32-lane broadcast scan;
- float gather ten unrolled iterations at width2560;
- half gather unrolled 10/5/2 stores at tg32/64/160, explicit 256+64 copy at tg256.
threadgroup sweeps ran through the same mtl::Dispatch interface in the standalone evaluator; the engine's tg256 was not changed merely to benchmark it. mm_scan and indirect row count were unchanged.
measurements #
three paired samples per cell, eight dispatch repetitions per sample, median GPU us/layer; 20ms warmup per gather pair in confirmation sweeps. baseline/candidate gather order alternates. seed7, topk10, 512 experts, tile16. measured snapshots were left untouched during runs. logs retain every attempted run, including contaminated runs.
a relatively stable repeated sweep (sweep3-T*.log):
| tokens | padded rows | baseline plan | candidate plan | baseline gather tg256 | candidate gather tg256 |
|---|---|---|---|---|---|
| 256 | 8096 | 7.625 | 8.016 | 254.609 | 253.953 |
| 512 | 8432 | 8.188 | 8.021 | 270.062 | 269.766 |
| 1024 | 14352 | 8.953 | 8.297 | 489.187 | 490.578 |
| 2048 | 24576 | 11.422 | 12.057 | 1161.984 | 1159.594 |
t2048 candidate tg32/64/160/256 in that sweep: 1185.151 / 1166.484 / 1163.359 / 1159.594 us. output-only effective bandwidth at tg256 is about 108.5 gb/s (125,829,120 output bytes); this is not a measurement of physical dram read+write traffic. apparent unused thread slots do not imply the copy is execution-bound.
sweep2 and sweep4 also failed the predeclared repeatable >5% copy win. plan deltas changed sign. final combined-path runs (final-candidate-T*.log) all passed parity, but timings became heavily contaminated (baseline tg256 t2048 1900us, and a later identical-source comparison reached 6408us). do not cherry-pick these as wins/losses. final combined candidate plan+gather was 889.875 / 360.042 / 661.167 / 1739.927 us for t256/512/1024/2048, respectively; these are explicitly not comparable clean performance estimates.
correctness and build evidence #
built moe_mm_parity and strata_window_cli successfully in scratch build-metal, release/ninja; final-build.log records the final successful build. cli was built, never executed. each standalone invocation had a five-second deadline; successful invocations were about 0.01–1.39 seconds.
candidate and baseline were compiled in the same process. byte comparisons passed for all allocated bytes of bcnt, boff, eoff, tiles, meta, srt_tok, srt_w, and pos. XS passed full bitwise memcmp against baseline float gather for all active/padded rows at each threadgroup, direct and indirect dispatch, and after repeated combined plan+gather. candidate float gather also matched baseline float gather. existing token/position uniqueness/padding checks passed.
coverage:
- t256/512/1024/2048, experts512, width2560;
- t1/experts64, t7/experts512, t64/experts1024, t256/experts64;
- generic width768, t7/experts64.
these kernels do not accept quantized weights or a quantization-format parameter, so this coverage is format-independent. no claim is made that all quantized gemm formats were separately executed.
negative control: a scratch baseline deliberately writing srt_w[row] = w[i] + 1 produced srt_w memcmp failed, exit1 (negative-comparator.log). invalid t0 exited2 (invalid.log). final production-baseline self-comparison passed (final-baseline.log).
reproduce #
cmake -S /tmp/sol_r20_gather/strata/metal -B /tmp/sol_r20_gather/strata/build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=<workspace>/vendor/strata/third_party/metal-cpp -DSTRATA_GGML_DIR=<workspace>/vendor/llama.cpp
cmake --build /tmp/sol_r20_gather/strata/build-metal --target moe_mm_parity strata_window_cli -j 4
MM_BASELINE_KERNEL=/tmp/sol_r20_gather/baseline/metal/kernels/moe_mm.metal STRATA_METAL_KERNELS=/tmp/sol_r20_gather/variants /tmp/sol_r20_gather/strata/build-metal/moe_mm_parity --gather-bench 2048 512 2560
repeat one t per invocation for 256/512/1024/2048. omit STRATA_METAL_KERNELS to test the final scratch candidate (identical to the variants snapshot). to test unchanged baseline, set STRATA_METAL_KERNELS=/tmp/sol_r20_gather/baseline/metal/kernels. final review used view_diff and contextual view_code; the main patch now includes the kernel candidate at the parent’s explicit request, with the measurement caveats above.