Round 40: 14-Item llama.cpp-feature-adaptive-kv-stream-rocm Audit & Apple M5 Metal Port #
Executive Summary #
Evaluated all 14 candidate optimizations (#1..#14) from llama.cpp-feature-adaptive-kv-stream-rocm (/Users/dawn/Downloads/llama.cpp-feature-adaptive-kv-stream-rocm.zip) against vendor/strata on Apple M5 (10-core GPU, 24 GB unified memory):
- 7 Optimizations Shipped / Verified Active (
#1, #2, #3, #6, #8, #11, #13+#7, #14already native in Strata) - 5 Micro-Architectural Candidates Rejected with Measured Negative-Result Proofs on M5 (
#4, #5, #9, #10, #12)
All shipped kernels pass their standalone parity suites (gdn_prompt_parity, comb_fold_parity 0 mtpdefer, comb_fold_parity 0 topk, route_parity --ab, gdn_parity, qsa_parity --paired) with exit_code = 0, and end-to-end 48-layer teacher-forced evaluation on K512-iq2_xs-metal.gguf (eval_tf_chat_json.txt, 159 prompt + 64 eval tokens) achieves mean_kld = 0.000479 (median_kld = 0.000004), 100.00% top-1 agreement (64/64), and 100% identical GEN_IDS (0.000830 with route_topk delayed-softmax, 100.00% top-1 agreement).
Summary Table (#1..#14) #
| # | Optimization | Target Files | Parity | M5 Measured Result | Verdict |
|---|---|---|---|---|---|
| #1 | Chunked WY GDN Prefill on Matrix Units (matmul2d + fp16 (hi, lo) split) |
gdn_prompt.metal, window.cpp, prompt_chain.inc |
o rel L2 4.0e-7, state rel L2 3.4e-7 vs gdn_scanr_4 & fp64; 48-layer KLD 0.000479, top-1 100.0% |
0.717–0.734x (-26.6% to -28.3%, -53.6 ms per 2K chunk; full GDN 269.3 -> 216.6 ms, 0.805x) |
SHIPPED |
| #2 | Combined (a + 1)-Row MTP Catch-Up + Draft Step-0 Batch (defer_catchup, QSA_LAST_Q) |
qsa.metal, window.cpp, comb_fold_parity.cpp |
same_tile=1 bitwise (max|dR|=0, max|dq|=0), same_tile=0 max|dR|=4.77e-7, KV bitwise |
a=1: -26.8% (635.8 -> 465.5 us, 19 -> 11 disp); a=2: -20.5%; a=3: -20.1% |
SHIPPED |
| #3 | Zero-Copy Dual-Slot GDN Recurrent State & History Rollback (dual2 / spec1) |
gdn.metal, window.cpp, prompt_chain.inc, gdn_parity.cpp |
100% bitwise exact across all 88 (ntok, commit, plane) cases + ring rollback bitwise: yes PASS |
Eliminates Window::fold() (2.0–2.2 ms) 100% of the time (n_keep=1 and n_keep=T); -0.14 to -0.58 ms/round net |
SHIPPED |
| #4 | 0x6400 IEEE-754 Mantissa Trick for int8 & uint4 (pd_i8x2 / pd_u4x2) |
qsa*.metal, dense_mm.metal, moe_mm.metal |
100% bitwise (memcmp == 0) |
1.015–1.036x slower on QSA (1.00x on dq_f16 / moe_mm): M5 hardware int8 -> float cvt is free |
REJECTED |
| #5 | In-Register Butterfly FWHT (D=256) for int4 QSA V-Cache & Q2_0 Outliers |
dev/fwht_bench.metal |
int4 rel L2 23.3% -> 8.85% on outliers (int8+FWHT: 1.68% -> 0.49%, 0 overhead) |
int4 V-cache still 15–18x worse than int8 (10.8% vs 0.59% Gaussian); retained int8 |
REJECTED (int4) |
| #6 | Phase-Switched Memory Trading (STRATA_METAL_PHASE_TRADE: Prefill Scratch <-> Decode Arena) |
stream.cpp, window.cpp |
Bitwise identical | Trades 1.12 GB (~720 expert slots) between PromptGemm and decode LRU (opt-in PHASE_TRADE=1/2) |
SHIPPED |
| #7 | Paged KV Streaming with Dual-Slot Ring & ROCr Async SDMA Copy Engine | src/kernels/cuda/kv_stream.cu, copy-engine.cu |
Capturable GPU clock-sweep resolve_kernel + copy_kernel already in Strata |
Documented ROCr SDMA engine 1u << (31 - __builtin_clz(mask)) for Desktop RX 7600 XT target |
VERIFIED |
| #8 | Prefill-Safe GPU MoE LRU Cache & Coalesced TENSOR_READ_LAZY Dense Mmap |
stream.cpp |
100% bitwise (mean_kld = 0.000000, GEN_IDS identical) |
Coalesced page-aligned align_range + ranges_complement: 1,079 syscalls -> 145 lazy + 146 mlock spans, never dropping shared 16 KiB boundary pages |
SHIPPED |
| #9 | Sub-Warp Column Segmentation in GDN Decode (gdn_step_t) |
gdn.metal |
100% bitwise (88/88 PASS on plane=0 and plane=49152) |
1.035–1.138x slower on M5: strided (r*128 + j) state loads lose 32-lane cacheline coalescing |
REJECTED |
| #10 | Time-Tiled Sliding-Window GDN Pre-Conv (gdn_pre_t<TT>) |
gdn_prompt.metal |
100% bitwise (memcmp == 0 on all 6 buffers, T=1..2048) |
1.004–1.018x slower at T=2048 (55.0 ms): L1/L2 already caches qkv[t-3..t] at ~95 GB/s |
REJECTED |
| #11 | Zero-Barrier 1-Simdgroup-per-Row qsa_prep (qsa_prep_sg, tg = 32 for ntok >= 3) |
qsa.metal, window.cpp, prompt_chain.inc |
100% bitwise (memcmp == 0) on all 13 QSA buffers (T=1..2048, M-RoPE, qf16/fp32) |
0.91–0.97x (-3% to -9%) at nt=3..8, 0.67x (-33%) at nt=64, 0.85x (-15%) at T=512, 0.91–0.92x at T=2048 |
SHIPPED |
| #12 | 2D Token-Tiled QSA Indexer Scoring (qsa_score_t<TT>) |
qsa.metal |
100% bitwise (memcmp == 0 for TT = 1, 4, 8) |
1.069–1.475x slower at decode nt=1..4 (pooled key cache is only 2 MB and stays in M5 SLC/L2) |
REJECTED |
| #13 | Logit-Domain / Delayed-Softmax MoE Top-K (route_topk, moe_route, router.metal) |
prompt.metal, moe.metal, router.metal |
100% bitwise ids (0 mismatches across 400 windows), max|dw| <= 5.96e-8 (1 ULP) |
route_topk 0.920–0.955x (-4.5% to -8.0%) across T=512..2048; moe_route 0.985–0.994x |
SHIPPED |
| #14 | Decorrelated Temperature-Synced Draft Sampling (spec_retune) |
coupled_draft.hpp, spec_prob.hpp |
Exact target distribution $p$ preserved under both coupled & rejection sampling | Strata already implements STRATA_SPEC_COUPLED=1 (shared Philox draw + Gumbel-max) and STRATA_SPEC_PROB=1 (disjoint Philox counters pos | 1<<61/62/63) |
VERIFIED |