6.6 kB 70 lines
4b0f233f
Round 2 blitz: gemv_cat+gr_norm_comb fusion, 1247 prefill barriers removed, M5 gemm_f16 tiles, slop4 MTP, P_GSLOT cache fix, parallel qsa_prep (32.4-41.2 tok/s)
dawn
⋯
audits/sol_slop4_mtp.md
sourcerendered
download raw history
at main Markdown

track c: acceptance-aligned slop4-aware mtp #

result #

implemented an opt-in, unapplied patch: /tmp/sol_slop4_mtp/sol_slop4_mtp.patch. set STRATA_METAL_MTP_SLOP=1 to enable after application. unset/0 keeps the old draft dispatches and picks. no tracked vendor files were edited; the baseline window file was verified unchanged afterward.

host syntax, real metal library compilation, policy regression tests, and patch applicability all pass. no acceptance or latency improvement has been measured. no gpu kernels were dispatched, no gpu/ssd benchmark ran, and port 8080 was untouched.

what the code establishes #

  • verifier finish adjusts logits with stateful lexical/structure penalties, end-token probability preservation (slop4::hold), then the empty-turn guard. draft selection previously used none of them.
  • shared slop4.h now owns token classes, state transitions, penalty rules, and held token ids. the verifier and metal draft kernel use the same source rather than separately maintained rules. cpu class lookup remains the existing lazy table; metal checks the bounded class arrays.
  • a 256-thread mtp_slop_pen dispatch runs after each chained draft head and before its argmax. step zero starts from the exact committed-prefix state; later steps advance a private state using dout[j-1]. this state never writes verifier history.
  • the chain still submits exactly one command buffer per round. each step adds one dispatch and its dependency barrier, not a host wait or another submission.
  • the non-chain path applies equivalent policy to the shared logits after its existing wait, rescans the argmax, and computes its existing probability from the adjusted row.
  • hold is calibrated on the draft head's distribution, not copied numerically from the target. absent subset ids contribute no mass. subtraction stays serial over at most 70 entries so overlapping penalties remain additive.
  • the inverse-vocabulary table costs 4*n_vocab bytes (993,280 bytes at 248,320 entries), plus a 12-byte draft state, only when opted in. lookup is o(1); loading is o(n_vocab + draft_vocab). duplicate subset ids produce an explicit load error in opt-in mode.

important correction to the proposed seed #

mtp_draft receives verifier output tokens, not verifier input tokens. see metal/src/strata_window_cli.cpp:1036,1164 and src/program/generate.cpp:12729,13890 in the baseline. residual row a belongs to input position p+a; tokens[a] is the output at p+a+1.

therefore the seed is:

slop4::step(slop_at(p + a), tokens[a])

stepping output tokens 0..a from slop_at(p-1) drops the verifier's first input token. the patch uses the corrected seed. this matters especially for the newline after </think> and for clause/line boundaries.

vocabulary check #

read the actual little-endian int32 files under vendor/strata/data/:

subset entries unique strictly ascending
draft_vocab_en.bin 40,525 40,525 yes
draft_vocab.bin 106,299 106,299 no
draft_vocab_fr.bin 46,211 46,211 no
draft_vocab_cyrillic.bin 58,963 58,963 no

a generic binary search would be wrong for three shipped subsets. also, a 40,525-entry binary search needs up to 16 comparisons, not 12. the inverse table avoids assuming ordering.

evidence and reproduction #

all artifacts live in /tmp/sol_slop4_mtp/:

  • manifest.json: baseline revision/file hashes, patch hash, exact host syntax command and test commands. the baseline was already dirty; baseline_window.cpp and baseline_slop4.h preserve what was inspected.
  • syntax.log: clang++ -std=c++20 -fsyntax-only ... window.cpp, exit 0, no diagnostics. includes the real metal compatibility headers; no cuda stub or fabricated engine types were needed.
  • compile_metal.mm, compiled_source.metal, metal_compile.log: the real metal compiler compiled the concatenated ggml/quant/units/shared-policy/mtp-chain source through newLibraryWithSource, exit 0. this compiles only; no command queue or dispatch is created. xcrun --find metal is unavailable here, so this uses the system runtime compiler instead. an initial compile caught unsupported metal log1p; the final kernel uses -log(1-b).
  • test_policy.cpp, policy_test.log: exit 0. checks every class id from -1 through 248319, 200,000 seeded state transitions, 5,632 synthetic states, and 150 full/shuffled-subset/missing-id policy rows across five knob configurations (including all zero). the actual extracted host helper is checked against baseline penalties/hold/guard; adjusted logits agree within 2e-6 and picks agree. tool/fence/thinking regions and the output-token seed regression are included.
  • patch_check.log: git apply --check /tmp/sol_slop4_mtp/sol_slop4_mtp.patch from vendor/strata, exit 0. do not apply against a subsequently changed window without rechecking.
  • review/: isolated git baseline plus final code used for the required visual diff/context review. new shared header reviewed in full.

quick cpu rerun:

clang++ -std=c++20 -O2 /tmp/sol_slop4_mtp/test_policy.cpp -o /tmp/sol_slop4_mtp/test_policy
/tmp/sol_slop4_mtp/test_policy
/tmp/sol_slop4_mtp/compile_metal /tmp/sol_slop4_mtp/compiled_source.metal

limits and decision #

this is a policy-alignment candidate, not a measured optimization winner. acceptance is not guaranteed: target and draft logits still differ, subset coverage still matters, and a finite penalty does not necessarily change the target's argmax. the supplied 65.5/71.9/93.8% figures are motivation, not evidence that the entire gap comes from slop4.

the hold correction depends on model logits, so its numeric value cannot be exactly the target's value on the draft. metal uses float reduction/math versus the host's double accumulation; near-tie differences require runtime parity testing. no metal kernel execution, chained state-buffer correctness run, min-p acceptance run, full engine link, or live persona benchmark was performed. existing host/chain min-p inclusion differences were deliberately left alone.

cost is also not established: the new kernel always reads the head for its maximum and conditionally computes a log-sum-exp when hold is relevant. that is o(draft_vocab), not merely o(number_of_penalties). claiming “under 2 microseconds” without measurement would be unjustified. keep this opt-in until paired persona/coding/control acceptance and end-to-end throughput tests show a net benefit. future optimization could fuse this reduction with existing head/argmax work, but that is separate from correctness.