round 3 track c: rank-1 moce and step-aware speculative retention #

result #

patch: /tmp/sol_moce_spec/sol_moce_spec.patch.
no tracked vendor files changed. git -C vendor/strata apply --check /tmp/sol_moce_spec/sol_moce_spec.patch passed against the current dirty working tree.
the suspected draft-moce/rank-1 interaction is consistent for every rank 0..9. rank-1 bonus is intentionally bypassed when r_prot=1; no draft interaction fix is needed.
a different weight bug is real: with r1_scale>1, r_prot>1, and moce_scale<=1, the remaining resident candidates retain the rank-1 multiplier after r_prot, but the old code stops dividing it out. both kernels get the same minimal inverse-factor fix. selected expert ids and group order are unchanged by this fix.
adds opt-in STRATA_METAL_SPEC_MIN_P_SCHEDULE and STRATA_METAL_SPEC_MIN_CUM_P. no threshold default is changed and no speedup is claimed.
cpu routing harness: 158,400 cases; original 10,033 weight failures, draft-row failures 0, patched failures 0; worst patched absolute weight error 8.94069672e-08.
schedule tests: 1,347,106 decisions plus parser/boundary checks passed, including an -DNDEBUG build. detached full window.cpp clang++ -fsyntax-only passed.
no gpu execution, ssd benchmark, server request, or access to :8080 occurred.

source and ownership #

baseline vendor head: 296b59d2f73c8f04a805df29c2fa97655adace78; the working tree was already substantially modified. this patch is incremental against that dirty tree, not a diff against vendor head. exact input snapshots are under /tmp/sol_moce_spec/original/; hashes, commands, and patch hash are in /tmp/sol_moce_spec/manifest.json.

owners are host-side draft prefix retention in metal/src/engine/window.cpp and selection-score unscaling in metal/kernels/moe.metal / router.metal. the new small header metal/src/engine/spec_min_p.hpp keeps parsing and gating testable without metal. a cpu-only regression test is included at metal/tests/spec_min_p.cpp; the existing cmake test-source glob picks it up as an executable.

no graph construction, model data, cache layout, kv format, interface signature, or function constant is changed. configuration is parsed once, with eight floats of bounded storage, not per token. settings are process-wide and require restarting after an environment change.

draft-moce/rank-1 proof #

write m for effective moce_scale, b for base scale, and r for r1_scale. on draft rows (sg>0 with the flag), m=b*b and r_prot=1. note that squaring the scale means doubling the log bonus, not multiplying the scale by two.

rank	selection multiplier for a remaining resident	inverse for a selected resident
0	1	1
1..9, m>1	m (applied exactly once at rank 1)	1/m
1..9, m<=1	1	1 (mask was never set)

r>1 && r_prot>1 is false, so no rank-1 multiplication occurs. min_scaled_rank is 1. rank<r_prot is false for every scaled winner, so rcp_moce is correct. rank 0 is unscaled and nonresident experts are never masked. r therefore has no effect on draft-row ids or weights. both route implementations have the same logic. on sg=0, the draft flag is ignored and configured protection remains in force.

for nondraft r_prot>1, resident scores receive r at rank 1. if m>1, remaining scores transition to m at rank r_prot using m/r, and existing inverse factors are correct. if m<=1, that transition is skipped, so the tail still has multiplier r. the patch retains rcp_r1 for that case instead of using rcp_moce=1. it changes only the recovered probabilities, not scores or selection.

edge protections were checked: r_prot=0 and 1 bypass the r1 branch; r_prot=10 never reaches the tail transition within ten picks. null residency never sets a scale mask. all-missing and all-resident maps, ties, and partial maps were exercised.

weights mean the original softmax probabilities renormalized over the selected set, with the existing max(sum,2^-14) floor. they are not the boosted selection probabilities. exact algebra is preserved; floating-point implementation is checked to a tolerance, not claimed bit-identical to infinite-precision arithmetic or gpu reductions.

schedule behavior #

example exploratory settings, not recommended new defaults:

STRATA_METAL_SPEC_MIN_P_SCHEDULE=0.10,0.35,0.55
STRATA_METAL_SPEC_MIN_CUM_P=0.20

schedule accepts one to eight finite probabilities in [0,1], comma-separated, allowing whitespace. the last threshold repeats for later steps. any order is allowed, including decreasing thresholds; increasing later thresholds is the hypothesis here.
a supplied schedule overrides caller min_p per step. without a schedule, caller min_p stays the per-step floor.
cumulative mode multiplies draft head probabilities in double, resetting to 1 each draft round. it adds a prefix gate product(p[0..j]) >= min_cum.
both settings compose by logical and. to use only the cumulative gate, set schedule 0 and a positive cumulative floor; otherwise caller min_p still applies.
equality passes; a failed gate terminates the contiguous prefix. later tokens cannot be retained across a rejected parent.
zero cumulative floor disables that gate. malformed, nonfinite, out-of-range, too-long, or empty settings reject the entire new configuration and print one diagnostic before falling back to caller min_p. a valid schedule 0 explicitly opts into the new policy while disabling its per-step floor.
nonfinite/out-of-range observed probabilities fail closed only when the new policy is enabled. unset mode preserves old comparison behavior.

important existing difference: the chained path rejects a low-confidence candidate before appending it; the nonchained host path currently appends that candidate and only then stops. unset settings preserve that behavior, including the old host count. when the new policy is enabled, both paths reject before writing drafts, probs, or incrementing n. this is deliberate opt-in cleanup rather than a silent default change. testing covered the pure policy and source ordering; neither full mtp runtime path was executed.

economics and confidence #

for the default chained path, the entire max_steps draft chain is already encoded and completed before the retention loop. retaining one more already-generated candidate adds no drafter gpu dispatch at that point. raising max_steps would not be free. in nonchain mode, stopping also avoids future draft steps, so the same economic argument does not transfer unchanged.

let a_i be conditional verifier acceptance for draft step i and A_i=product(a_0..a_i). with k drafts, expected committed tokens are approximately

R_k = 1 + sum(i=0..k-1, A_i)
C_k = fixed_draft_cost + verifier_cost(k+1)

retaining draft k improves the local ratio iff

A_k / delta_verifier_cost > R_k / C_k

when the marginal verifier cost is positive. when it is effectively zero, any positive acceptance is locally beneficial. this explains why the cheap first suffix can warrant a lower threshold while later suffixes need stronger evidence. even equal per-step probabilities imply decaying path survival: (0.5,0.5,0.5) has prefix survival (0.5,0.25,0.125); a flat 0.35 gate retains all three. a cumulative floor 0.2 retains two, while (0.1,0.35,0.55) also retains two on this example. conversely a first probability 0.2 is rejected by flat 0.35 but can pass the cheap-first schedule.

caveats:

draft head softmax confidence is not calibrated verifier acceptance. product(p_j) is a heuristic, not an unbiased probability of acceptance. this remains true with argmax drafting, vocabulary restrictions, or different routing on verifier suffix rows.
draft-moce can reduce incremental misses, but r_prot=1 and a doubled log bonus do not prove zero new misses or equal verifier semantics. window shapes, alignment, residency, rollback, and subsequent catch-up costs may differ. cost figures in the task were not remeasured here.
C_k is a local approximation: the next round's draft catch-up and cache state may depend on accepted length. confirmation needs end-to-end throughput, not just per-window timings.
a single cumulative floor does not encode arbitrary marginal costs; a per-step schedule cannot distinguish a shaky parent prefix from a confident one with the same current p_j. both controls together are useful ablation knobs, not an established optimum.

synthetic cpu illustration, not hardware evidence #

/tmp/sol_moce_spec/schedule_math.py enumerates 1,000 triples on the uniform grid 0.1..1.0, assumes acceptance equals draft confidence, assumes verifier costs [1,1.02,1.25,1.60], and fixed draft cost 0.15. the numbers are made-up normalized costs, not measurements:

policy	mean retained drafts	synthetic expected tokens / cost
flat 0.35	1.533	1.322892
step 0.10,0.35,0.55	2.050	1.325972
cumulative 0.25 only	1.512	1.370152
step plus cumulative 0.20	1.657	1.376354

this establishes only that the two heuristics make materially different choices under stated assumptions. it does not select thresholds, validate calibration, or establish a speedup. no default should change from this exercise.

validation and limits #

all jobs ran under /tmp/sol_moce_spec/, cpu only:

clang++ -std=c++20 -O2 -Wall -Wextra route_cpu.cpp -o route_cpu, then ./route_cpu: pass. seed 20260605. 12 trials; 32/256/512 experts; base scales 0.5,1,2,4,exp(0.85); r1 scales 0.5,1,2,4,exp(0.4); protections 0..10; draft flag off/on; head and suffix row; null/all-miss/all-hit/mixed residency. exact tied and untied inputs. all ten ranks checked.
routing adapter uses selection/scaling and unscaling blocks extracted from both original and patched kernels, an independent per-rank score reference, and original selected probabilities for reference weights. checks kernel parity, expected ids, weights within 2e-7, and exact unchanged output when r1<=1 or the row is a draft. cpu argmax models lowest-id tie breaking. this is not an execution of metal simd instructions or a bitwise gpu softmax test.
group sorting uses a cpu model of the existing plan_body per-lane/per-simdgroup prefix construction, checked against an independent stable resident-first/id sort for split off/on. entry counts, group offsets, token/entry traversal order, and complete unique mapping are checked. this is a model, not execution of metal barriers, atomics, or the actual plan buffer on gpu. the patch does not change the planner.
clang++ -std=c++20 -O2 -DNDEBUG -Wall -Wextra -Werror patched/metal/tests/spec_min_p.cpp -o spec_min_p_release, then ./spec_min_p_release: pass. checks are active in release builds. seed 12345; 100,000 probability sequences, legacy flat comparisons, schedule/cumulative comparisons, parser failures, equal boundaries, repeat-last behavior, and invalid observations.
detached full window.cpp clang++ -std=c++20 -fsyntax-only ...: pass. headers copied into the temp tree to resolve relative includes; ordinary vendor include paths and build path macros supplied. the first attempt failed only because those normal cmake macros were omitted; retry and final check passed. exact argv in manifest.json.
git diff --check in the temp review repo: pass. patch apply check against vendor/strata: pass. final source snapshots compared equal to all three untouched vendor inputs.

logs: route.log, spec_release.log, window_syntax_final.log, apply_check.log, schedule_math.log under /tmp/sol_moce_spec/. plan in plan.md; review-only temporary git repo in review/; shipped source copies in patched/.

not tested: actual metal compilation/execution, simd reduction rounding, gpu/group synchronization, streamed residency races, real mtp acceptance or throughput, server behavior, or sampled-output equivalence. analysis assumes finite scales that do not overflow their products; existing unbounded exp(atof(env)) bonus parsing can overflow for extreme input and was not changed here. no cpu sample can exhaust the continuum of floating-point scale combinations; the rank-case proof supplies the general finite-scale argument.

next decision #

keep defaults. apply the two-kernel weight fix as correctness work and expose the schedule only as opt-in experimentation. if gpu testing is later authorized, first run metal route parity for r1>1, m=1, then same-day held-out sweeps of flat / step / cumulative / composed policies, recording per-step p_j, actual matched-prefix acceptance, verifier shape cost, expert misses, and end-to-end committed tokens/sec. do not treat higher draft count as a win by itself.