track 3: streaming publication, two-layer prompt lookahead, warm scratch #

deliverables / status #

candidate only: /tmp/sol_async_stream/sol_async_stream.patch (three engine files; not applied to the shared checkout).
source and cpu-only regression harness: /tmp/sol_async_stream/vendor/strata/metal/src/engine/, /tmp/sol_async_stream/test_stream.cpp.
exact compiler invocations and outputs: /tmp/sol_async_stream/verification.json.
no gpu execution, model reads, ssd benchmarks, or port 8080 access occurred.
this is not an implementation or validation of live cpu-to-gpu lock-free publication. that part of the requested design is rejected pending a real metal synchronization contract. no performance or quality win is claimed.

ownership / invariant #

ExpertStream owns expert lifetime, pending io, slot residency, and prompt scratch backing. Window owns router prediction dispatches and the metal views over scratch. the existing shared-event handshake is the publication boundary. selected expert bytes, expert arithmetic, and resident/missing plan membership must remain valid until that layer finishes. no new kernel interface or weight format is needed for this safe candidate.

why immediate io_main smap writes are not just nondeterminism #

io_main finishes each pread outside mu_, then decrements Slot::left under mu_. only the last completed part sets state = 0; that existing seq-cst atomic store already gives the host an acquire/release completion boundary for every piece. urgent jobs publish smap, speculative jobs intentionally do not.

changing only that speculative condition to atomic_ref<uint32_t>(smap[key]).store(..., release) does not establish a synchronization relationship with the existing metal device const uint* reads. a c++ release store is not a gpu acquire, and replacing metal reads with relaxed atomics alone would not prove host/gpu data visibility either. the issue is full expert payload visibility, not merely a tear-free 32-bit slot number. this needs an apple-supported shared-resource synchronization scheme (or existing shared-event boundary), not an assumption about apple silicon cache coherence.

related lifetime traps:

victim() invalidates smap and immediately makes backing bytes reusable. the current layer must never be a victim while its resident pass executes. during decode after_layer(l), the gpu can also reach layer l+1's router. existing ahead(l+2) only avoided its target layer, not the router currently in flight. the candidate protects current layer throughout a turn and additionally next layer during after-turn work.
cancel_slot() is safe only for state-1 queued reads: no piece has begun, no completion may publish it, and its map entry must still be absent. the existing queue callers check this. it is not a completed-slot eviction primitive.
retire_unused_pre() parks queued, reading, or completed unchosen speculative slots in limbo. the old release_limbo() cleared where_ but never smap, because those slots were never published. once speculative residency is published, that omission becomes a stale address into recycled expert bytes. the candidate clears smap before recycling.
route_topk in prompt.metal uses smap both for resident probability scaling and the cold-expert pruning mode. moe_route and fused route_body in router.metal use it for moce; plan_body in moe.metal snapshots each expert's resident/missing classification once into the plan. gu/down then resolve the slot again from smap. the plan is fixed, so adding residency need not inherently duplicate work, but removing/recycling an expert the plan already considered resident corrupts computation.
moe_mm.metal also resolves each tile's expert address from smap. its existing prompt handshake guarantees the required experts exist before those dispatches. no atomic kernel rewrites were made because the candidate keeps that contract.

candidate behavior #

handshake publication #

STRATA_METAL_PUBLISH_PREFETCH=1 (candidate default) publishes completed, non-limbo slots for layers l+1..l+2 at at_layer(l), before the gpu is released. it uses the existing atomic state completion marker, under the slot metadata mutex. the current layer is excluded because decode's resident expert pass can overlap this host callback. no io thread writes speculative smap entries while gpu routers are live.

already-published entries stay published; victim and limbo recycling explicitly invalidate them. the old non-split moce path's determinism-only wait/evict block is retained when this flag is zero and skipped when enabled. setting the flag to zero restores the old speculative publication policy (the strengthened lifetime guards remain).

this is bounded handshake publication, not lock-free io publication. publication scans at most 2 * ne_ map entries per layer, independent of conversation length. benefit requires reads to have completed by an earlier handshake; it cannot expose a completion that arrives after that boundary to the immediately following router.

two-layer prompt lookahead #

STRATA_METAL_PROMPT_PRED2=1 adds the l+2 router pass on layer l's existing FFN input. the existing router-weight/logit/weight scratch is reused sequentially; only another MT * TOPK int32 id buffer is allocated. measured and actual buffer layout account for it identically. no model input is altered and prediction weights are never used as actual expert mixture weights.

ids are consumed at the current handshake, ordered by token frequency, then queued behind urgent work and one-layer predictions. STRATA_METAL_PROMPT_PRED2_MAX caps newly submitted two-layer predictions (default 64; clamped 0..512). all slots remain bounded by the arena. current-layer and next-layer residency are protected during selection; no speculative submission waits for space. absent capacity, it stops. layers 46/47 skip out-of-range lookahead; prediction-off leaves existing profile fallback.

existing decode two-layer prediction already exists through STRATA_METAL_PREFETCH2; no redundant decode router pass was added. its eviction protection is strengthened as above.

STRATA_METAL_PROMPT_START_LAYERS=4 optionally queues profile-ranked startup experts for layers 0..3. default remains one layer. layer 0 keeps its existing cap; later startup layers each queue at most 32 nonresident experts and cannot evict layer 0. this is deliberately bounded, not a full-model upfront prediction sweep. the extra router pass and wrong predictions may cost more than overlap saves; measurement is still required.

warm prompt scratch #

STRATA_METAL_PROMPT_BUF_REUSE=1 retains the single existing separate scratch mapping and its metal buffer views across turns, skipping the per-turn free/delete/munmap/purge_device_allocations path. memory is bounded by one configured prompt chunk allocation; this retains physical memory and may reduce expert-cache headroom on a 24 gb machine. default stays off. the arena-backed path is unchanged. prompt teardown is shared in one helper and runs at window destruction before the separate mapping is released.

this avoids the suspected lifecycle overhead structurally, but the task's roughly 300 ms claim was not remeasured and must not be reported as a measured win. MADV_FREE was not shipped on live no-copy metal buffers: physical reclaimability while metal retains/wires those pages and gpu write/reuse semantics need verification. neither madvise nor warm retention is a magic smaller working set. a truly smaller 512-row resident scratch design needs row-capacity selection at initialization; retaining a 1024-row allocation because the current T is 512 does not shrink it.

verification actually run #

requested flags alone: stream.cpp syntax passed; window.cpp failed on the pre-existing missing cuda_runtime.h include path.
with the actual CMake metal compatibility include directories and the two required CMake string defines: both modified translation units passed clang++ -fsyntax-only -std=gnu++20 -arch arm64 -DSTRATA_GPU_METAL=1 (all other include flags retained).
cpu-only regression executable linked against the actual modified stream.cpp, with unused gpu-dependent functions dead stripped. it tests incomplete and limbo slots excluded from publication; completed publication; executing-layer and next-router victim exclusion; map invalidation before recycling; queued cancellation; separate-scratch retain/release; real at_layer submission of both prediction distances; final-layer bounds and queued cancellation.
the final executable passed with address sanitizer + undefined behavior sanitizer, exit 0, no diagnostics.
git diff --check passed in the isolated temporary git repository.
final diff reviewed visually; repeated router dispatch code was consolidated into one distance loop and prompt cleanup into one helper.

merge gate #

this is a candidate, not a production-ready speed claim. integration needs gpu correctness/parity on cold/warm, prompt/decode transitions, partial chunks, layer 47 rollover, split-turn on/off, and active moce modes; quality checks matter because changed residency changes existing moce routing. also measure memory pressure and warm-turn latency with retention enabled. none were run under this track's explicit no-gpu/no-benchmark rule. do not enable a live async smap mode merely because nondeterminism is allowed: arithmetic races and stale expert addresses are not nondeterminism.