13 kB 191 lines
6fdbee92
keep 00-..04- docs and non-persona engine/pruning files at top level; keep only persona/anti-slop in persona/
dawn
⋯
02-hardware-and-streaming-model.md
sourcerendered
download raw history
at main Markdown

02 — sizes, streaming speed evidence, performance model & measurement plan #

3. sizes [est] #

expert = 4.915M params. totals across 48 layers:

experts/layer IQ2_XS (2.31 bpw) Q2_0 (2.25 bpw [src: ggml.h type 42, 64w=18B]) Q3_K / IQ3_S (3.44 bpw) IQ3_XXS (3.06 bpw)
512 34.9 GB 34.0 GB 51.9 46.2
320 21.8 21.2 32.4 28.9
256 17.5 17.0 26.0 23.1
224 15.3 14.9 22.7 20.2

other fixed costs:

  • dense ≈ 3.2 GB
  • KV at 128K: fp16 3.0 GiB, q8 1.6 GiB, q4 0.84 GiB
  • GDN state ≈ 110 MiB (fp32)
  • mmproj ≈ 0.5–0.9 GB

fast-memory budget for experts:

  • desktop: ~4.3 GB VRAM + ~7.5 GB RAM ≈ 11.8 GB ≈ 178 experts/layer at Q2_0 (1.3824 MB/expert)
  • laptop: ~7.5 GB ≈ 113 experts/layer at Q2_0
  • so on both machines, even 224 is partly streamed from SSD

desktop note: Strata needs ≥32 GB system RAM [src: Strata MODELS.md, web], and the measured 7600 XT run had 64 GB. 16 GB RAM is the desktop bottleneck, not the GPU. options:

  • (a) more RAM
  • (b) Strata/llama.cpp low-RAM mode with heavier SSD streaming
  • (c) prune harder for the desktop

4. what's known about streaming speed #

4.1 Mac evidence [src: Strata docs/MAC.md, the only streamed-Mac measurement] #

M4 Mac mini, 32 GB, experts streamed, 11.5–14.7 GB arena, no MTP, medians of 5:

Q2_0 IQ3_XXS
decode, short 14.1 10.0
decode after 6K doc 13.1 7.7
prompt, 4K tokens 93.6 65.8 (84.1 with experts' copy)
  • M5 Max 128 GB, resident: Q2_0 prompt ~1,430 tok/s at 4K, decode 87 tok/s without MTP.
  • Slotstream [src]: 24–48 GB Macs, decode ~5–16 tok/s (est. table), 20.9–23.7 tok/s on a 48 GB M5 Pro. no prefill numbers.

4.2 ds4-recipe evidence [src: tangled.org/astrra.space/ds4-recipe] #

DeepSeek V4.1 Flash, Strix Halo 128 GB UMA, 3 NVMe, ~31% of experts resident:

  • cold prefill 354–391 tok/s (README says ~450 now), from:
    • 4 lanes × 8,192-token chunks sharing one read of each layer's expert union
    • hipBLASLt prepared GEMMs
  • p50 prefill by turn size:
new tokens in turn p50 tok/s
64–256 19
256–1k 46
1k–4k 106
4k–16k 236
16k+ ~340

small turns are SSD-bound.

4.3 Air thermals & local M5 hardware microbench [src: local bench/README.md + web] #

  • measured on this 13" M5 Air (Mac17,3, 10-core GPU, 1 TB SSD, bench/run_01..03):
    • M5 10-core GPU FP16 GEMM (K=2560): 12.60 TFLOP/s at M=512, 13.66 TFLOP/s at M=1024, 14.07 TFLOP/s at M=4096 (recommendedMaxWorkingSetSize = 17.76 GiB). exactly 1/4 of the 40-core M5 Max, confirming the ~356–360 tok/s resident prefill compute ceiling.
    • 60-second continuous GPU thermal sag test (M=4096, K=2560, N=2560 FP16): started at 14.00 TFLOP/s (t=0..15s) and ended at 13.88 TFLOP/s at t=55..60s (−0.86% sag over a full minute). so single-prompt cold prefills (even 15–30 s) do not hit a thermal cliff; multi-minute sustained loads [web: Notebookcheck ~9 W sustained] still favor KV reuse.
    • 1 TB SSD uncached read bandwidth (F_NOCACHE=1, 100.0% physical disk0 verified via iostat -Id on virgin 1 GiB slices):
      • standard GGUF (512 KiB × 3 reads/expert): 2.41 GB/s (1 thread) → 5.52 GB/s (2 threads) → 7.03 GB/s (4 threads)
      • contiguous .experts copy (1.5 MiB × 1 read/expert): 3.85–3.93 GB/s (1 thread) → 6.84 GB/s (2 threads) → 6.60–6.89 GB/s (4 threads)
      • PLE n-gram table (4 KiB random page reads): 0.04 GB/s (~10k IOPS, ~1.6 ms/tok for 16 heads) at 1 thread → 0.33 GB/s (~82k IOPS, ~0.2 ms/tok) at 8 threads
    • methodological trap caught (run_02 → run_03): on macOS APFS, repeating pread on identical file offsets across benchmark iterations can partially hit APFS cache on pass 2+ even with F_NOCACHE=1. always use virgin slices and check iostat -Id (disk0 MiB delta == 100% of requested MiB).

5. performance model [est, to be replaced by measurements] #

5.1 the metric: per-turn floor #

  • each prefill chunk (or each turn, however short) reads, for every layer, the union of experts its tokens picked. with top-10 of 512, even ~256 tokens touch most of a layer (uniform-routing approximation; real routing is skewed, so less).
  • so per-turn floor ≈ non-resident expert bytes ÷ SSD read rate, plus compute.
  • kernel opts move the ceiling. pruning, residency and MoCE move the floor.

5.2 laptop model (anchored to measured M5 Air SSD & GPU constants) #

inputs:

  • Q2_0 (2.25 bpw = 1.3824 MB/expert [src: ggml.h type 42]), ~7.5 GB experts resident (~113 experts/layer)
  • compute ceiling 356 tok/s [src: local Metal FP16 GEMM 14.07 TFLOP/s on M5 10-core GPU, scaled from M5 Max 1,430 tok/s @ 40c]
  • measured SSD rates on this 13" M5 Air (iostat -Id 100% physical disk0 verified) [src: bench/README.md]:
    • 2.41 GB/s: 1-thread standard GGUF (3 × 512 KiB/expert)
    • 3.90 GB/s: 1-thread contiguous .experts copy (1 × 1.5 MiB/expert)
    • 6.80 GB/s: 2–4 thread pread fan-out (saturates PCIe Gen4 x4 NVMe)
  • no IO/compute overlap (pessimistic), before MoCE (measured Phase 1b OOS lru λ = 0 -> 0.09 with protect_top_j = 3 cuts decode misses 17.4–19.9% on its own, or −36.4% at K320_reap_hybrid / −47.4% at K256_reap_l2_sum combined with pruning; see §7.1 for exact OOS decode tok/s)
experts (non-resident Q2_0 @ 2.25 bpw) SSD mode (measured) 1024 chunk 4096 8192 256-tok cold prefill floor (pre-MoCE)
512 (26.5 GB) 2.41 GB/s (std 1-th) 74 tok/s 182 241 ~11.7 s
512 (26.5 GB) 3.90 GB/s (.experts 1-th) 106 224 275 ~7.5 s
512 (26.5 GB) 6.80 GB/s (2–4 th) 151 266 304 ~4.6 s
320 (13.7 GB) 3.90 GB/s (.experts 1-th) 160 273 309 ~4.2 s
320 (13.7 GB) 6.80 GB/s (2–4 th) 209 303 327 ~2.7 s (warm 256-tok decode: 5.88 s @ 43.5 tok/s w/ MoCE)
256 (9.5 GB) 3.90 GB/s (.experts 1-th) 193 294 322 ~3.2 s
256 (9.5 GB) 6.80 GB/s (2–4 th) 240 318 336 ~2.1 s (warm 256-tok decode: 5.40 s @ 47.4 tok/s w/ MoCE)
224 (7.4 GB) 3.90 GB/s (.experts 1-th) 215 306 329 ~2.6 s
224 (7.4 GB) 6.80 GB/s (2–4 th) 259 325 340 ~1.8 s (warm 256-tok decode: 5.01 s @ 51.0 tok/s w/ MoCE)

reading:

  • 200 tok/s on cold prompts is mostly a chunk-size question
  • pruning mostly buys the per-turn floor
  • Air throttling will pull sustained cold prefill down (guess 150–250 tok/s, unmeasured)
  • design implication for the Air: maximize reuse so the GPU only works in short bursts (disk KV cache, SCR, preserved thinking)

5.3 decode #

  • per token reads ~1.5–2 GB of weights [est] vs ~150 MB of context-dependent traffic at 128K (QSA KV ~50 MB + indexer scan ~100 MB)
  • Strata RTX 5070: 94 → 76 tok/s from short to 128K [src]
  • on the Air, decode is dominated by SSD misses (~14 tok/s measured on M4 vs ~80 tok/s bandwidth bound)
  • so MoCE / residency / prefetch matter. attention tricks don't.

6. measurement plan #

principle:

  • Colab measures hardware-independent quantities (quality, routing, misses per token/turn under each policy)
  • the local machines measure hardware constants (SSD GB/s with the real read pattern, resident compute ceiling, thermals)
  • multiply them to predict latency for every configuration, then validate only the top 2–3 for real

6.1 Colab (A100 80 GB) #

  1. setup:
    • HF token (Swift repos are gated)
    • download Swift GSQ-RCO IQ3_S (≈ 78 GB total; shard 1 resident in VRAM, PLE shard on disk via mmap/lazy)
    • build mainline llama.cpp with CUDA
  2. routing-trace logger:
    • find the router-logit tensors in the qwen4exp graph (name unverified), hook them via the eval callback
    • log real agent sessions, prefill and decode, marked separately
    • format [est]: per token per layer, top-64 (expert id u16, logit fp16) = ~12 KB/token, so 1M tokens ≈ 12 GB. full 512-wide logits for a sampled subset only.
    • also log token ids, turn boundaries, session ids
  3. offline cache replayer (CPU, can be written locally now):
    • policies: LRU, LFU, popularity EMA, EMA + near-pick credit, inverse-router score, Belady MIN (ceiling)
    • cache sizes matching each machine × each prune
    • output: misses per decoded token and per turn (by turn size), prefetch recall / read amplification for next-layer prediction
  4. pruning variants 224 / 256 / 320:
    • saliency from our traces and/or RCO
    • GGUF slicing script
    • each variant ~40–60 GB: build → eval → upload to private HF → delete (disk is ~236 GB)
  5. quality evals for each prune and each λ:
    • KL vs unpruned on held-out traces
    • JA probe
    • world-knowledge fabrication set and "said idk when it knew" set
    • persistent-agent evals (§6.4)
    • vision check (ISTA lesson)
    • persona style score (after the tune)
  6. MoCE λ sweep, closed loop: simulate the cache inside the forward pass (routing depends on residency, so open-loop replay is wrong for λ > 0, as ds4 notes). λ ∈ {0, 0.05, 0.075, 0.1}, protect top-J ∈ {2, 3}.
  7. SCR staleness on GDN: nats/token of SCR splices vs fresh prefill on real edited contexts.
  8. prefix-sliding toggle and top-k 8 override: quality on the same evals.
  9. teacher top-k logits + known/unknown labels for healing (phase 3).

6.2 laptop (phase 0, no training) #

  1. Strata-Metal (or Slotstream) running Swift GSQ-RCO Q2_0 / IQ2_XS with a persona system prompt
  2. raw SSD read rate with expert-sized reads (and Strata's actual pattern)
  3. resident compute ceiling: prefill on a small model / short prompt (gives the ~360 tok/s check)
  4. cold prefill at 4K and 32K, chunk 1024 / 4096 / 8192 (if memory allows)
  5. thermals: 3 back-to-back 32K cold prefills with sudo powermetrics --samplers gpu_power. watch tok/s sag and watts
  6. decode at 4K / 64K / 128K context (tells us if prefix sliding has anything to win)
  7. seconds per agent turn at 500 / 2K / 8K fresh tokens, warm cache
  8. log two clocks: wall + server counters (REAP card lesson)
  • early warning: if per-turn time is unusable even with reuse, revisit "same model on laptop" (§10)

6.3 desktop (phase 0) #

  • same list, on Strata HIP (gfx1102) or llama.cpp --n-cpu-moe + mmap, in low-RAM mode
  • answers how bad 16 GB RAM really is

6.4 persistent-agent eval #

what it's for: picking K, λ / top-J, the top-k 8 and prefix-sliding flags, accepting SCR's GDN error, and regression checks after healing / persona / FTPO. model-level metrics miss compounding errors over long tool loops (ISTA Coder: LiveCodeBench kept 98.7%, SWE-bench only 91.3% [src]).

  1. tier 1, fast screen (every variant):
    • teacher-forced KL and top-1 agreement vs unpruned Swift on a fixed set of synthetic/curated persistent-agent traces (not personal session transcripts)
    • minutes per variant, no generation
  2. tier 2, task suite (finalists only), ~30–50 tasks with scripted pass checks, run through the real harness:
    • ops: in a throwaway VM/container. e.g.: find why a systemd unit fails, fix a cron job, free disk, read logs and report, install and configure a package
    • python-tool: data wrangling, file inspection, quick calculations, the albedo-style kernel use pattern
    • memory across sessions: plant facts/preferences, run unrelated turns + a compaction, then check recall and correct use later. this exercises SCR + compaction.
    • long-horizon persona consistency: voice still right after 100K+ tokens and many tool outputs. judged by a rubric, plus the slop rate.
    • abstention: world-knowledge questions where the right move is "not sure" or a search call, plus ones it should just answer
    • context-instruction following: does it obey what the harness injects (memories, its own self-edited prompt rules, LCM/compaction summaries, tool protocols), including instructions that override the voice (exact formats, writing for other people, files/commits in a specified style). this replaces generic IFBench as the category that matters.
    • JA conversation and tool use
    • vision spot checks
    • existing benchmarks worth borrowing from (check licenses/fit): Terminal-Bench (ops), tau-bench (tool + user loops), LongMemEval / LoCoMo (long-term memory)
  3. report per variant:
    • pass rate per category
    • the speed side from the latency model: seconds per turn for a typical turn mix, i.e. many small turns (user message, tool output), occasional compaction, rare cold starts