# 02 — sizes, streaming speed evidence, performance model & measurement plan

## 3. sizes [est]

expert = 4.915M params. totals across 48 layers:

| experts/layer | IQ2_XS (2.31 bpw) | Q2_0 (**2.25 bpw** [src: `ggml.h` type 42, 64w=18B]) | Q3_K / IQ3_S (3.44 bpw) | IQ3_XXS (3.06 bpw) |
|---|---|---|---|---|
| 512 | 34.9 GB | **34.0 GB** | 51.9 | 46.2 |
| 320 | 21.8 | **21.2** | 32.4 | 28.9 |
| 256 | 17.5 | **17.0** | 26.0 | 23.1 |
| 224 | 15.3 | **14.9** | 22.7 | 20.2 |

other fixed costs:
- dense ≈ 3.2 GB
- KV at 128K: fp16 3.0 GiB, q8 1.6 GiB, q4 0.84 GiB
- GDN state ≈ 110 MiB (fp32)
- mmproj ≈ 0.5–0.9 GB

fast-memory budget for experts:
- desktop: ~4.3 GB VRAM + ~7.5 GB RAM ≈ 11.8 GB ≈ 178 experts/layer at Q2_0 (1.3824 MB/expert)
- laptop: ~7.5 GB ≈ 113 experts/layer at Q2_0
- so on both machines, even 224 is partly streamed from SSD

desktop note: Strata needs ≥32 GB system RAM [src: Strata MODELS.md, web], and the measured 7600 XT run had 64 GB. **16 GB RAM is the desktop bottleneck, not the GPU.** options:
- (a) more RAM
- (b) Strata/llama.cpp low-RAM mode with heavier SSD streaming
- (c) prune harder for the desktop

---

## 4. what's known about streaming speed

### 4.1 Mac evidence [src: Strata docs/MAC.md, the only streamed-Mac measurement]
M4 Mac mini, 32 GB, experts streamed, 11.5–14.7 GB arena, no MTP, medians of 5:

| | Q2_0 | IQ3_XXS |
|---|---|---|
| decode, short | 14.1 | 10.0 |
| decode after 6K doc | 13.1 | 7.7 |
| prompt, 4K tokens | **93.6** | **65.8** (84.1 with experts' copy) |

- M5 Max 128 GB, resident: Q2_0 prompt ~1,430 tok/s at 4K, decode 87 tok/s without MTP.
- Slotstream [src]: 24–48 GB Macs, decode ~5–16 tok/s (est. table), 20.9–23.7 tok/s on a 48 GB M5 Pro. no prefill numbers.

### 4.2 ds4-recipe evidence [src: tangled.org/astrra.space/ds4-recipe]
DeepSeek V4.1 Flash, Strix Halo 128 GB UMA, 3 NVMe, ~31% of experts resident:
- cold prefill 354–391 tok/s (README says ~450 now), from:
  - 4 lanes × 8,192-token chunks sharing one read of each layer's expert union
  - hipBLASLt prepared GEMMs
- **p50 prefill by turn size**:

| new tokens in turn | p50 tok/s |
|---|---|
| 64–256 | 19 |
| 256–1k | 46 |
| 1k–4k | 106 |
| 4k–16k | 236 |
| 16k+ | ~340 |

small turns are SSD-bound.

### 4.3 Air thermals & local M5 hardware microbench [src: local `bench/README.md` + web]
- **measured on this 13" M5 Air (`Mac17,3`, 10-core GPU, 1 TB SSD, `bench/run_01..03`):**
  - **M5 10-core GPU FP16 GEMM (`K=2560`):** **12.60 TFLOP/s** at M=512, **13.66 TFLOP/s** at M=1024, **14.07 TFLOP/s** at M=4096 (`recommendedMaxWorkingSetSize = 17.76 GiB`). exactly 1/4 of the 40-core M5 Max, confirming the **~356–360 tok/s** resident prefill compute ceiling.
  - **60-second continuous GPU thermal sag test (`M=4096, K=2560, N=2560` FP16):** started at **14.00 TFLOP/s** (`t=0..15s`) and ended at **13.88 TFLOP/s** at `t=55..60s` (**−0.86% sag over a full minute**). so single-prompt cold prefills (even 15–30 s) do not hit a thermal cliff; multi-minute sustained loads [web: Notebookcheck ~9 W sustained] still favor KV reuse.
  - **1 TB SSD uncached read bandwidth (`F_NOCACHE=1`, 100.0% physical `disk0` verified via `iostat -Id` on virgin 1 GiB slices):**
    - standard GGUF (`512 KiB` × 3 reads/expert): **2.41 GB/s** (1 thread) → **5.52 GB/s** (2 threads) → **7.03 GB/s** (4 threads)
    - contiguous `.experts` copy (`1.5 MiB` × 1 read/expert): **3.85–3.93 GB/s** (1 thread) → **6.84 GB/s** (2 threads) → **6.60–6.89 GB/s** (4 threads)
    - PLE n-gram table (`4 KiB` random page reads): **0.04 GB/s** (~10k IOPS, ~1.6 ms/tok for 16 heads) at 1 thread → **0.33 GB/s** (~82k IOPS, ~0.2 ms/tok) at 8 threads
  - **methodological trap caught (`run_02` → `run_03`):** on macOS APFS, repeating `pread` on identical file offsets across benchmark iterations can partially hit APFS cache on pass 2+ even with `F_NOCACHE=1`. always use virgin slices and check `iostat -Id` (`disk0` MiB delta == 100% of requested MiB).

---

## 5. performance model [est, to be replaced by measurements]

### 5.1 the metric: per-turn floor
- each prefill chunk (or each turn, however short) reads, for every layer, the union of experts its tokens picked. with top-10 of 512, even ~256 tokens touch most of a layer (uniform-routing approximation; real routing is skewed, so less).
- so **per-turn floor ≈ non-resident expert bytes ÷ SSD read rate**, plus compute.
- kernel opts move the ceiling. pruning, residency and MoCE move the floor.

### 5.2 laptop model (anchored to measured M5 Air SSD & GPU constants)
inputs:
- Q2_0 (2.25 bpw = **1.3824 MB/expert** [src: `ggml.h` type 42]), ~7.5 GB experts resident (~113 experts/layer)
- compute ceiling **356 tok/s** [src: local Metal FP16 GEMM **14.07 TFLOP/s** on M5 10-core GPU, scaled from M5 Max 1,430 tok/s @ 40c]
- measured SSD rates on this 13" M5 Air (`iostat -Id` 100% physical `disk0` verified) [src: `bench/README.md`]:
  - **2.41 GB/s**: 1-thread standard GGUF (3 × 512 KiB/expert)
  - **3.90 GB/s**: 1-thread contiguous `.experts` copy (1 × 1.5 MiB/expert)
  - **6.80 GB/s**: 2–4 thread `pread` fan-out (saturates PCIe Gen4 x4 NVMe)
- no IO/compute overlap (pessimistic), before MoCE (measured Phase 1b OOS `lru` `λ = 0 -> 0.09` with `protect_top_j = 3` cuts decode misses **17.4–19.9%** on its own, or **−36.4% at `K320_reap_hybrid` / −47.4% at `K256_reap_l2_sum`** combined with pruning; see §7.1 for exact OOS decode tok/s)

| experts (non-resident Q2_0 @ 2.25 bpw) | SSD mode (measured) | 1024 chunk | 4096 | 8192 | 256-tok cold prefill floor (pre-MoCE) |
|---|---|---|---|---|---|
| 512 (26.5 GB) | 2.41 GB/s (std 1-th) | 74 tok/s | 182 | 241 | ~11.7 s |
| 512 (26.5 GB) | 3.90 GB/s (`.experts` 1-th) | 106 | 224 | 275 | ~7.5 s |
| 512 (26.5 GB) | 6.80 GB/s (2–4 th) | 151 | 266 | 304 | ~4.6 s |
| 320 (13.7 GB) | 3.90 GB/s (`.experts` 1-th) | 160 | 273 | 309 | ~4.2 s |
| 320 (13.7 GB) | 6.80 GB/s (2–4 th) | 209 | **303** | 327 | **~2.7 s** (warm 256-tok decode: `5.88 s` @ `43.5 tok/s` w/ MoCE) |
| 256 (9.5 GB) | 3.90 GB/s (`.experts` 1-th) | 193 | 294 | 322 | ~3.2 s |
| 256 (9.5 GB) | 6.80 GB/s (2–4 th) | 240 | **318** | 336 | **~2.1 s** (warm 256-tok decode: `5.40 s` @ `47.4 tok/s` w/ MoCE) |
| 224 (7.4 GB) | 3.90 GB/s (`.experts` 1-th) | 215 | 306 | 329 | ~2.6 s |
| 224 (7.4 GB) | 6.80 GB/s (2–4 th) | 259 | **325** | 340 | **~1.8 s** (warm 256-tok decode: `5.01 s` @ `51.0 tok/s` w/ MoCE) |

reading:
- 200 tok/s on cold prompts is mostly a chunk-size question
- pruning mostly buys the per-turn floor
- Air throttling will pull sustained cold prefill down (guess 150–250 tok/s, unmeasured)
- design implication for the Air: maximize reuse so the GPU only works in short bursts (disk KV cache, SCR, preserved thinking)

### 5.3 decode
- per token reads ~1.5–2 GB of weights [est] vs ~150 MB of context-dependent traffic at 128K (QSA KV ~50 MB + indexer scan ~100 MB)
- Strata RTX 5070: 94 → 76 tok/s from short to 128K [src]
- on the Air, decode is dominated by SSD misses (~14 tok/s measured on M4 vs ~80 tok/s bandwidth bound)
- so MoCE / residency / prefetch matter. attention tricks don't.

---

## 6. measurement plan

principle:
- **Colab measures hardware-independent quantities** (quality, routing, misses per token/turn under each policy)
- **the local machines measure hardware constants** (SSD GB/s with the real read pattern, resident compute ceiling, thermals)
- multiply them to predict latency for every configuration, then validate only the top 2–3 for real

### 6.1 Colab (A100 80 GB)
1. setup:
   - HF token (Swift repos are gated)
   - download Swift GSQ-RCO IQ3_S (≈ 78 GB total; shard 1 resident in VRAM, PLE shard on disk via mmap/lazy)
   - build mainline llama.cpp with CUDA
2. **routing-trace logger**:
   - find the router-logit tensors in the qwen4exp graph (name unverified), hook them via the eval callback
   - log real agent sessions, prefill and decode, marked separately
   - format [est]: per token per layer, top-64 (expert id u16, logit fp16) = ~12 KB/token, so 1M tokens ≈ 12 GB. full 512-wide logits for a sampled subset only.
   - also log token ids, turn boundaries, session ids
3. **offline cache replayer** (CPU, can be written locally now):
   - policies: LRU, LFU, popularity EMA, EMA + near-pick credit, **inverse-router score**, Belady MIN (ceiling)
   - cache sizes matching each machine × each prune
   - output: misses per decoded token and per turn (by turn size), prefetch recall / read amplification for next-layer prediction
4. **pruning variants** 224 / 256 / 320:
   - saliency from our traces and/or RCO
   - GGUF slicing script
   - each variant ~40–60 GB: build → eval → upload to private HF → delete (disk is ~236 GB)
5. **quality evals** for each prune and each λ:
   - KL vs unpruned on held-out traces
   - JA probe
   - world-knowledge fabrication set **and** "said idk when it knew" set
   - persistent-agent evals (§6.4)
   - vision check (ISTA lesson)
   - persona style score (after the tune)
6. **MoCE λ sweep, closed loop**: simulate the cache inside the forward pass (routing depends on residency, so open-loop replay is wrong for λ > 0, as ds4 notes). λ ∈ {0, 0.05, 0.075, 0.1}, protect top-J ∈ {2, 3}.
7. **SCR staleness on GDN**: nats/token of SCR splices vs fresh prefill on real edited contexts.
8. prefix-sliding toggle and top-k 8 override: quality on the same evals.
9. teacher top-k logits + known/unknown labels for healing (phase 3).

### 6.2 laptop (phase 0, no training)
1. Strata-Metal (or Slotstream) running Swift GSQ-RCO Q2_0 / IQ2_XS with a persona system prompt
2. raw SSD read rate with expert-sized reads (and Strata's actual pattern)
3. resident compute ceiling: prefill on a small model / short prompt (gives the ~360 tok/s check)
4. cold prefill at 4K and 32K, chunk 1024 / 4096 / 8192 (if memory allows)
5. thermals: 3 back-to-back 32K cold prefills with `sudo powermetrics --samplers gpu_power`. watch tok/s sag and watts
6. decode at 4K / 64K / 128K context (tells us if prefix sliding has anything to win)
7. **seconds per agent turn at 500 / 2K / 8K fresh tokens**, warm cache
8. log two clocks: wall + server counters (REAP card lesson)
- early warning: if per-turn time is unusable even with reuse, revisit "same model on laptop" (§10)

### 6.3 desktop (phase 0)
- same list, on Strata HIP (gfx1102) or llama.cpp `--n-cpu-moe` + mmap, in low-RAM mode
- answers how bad 16 GB RAM really is


### 6.4 persistent-agent eval
what it's for: picking K, λ / top-J, the top-k 8 and prefix-sliding flags, accepting SCR's GDN error, and regression checks after healing / persona / FTPO. model-level metrics miss compounding errors over long tool loops (ISTA Coder: LiveCodeBench kept 98.7%, SWE-bench only 91.3% [src]).

1. **tier 1, fast screen (every variant):**
   - teacher-forced KL and top-1 agreement vs unpruned Swift on a fixed set of synthetic/curated persistent-agent traces (not personal session transcripts)
   - minutes per variant, no generation
2. **tier 2, task suite (finalists only), ~30–50 tasks with scripted pass checks, run through the real harness:**
   - **ops:** in a throwaway VM/container. e.g.: find why a systemd unit fails, fix a cron job, free disk, read logs and report, install and configure a package
   - **python-tool:** data wrangling, file inspection, quick calculations, the albedo-style kernel use pattern
   - **memory across sessions:** plant facts/preferences, run unrelated turns + a compaction, then check recall and correct use later. this exercises SCR + compaction.
   - **long-horizon persona consistency:** voice still right after 100K+ tokens and many tool outputs. judged by a rubric, plus the slop rate.
   - **abstention:** world-knowledge questions where the right move is "not sure" or a search call, plus ones it should just answer
   - **context-instruction following:** does it obey what the harness injects (memories, its own self-edited prompt rules, LCM/compaction summaries, tool protocols), including instructions that override the voice (exact formats, writing for other people, files/commits in a specified style). this replaces generic IFBench as the category that matters.
   - **JA** conversation and tool use
   - **vision** spot checks
   - existing benchmarks worth borrowing from (check licenses/fit): Terminal-Bench (ops), tau-bench (tool + user loops), LongMemEval / LoCoMo (long-term memory)
3. **report per variant:**
   - pass rate per category
   - the speed side from the latency model: seconds per turn for a typical turn mix, i.e. many small turns (user message, tool output), occasional compaction, rare cold starts

---