7.3 kB 88 lines
6fdbee92
keep 00-..04- docs and non-persona engine/pruning files at top level; keep only persona/anti-slop in persona/
dawn
⋯
audits/opus_audit_phase0_2_m5_v5.md
sourcerendered
download raw history
at main Markdown

opus audit v5: final check of the v4 doc fixes (audits/opus_audit_phase0_2_m5_v4.md §what to fix) #

scope: read-only. i checked every number on PLAN L274–282 (§2.7.A #29, #30) and L571–586 (§7.1 #5) against results/phase0_phase2_m5_validation.json (V), results/phase0_phase2_m5_unpatched_baseline.json (U) and audits/opus_audit_phase0_2_m5_v3.md (v3). i used a strict pass over the 5 strata perf dicts, 9 TF runs, 4 + 4 llama runs and the raw STRATA_RESULT tails, plus a numeric sweep of every token on those lines. no repo files were edited except this report.

what i ran:

  • scripts/run_phase0_phase2_m5_validation.py --self-test → PASS
  • vendor/strata/build-metal/route_parity, default math and STRATA_METAL_MATH=fast → 0 mismatches, 2.16e-07 / 3.06e-07 and 2.15e-07 / 4.00e-07, PASS (same as PLAN L278)
  • SHA-256 of the 5 scripts/binaries and 7 prompt files against provenance
  • a GGUF-vocab detokenization of the last 64 ids of each eval_tf_*.txt (vocab length 248,320 ✓)

final verdict: PASS (after the fixes, see the addendum). the first pass was a FALSIFIER HIT: 3 numbers had no source, 2 of them regressions from the v5 rewrite. the 5 requested fixes themselves landed.

falsifier hits #

PLAN claim source says status
L579 IQ3_S MoCE=0 4.88 tok/s (204.78 ms/tok) V strata_iq3s_moce0_short_43tok: decode_ms_per_tok=205.02 (STRATA_RESULT, perf, summary_comparison all say 205.02; T=1 window line 205.00) wrong, regression. 204.78 is in no file on disk. v4 passed every ms/tok on this line, and parent transcript row 177171 (the v5 PLAN rewrite) is the first place 204.78 shows up.
L580 Q2_0 MoCE=0 6.32 tok/s (158.13 ms/tok) V: 158.22 (T=1 window 158.20) wrong, regression, same origin
L275 175.4 MiB > 64.0 MiB on step1_ngl20_cmoe_smoke before -nr not in U and not in V or any file under results/, audits/, scripts/, bench/. all 4 runs in U (and V's 4 unpatched runs) already pass -nr and have max_swapout_delta_mib = 0.0 unarchived. the value is real but lives only in parent transcript row 175706 (trip_reason: swapouts increased by 175.4 MiB (> 64.0 MiB cap)) from a tripped run that was never saved. so the brief's claim that it comes "from results/phase0_phase2_m5_unpatched_baseline.json" is false.

fix:

  • L579 204.78 → 205.02
  • L580 158.13 → 158.22
  • L275: archive the tripped run's trip_reason + vm deltas (for example as an extra entry in U), or cite it as "[unarchived: parent transcript row 175706]".

the 5 requested fixes #

  1. the 8 "cool in v3" values: PASS. none of 7.27, 137.56, 105.12, 73.94, 13.63, 73.37, 13.22, 75.62 appears anywhere in PLAN.md. every remaining v3-attributed number is in v3:
    • 320.58 / 353.49
    • 3,194.19 / 5,793.65
    • 339.50–356.06 (MoCE speed init)
    • 9.51 → 13.52 (the MoCE 0 → 0.85 pair)
    • 16.11 + 56.29
  2. #29 / #30 stale claims: PASS for the removals.
    • 17.0 GiB, L1914, 96 graph splits, 93..163 pages, "63 generated tokens" and "generating 63" are all gone.
    • L276 now says -n 64, ~64 generated tokens / 63 decode steps, which matches the cmd (-n 64) and v4's ≈64-token detok.
    • the 175.4 MiB that replaced 17.0 GiB is unarchived (see above).
  3. thermal label: PASS.
    • L577 and L282 carry [hypothesis: passive SoC thermal throttling + concurrent background CPU load].
    • "thermal soak", "soaked" and "fanless thermal" no longer appear.
    • the runner comment on L506 still says "thermally unthrottled" (v4 fix 4b). it can't be fixed without touching scripts, which this round kept frozen. it's an open nit, not a falsifier.
  4. L583 scoring wording: PASS, verified on tokens.
    • the last 64 ids of eval_tf_ops_tools.txt contain no <think> (248068) or </think> (248069) and detokenize to the tail of the visible JSON (…"commands": [], "task_complete": true}).
    • chat_json and ja each have exactly one </think> in the scored span, with reasoning before it and visible JSON / Japanese after it.
    • no <think> opening tag is in any scored span.
    • prompt lengths: 308−64=244, 223−64=159, 520−64=456 ✓.
  5. untouched since v4: PASS.
    • V still has generated_at_utc 2026-10-15T13:00:39Z and mtime 16:00:39, which is before v4.md (16:15:42). its sha256 is 6bdd729f…ffca6.
    • all 5 provenance hashes (run script, patch_strata_metal, patch_m5_lazy_moe, prepare_strata_k256, strata_window_cli) and all 7 prompt hashes match.
    • mtimes: route_parity and the CLI are 15:57, the runner 15:56:59, all before the run.
    • the v4 checks still hold: the self-test and route_parity were re-run above.

everything else that recomputes (✓) #

  • run count and watchdog: 22 runs = 5 + 9 + 4 + 4, max swapout delta over all 22 = 0.0, watchdog 0.25 / 2.2 / 64.0.
  • llama:
    • unpatched 1.4 / 14.3 / 21.7, disk0 105,851.73 / 85,725.73
    • patched 3.22 / 38.11 / 63.27 / 0.28, disk0 33,428.34 / 46,808.58 / 28,272.94; 22 + 63 decode tokens, 1,264 prompt tokens
    • flags -ngl 99 -cmoe -nr -lzm on -n 64 ✓
    • /tmp/m5_prompt_1600tok.txt still exists and is byte-identical to llama_prompt_1264tok.txt
    • nit (carried over): the unpatched ub2048 run decoded at 1.3, so 1.40 is the ub1024 run only
  • strata init: 424.45–504.58 (MoCE speed), 424.45–506.90 (all 14)
  • strata fill: 0.6–1.3 s; 2.69–5.92 GB/s (10 Q2_0 runs), 3.98–5.04 (4 IQ3_S runs); slots 1,550 / 2,441
  • prefill: 229.99 / 268.23 tok/s, 4,452.38 / 7,635.20 ms
  • steady-state decode: 8.04 / 124.31 (34.15 + 88.35), 9.33 / 107.20 (30.78 + 73.87), 8.37 / 119.51 (24.38 + 92.44)
  • hit rates: every hit split, misses/tok and MB/tok on L579–582; 51.5% / 62.4%
  • memory: 4.73→4.68, 7.41→7.36, 1.36, 2.90 / 0.34 / 1.28 / 4.53
  • KLD blocks:
    • 0.0588 / 96.88 (0.1477 / 0.0052 / 0.0234)
    • 0.2032 / 90.63 (0.4621 / 0.0395 / 0.1081)
    • 0.2386 / 89.58 (0.5640 / 0.0406 / 0.1112)
    • all PPL pairs and per-domain top-1 match

addendum: re-check after the parent's fixes → final verdict: PASS #

  • L579: 4.88 tok/s (205.02 ms/tok). this equals V's decode_ms_per_tok ✓
  • L580: 6.32 tok/s (158.22 ms/tok). equals V ✓
  • L275: the 175.4 MiB figure is gone. the line now says "tripping the 64.0 MiB swapout watchdog before adding --no-repack / -nr ([unarchived pre-baseline smoke run])". 64.0 MiB = watchdog_policy.max_swapout_delta_mib ✓, and the claim is honestly labelled unarchived ✓
  • 204.78, 158.13 and 175.4 no longer occur anywhere in PLAN.md.
  • i re-ran the numeric sweep over L274–282 and L571–586. the only tokens it can't match are non-measurements: Niko1221 (a repo name), 256..511 (route_parity trap range, printed by the binary), top-3, and 248,320 (GGUF vocab length, verified). so there are 0 unsourced numbers.
  • V's sha256 is still 6bdd729f…ffca6, and all 5 provenance hashes still match.

the falsifier is cleared. the only open item is the runner's L506 "thermally unthrottled" comment, which is a script-side nit.