12 kB 79 lines
6fdbee92
keep 00-..04- docs and non-persona engine/pruning files at top level; keep only persona/anti-slop in persona/
dawn
⋯
audits/opus_audit_phase1c_v2.md
sourcerendered
download raw history
at main Markdown

opus audit v2: phase 1c fixes + phase 2 pareto #

read-only · auditor: opus-verify-1c-2 · scope: fixes for audits/opus_audit_phase1c.md, results/phase2_pareto_analysis.json, PLAN.md §2.7.A #25 and §7.1

verdict: ISSUES. every number in results/phase2_pareto_analysis.json recomputes from phase 0 + phase 1b/1c files (0 mismatches), and the Tier-1 numbers in PLAN §7.1 are exact. three things still fail the falsifier:

  1. PLAN §7.1 item 3 (phase 1b) quotes MoCE numbers that aren't in any committed JSON (55.2 / 88.58% / 45.6 / 90.51%), and its "beats Belady" claim is false for the policy it names.
  2. PLAN §7.1 item 2 (phase 1a) labels in-sample retention as OOS, and "matches in-sample within 0.3%" is contradicted by results/oos_prune_retention.json (the real gap is 4.6–5.8 pp).
  3. the Tier-2 results tagged v2_opus_1c_fixes were never re-run. they're a local rescore of the old 2048-token v1 run over 240-char previews. only the JA keyword groups and the truncation rule were applied. the tool-call, abstention, and comp_2 rubric changes and the 4096 budget never reached the data, so the label overclaims.

1. were the opus_audit_phase1c.md findings addressed? #

audit item status evidence
1.1 commit perplexity patch + tokenizer self-check partial scripts/patch_perplexity.py is committed and fail-closed. run_phase1c_kld.py L5 asserts the patched source text, not the tokenizer at runtime (no `<
1.2 report exact Δppl, rank by KLD/top-1 done exact_delta_ppl == ppl − K512.ppl for all 18 rows (recomputed: e.g. K320_l2_sum ja −6.8913, K256_shrink ja −11.725). script emits it too. PLAN #25 states the rule. clamped_delta_ppl is in the JSON but the script doesn't emit it (post-hoc local edit, harmless).
1.3a strip TB execution-contract boilerplate, put ukisai first done in code, not in data assemble_corpus.py L103–106 clean_tb_prompt + L249 uki_eval_docs[:25] + tb_eval_docs. but the KLD results weren't regenerated, so every ops_tools KLD in the JSON/PLAN still comes from the boilerplate-heavy corpus, and PLAN #25 says "fixed" without saying so. side effect: clean_tb_prompt is also applied to tb_calib_docs (L112), so the committed script no longer reproduces the calibration_mix_v2.txt behind phase 1a REAP / 1b traces. there's no git history here to recover the old version.
1.3b raise --chunks ≥16 not done run_phase1c_kld.py still passes --chunks 4 (L37, L85).
1.3c K512 self-KLD sanity row not done not in the script or phase1c_kld_results.json. (the parent's reasoning at seq 174867 says "included a self-check" but no code was added.)
1.4 honest significance claims done PLAN §7.1 item 4 restates all 4 comparisons with the audit's z-scores (recomputed exactly: −0.77/+0.07/−2.63, −8.67/+0.42/−3.48, +2.27/−2.17/−8.03, −3.94/−1.34/−3.92). it says K320 hybrid vs l2_sum is ja-only and marginal, and that shrink wins ops KLD.
2.1 docstring variant count done docstring lists the 5 evaluated variants.
2.2 tier-2 has no ranking power partial PLAN calls it a "smoke suite" but never says pairwise differences are noise (Fisher p ≥ 0.23).
2.3.1–2 case-insensitive JA alias groups done (script + rescore) expect_ja_keyword_groups with .lower(). rescore (seq 174871) uses orig_full_text_kw_ok OR alias_kw_ok(preview).
2.3.3 enforce forbidden_confident_patterns script only enforced in score_task. the rescore never re-ran abstention scoring, so the results still carry v1 verdicts.
2.3.4 rubric_version + note on rescoring partial / misleading rubric_version: v2_opus_1c_fixes is on every row, but nothing says it's a rescore. the JSON lacks the script's own output fields (finish_reason, visible_full, reasoning_preview, tool_calls_raw), and notes carry v1 strings (e.g. no missing_ja_keyword_group, while v1 missing_ja_keyword remains on stalled rows), which proves the committed script didn't produce this file.
2.3.5 record finish_reason, fail truncations script ok; results use a proxy rescore sets truncated = completion_tokens >= 2048 (sound proxy for a 2048 cap). K224 ja_4 is now failed correctly.
2.3.6 raise max_tokens script only script has 4096, but every result still caps at 2048 (K320_hybrid ja_4, K320_l2_sum ja_3, K224 ja_1, K224 ja_4 = 2048 tokens). 4 of 7 pruned-variant failures are budget-bound truncations that a 4096 re-run might flip.
2.3.7 weak checks (ops_4, comp_2 zero, raw <tool_call>) script only fixed in TASKS (ops_4 any-of tool names, comp_2 zero removed, has_tc needs open+close tags). not applied to results.
2.3.8 stepwise tool-call FN script only ops_1 relaxed to ["wg0"], ops_3 to any-of nix-store/gcroots. tool args were never saved in the v1 run, so these can't be rescored. K320_l2_sum ops_1 and K320_hybrid ops_3 failures are v1-rubric verdicts under a v2 label. under the real v2 rubric, either K320 variant could be 19/20.
2.5 store raw outputs, ≥3 seeds raw: script only; seeds: not done single seed 42 remains.
3 record phase 1c outcome/finalists in PLAN done PLAN §7.1 item 4 + finalist list.

2. phase 2 pareto: arithmetic recomputation #

constants: expert = 4.915M × 2.25 / 8 = 1.38234 MB; resident = 113 × 48 × 1.38234 = 7.498 GB (≈ 7.5). cold model (no overlap, from PLAN §5.2): t = nonres/6.80 GB/s + n/356 tok/s.

variant total GB (K×48×MB) nonres (−7.5) 4K cold tok/s 4096/(nr/6.8+4096/356) 256-tok floor nr/6.8+256/356 JSON
K512 33.972 26.472 266.00 4.612 33.97 / 26.47 / 266.0 / 4.61 ✓
K320 (both) 21.233 13.733 302.84 (302.85 from rounded 13.73) 2.739 21.23 / 13.73 / 302.9 / 2.74 ✓ (rounding)
K256_l2_sum 16.986 9.486 317.50 2.114 16.99 / 9.49 / 317.5 / 2.11 ✓
K224_l2_sum 14.863 7.363 325.38 1.802 14.86 / 7.36 / 325.4 / 1.80 ✓

phase 1b fields pick, for each variant, the OOS row policy=near_pick_ema, λ=0.09, protect_top_j=3 plus the OOS lru, λ=0 hit rate from phase1b_heldout_cache_moce_sweep.json. every value matches exactly: K512 0.8232/0.8249/84.03/23.93/34.38; K320_hybrid 0.8639/0.8672/63.74/28.91/40.07; K320_l2_sum 0.8581/0.86/67.2/27.92/38.97; K256_l2_sum 0.8869/0.8905/52.55/32.65/44.08; K224 0.9055/0.9101/43.16/36.63/48.14. sweep internals check out too: ssd_mb_per_tok = misses × 1.3823, and decode = 1/(ssd_MB/3900 + 0.012 s) (1-th) or /6800 (4-th), constant across rows.

warm_256tok_turn_sec_moce_pread4 = 256 × ssd_mb_per_tok / 6800 + 256/356 recomputes (5.092 / 4.037 / 4.217 / 3.454 / 2.965 ✓). but the field is mislabeled: it charges every token's misses sequentially (a 256-token decode model), so it comes out larger than the "cold" floor for every variant. 256 × 88.12 MB = 22.6 GB for K320_hybrid, more than the entire 13.73 GB non-resident set. as a prefill/turn floor that's physically impossible. rename it (e.g. decode_256tok_sec_moce_4th) or drop it next to the cold floor. it's also why PLAN §5.2's "~1.3 s w/ MoCE" estimate isn't supported by anything measured.

Tier-1 (macro = mean over 3 domains): KLD 0.3941 / 0.4178 / 0.5642 / 0.6667, top-1 83.20 / 82.35 / 79.76 / 77.62 ✓. PPL fields copy the JSON ✓. K256_hybrid 0.7485 and K256_shrink 0.6069 (PLAN) ✓. ops top-1 76.64 vs 76.37 ✓.

Tier-2: tier2_pass_str / stalls / truncated recompute from phase1c_tier2_results.json (20/0/0, 18/1/1, 18/0/1, 19/1/0, 18/1/2) ✓. total_passed, pass_rate, by_category, and mean_completion_tokens recompute from task rows for all 5 variants ✓. (provenance caveat in §1 above.)

pareto arithmetic: PASS. 0 mismatches.

3. PLAN §7.1 claim check #

item claim source value status
1 phase 0 2.41 / 3.90 / 6.80 GB/s, 14.07 TFLOP/s, 356 tok/s, sag 14.00→13.88 bench/README: 2.41; 3.85–3.93; 6.60–7.03; 14.07; −0.857% ✓ (3.90 and 6.80 are representative picks; "on 12 GiB virgin slices" should be "1 GiB virgin slices of a 12 GiB file"; "<0.86%" is really "0.86%")
2 phase 1a 1,310,720 picks/layer (131,072 × 10), 99.97% coverage transcript seq 173802/173819 ✓
2 phase 1a "OOS REAP L2 retention … matches in-sample within 0.3% … (91.76% / 85.28% / 81.11% for reap_l2_sum)" oos_prune_retention.json: 87.18% / 79.71% / 75.27%. 91.76/85.28/81.11 are the in-sample v2 table (seq 173802/173894) ✗ mislabeled; gap 4.6–5.8 pp, not ≤0.3%. OOS ordering l2_sum > hybrid > shrink at every K does hold.
3 phase 1b K512 LRU 84.8 misses/tok; Belady 46.1 OOS 84.84; 46.13 ✓
3 phase 1b near_pick_ema+MoCE → 55.2 (88.58%) K320_hybrid, 45.6 (90.51%) K256_l2_sum OOS near_pick λ=.09: 63.74 (86.72%), 52.55 (89.05%). these values appear nowhere in either sweep JSON; they come from an earlier table at seq 174822 (also "88.13% / 57.3" for K320_l2_sum, also absent). ✗ unreproducible
3 phase 1b K256 "beating unpruned Belady 46.1" near_pick+MoCE 52.55 > 46.13. only lru+MoCE 44.65 beats it ✗ for the stated policy
3 phase 1b decode 28.9–40.1 / 32.6–44.1 near_pick rows 28.91/40.07, 32.65/44.08 ✓, but they come from different rows than the 55.2/88.58 quoted beside them
4 phase 1c KLD, top-1, z-scores, Tier-2 counts, 13.7/9.5/7.4 GB, 302.9/317.5 tok/s §2 above ✓
4 finalists 2.74 s / 2.11 s floors, decode ranges, K256 int8 fits 80 GB (256×48×4.915M ≈ 60.4 GB + 7.3 GB dense) ✓ ✓
#25 "max_tokens raised to 4096 … raw tool_calls/reasoning_preview/visible_full saved" true of the script; false of the reported results (still the 2048 v1 run) misleading: say the reported numbers are a v1-run rescore

policy-choice observation (not arithmetic): in the OOS sweep, lru+MoCE λ=0.09 beats near_pick_ema+MoCE λ=0.09 for every variant (K320_hybrid 53.95 vs 63.74; K256_l2_sum 44.65 vs 52.55; K224_l2_sum 37.34 vs 43.16; K512 70.07 vs 84.03). the pareto table and PLAN pick the worse policy without saying why. if near_pick is meant to be the runtime policy, that needs a reason. otherwise use lru+MoCE, which is also what makes the "beats Belady" line true.

4. remaining fixes (ordered) #

  1. PLAN §7.1 item 3: replace 55.2/88.58%/45.6/90.51% with the OOS JSON rows, and either switch to lru+MoCE (53.95/88.76%, 44.65/90.70%, which beats Belady 46.13) or drop the Belady claim.
  2. PLAN §7.1 item 2: quote the OOS 87.18/79.71/75.27% (or label 91.76/85.28/81.11 as in-sample) and delete "within 0.3%".
  3. Tier-2: either re-run run_phase1c_tier2_eval.py (4096 tokens, raw outputs, ≥3 seeds), or relabel the JSON (e.g. rubric_version: "v1_run_rescored_ja_trunc_only" + a top-level note) and add a line to PLAN #25/§7.1 saying the ops/abstention/comp verdicts are v1 and 4 failures are 2048-cap truncations.
  4. rename or drop warm_256tok_turn_sec_moce_pread4 (it's a decode model, and exceeds the cold floor).
  5. KLD: add the runtime tokenizer self-check and the K512 self-KLD row, raise --chunks to ≥16, re-run on the deduped ops corpus, and note in PLAN #25 that the current ops KLD predates the dedupe.
  6. assemble_corpus.py: apply clean_tb_prompt only to eval docs, or flag that calibration output changed versus phase 1a.