# opus audit v2: phase 1c fixes + phase 2 pareto

read-only · auditor: opus-verify-1c-2 · scope: fixes for `audits/opus_audit_phase1c.md`, `results/phase2_pareto_analysis.json`, `PLAN.md` §2.7.A #25 and §7.1

**verdict: ISSUES.** every number in `results/phase2_pareto_analysis.json` recomputes from phase 0 + phase 1b/1c files (0 mismatches), and the Tier-1 numbers in PLAN §7.1 are exact. three things still fail the falsifier:

1. **PLAN §7.1 item 3 (phase 1b) quotes MoCE numbers that aren't in any committed JSON** (55.2 / 88.58% / 45.6 / 90.51%), and its "beats Belady" claim is false for the policy it names.
2. **PLAN §7.1 item 2 (phase 1a) labels in-sample retention as OOS**, and "matches in-sample within 0.3%" is contradicted by `results/oos_prune_retention.json` (the real gap is 4.6–5.8 pp).
3. **the Tier-2 results tagged `v2_opus_1c_fixes` were never re-run.** they're a local rescore of the old 2048-token v1 run over 240-char previews. only the JA keyword groups and the truncation rule were applied. the tool-call, abstention, and comp_2 rubric changes and the 4096 budget never reached the data, so the label overclaims.

---

## 1. were the opus_audit_phase1c.md findings addressed?

| audit item | status | evidence |
|---|---|---|
| 1.1 commit perplexity patch + tokenizer self-check | **partial** | `scripts/patch_perplexity.py` is committed and fail-closed. `run_phase1c_kld.py` L5 asserts the patched *source text*, not the tokenizer at runtime (no `<|im_start|>` → 1 token check), so an un-rebuilt binary would still pass. the VM setup script isn't committed. |
| 1.2 report exact Δppl, rank by KLD/top-1 | **done** | `exact_delta_ppl == ppl − K512.ppl` for all 18 rows (recomputed: e.g. K320_l2_sum ja −6.8913, K256_shrink ja −11.725). script emits it too. PLAN #25 states the rule. `clamped_delta_ppl` is in the JSON but the script doesn't emit it (post-hoc local edit, harmless). |
| 1.3a strip TB execution-contract boilerplate, put ukisai first | **done in code, not in data** | `assemble_corpus.py` L103–106 `clean_tb_prompt` + L249 `uki_eval_docs[:25] + tb_eval_docs`. but the KLD results weren't regenerated, so every ops_tools KLD in the JSON/PLAN still comes from the boilerplate-heavy corpus, and PLAN #25 says "fixed" without saying so. **side effect:** `clean_tb_prompt` is also applied to `tb_calib_docs` (L112), so the committed script no longer reproduces the `calibration_mix_v2.txt` behind phase 1a REAP / 1b traces. there's no git history here to recover the old version. |
| 1.3b raise `--chunks` ≥16 | **not done** | `run_phase1c_kld.py` still passes `--chunks 4` (L37, L85). |
| 1.3c K512 self-KLD sanity row | **not done** | not in the script or `phase1c_kld_results.json`. (the parent's reasoning at seq 174867 says "included a self-check" but no code was added.) |
| 1.4 honest significance claims | **done** | PLAN §7.1 item 4 restates all 4 comparisons with the audit's z-scores (recomputed exactly: −0.77/+0.07/−2.63, −8.67/+0.42/−3.48, +2.27/−2.17/−8.03, −3.94/−1.34/−3.92). it says K320 hybrid vs l2_sum is ja-only and marginal, and that shrink wins ops KLD. |
| 2.1 docstring variant count | **done** | docstring lists the 5 evaluated variants. |
| 2.2 tier-2 has no ranking power | **partial** | PLAN calls it a "smoke suite" but never says pairwise differences are noise (Fisher p ≥ 0.23). |
| 2.3.1–2 case-insensitive JA alias groups | **done (script + rescore)** | `expect_ja_keyword_groups` with `.lower()`. rescore (seq 174871) uses `orig_full_text_kw_ok OR alias_kw_ok(preview)`. |
| 2.3.3 enforce `forbidden_confident_patterns` | **script only** | enforced in `score_task`. the rescore never re-ran abstention scoring, so the results still carry v1 verdicts. |
| 2.3.4 `rubric_version` + note on rescoring | **partial / misleading** | `rubric_version: v2_opus_1c_fixes` is on every row, but nothing says it's a rescore. the JSON lacks the script's own output fields (`finish_reason`, `visible_full`, `reasoning_preview`, `tool_calls_raw`), and notes carry v1 strings (e.g. no `missing_ja_keyword_group`, while v1 `missing_ja_keyword` remains on stalled rows), which proves the committed script didn't produce this file. |
| 2.3.5 record finish_reason, fail truncations | **script ok; results use a proxy** | rescore sets `truncated = completion_tokens >= 2048` (sound proxy for a 2048 cap). K224 ja_4 is now failed correctly. |
| 2.3.6 raise max_tokens | **script only** | script has 4096, but every result still caps at 2048 (K320_hybrid ja_4, K320_l2_sum ja_3, K224 ja_1, K224 ja_4 = 2048 tokens). 4 of 7 pruned-variant failures are budget-bound truncations that a 4096 re-run might flip. |
| 2.3.7 weak checks (ops_4, comp_2 `zero`, raw `<tool_call>`) | **script only** | fixed in TASKS (ops_4 any-of tool names, comp_2 `zero` removed, has_tc needs open+close tags). not applied to results. |
| 2.3.8 stepwise tool-call FN | **script only** | ops_1 relaxed to `["wg0"]`, ops_3 to any-of `nix-store`/`gcroots`. **tool args were never saved in the v1 run, so these can't be rescored.** K320_l2_sum `ops_1` and K320_hybrid `ops_3` failures are v1-rubric verdicts under a v2 label. under the real v2 rubric, either K320 variant could be 19/20. |
| 2.5 store raw outputs, ≥3 seeds | **raw: script only; seeds: not done** | single seed 42 remains. |
| 3 record phase 1c outcome/finalists in PLAN | **done** | PLAN §7.1 item 4 + finalist list. |

## 2. phase 2 pareto: arithmetic recomputation

constants: expert = 4.915M × 2.25 / 8 = **1.38234 MB**; resident = 113 × 48 × 1.38234 = 7.498 GB (≈ 7.5). cold model (no overlap, from PLAN §5.2): `t = nonres/6.80 GB/s + n/356 tok/s`.

| variant | total GB (K×48×MB) | nonres (−7.5) | 4K cold tok/s `4096/(nr/6.8+4096/356)` | 256-tok floor `nr/6.8+256/356` | JSON |
|---|---|---|---|---|---|
| K512 | 33.972 | 26.472 | 266.00 | 4.612 | 33.97 / 26.47 / 266.0 / 4.61 ✓ |
| K320 (both) | 21.233 | 13.733 | 302.84 (302.85 from rounded 13.73) | 2.739 | 21.23 / 13.73 / 302.9 / 2.74 ✓ (rounding) |
| K256_l2_sum | 16.986 | 9.486 | 317.50 | 2.114 | 16.99 / 9.49 / 317.5 / 2.11 ✓ |
| K224_l2_sum | 14.863 | 7.363 | 325.38 | 1.802 | 14.86 / 7.36 / 325.4 / 1.80 ✓ |

phase 1b fields pick, for each variant, the OOS row `policy=near_pick_ema, λ=0.09, protect_top_j=3` plus the OOS `lru, λ=0` hit rate from `phase1b_heldout_cache_moce_sweep.json`. every value matches exactly: K512 0.8232/0.8249/84.03/23.93/34.38; K320_hybrid 0.8639/0.8672/63.74/28.91/40.07; K320_l2_sum 0.8581/0.86/67.2/27.92/38.97; K256_l2_sum 0.8869/0.8905/52.55/32.65/44.08; K224 0.9055/0.9101/43.16/36.63/48.14. sweep internals check out too: `ssd_mb_per_tok = misses × 1.3823`, and `decode = 1/(ssd_MB/3900 + 0.012 s)` (1-th) or `/6800` (4-th), constant across rows.

`warm_256tok_turn_sec_moce_pread4 = 256 × ssd_mb_per_tok / 6800 + 256/356` recomputes (5.092 / 4.037 / 4.217 / 3.454 / 2.965 ✓). **but the field is mislabeled:** it charges every token's misses sequentially (a 256-token *decode* model), so it comes out *larger* than the "cold" floor for every variant. 256 × 88.12 MB = 22.6 GB for K320_hybrid, more than the entire 13.73 GB non-resident set. as a prefill/turn floor that's physically impossible. rename it (e.g. `decode_256tok_sec_moce_4th`) or drop it next to the cold floor. it's also why PLAN §5.2's "~1.3 s w/ MoCE" estimate isn't supported by anything measured.

Tier-1 (macro = mean over 3 domains): KLD 0.3941 / 0.4178 / 0.5642 / 0.6667, top-1 83.20 / 82.35 / 79.76 / 77.62 ✓. PPL fields copy the JSON ✓. K256_hybrid 0.7485 and K256_shrink 0.6069 (PLAN) ✓. ops top-1 76.64 vs 76.37 ✓.

Tier-2: `tier2_pass_str` / stalls / truncated recompute from `phase1c_tier2_results.json` (20/0/0, 18/1/1, 18/0/1, 19/1/0, 18/1/2) ✓. `total_passed`, `pass_rate`, `by_category`, and `mean_completion_tokens` recompute from task rows for all 5 variants ✓. (provenance caveat in §1 above.)

**pareto arithmetic: PASS. 0 mismatches.**

## 3. PLAN §7.1 claim check

| item | claim | source value | status |
|---|---|---|---|
| 1 phase 0 | 2.41 / 3.90 / 6.80 GB/s, 14.07 TFLOP/s, 356 tok/s, sag 14.00→13.88 | bench/README: 2.41; 3.85–3.93; 6.60–7.03; 14.07; −0.857% | ✓ (3.90 and 6.80 are representative picks; "on 12 GiB virgin slices" should be "1 GiB virgin slices of a 12 GiB file"; "<0.86%" is really "0.86%") |
| 2 phase 1a | 1,310,720 picks/layer (131,072 × 10), 99.97% coverage | transcript seq 173802/173819 | ✓ |
| 2 phase 1a | "OOS REAP L2 retention … matches in-sample within 0.3% … (91.76% / 85.28% / 81.11% for reap_l2_sum)" | `oos_prune_retention.json`: **87.18% / 79.71% / 75.27%**. 91.76/85.28/81.11 are the *in-sample* v2 table (seq 173802/173894) | **✗ mislabeled; gap 4.6–5.8 pp, not ≤0.3%**. OOS ordering l2_sum > hybrid > shrink at every K does hold. |
| 3 phase 1b | K512 LRU 84.8 misses/tok; Belady 46.1 | OOS 84.84; 46.13 | ✓ |
| 3 phase 1b | near_pick_ema+MoCE → **55.2 (88.58%)** K320_hybrid, **45.6 (90.51%)** K256_l2_sum | OOS near_pick λ=.09: **63.74 (86.72%)**, **52.55 (89.05%)**. these values appear nowhere in either sweep JSON; they come from an earlier table at seq 174822 (also "88.13% / 57.3" for K320_l2_sum, also absent). | **✗ unreproducible** |
| 3 phase 1b | K256 "beating unpruned Belady 46.1" | near_pick+MoCE 52.55 > 46.13. only **lru**+MoCE 44.65 beats it | **✗ for the stated policy** |
| 3 phase 1b | decode 28.9–40.1 / 32.6–44.1 | near_pick rows 28.91/40.07, 32.65/44.08 | ✓, but they come from different rows than the 55.2/88.58 quoted beside them |
| 4 phase 1c | KLD, top-1, z-scores, Tier-2 counts, 13.7/9.5/7.4 GB, 302.9/317.5 tok/s | §2 above | ✓ |
| 4 finalists | 2.74 s / 2.11 s floors, decode ranges, K256 int8 fits 80 GB (256×48×4.915M ≈ 60.4 GB + 7.3 GB dense) | ✓ | ✓ |
| #25 | "max_tokens raised to 4096 … raw tool_calls/reasoning_preview/visible_full saved" | true of the script; false of the reported results (still the 2048 v1 run) | **misleading**: say the reported numbers are a v1-run rescore |

**policy-choice observation (not arithmetic):** in the OOS sweep, `lru`+MoCE λ=0.09 beats `near_pick_ema`+MoCE λ=0.09 for every variant (K320_hybrid 53.95 vs 63.74; K256_l2_sum 44.65 vs 52.55; K224_l2_sum 37.34 vs 43.16; K512 70.07 vs 84.03). the pareto table and PLAN pick the worse policy without saying why. if near_pick is meant to be the runtime policy, that needs a reason. otherwise use lru+MoCE, which is also what makes the "beats Belady" line true.

## 4. remaining fixes (ordered)
1. PLAN §7.1 item 3: replace 55.2/88.58%/45.6/90.51% with the OOS JSON rows, and either switch to lru+MoCE (53.95/88.76%, 44.65/90.70%, which beats Belady 46.13) or drop the Belady claim.
2. PLAN §7.1 item 2: quote the OOS 87.18/79.71/75.27% (or label 91.76/85.28/81.11 as in-sample) and delete "within 0.3%".
3. Tier-2: either re-run `run_phase1c_tier2_eval.py` (4096 tokens, raw outputs, ≥3 seeds), or relabel the JSON (e.g. `rubric_version: "v1_run_rescored_ja_trunc_only"` + a top-level note) and add a line to PLAN #25/§7.1 saying the ops/abstention/comp verdicts are v1 and 4 failures are 2048-cap truncations.
4. rename or drop `warm_256tok_turn_sec_moce_pread4` (it's a decode model, and exceeds the cold floor).
5. KLD: add the runtime tokenizer self-check and the K512 self-KLD row, raise `--chunks` to ≥16, re-run on the deduped ops corpus, and note in PLAN #25 that the current ops KLD predates the dedupe.
6. `assemble_corpus.py`: apply `clean_tb_prompt` only to eval docs, or flag that calibration output changed versus phase 1a.