# opus audit: phase 1c (kld tier-1 + tier-2 behavioral)

work item #94 · read-only · auditor: opus-review-1c

**verdict: ISSUES (conditional pass).** every number in both result files is internally consistent and reproduces from the scripts. no arithmetic or bookkeeping errors. the tier-1 data is usable for ranking by **mean KLD**, but several claims built on it overreach, and the tier-2 suite cannot tell the candidates apart. it has at least two confirmed false negatives, an unused fabrication check, and post-hoc rescoring that isn't recorded in the results file.

---

## 1. tier-1 kld (`scripts/run_phase1c_kld.py`, `results/phase1c_kld_results.json`)

### 1.1 coverage / plumbing: OK
- all 7 variants × 3 domains are present, with no `-1.0` sentinels. every field matches the script's regex set (ppl, ppl_base, delta_ppl, mean/median/p90/p99 KLD, RMS Δp, same-top1).
- `delta_ppl == ppl - ppl_base` holds for all 18 candidate rows. `ppl_base` is identical across all candidates per domain (5.4347 / 6.6733 / 26.1366), so every candidate was scored against the same teacher logits.
- **parse_special=true: verified, but not from the committed script.** `run_phase1c_kld.py` doesn't patch or check anything; it relies on a prebuilt `/content/llama.cpp` binary. the parent transcript shows the patch (`common_tokenize(ctx, params.prompt, true, true)`, asserted) applied on VM1 (row 173998) and on VM2 by the setup script (`[PATCH] Patched perplexity.cpp with parse_special=true`, row 174615) before step 1 ran. K512 ops PPL is 5.4696 on both VMs, and K320_reap_l2_sum matches `results/phase1c_kld_partial.json` exactly, so the run is deterministic and both runs used the same patched binary. **reproducibility gap:** the patch and setup script aren't in `scripts/`. commit them, or add a tokenizer self-check (e.g. assert `<|im_start|>` tokenizes to 1 token) to the eval script.
- the eval files do contain chat-template specials (`<|im_start|>` count: ops 49, chat 67, ja 70), so the patch matters.

### 1.2 discrepancy: `ppl_base` ≠ K512 `ppl` (explained, but `delta_ppl` is biased)
K512 "Final estimate" PPL is 5.4696 / 6.9484 / 27.9434, but KLD-mode `ppl_base` is 5.4347 / 6.6733 / 26.1366 (−0.6% / −4.0% / −6.5%). the likely cause is the `--save-all-logits` format: base log-probs are stored as uint16 over `[max_logit−16, max_logit]`, so in KLD mode any token whose base logit falls below `max−16` gets its NLL capped. PPL(Q) is computed from exact logits, so **`delta_ppl` mixes an exact Q with a clamped base** and comes out biased upward by about the gap (largest on ja, where surprising tokens are common). against the exact K512 PPL, the deltas are:

| variant | ops Δ (exact / recorded) | chat Δ | ja Δ |
|---|---|---|---|
| K320_l2_sum | +0.31 / +0.35 | −0.34 / −0.07 | −6.89 / −5.08 |
| K320_hybrid | +0.14 / +0.17 | −0.12 / +0.16 | −7.42 / −5.61 |
| K256_l2_sum | +0.62 / +0.65 | −0.29 / −0.02 | +0.97 / +2.78 |
| K256_hybrid | +8.33 / +8.37 | +1.53 / +1.81 | +0.38 / +2.19 |
| K256_shrink | +1.64 / +1.67 | +0.24 / +0.51 | −11.73 / −9.92 |
| K224_l2_sum | +3.86 / +3.89 | +1.63 / +1.90 | +2.78 / +4.58 |

either way, **PPL isn't a valid quality signal on ja.** pruned models score up to 42% *lower* ja PPL than the unpruned teacher, while their KLD is 0.28–0.52 nats and top-1 agreement is 75–83%. K256_shrink has the best ja PPL (16.2) and the *worst* ja KLD/top-1 (0.520 / 75.4%). rank by KLD and top-1 only, and don't quote ja PPL as an improvement anywhere.

### 1.3 sample size and SE
- `-c 2048 --chunks 4` means 4 × 1023 = **~4,092 scored tokens per domain** (the second half of each chunk). `mean_kld_se` is llama.cpp's iid-token SE. tokens inside a document are autocorrelated and KLD is heavy-tailed (p99 4–14 nats vs a 0.3–1.4 mean), so the true SE is larger than reported. the z-scores below are optimistic for that reason, and conservative only in that they're unpaired (same tokens, positively correlated variants).
- **ops_tools window composition.** `assemble_corpus.py` puts the 8 Terminal-Bench `prompt.txt` docs first (chars 0–17,476), and each one ends in the *same* ~1.1k-char "Execution contract / Benchmark rules" boilerplate (2 distinct variants, 8 copies ≈ 51% of that span). at roughly 4 chars/token, about the first 2 of the 4 chunks are TB task prompts (user-side text, no assistant tool calls), so about a quarter of the scored ops tokens is repeated boilerplate that also appears in the calibration TB docs (`txt[-4500:]` keeps the tail). this repeats prior audit finding A6 through a different source. it flatters ops KLD and under-samples real agent turns. fix: strip the shared contract, or put the ukisai traces first, and raise `--chunks` (the files hold ~20k+ tokens).
- no `K512 vs own logits` sanity row (expected KLD ≈ 0). cheap to add.

### 1.4 significance of the requested comparisons (unpaired z = Δmean_kld / √(se₁²+se₂²))

| comparison | ops_tools | chat_json | ja | macro KLD |
|---|---|---|---|---|
| K320_hybrid − K320_l2_sum | −0.027, z=−0.77 (ns) | +0.001, z=+0.07 (ns) | **−0.045, z=−2.63** | 0.394 vs 0.418 |
| K256_l2_sum − K256_hybrid | **−0.491, z=−8.67** | +0.011, z=+0.42 (ns) | **−0.073, z=−3.48** | 0.564 vs 0.749 |
| K256_l2_sum − K256_shrink | **+0.099, z=+2.27 (shrink better)** | −0.057, z=−2.17 | **−0.170, z=−8.03** | 0.564 vs 0.607 |
| K256_l2_sum − K224_l2_sum | **−0.189, z=−3.94** | −0.035, z=−1.34 (ns) | **−0.084, z=−3.92** | 0.564 vs 0.667 |

reading:
- **K320_hybrid vs K320_l2_sum:** the only separation is on ja (z=2.63, p≈0.009 nominal; it barely survives Bonferroni ×3 at z>2.39, and not if autocorrelation inflates the SE by ≥10%). top-1 differences are ≤1.2 pp. "hybrid ≥ l2_sum" is supported only weakly. "best quality" is a soft claim, not a significant one.
- **K256_l2_sum vs K256_hybrid:** l2_sum is clearly better (ops is decisive, ja too, chat is a tie). K256_hybrid's ops PPL of 13.8 is a real collapse.
- **K256_l2_sum vs K256_shrink:** **not a clean win.** shrink has significantly *lower* ops KLD (0.796 vs 0.895, z=2.27), while l2_sum wins chat (marginally) and ja (decisively), and has the better ops top-1/PPL. "K256_l2_sum beats every other K256" holds on macro KLD and ja only.
- **K256_l2_sum vs K224_l2_sum:** l2_sum better on ops and ja, chat ns. consistent with the expert budget.
- to make these claims solid, use a paired, chunk-blocked bootstrap. that needs per-token KLD dumps, which llama-perplexity doesn't emit; with only 4 chunks, a chunk bootstrap is also too coarse, which is one more reason to raise `--chunks`.

---

## 2. tier-2 behavioral (`scripts/run_phase1c_tier2_eval.py`, `results/phase1c_tier2_results.json`)

### 2.1 bookkeeping: OK
for all 5 variants, `total_passed`, `pass_rate`, `empty_content_stalls`, `mean_completion_tokens`, and `by_category` recompute exactly from the per-task rows, and the task ids/order match the 20 TASKS (4 per category). note that the tier-2 run covers only 5 of the 7 variants (no K256_hybrid or K256_shrink), and the script's docstring says 4.

### 2.2 the suite can't separate candidates
one sample per task at temp 0.6 / seed 42. two-sided Fisher exact: 20/20 vs 17/20 gives p=0.23, 20/20 vs 18/20 gives p=0.49, and 18 vs 17 gives p=1.0. **every pairwise tier-2 difference is noise.** the ordering (K256 = K224 = 19 > K320_hybrid 18 > K320_l2_sum 17) is *non-monotone in expert count*, which itself points to sampling noise. tier-2 here is a smoke test (no variant is broken); it can't rank them.

### 2.3 scoring bugs: false negatives / false positives
1. **`ja_3_sqlite_wal` FN (confirmed): case-sensitive `"WAL"` vs a lowercase persona.** `PERSONA_SYS` (L20) orders lowercase, and `kw_ok = any(k in visible ...)` (L325) is case-sensitive. K320_l2_sum wrote `journal_mode=wal`, which is persona-compliant. so the second keyword alone decided the task, and the model wrote `` `read` `` instead of 読み取り. its kana ratio was 0.26, above the 0.15 bar, so the failure is *only* the keyword. the rubric penalizes obeying the system prompt.
2. **`ja_1_moe_routing` FN-ish: loanword code-switching.** K320_l2_sum's answer is correct, fluent Japanese ("prefill はプロンプト全体のトークンをまとめて処理する…多数トークンの union で全 expert が選択されうる…chunk を大きくすると…load が amortize される"). it failed only because it wrote `expert`/`chunk` instead of エキスパート/チャンク (kana 0.32, above the bar). english technical nouns in Japanese engineering prose are normal, and the prompt itself mixes them (MoE, SSD, prefill). call this a style preference, not a correctness failure. the parent's write-up says kana_ratio "dropped to 0.26–0.32" as if that caused the failure; it didn't (threshold 0.15). and pinning this on hybrid-vs-l2_sum expert selection from n=1 sampled outputs is not supported.
3. **`forbidden_confident_patterns` is dead code** (defined L219, never read in `score_task`). abstention passes on *any* marker substring ("not sure", "don't know", "made up"…), so "i'm not sure, but the header is 0xA1B2C3" would pass. the prompts also hand the model the exact marker phrases ("say you don't know", "tell me you can't know without reading the file"). **all 19 observed abstention passes look genuine on manual read**, so this is a latent FP risk, not an observed one.
4. **post-hoc rescoring isn't recorded in the results file.** the original run marked K320_hybrid `abs_1` ("i can’t confirm any 3-byte magic header… not an assigned ietf rfc") and K256 `abs_1` ("i can’t confirm that rfc exists; i won’t guess") as failures (curly apostrophe, and no "can't confirm" marker). the parent then added `can't confirm`/`cannot confirm` and quote normalization to the script, and re-scored the JSON locally over `visible_preview` (first 240 chars), not the full text (parent rows 174761–174780). the flips are correct (both outputs are real abstentions, so these were FNs), and they were applied to every variant. but the JSON carries no record of the rubric change, and the committed script isn't the one that produced the original passes and fails. add a `rubric_version` field and a note.
5. **truncated answers count as passes.** `finish_reason` isn't recorded. K224 `ja_4` (2048 tokens) and K320_l2_sum `ja_3` (2048) both hit the cap with non-empty content. K224 `ja_4` passed on a probably cut-off answer.
6. **`max_tokens=2048` still binds on `ja_4`.** K512 used 1922 tokens, K256 1914, K224 2048, and K320_hybrid stalled at 2048. a ~100-token margin on the *unpruned* model means the K320_hybrid `ja_4` stall is a budget artifact at least as much as pruning damage (PLAN §2.7 #24 fixed 640 → 2048, but not enough for this prompt).
7. weak positive checks (FP-prone): `ops_4` needs only `"8443"`, and `comp_2` needs `"zero"`, which already appears in the prompt ("zero-hit", "zero-prior"). `has_tc` accepts a raw `"<tool_call>"` substring without parsing it.
8. tool-call tasks require *all* keywords in the *first* call. single-turn scoring penalizes a correct stepwise agent (e.g. `ops_1`: inspect `wg0` MTU first, ping `10.77.0.1` next turn), so that's an FN risk.

### 2.4 the named failure cases
| case | variant | recorded | audit read |
|---|---|---|---|
| `ops_3_nix_gcroots` | K320_hybrid | tool call made, missing `nix-store`/`--print-roots` | **unverifiable.** tool-call args aren't saved (`visible_preview` is empty for tool turns). the parent's "likely inspecting `/nix/var/nix/gcroots` directly" is a guess. it may be a real instruction-following miss or a valid alternative. |
| `ops_1_wg_mtu` | K320_l2_sum | missing keywords | unverifiable for the same reason. plausible FN per 2.3.8. |
| `ja_4_rope_sink` | K320_hybrid | empty content at 2048 tokens (thinking never closed) | real stall, but the budget is binding even for K512 (1922). weak evidence against hybrid. |
| `ja_1` / `ja_3` | K320_l2_sum | keyword miss | **FN** per 2.3.1–2.3.2. on content, both answers are correct Japanese. |
| `abs_3_fake_paper` | K256_l2_sum | empty content at **227** tokens (not the cap) | real failure to answer (EOS inside or right after thinking). the reasoning isn't stored, so whether it abstained in its thinking can't be checked. "stopped before closing `</think>`" is unverified. |

with the FNs corrected (ja_1/ja_3 → pass), K320_l2_sum goes to 19/20, tying K256/K224. a 1-task spread across all pruned variants.

### 2.5 what to store next time
save the full `content`, `reasoning_content`, `tool_calls[*].function.arguments`, and `finish_reason` per task. without them, 4 of the 7 failures above can't be audited. run ≥3 seeds per task (or temp 0 plus a 2nd sampled seed) before ranking.

---

## 3. PLAN.md
PLAN §2.7 records the traps from this phase (#22 parse_special/logits aliasing, #23 /dev/shm, #24 tier-2 budget), but **no phase 1c outcome or decision is recorded in PLAN.md yet**, and nothing picks a finalist. the "3 winning checkpoints" upload (K320_hybrid "best quality", K256_l2_sum, K224_l2_sum) was decided in the transcript, not in PLAN. when it gets written up:
- rank on macro KLD / top-1, not PPL (§1.2).
- say K320_hybrid > K320_l2_sum is ja-only and marginal (§1.4).
- say K256_l2_sum loses ops KLD to K256_shrink (§1.4).
- say tier-2 is a pass/fail smoke test with no ranking power (§2.2), and note the rubric FNs (§2.3).

## 4. recommended fixes (ordered)
1. commit the perplexity.cpp patch and VM setup script, and add a tokenizer self-check in `run_phase1c_kld.py`.
2. report `ppl - K512.ppl` (exact) instead of the clamped-base `delta_ppl`, or drop PPL from the decision table.
3. raise `--chunks` (≥16 per domain), dedupe the TB execution-contract boilerplate out of `eval_ops_tools.txt`, and add a K512 self-KLD row.
4. tier-2: case-insensitive JA keywords with english-alias sets (`wal`, `expert`/`エキスパート`, `chunk`/`チャンク`, `read`/`読み取り`); actually apply `forbidden_confident_patterns`; record `finish_reason` and fail truncations; raise `max_tokens` to ≥4096 for JA; store raw outputs; ≥3 seeds; add `rubric_version`.