opus audit: phase 1c (kld tier-1 + tier-2 behavioral) #

work item #94 · read-only · auditor: opus-review-1c

verdict: ISSUES (conditional pass). every number in both result files is internally consistent and reproduces from the scripts. no arithmetic or bookkeeping errors. the tier-1 data is usable for ranking by mean KLD, but several claims built on it overreach, and the tier-2 suite cannot tell the candidates apart. it has at least two confirmed false negatives, an unused fabrication check, and post-hoc rescoring that isn't recorded in the results file.

---

1. tier-1 kld (scripts/run_phase1c_kld.py, results/phase1c_kld_results.json) #

1.1 coverage / plumbing: OK #

• all 7 variants × 3 domains are present, with no -1.0 sentinels. every field matches the script's regex set (ppl, ppl_base, delta_ppl, mean/median/p90/p99 KLD, RMS Δp, same-top1).
• delta_ppl == ppl - ppl_base holds for all 18 candidate rows. ppl_base is identical across all candidates per domain (5.4347 / 6.6733 / 26.1366), so every candidate was scored against the same teacher logits.
• parse_special=true: verified, but not from the committed script. run_phase1c_kld.py doesn't patch or check anything; it relies on a prebuilt /content/llama.cpp binary. the parent transcript shows the patch (common_tokenize(ctx, params.prompt, true, true), asserted) applied on VM1 (row 173998) and on VM2 by the setup script ([PATCH] Patched perplexity.cpp with parse_special=true, row 174615) before step 1 ran. K512 ops PPL is 5.4696 on both VMs, and K320_reap_l2_sum matches results/phase1c_kld_partial.json exactly, so the run is deterministic and both runs used the same patched binary. reproducibility gap: the patch and setup script aren't in scripts/. commit them, or add a tokenizer self-check (e.g. assert <|im_start|> tokenizes to 1 token) to the eval script.
• the eval files do contain chat-template specials (<|im_start|> count: ops 49, chat 67, ja 70), so the patch matters.

1.2 discrepancy: ppl_base ≠ K512 ppl (explained, but delta_ppl is biased) #

K512 "Final estimate" PPL is 5.4696 / 6.9484 / 27.9434, but KLD-mode ppl_base is 5.4347 / 6.6733 / 26.1366 (−0.6% / −4.0% / −6.5%). the likely cause is the --save-all-logits format: base log-probs are stored as uint16 over [max_logit−16, max_logit], so in KLD mode any token whose base logit falls below max−16 gets its NLL capped. PPL(Q) is computed from exact logits, so delta_ppl mixes an exact Q with a clamped base and comes out biased upward by about the gap (largest on ja, where surprising tokens are common). against the exact K512 PPL, the deltas are:

variant	ops Δ (exact / recorded)	chat Δ	ja Δ
K320_l2_sum	+0.31 / +0.35	−0.34 / −0.07	−6.89 / −5.08
K320_hybrid	+0.14 / +0.17	−0.12 / +0.16	−7.42 / −5.61
K256_l2_sum	+0.62 / +0.65	−0.29 / −0.02	+0.97 / +2.78
K256_hybrid	+8.33 / +8.37	+1.53 / +1.81	+0.38 / +2.19
K256_shrink	+1.64 / +1.67	+0.24 / +0.51	−11.73 / −9.92
K224_l2_sum	+3.86 / +3.89	+1.63 / +1.90	+2.78 / +4.58

either way, PPL isn't a valid quality signal on ja. pruned models score up to 42% lower ja PPL than the unpruned teacher, while their KLD is 0.28–0.52 nats and top-1 agreement is 75–83%. K256_shrink has the best ja PPL (16.2) and the worst ja KLD/top-1 (0.520 / 75.4%). rank by KLD and top-1 only, and don't quote ja PPL as an improvement anywhere.

1.3 sample size and SE #

• -c 2048 --chunks 4 means 4 × 1023 = ~4,092 scored tokens per domain (the second half of each chunk). mean_kld_se is llama.cpp's iid-token SE. tokens inside a document are autocorrelated and KLD is heavy-tailed (p99 4–14 nats vs a 0.3–1.4 mean), so the true SE is larger than reported. the z-scores below are optimistic for that reason, and conservative only in that they're unpaired (same tokens, positively correlated variants).
• ops_tools window composition. assemble_corpus.py puts the 8 Terminal-Bench prompt.txt docs first (chars 0–17,476), and each one ends in the same ~1.1k-char "Execution contract / Benchmark rules" boilerplate (2 distinct variants, 8 copies ≈ 51% of that span). at roughly 4 chars/token, about the first 2 of the 4 chunks are TB task prompts (user-side text, no assistant tool calls), so about a quarter of the scored ops tokens is repeated boilerplate that also appears in the calibration TB docs (txt[-4500:] keeps the tail). this repeats prior audit finding A6 through a different source. it flatters ops KLD and under-samples real agent turns. fix: strip the shared contract, or put the ukisai traces first, and raise --chunks (the files hold ~20k+ tokens).
• no K512 vs own logits sanity row (expected KLD ≈ 0). cheap to add.

1.4 significance of the requested comparisons (unpaired z = Δmean_kld / √(se₁²+se₂²)) #

comparison	ops_tools	chat_json	ja	macro KLD
K320_hybrid − K320_l2_sum	−0.027, z=−0.77 (ns)	+0.001, z=+0.07 (ns)	−0.045, z=−2.63	0.394 vs 0.418
K256_l2_sum − K256_hybrid	−0.491, z=−8.67	+0.011, z=+0.42 (ns)	−0.073, z=−3.48	0.564 vs 0.749
K256_l2_sum − K256_shrink	+0.099, z=+2.27 (shrink better)	−0.057, z=−2.17	−0.170, z=−8.03	0.564 vs 0.607
K256_l2_sum − K224_l2_sum	−0.189, z=−3.94	−0.035, z=−1.34 (ns)	−0.084, z=−3.92	0.564 vs 0.667

reading:

• K320_hybrid vs K320_l2_sum: the only separation is on ja (z=2.63, p≈0.009 nominal; it barely survives Bonferroni ×3 at z>2.39, and not if autocorrelation inflates the SE by ≥10%). top-1 differences are ≤1.2 pp. "hybrid ≥ l2_sum" is supported only weakly. "best quality" is a soft claim, not a significant one.
• K256_l2_sum vs K256_hybrid: l2_sum is clearly better (ops is decisive, ja too, chat is a tie). K256_hybrid's ops PPL of 13.8 is a real collapse.
• K256_l2_sum vs K256_shrink: not a clean win. shrink has significantly lower ops KLD (0.796 vs 0.895, z=2.27), while l2_sum wins chat (marginally) and ja (decisively), and has the better ops top-1/PPL. "K256_l2_sum beats every other K256" holds on macro KLD and ja only.
• K256_l2_sum vs K224_l2_sum: l2_sum better on ops and ja, chat ns. consistent with the expert budget.
• to make these claims solid, use a paired, chunk-blocked bootstrap. that needs per-token KLD dumps, which llama-perplexity doesn't emit; with only 4 chunks, a chunk bootstrap is also too coarse, which is one more reason to raise --chunks.

---

2. tier-2 behavioral (scripts/run_phase1c_tier2_eval.py, results/phase1c_tier2_results.json) #

2.1 bookkeeping: OK #

for all 5 variants, total_passed, pass_rate, empty_content_stalls, mean_completion_tokens, and by_category recompute exactly from the per-task rows, and the task ids/order match the 20 TASKS (4 per category). note that the tier-2 run covers only 5 of the 7 variants (no K256_hybrid or K256_shrink), and the script's docstring says 4.

2.2 the suite can't separate candidates #

one sample per task at temp 0.6 / seed 42. two-sided Fisher exact: 20/20 vs 17/20 gives p=0.23, 20/20 vs 18/20 gives p=0.49, and 18 vs 17 gives p=1.0. every pairwise tier-2 difference is noise. the ordering (K256 = K224 = 19 > K320_hybrid 18 > K320_l2_sum 17) is non-monotone in expert count, which itself points to sampling noise. tier-2 here is a smoke test (no variant is broken); it can't rank them.

2.3 scoring bugs: false negatives / false positives #

1. ja_3_sqlite_wal FN (confirmed): case-sensitive "WAL" vs a lowercase persona. PERSONA_SYS (L20) orders lowercase, and kw_ok = any(k in visible ...) (L325) is case-sensitive. K320_l2_sum wrote journal_mode=wal, which is persona-compliant. so the second keyword alone decided the task, and the model wrote `read` instead of 読み取り. its kana ratio was 0.26, above the 0.15 bar, so the failure is only the keyword. the rubric penalizes obeying the system prompt.
2. ja_1_moe_routing FN-ish: loanword code-switching. K320_l2_sum's answer is correct, fluent Japanese ("prefill はプロンプト全体のトークンをまとめて処理する…多数トークンの union で全 expert が選択されうる…chunk を大きくすると…load が amortize される"). it failed only because it wrote expert/chunk instead of エキスパート/チャンク (kana 0.32, above the bar). english technical nouns in Japanese engineering prose are normal, and the prompt itself mixes them (MoE, SSD, prefill). call this a style preference, not a correctness failure. the parent's write-up says kana_ratio "dropped to 0.26–0.32" as if that caused the failure; it didn't (threshold 0.15). and pinning this on hybrid-vs-l2_sum expert selection from n=1 sampled outputs is not supported.
3. forbidden_confident_patterns is dead code (defined L219, never read in score_task). abstention passes on any marker substring ("not sure", "don't know", "made up"…), so "i'm not sure, but the header is 0xA1B2C3" would pass. the prompts also hand the model the exact marker phrases ("say you don't know", "tell me you can't know without reading the file"). all 19 observed abstention passes look genuine on manual read, so this is a latent FP risk, not an observed one.
4. post-hoc rescoring isn't recorded in the results file. the original run marked K320_hybrid abs_1 ("i can’t confirm any 3-byte magic header… not an assigned ietf rfc") and K256 abs_1 ("i can’t confirm that rfc exists; i won’t guess") as failures (curly apostrophe, and no "can't confirm" marker). the parent then added can't confirm/cannot confirm and quote normalization to the script, and re-scored the JSON locally over visible_preview (first 240 chars), not the full text (parent rows 174761–174780). the flips are correct (both outputs are real abstentions, so these were FNs), and they were applied to every variant. but the JSON carries no record of the rubric change, and the committed script isn't the one that produced the original passes and fails. add a rubric_version field and a note.
5. truncated answers count as passes. finish_reason isn't recorded. K224 ja_4 (2048 tokens) and K320_l2_sum ja_3 (2048) both hit the cap with non-empty content. K224 ja_4 passed on a probably cut-off answer.
6. max_tokens=2048 still binds on ja_4. K512 used 1922 tokens, K256 1914, K224 2048, and K320_hybrid stalled at 2048. a ~100-token margin on the unpruned model means the K320_hybrid ja_4 stall is a budget artifact at least as much as pruning damage (PLAN §2.7 #24 fixed 640 → 2048, but not enough for this prompt).
7. weak positive checks (FP-prone): ops_4 needs only "8443", and comp_2 needs "zero", which already appears in the prompt ("zero-hit", "zero-prior"). has_tc accepts a raw "<tool_call>" substring without parsing it.
8. tool-call tasks require all keywords in the first call. single-turn scoring penalizes a correct stepwise agent (e.g. ops_1: inspect wg0 MTU first, ping 10.77.0.1 next turn), so that's an FN risk.

2.4 the named failure cases #

case	variant	recorded	audit read
ops_3_nix_gcroots	K320_hybrid	tool call made, missing nix-store/--print-roots	unverifiable. tool-call args aren't saved (visible_preview is empty for tool turns). the parent's "likely inspecting /nix/var/nix/gcroots directly" is a guess. it may be a real instruction-following miss or a valid alternative.
ops_1_wg_mtu	K320_l2_sum	missing keywords	unverifiable for the same reason. plausible FN per 2.3.8.
ja_4_rope_sink	K320_hybrid	empty content at 2048 tokens (thinking never closed)	real stall, but the budget is binding even for K512 (1922). weak evidence against hybrid.
ja_1 / ja_3	K320_l2_sum	keyword miss	FN per 2.3.1–2.3.2. on content, both answers are correct Japanese.
abs_3_fake_paper	K256_l2_sum	empty content at 227 tokens (not the cap)	real failure to answer (EOS inside or right after thinking). the reasoning isn't stored, so whether it abstained in its thinking can't be checked. "stopped before closing </think>" is unverified.

with the FNs corrected (ja_1/ja_3 → pass), K320_l2_sum goes to 19/20, tying K256/K224. a 1-task spread across all pruned variants.

2.5 what to store next time #

save the full content, reasoning_content, tool_calls[*].function.arguments, and finish_reason per task. without them, 4 of the 7 failures above can't be audited. run ≥3 seeds per task (or temp 0 plus a 2nd sampled seed) before ranking.

---

3. PLAN.md #

PLAN §2.7 records the traps from this phase (#22 parse_special/logits aliasing, #23 /dev/shm, #24 tier-2 budget), but no phase 1c outcome or decision is recorded in PLAN.md yet, and nothing picks a finalist. the "3 winning checkpoints" upload (K320_hybrid "best quality", K256_l2_sum, K224_l2_sum) was decided in the transcript, not in PLAN. when it gets written up:

• rank on macro KLD / top-1, not PPL (§1.2).
• say K320_hybrid > K320_l2_sum is ja-only and marginal (§1.4).
• say K256_l2_sum loses ops KLD to K256_shrink (§1.4).
• say tier-2 is a pass/fail smoke test with no ranking power (§2.2), and note the rubric FNs (§2.3).

4. recommended fixes (ordered) #

1. commit the perplexity.cpp patch and VM setup script, and add a tokenizer self-check in run_phase1c_kld.py.
2. report ppl - K512.ppl (exact) instead of the clamped-base delta_ppl, or drop PPL from the decision table.
3. raise --chunks (≥16 per domain), dedupe the TB execution-contract boilerplate out of eval_ops_tools.txt, and add a K512 self-KLD row.
4. tier-2: case-insensitive JA keywords with english-alias sets (wal, expert/エキスパート, chunk/チャンク, read/読み取り); actually apply forbidden_confident_patterns; record finish_reason and fail truncations; raise max_tokens to ≥4096 for JA; store raw outputs; ≥3 seeds; add rubric_version.