# anti-slop v4: honest framing, markdown structure, the 7 cliches (K512 IQ2_XS) opus, `anti-slop-v4-opus`. everything that runs is in `/tmp/opus_antislop_v4/`. no repo file was edited except this report. ## 0. what i found first - **the obvious version of this leaks into thinking on every turn.** the generation prompt ends in `<|im_start|>assistant\n\n`, and the prompt goes through `prompt_run_rows` / `prompt_chunk` / `Window::prompt`, so `finish()` never sees ``. a flag set from `finish()`'s tokens starts every decode as "not thinking", and the penalties land in the whole reasoning trace. the fix used here: the state lives **per position**, and `prompt_run_rows` scans the prompt (1 line). there's also a conservative default: the shaper is on only after a `` seen in the current turn. `<|im_start|>`, `<|im_end|>` and `` reset it, and an unknown state means off. - **a few IDs in the brief don't match this vocab** (checked against `vocab.json`, which is byte-identical in K256 and K512): ``/`` = **248058/248059** (248056/248057 are `<|image_pad|>`/`<|video_pad|>`). `510` = `'`.** on real K512 logits, `"roll back right"` → ` away` (p 0.85), `"If I'm being"` → ` honest` (0.95), and `"merge rather"` → ` than` (0.995) all switched their argmax to `<|im_end|>` after the penalty: the reply would end mid-sentence. v4 holds `<|im_end|>`, ``, `<|im_start|>` and eos at their old probability. it subtracts log(1/(1−b)), where b is the share the penalty removed. after that, the new argmaxes are ` there`, ` real` and ` than`. - **the v3 code has a latent argmax bug once more than one token per row is penalized.** `apply_pen` rescans a partition only when the penalized token is the *global* argmax. if it's only its partition's max, `apv_` keeps the stale value, and a later rescan can pick it. v4 rescans every partition whose max was pushed down. - **layer B (weight-level WFLD) can't do this job.** §5 has the numbers: a linear direction in the L47 shared-expert input can't tell "line start in prose" from "line start in a fence or in thinking" on held-out text. the best fit leaks **28–68 % of the prose penalty into fenced-code line starts** and 10–14 % into thinking line starts. the falsifier fails by construction, so the stack is layer A only. ## 1. evidence: how often this slop shows up (read-only mining) `mine_slop.py` reads `~/.prime/agent/sessions/*.jsonl` and `~/.albedo/albedo.sqlite`, assistant text with fenced code removed: | family | turns | turns w/ honesty frame | "honestly" | "the honest" | "honest take/answer/framing" | turns w/ bullets | w/ bold/# headers | w/ ":\n- " list | |---|---|---|---|---|---|---|---|---| | claude | 13449 | 1.3% | 90 | 60 | 17 | 27.9% | 11.2% | 18.8% | | glm | 9822 | 1.9% | 81 | 71 | 17 | 7.7% | 2.3% | 6.1% | | deepseek | 7889 | 4.4% | 98 | 224 | 25 | 14.5% | 8.2% | 10.6% | | gpt | 3647 | 0.7% | 22 | 4 | 2 | 27.2% | 2.9% | 24.5% | | gemini | 2556 | 1.8% | 41 | 2 | 3 | 52.3% | 53.1% | 46.9% | | mimo | 972 | 1.2% | 4 | 6 | 1 | 8.7% | 2.7% | 7.1% | | qwen | 105 | 1.9% | 1 | 1 | 0 | 21.0% | 8.6% | 15.2% | (the albedo DB tags few sessions as qwen, so the qwen row is small. the qwen behaviour is measured directly on K512 in §3.) gemini puts bullets in **52 %** of its replies, headers in **53 %**, and colon-launched lists in **47 %**. a real example has all three at once: `"so the honest version of the idea, in order of how much of the app survives:\n\n1. **node start…"`. ## 2. token tables (Qwen3.8-Flash-Next vocab, 248,320) | slop | rejected next token(s) | condition (on the last token p0, the one before p1, and the line state) | |---|---|---| | "the honest answer / framing / take…" | `' honest'` 10429 | p0 ∈ the 1719, The 760, ` the` 279, ` The` 561, an 276, ` an` 449, ` An` 1473, An 2014, my 2329, ` my` 821, ` My` 2921, My 4888, be 1323, ` be` 381, ` being` 1602, being 32985, ` most` 1379, most 3444, ` Most` 7261, more 6183, ` more` 777, ` More` 4254, ` your` 678; or a sentence start (`.` 13 `!` 0 `?` 30 `:` 25 `;` 26 `...` 1076 `—` 2218 ` —` 1892 `)` 8, a newline, a line start) | | "Honestly," / "Honest take" | `'Honestly'` 92115, `' Honestly'` 68540, `' Honest'` 51397 | any prose row | | "…, honestly" / "but honestly" | `' honestly'` 25211 | a clause start: the sentence set + `,` 11, `-` 12, ` -` 471, ` and` 321, ` but` 694, ` so` 748, and 429, but 7834, so 684, ` And` 1541, ` But` 1921, ` So` 1987, And 2939, But 3850, So 4272 | | lowercase line-initial "honestly" / "honest" (`h`+`onestly`) | `'onestly'` 77696, `'onest'` 33961 (1.5×) | p0 ∈ `h` 71, `H` 39, `(h` 3101, `"h` 75833 | | "in all honesty" | `' honesty'` 46253, `' honestly'` 25211 | p0 = ` all` 660 | | bullets / headers / rules at a line start | `-` 12, `*` 9, `•` 6458, ` -` 471, ` *` 348, ` •` 7061, `**` 332, ` **` 2972, `#` 2, `##` 550, `###` 13962, `####` 794, ` #` 653, ` ##` 7460, ` ###` 16112, `---` 4277, `***` 11862, `–` 4006, `—` 2218, `+` 10, `→` 49300 | the line start (after any of the 200 newline tokens, or after ``, with blank tokens in between); +colon bonus if the line above ended in `:` 25 / `:**` 64700 / `**:` 92217 | | numbered lists | digits `1`–`9` 16–24 (¼ strength, full after a `:` line); `.` 13 / `)` 8 after a line-initial digit | | | short label line + colon | `\n` 198, `\n\n` 271 | p0 ∈ {25, 64700, 92217} and ≤ 6 content tokens on the line | | v4.1 candor siblings (the "If I'm being real" shift) | real 1865, blunt 47045, frank 24969, candid 5507, transparent 17290, straight 7435, upfront 62179, brutally 66842, truthful 86775, totally 12116, completely 6379, fully 6999, perfectly 13542, entirely 11040, painfully 73018 | after ` being` 1602/being 32985; after ` be` 381/be 1323 when p1 ∈ to 310/951, 'll 3172, me 728, just 1066, gonna 16032 (+` direct` 2050; + ` fair` 6415 / ` clear` 2708 after "to be"); ` honest` + siblings after an intensifier that follows being/be | | v4.1 branch points (one token before the frame closes) | ` being` 1602 (½) and ` honest` after "I'm" (`'m` 2688 after I 353/585/40/72); ` real/blunt/frank/candid/straight/short/brutal` (½) after a clause-initial the/The 1719/760/279/561; ` be` 381 (½) after a clause-initial To/to 1206/1946/951/310; `short`/`Short`/`real`/`Real` 8412/12115/7700/12402 (½) at a line start | | | v4.1 frame nouns | answer 4087, take 1831, framing 56004, version 2243, truth 7793, read 1301, summary 11782, assessment 14996, question 3296 | after a frame word (honest, real, blunt, frank, candid, straight, short, simple, quick, brutal, plain, actual + no-space/capital forms) that follows a det (the/The/my/My/an/An) or a sentence/line start | | v4.1 misc | ` frankly` 40285 at a clause start; ` fairness` 49034 / ` seriousness` 71853 after ` all` | | | 7 cliches (v3 rules, unchanged) | ` than` 1056, ` away` 3023, ` end` 809, ` version` 2243, ` what` 1092, ` actually` 3404 | as v3: rather 4598/24944/61745/70714 (+`ather` 1792 after `(r` 2515/`(R` 2697/`[r` 15656/`[R` 43487); right 1245/1246/5791/9785/26806/51140; "end to" 310 after 400/809/3601/3839/14637/55293; short 2716/8412/10382/12115/36639; exactly 6681/63151/66172 (+`actly` 31654 after `ex` 325/`(ex` 5291); `'s` 579 after here 1532/5514/6527/8160/475 → ` what`, after what 1092/3437/3710/12195/29348 → ` actually` | kaomoji survive the line-start rules: `*:・ﾟ✧` starts with `'*:'` 77340 and `*.+:｡` with `'*.'` 19315, not `'*'` 9. the known casualty is `-_-` at a line start (`'-'` + `'_-'`). ## 3. evaluation on real L47 dumps (`eval_l47.cpp`, which `#include`s the exact `antislop_v4.inc` that ships) each row goes through the head: `output_hc_norm/down/up` (Q8 round-trip, like the runtime) and `output.weight`. the shipped `slop4::step`/`penalties`/`hold` are applied, and the result is scored exactly. KLD(P‖Q) = Σ_S p_i·pen_i + log(1 − Σ_S p_i(1−e^(−pen_i))), over the full 248k vocab, with the argmax taken from the top 16. "ungated" = the same rules with the think/tool/fence gate forced open (the counterfactual). **final defaults: lex 9, honest 6, md 4, colon 3**, with the v4.1 candor rules (§8) included. **new real K512 dumps** (`K512-iq2_xs-metal.gguf`, `STRATA_METAL_DUMP_MOE`, 2 slop-dense docs in dawn's persona. they include bullets and "honestly" inside ``, a `` with `# - comment` lines, and ```` ```nix ```` / ```` ```yaml ```` / ```` ```bash ```` fences with `#`, `- `, `## not a header`, `**not bold**`): | doc | region | rows | mean KLD | max KLD | ordinary-prose KLD¹ | ungated mean KLD | argmax changed | |---|---|---|---|---|---|---|---| | docA | prose | 324 | 0.151514 | 5.7390 | 0.028359 | 0.15151 | 18 | | docA | think/other | 219 | 0.000000 | 0.0000 | — | 0.09265 | 0 | | docA | tool | 49 | 0.000000 | 0.0000 | — | 0.01750 | 0 | | docA | fence | 90 | 0.000000 | 0.0000 | — | 0.07890 | 0 | | docB | prose | 329 | 0.169362 | 5.3991 | 0.038402 | 0.16936 | 18 | | docB | think/other | 146 | 0.000000 | 0.0000 | — | 0.09600 | 0 | | docB | fence | 31 | 0.000000 | 0.0000 | — | 0.06158 | 0 | ¹ rows whose forced next token is not penalized. in these slop-dense docs, that still includes rows where the model *wanted* slop (`'**'` at p 0.745 or 0.973 after a kaomoji line, `' honestly'` at 0.175 after `"a third full."`), so this number is mostly the intended effect. ordinary-prose collateral is measured on the clean corpora below. **K256-trunk dumps** (cal_cache, the pre-existing ones), through the **K512 head**: | trace | region | rows | mean KLD | max KLD | ordinary-prose KLD | ungated mean KLD | argmax changed | |---|---|---|---|---|---|---|---| | blind | prose | 452 | 0.172000 | 8.0535 | 0.004816 | 0.17200 | 17 | | blind | think/other | 646 | 0.000000 | 0.0000 | — | 0.00241 | 0 | | calp3 | prose | 147 | 0.001696 | 0.1683 | 0.001696 | 0.00170 | 0 | | calp3 | think/other | 702 | 0.000000 | 0.0000 | — | 0.00418 | 0 | | calp3 | tool | 173 | 0.000000 | 0.0000 | — | 0.00000 | 0 | | collat | think/other | 1638 | 0.000000 | 0.0000 | — | 0.00241 | 0 | | ftpoho | prose | 455 | 0.143291 | 8.4996 | 0.002496 | 0.14329 | 16 | | ftpoho | think/other | 643 | 0.000000 | 0.0000 | — | 0.00100 | 0 | | ftposft | prose | 808 | 0.245807 | 11.2758 | 0.003234 | 0.24581 | 37 | | ftposft | think/other | 55 | 0.000000 | 0.0000 | — | 0.00278 | 0 | | ftposft | tool | 159 | 0.000000 | 0.0000 | — | 0.00825 | 0 | | vn | prose | 7813 | 0.001203 | 2.4227 | 0.000689 | 0.00120 | 3 | | vn | think/other | 9061 | 0.000000 | 0.0000 | — | 0.00069 | 0 | | vn | tool | 478 | 0.000000 | 0.0000 | — | 0.00334 | 0 | the same traces through their own K256 head: | trace | region | rows | mean KLD | max KLD | ordinary-prose KLD | ungated mean KLD | argmax changed | |---|---|---|---|---|---|---|---| | blind | prose | 452 | 0.170271 | 8.6093 | 0.004725 | 0.17027 | 17 | | blind | think/other | 646 | 0.000000 | 0.0000 | — | 0.00198 | 0 | | calp3 | prose | 147 | 0.001665 | 0.1511 | 0.001665 | 0.00167 | 0 | | calp3 | think/other | 702 | 0.000000 | 0.0000 | — | 0.00426 | 0 | | calp3 | tool | 173 | 0.000000 | 0.0000 | — | 0.00000 | 0 | | collat | think/other | 1638 | 0.000000 | 0.0000 | — | 0.00240 | 0 | | ftpoho | prose | 455 | 0.146890 | 9.0405 | 0.002337 | 0.14689 | 17 | | ftpoho | think/other | 643 | 0.000000 | 0.0000 | — | 0.00084 | 0 | | ftposft | prose | 808 | 0.248088 | 11.2103 | 0.003223 | 0.24809 | 36 | | ftposft | think/other | 55 | 0.000000 | 0.0000 | — | 0.00238 | 0 | | ftposft | tool | 159 | 0.000000 | 0.0000 | — | 0.00802 | 0 | | vn | prose | 7813 | 0.001051 | 2.1890 | 0.000605 | 0.00105 | 3 | | vn | think/other | 9061 | 0.000000 | 0.0000 | — | 0.00066 | 0 | | vn | tool | 478 | 0.000000 | 0.0000 | — | 0.00373 | 0 | **falsifier check:** across **35,320 rows**, every think/other, tool and fence row (12,745+365 think, 810+49 tool, 121 fence) has **0 penalties and KLD exactly 0**: the logits are not touched. the same rules ungated would have cost 0.06–0.09 nats/row in K512 thinking and fences. ordinary prose KLD: **VN chat 0.00069 (7,806 rows), cal_p3 0.0017, ftpo 0.0025–0.0032, blind 0.0048**. all are under 0.005, but blind's margin is thin. most of blind's number is one row: the K512 head wants `'#'` at p 0.61 right after `\n` (a markdown header at turn start), which is exactly the target. if the v4.1 rules or a higher md ever push it over, the knobs are per family. **the 7 cliches on the untouched blind test** (21 positions): p(slop) **0.5505 → 0.0069 (−98.7 %), 21/21 argmax clean**, with the K512 head and no WFLD. v3's bar was 0.5543 → 0.0056 *with* WFLD. ftpo held-out: 0.452 → 0.0077 (21/21). ftpo sft: 0.693 → 0.0027 (42/42). the lex default went 6 → 9: at 6, `"end to"` → ` end` (p 0.98) and `"merge rather"` → ` than` (0.995) stayed argmax on K512, and the VN collateral cost of 9 is +0.00001. **`models/K512_iq2_xs/strata-k512-iq2xs.json` sets `STRATA_ANTI_SLOP_PENALTY=6.0`, so raise it to 9 or drop the line.** sweeps: md 6 pushed blind ordinary to 0.0077 (> 0.005), so md stays 4. colon 1.5 → 3 lifted the colon-launched list cut from 26/59 % to 60/88 % for +0.0001 on blind and +0.00003 on VN. slop probability mass per 1000 prose tokens, before → after (K512 docs; `?` = the v4.1 candor/frame ids): | doc | family | before | after | change | |---|---|---|---|---| | docA | hon | 8.32 | 0.629 | −92.4% | | docA | md | 32.81 | 12.751 | −61.1% | | docA | colon | 9.14 | 3.661 | −60.0% | | docA | lex | 8.63 | 0.096 | −98.9% | | docB | hon | 7.72 | 0.451 | −94.2% | | docB | md | 35.91 | 8.485 | −76.4% | | docB | colon | 3.51 | 0.408 | −88.4% | | docB | lex | 8.66 | 0.019 | −99.8% | md shows less because the docs force mid-list items (`"- a\n"` → `'-'` at p 0.97–0.999). once a list exists, the model is committed. the decision row is the **first** bullet, and that one drops hard: `"**Key points:**\n\n"` → `'-'` 0.713 → 0.02, `"the fix is boring:\n\n"` → `'-'` 0.596 → 0.016. **every slop target in the K512 docs** (forced next token, p before → after, argmax before → after): | doc | family | context | next | p before | p after | argmax | |---|---|---|---|---|---|---| | docA | colon | `'.⏎⏎⏎the usual roster:'` | `'\n\n'` | 0.644 | 0.1492 | `'\n\n'` → `' ('` | | docA | md | `'⏎⏎⏎the usual roster:⏎⏎'` | `'**'` | 0.009 | 0.0000 | `'('` → `'('` | | docA | colon | `' roster:⏎⏎**happy / soft:**'` | `'\n'` | 0.909 | 0.4188 | `'\n'` → `' ('` | | docA | md | `':⏎⏎**happy / soft:**⏎'` | `'-'` | 0.033 | 0.0000 | `'('` → `'('` | | docA | md | `'｡•��•｡)⏎'` | `'-'` | 0.999 | 0.9363 | `'-'` → `'-'` | | docA | hon | `'�� for when something ships⏎⏎h'` | `'onestly'` | 0.874 | 0.0014 | `'onestly'` → `'mm'` | | docA | hon | `' answer is no.⏎⏎⏎the'` | `' honest'` | 0.056 | 0.0004 | `' short'` → `' short'` | | docA | ? | `' is no.⏎⏎⏎the honest'` | `' answer'` | 0.823 | 0.0142 | `' answer'` → `' short'` | | docA | ? | `' old ones ends up with duplicates. to'` | `' be'` | 0.115 | 0.0064 | `' keep'` → `' keep'` | | docA | hon | `' ones ends up with duplicates. to be'` | `' honest'` | 0.051 | 0.0001 | `' safe'` → `' safe'` | | docA | lex | `' duplicates. to be honest, the short'` | `' version'` | 0.225 | 0.0000 | `' answer'` → `' rule'` | | docA | lex | `' short version is that you should merge rather'` | `' than'` | 0.995 | 0.0813 | `' than'` → `'.'` | | docA | md | `'ake, many hosts.⏎⏎⏎'` | `'###'` | 0.000 | 0.0000 | `'the'` → `'the'` | | docA | md | `' hosts.⏎⏎⏎### Layout⏎⏎'` | `'**'` | 0.135 | 0.0041 | `'-'` → `'the'` | | docA | colon | `'⏎⏎### Layout⏎⏎**Key points:**'` | `'\n\n'` | 0.280 | 0.1657 | `'\n'` → `'\n'` | | docA | md | `'### Layout⏎⏎**Key points:**⏎⏎'` | `'-'` | 0.713 | 0.0078 | `'-'` → `'the'` | | docA | md | `'flake.nix` at the root⏎'` | `'-'` | 0.974 | 0.5748 | `'-'` → `'-'` | | docA | md | `'/` with one dir per machine⏎'` | `'-'` | 0.954 | 0.6382 | `'-'` → `'-'` | | docA | md | `' `modules/` for shared bits⏎⏎'` | `'1'` | 0.001 | 0.0003 | `'**'` → `'```'` | | docA | num | `'modules/` for shared bits⏎⏎1'` | `'.'` | 0.932 | 0.0142 | `'.'` → `' fl'` | | docA | md | `' pin inputs in `flake.lock`⏎'` | `'2'` | 0.767 | 0.9353 | `'2'` → `'2'` | | docA | num | `' inputs in `flake.lock`⏎2'` | `'.'` | 0.999 | 0.6308 | `'.'` → `'.'` | | docA | hon | `' flake check` in ci⏎⏎h'` | `'onest'` | 0.001 | 0.0003 | `'onestly'` → `'mm'` | | docA | lex | `" take: keep it boring. here's"` | `' what'` | 0.035 | 0.0000 | `' the'` → `' the'` | | docA | lex | `"that's it. it works end to"` | `' end'` | 0.981 | 0.1192 | `' end'` → `'<|im_end|>'` | | docA | lex | `' end to end and you can see exactly'` | `' what'` | 0.554 | 0.0002 | `' what'` → `' where'` | | docB | lex | `"-bound work just takes turns. what's"` | `' actually'` | 0.021 | 0.0000 | `' speeding'` → `' speeding'` | | docB | hon | `' interpreter lock, not your pool.⏎⏎'` | `'Honestly'` | 0.000 | 0.0000 | `'for'` → `'for'` | | docB | md | `'Honestly, the fix is boring:⏎⏎'` | `'-'` | 0.596 | 0.0039 | `'-'` → `'```'` | | docB | md | `'- use `ProcessPoolExecutor`⏎'` | `'-'` | 0.905 | 0.1529 | `'-'` → `' '` | | docB | md | `' or move the hot loop into numpy⏎⏎'` | `'**'` | 0.004 | 0.0001 | `'-'` → `'If'` | | docB | lex | `" not threads. i'd try that right"` | `' away'` | 0.402 | 0.0001 | `' now'` → `' now'` | | docB | lex | `'⏎⏎⏎the syscall log shows exactly'` | `' what'` | 0.456 | 0.0001 | `' what'` → `' where'` | | docB | ? | `" thinks it is.⏎⏎If I'm"` | `' being'` | 0.089 | 0.0053 | `' reading'` → `' reading'` | | docB | hon | `" it is.⏎⏎If I'm being"` | `' honest'` | 0.950 | 0.1981 | `' honest'` → `' honest'` | | docB | hon | `"⏎⏎If I'm being honest, my"` | `' honest'` | 0.000 | 0.0000 | `' first'` → `' first'` | | docB | md | `'⏎⏎summary.⏎⏎⏎'` | `'##'` | 0.002 | 0.0000 | `'the'` → `'the'` | | docB | lex | `"⏎⏎## Migration plan⏎⏎Here's"` | `' what'` | 0.117 | 0.0000 | `' the'` → `' the'` | | docB | colon | `" plan⏎⏎Here's what we do:"` | `'\n\n'` | 0.602 | 0.1606 | `'\n\n'` → `'<|im_end|>'` | | docB | md | `"⏎⏎Here's what we do:⏎⏎"` | `'1'` | 0.197 | 0.0039 | `'-'` → `'Phase'` | | docB | num | `"Here's what we do:⏎⏎1"` | `'.'` | 0.983 | 0.1465 | `'.'` → `'.'` | | docB | md | `':⏎⏎1. snapshot the db⏎'` | `'2'` | 0.859 | 0.8868 | `'2'` → `'2'` | | docB | num | `'⏎⏎1. snapshot the db⏎2'` | `'.'` | 0.992 | 0.1850 | `'.'` → `'.'` | | docB | md | `'. run the migration in a transaction⏎'` | `'3'` | 0.952 | 0.9677 | `'3'` → `'3'` | | docB | num | `' run the migration in a transaction⏎3'` | `'.'` | 0.925 | 0.0828 | `'.'` → `'<|im_end|>'` | | docB | lex | `' if a statement fails, roll back right'` | `' away'` | 0.853 | 0.0012 | `' away'` → `' there'` | | docB | md | `' statement fails, roll back right away⏎⏎'` | `'•'` | 0.000 | 0.0000 | `'Honestly'` → `'The'` | | docB | md | `' the old schema around for a day⏎'` | `'•'` | 0.188 | 0.0389 | `'*'` → `' '` | | docB | hon | `' day⏎• then drop it⏎⏎The'` | `' honest'` | 0.099 | 0.0003 | `' plan'` → `' plan'` | | docB | ? | `'⏎• then drop it⏎⏎The honest'` | `' truth'` | 0.183 | 0.0010 | `' answer'` → `' plan'` | | docB | hon | `' in step 2, and in all'` | `' honesty'` | 0.814 | 0.0381 | `' honesty'` → `' likelihood'` | | docB | lex | `' bookkeeping. it should go end to'` | `' end'` | 0.977 | 0.0444 | `' end'` → `'<|im_end|>'` | ## 4. the stack (layer A only) - `slop4::State` (12 bytes) per position: the last two tokens, flags (PROSE, TOOL, FENCE, BOL = line start, COLON = this line follows a `…:` line, PBOL = the last token opened a line), and the content tokens on the current line. `step()` is pure, so a row's state is `step(state[pos0+t−1], tokens[t])`. speculative windows, rejected drafts, cached prefixes and re-reads all compose. `test_state.cpp` checks this: 200 random runs of prompt split + windows of 1–6 + random rejected drafts against one straight pass, on docA and on the 17k-token VN trace. **0 mismatches.** - `penalties()` returns nothing unless the state is PROSE and not TOOL or FENCE. after the penalties, `hold()` keeps the specials at their old probability. `hold()` needs the row's log-sum-exp: one 248k pass, done only when a penalized token is within 12 nats of the row max (otherwise b < 1e-5). most rows skip it. - knobs: `STRATA_ANTI_SLOP_PENALTY` (lex, default 9), `STRATA_ANTI_SLOP_HONEST` (6), `STRATA_ANTI_SLOP_MD` (4), `STRATA_ANTI_SLOP_COLON` (3). 0 turns that family off. the v3 "no empty turn after `\n\n`" guard is kept, on the lex knob as before. - hooks: `prompt_run_rows` (1 line, covers all three prompt paths), `session_load` (scans the loaded ids), and `scr_relocate` (moves the states with their cells). - caveats: (a) `checkpoint_restore` doesn't carry the states. that's harmless, because the re-prefilled suffix always has the new turn's `<|im_start|>`, which resets everything. (b) with `STRATA_LOGPOS` set, prompt rows get head logits and so get shaped too, same as v3. set the knobs to 0 for KL-harness runs. (c) MTP drafts come from the unshaped draft head, so a few more rejections near slop. (d) unfenced code in prose (VN has some raw yaml and nginx config) does get the line-start rules. the measured cost is in the VN number above. ## 5. layer B (WFLD on `blk.47.ffn_down_shexp`, IQ4_NL) — rejected, with numbers any rank-r change to the shared-expert down-projection moves the logits by a linear function of z = σ(sgv)·hh (640-d). to put a line-start penalty on prose and nowhere else, a direction v would need v·z ≈ 1 on prose line starts and ≈ 0 on fence, think and tool line starts. `layerB_separability.py` fits v by ridge on one K512 doc and tests on the other: - train docA → test docB: think-bol leak **+0.14** (p90 |·| 0.58), fence-bol **+0.68** (p90 1.07), all other rows p90 0.32 - train docB → test docA: think-bol **+0.10** (p90 0.48), fence-bol **+0.28** (p90 0.63), tool-bol **+0.27** (p90 0.45) - even the in-sample overfit leaves |v·z| up to 0.33 on rows that must be 0 at md = 4 nats, that's 1–4 nats of bullet/`#` penalty inside code fences, and the update also perturbs every other row (code, tools, math). the honesty frames have the same problem: "honestly" inside `` shares the direction. the stateful shaper is exactly 0 outside prose, so there's no reason to touch the weights. the existing v3.2 WFLD was a K256 artifact, and K512's shexp47 is the untouched IQ4_NL. ## 8. the synonym treadmill and where the model actually gives up the frame (dawn's question) **short version:** inside a committed prefix, it moves to the next sibling every time. one token earlier, at the clause entry, the model still has real content mass, and pushing there makes it drop the frame. the exact top-k on K512 (`eval_k512_docs_top.tsv`, `v41_docs.tsv`, `v41b_docs.tsv`; p is exact, after the penalty and the hold): | slot | v4 (word-level) | v4.1 (+ siblings, branch points) | |---|---|---| | `"If I'm being" ___` (committed) | before: ` honest` 0.95, then real .009, totally .005, frank .004, straight .003, blunt, candid, completely, fully, direct, brutally. after v4: ` real` 0.17, ` totally` 0.09, ` frank` 0.08, ` straight` 0.05, ` blunt` 0.05, … **64 % of the remaining mass stays in the frame** | 15 siblings banned, and the mass just moves on: ` honest` 0.10, ` direct` 0.045, ` fair` 0.027, ` straightforward` 0.016. **still a treadmill** | | `"If I'm" ___` (branch point) | ` reading` 0.30, ` right` 0.13, ` being` 0.089, ` honest` 0.072, ` looking` 0.055, … frame entries 0.16 | ` reading` 0.36, ` right` 0.15, ` looking` 0.07, ` seeing`, ` debugging`, ` guessing`. **frame mass 0.161 → 0.005: it gives up the frame** ("If I'm reading this right…") | | `"\n\nthe" ___` | ` short` 0.65 (the short answer), ` rule`, ` usual`, ` general`, ` honest` .056, ` real` .013 | v4.1b: ` short` 0.087, ` rule` .084, ` usual` .07, ` general` .06, ` golden` .05. meta mass 0.66 → 0.09 | | `"…duplicates. to be" ___` | ` safe` 0.74 already dominates; frame siblings ` fair`, ` frank`, ` blunt`, ` real` 0.056 | 0.000 meta, ` safe` 0.79 | | `"and in all" ___` | ` honesty` .81 → v4: ` fairness` 0.37, ` seriousness` .08, ` cand(or)` .06 (sideways) | ` likelihood` .22, ` my`, ` cand` .11, `,`: meta 0.50 → 0.09; ` cand`+`or` is the next leak | **end to end (greedy, final v4.1b) the treadmill shows up exactly where the analysis predicts.** base and v4 rebase both said "the real answer". v4.1 (no ` actual` yet) said "so **the actual question** is". with ` actual` added and ` question` made a frame noun, the final build says "so **the actual safety question** is". the frame was committed at "so the actual", so the model put a modifier in front of the banned noun. review: "the real thing", "the actual bottleneck". **a forward-only shaper can't remove a frame the model has already entered. it can only move the entry point earlier, to where the content alternatives already carry mass.** v4.1 does that for the measured branch points ("I'm" → being/honest, clause-initial "the"/"to", line-initial short/real). a committed frame still finds a sibling. **what the literature says** - **Antislop** (Paech, Roush, Goldfeder, Shwartz-Ziv, arXiv:2510.15061, ICLR 2026) handles this with a **backtracking sampler**. when a banned pattern completes, it rolls back to the pattern's **first token**, lowers that token's probability (p ← p·10^(−10s)), and resamples. per the paper, that suppresses 8,000+ patterns without quality loss, where plain token banning becomes unusable around 2,000. the cost is throughput: −69 % to −96 % with frequent rollbacks. that's why they distil the rollback pairs into the weights with FTPO (final-token preference optimization: a margin loss on the rejected first token versus min-p alternatives, with MSE ties to the base logits). this is the principled fix for the treadmill. v4's lookup is the forward-only approximation of it. - **verbal tics** (arXiv:2604.19139v2, 2026 preprint, which i've only seen through search snippets): it catalogs "I have to be honest" / "to be honest" as hedging tics and ties them to alignment training (Constitutional AI pushing a "thoughtful" persona). the parent's summary also cites its finding that tics accumulate over a conversation (+110 % from turn 1 to 20, in-context repetition). that's one more reason to kill them early: each one that gets through primes the next. - the persona prompt itself contains **"the actual thought; the disagreement…"**. that line plausibly primes "the actual X" frames in context. dawn's point about "lead with your actual thought" becoming candor announcements fits it. rewording the prompt so it shows the behaviour rather than names it ("say the disagreement first" instead of "the actual thought") is free, and complements the shaper. **what would actually finish the job, in order of cost:** 1. **reword the system prompt** as above: free, and it removes an in-context primer. 2. **a short-horizon rollback in strata.** the decode window already commits speculatively (`commit(n_keep)`) and the server has conversation checkpoints, so a rollback of ≤ 6 tokens at a frame match ("the actual X question", "If I'm being Y") is plausible. re-decode from the frame's first token with it penalized there. the per-position `slop_` store already makes the shaper's state rollback-safe. this is the Antislop sampler restricted to a small pattern set, at a cost of a few extra decode steps per hit. 3. **FTPO/LoRA on the rollback pairs**, through the whole network rather than an L47 linear edit (§5 shows that one can't separate the contexts). this is the Antislop paper's own route. ## 9. end-to-end greedy A/B on K512 (private CLIs in `/tmp/opus_antislop_v4/build/`, 8 prompts, 700 tokens) the base CLI is the current `window.cpp` with `STRATA_ANTI_SLOP_PENALTY=0`. v4 is the same source patched. both are linked against the current libs. replies were counted outside fences, after ``: | prompt | bullet lines | bold/# header lines | honest* in reply | honest* in think (must not change) | ``` | |---|---|---|---|---|---| | kaomoji | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | | flake | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | 4 → 3 | | dbchoice | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | | rebase | 3 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | | backup | 0 → 0 | 0 → 0 | 0 → 0 | 0 → 0 | 2 → 4 | | wireguard | 6 → 0 | 3 → 0 | 0 → 0 | 0 → 0 | 6 → 0 | | review | 0 → 0 | 0 → 0 | 0 → 0 | 2 → 2 | 0 → 0 | | gil | 0 → 4 | 0 → 0 | 0 → 0 | 0 → 0 | 2 → 2 | - wireguard is the clearest case: base used `**the mental model first:**`, `**step 1: install**` and 6 bullets. v4 writes "the mental model that trips people up: …" and "step 1: generate keypairs …" as paragraphs, with commands inline in backticks instead of fences. - **the colon-launched list still gets through when the model is very sure.** v4 gil had "other escape hatches:\n- numpy …" and review had "is it:\n- a specific endpoint …". that's why colon went 1.5 → 3. the final build's gil came out with 0 bullets, but review still had 3 ("- you have a measured…"). the model's confidence in a list after "is it:" exceeds 7 nats. as with the frames, this needs either a higher md, which breaks the KLD bar, or a rollback. - **think is never shaped**, but the A/B can't show "think token-identical": **the decode isn't deterministic run to run.** the same base binary with the same env and `STRATA_METAL_MOCE_BONUS=0` diverged from itself at token 53 (dbchoice). so instead i replayed every v4 generation (prompt + output) through the shipped `step()`/`penalties()` (`replay_gen.cpp`). **think rows with any penalty: 0 of 2,484. fence rows with any penalty: 0 of 606.** this matches the exact-logit leak result in §3. - honesty in the replies: this persona prompt on greedy K512 rarely says "honestly" in the *reply*. it shows up in think instead ("The honest answer:" in review's think, untouched by design). the reply-side leaks are the candor siblings above. ## 6. drop-in patch apply with `uv run python3 /tmp/opus_antislop_v4/apply_patch.py vendor/strata/metal/src/engine/window.cpp`. it's anchor-based and asserts each anchor is unique. the unified diff against the current `window.cpp` is `/tmp/opus_antislop_v4/window_cpp_antislop_v4.patch`. the patched file compiles with the build's own flags (`build.ninja`'s `window.cpp.o` line), and links into `/tmp/opus_antislop_v4/build/strata_window_cli_v4`. the end-to-end A/B with this build is in §9. ```diff --- a/vendor/strata/metal/src/engine/window.cpp +++ b/vendor/strata/metal/src/engine/window.cpp @@ -124,6 +124,223 @@ const int32_t* g_mrope_host = nullptr; const int32_t* g_mrope_dev = nullptr; +// Anti-slop v4: a sampler shaper over the head logits, outside , and ``` fences only. +// Each position's state (the last two tokens, the region, where the line stands) follows from the one before it, so +// it is kept per position: a draft row, a rejected suffix, a prompt read from a cached prefix all start from the +// state of the position before them. Tables: the Qwen3.8-Flash-Next vocab (models/*/tokenizer/vocab.json). +namespace slop4 { + +enum : uint8_t { PROSE = 1, TOOL = 2, FENCE = 4, BOL = 8, COLON = 16, PBOL = 32, KNOWN = 128 }; + +struct State { + int32_t t1 = -1, t2 = -1; // the last token and the one before it + uint8_t f = 0; // PROSE: after in this turn; BOL: at a line start; COLON: that line follows "...:" + uint8_t len = 0; // content tokens on the current line (saturating) + uint8_t pad[2] = {}; +}; + +struct Knobs { float lex, hon, md, colon; }; + +struct Pen { int32_t id; float v; }; + +// token classes: 1 a newline run ("\n", "\n\n", " \n", ...), 2 blanks only (indentation) +static const int32_t kNewline[] = { + 198, 201, 271, 317, 695, 845, 987, 1031, 1358, 1517, 1726, 2228, 2317, 2449, 3265, 4326, 4408, 4558, 4969, 5690, + 5744, 5773, 6010, 6159, 6219, 6710, 6987, 7388, 8301, 8675, 8726, 9122, 10109, 10367, 10628, 10797, 11463, + 11953, 12213, 13488, 14200, 14305, 14379, 15216, 15334, 15398, 15602, 15684, 16391, 16750, 16957, 17116, 17694, + 17817, 18015, 18050, 18438, 19075, 19721, 21996, 22616, 22727, 23574, 23745, 23840, 24630, 25263, 25401, 25445, + 25698, 25952, 26467, 26480, 26771, 27330, 29644, 29773, 30858, 31472, 32077, 32527, 32811, 33006, 33019, 33339, + 33438, 33496, 33954, 34127, 35369, 35412, 35624, 35973, 36444, 36826, 37050, 37528, 38336, 38544, 39000, 39598, + 40260, 40315, 41210, 41585, 43622, 43916, 44058, 44281, 44879, 44902, 46813, 47458, 47947, 48305, 49404, 49420, + 49746, 50946, 52245, 53585, 54701, 56137, 56580, 56955, 56963, 57103, 57633, 58943, 59366, 59776, 60121, 61332, + 62386, 62591, 63063, 63111, 63454, 64137, 64576, 65117, 65382, 67511, 68830, 69229, 69340, 69896, 70273, 70707, + 70866, 70929, 71356, 71848, 71990, 72666, 73714, 75128, 75321, 75668, 75981, 76294, 76374, 76421, 76507, 76909, + 77670, 77933, 78035, 78254, 79519, 81018, 81927, 82471, 82946, 83774, 83829, 84075, 85709, 85748, 85809, 85844, + 85935, 86092, 86182, 87199, 87251, 87328, 88065, 88574, 88927, 88985, 89803, 91421, 91668, 91804, 92699, 92766, + 93957, 94820, 94861 +}; +static const int32_t kBlank[] = { + 197, 220, 256, 257, 260, 262, 285, 297, 309, 336, 388, 406, 454, 493, 556, 646, 670, 761, 950, 967, 1025, 1107, + 1335, 1604, 1638, 1734, 1779, 1855, 2118, 2215, 2467, 2475, 2596, 2672, 3235, 3388, 3551, 3696, 4152, 4420, + 4448, 4946, 5015, 5071, 5233, 5780, 6173, 6247, 6323, 6447, 6800, 7216, 7237, 7322, 7536, 7599, 7879, 8074, + 8424, 8830, 9081, 9411, 9876, 10190, 10275, 10747, 11449, 11530, 11611, 12473, 12670, 12687, 13076, 13152, + 13300, 13857, 14221, 14826, 14843, 14978, 16203, 16754, 16851, 17025, 17122, 17468, 17780, 17895, 18186, 18625, + 18688, 18690, 19679, 20200, 20334, 20839, 20851, 21519, 21641, 22539, 22568, 22691, 23420, 23581, 24952, 25232, + 25870, 26058, 27729, 27860, 28959, 29034, 29374, 29427, 29708, 29915, 30026, 30631, 30927, 31593, 31632, 31726, + 33368, 34158, 34381, 34450, 35315, 35696, 35853, 35914, 36909, 37181, 37207, 38177, 38448, 38772, 39700, 40595, + 41293, 41785, 44606, 44714, 44914, 45043, 45207, 45550, 45964, 46330, 46572, 47256, 48546, 48742, 48835, 49349, + 49373, 49642, 49741, 50328, 52872, 52989, 53928, 54652, 55036, 55133, 55720, 56288, 56612, 56683, 57926, 58096, + 60270, 62037, 63023, 63067, 63287, 63288, 64278, 64571, 65897, 66227, 66639, 67204, 67685, 68190, 68977, 71032, + 71400, 72030, 72352, 72738, 73753, 74019, 75424, 76010, 76332, 76785, 76937, 77120, 77968, 78052, 78420, 80396, + 80905, 82130, 83375, 83770, 84703, 85012, 86978, 88011, 88074, 88531, 91507, 92130, 92492, 92770, 93340, 93457, + 93647, 94057, 95323, 95459 +}; + +inline uint8_t cls(int32_t t) { + static const std::vector tab = [] { + std::vector v(248320, 0); + for (int32_t i : kNewline) v[(size_t) i] = 1; + for (int32_t i : kBlank) v[(size_t) i] = 2; + return v; + }(); + return t >= 0 && (size_t) t < tab.size() ? tab[(size_t) t] : 0; +} + +template inline bool in(int32_t t, const int32_t (&s)[N]) { + for (int32_t x : s) if (x == t) return true; + return false; +} + +enum : int32_t { + IM_START = 248045, IM_END = 248046, THINK = 248068, ETHINK = 248069, TOOL_CALL = 248058, ETOOL_CALL = 248059, + NL = 198, NL2 = 271, +}; +static const int32_t kFence[] = {71093, 52451}; // "```", " ```" +static const int32_t kColon[] = {25, 64700, 92217}; // ":", ":**", "**:" +static const int32_t kDigit[] = {16, 17, 18, 19, 20, 21, 22, 23, 24}; // "1".."9" + +inline State step(State s, int32_t tok) { + s.t2 = s.t1; + s.t1 = tok; + s.f |= KNOWN; + const uint8_t bol = s.f & BOL; + s.f &= (uint8_t) ~PBOL; + if (tok == IM_START || tok == IM_END || tok == THINK) { + s.f = KNOWN | BOL; + s.len = 0; + } else if (tok == ETHINK) { + s.f = KNOWN | PROSE | BOL; + s.len = 0; + } else if (tok == TOOL_CALL) { + s.f |= TOOL; + } else if (tok == ETOOL_CALL) { + s.f = (uint8_t) ((s.f & ~(TOOL | COLON)) | BOL); + s.len = 0; + } else if (cls(tok) == 1) { + if (!bol) s.f = (uint8_t) ((s.f & ~COLON) | (in(s.t2, kColon) ? COLON : 0)); + s.f |= BOL; + s.len = 0; + } else if (cls(tok) == 2) { + // indentation: the line has not started yet + } else { + if (in(tok, kFence) && (s.f & PROSE) && !(s.f & TOOL)) s.f ^= FENCE; + s.f = (uint8_t) ((s.f & ~(BOL | COLON)) | (bol ? PBOL : 0)); + if (s.len < 255) ++s.len; + } + return s; +} + +// the penalties on the next token after state s; out holds at least 64 +inline int penalties(const State& s, const Knobs& k, Pen* out) { + if ((s.f & (KNOWN | PROSE | TOOL | FENCE)) != (KNOWN | PROSE)) return 0; + int n = 0; + auto pen = [&](int32_t id, float v) { if (v > 0.0f && n < 64) out[n++] = {id, v}; }; + const int32_t p0 = s.t1, p1 = s.t2; + // the 7 lexical cliches (results/banned_llm_isms.json), as in v3 + static const int32_t rather[] = {4598, 24944, 61745, 70714}, ather_pre[] = {2515, 2697, 15656, 43487}; + static const int32_t right[] = {1245, 1246, 5791, 9785, 26806, 51140}; + static const int32_t end_pre[] = {400, 809, 3601, 3839, 14637, 55293}; + static const int32_t shrt[] = {2716, 8412, 10382, 12115, 36639}; + static const int32_t exactly[] = {6681, 63151, 66172}, ex_pre[] = {325, 5291}; + static const int32_t here[] = {1532, 5514, 6527, 8160, 475}, what[] = {1092, 3437, 3710, 12195, 29348}; + if (in(p0, rather) || (p0 == 1792 && in(p1, ather_pre))) pen(1056, k.lex); // " than" + if (in(p0, right) || (p0 == 481 && p1 == 2697)) pen(3023, k.lex); // " away" + if (p0 == 310 && in(p1, end_pre)) pen(809, k.lex); // "end to" -> " end" + if (in(p0, shrt)) pen(2243, k.lex); // " version" + if (in(p0, exactly) || (p0 == 31654 && in(p1, ex_pre))) pen(1092, k.lex); // " what" + if (p0 == 579 && in(p1, here)) pen(1092, k.lex); // "here's" -> " what" + if (p0 == 579 && in(p1, what)) pen(3404, k.lex); // "what's" -> " actually" + // performative honesty + static const int32_t det[] = {1719, 760, 279, 561, 276, 449, 1473, 2014, 2329, 821, 2921, 4888, 1323, 381, + 1602, 32985, 1379, 3444, 7261, 6183, 777, 4254, 678}; // the/an/my/be/being/most/more/your + static const int32_t clause[] = {13, 11, 0, 30, 25, 26, 1076, 2218, 1892, 12, 471, 8, + 321, 694, 748, 429, 7834, 684, 1541, 1921, 1987, 2939, 3850, 4272}; + static const int32_t h_[] = {71, 39, 3101, 75833}; // "h", "H", "(h", "\"h" + static const int32_t sentence[] = {13, 0, 30, 25, 26, 1076, 2218, 1892, 8}; // . ! ? : ; ... — ) before " honest take" + const bool line_start = cls(p0) == 1 || (s.f & BOL); + const bool sentence_start = in(p0, sentence) || line_start; + const bool clause_start = in(p0, clause) || line_start; + pen(92115, k.hon); // "Honestly" + pen(68540, k.hon); // " Honestly" + pen(51397, k.hon); // " Honest" + if (in(p0, det) || sentence_start) pen(10429, k.hon); // " honest" + if (clause_start) pen(25211, k.hon); // " honestly" + if (in(p0, h_)) { pen(33961, 1.5f * k.hon); pen(77696, 1.5f * k.hon); } // "h|onest", "h|onestly": committed + if (p0 == 660) { pen(46253, k.hon); pen(25211, k.hon); } // "in all" -> " honesty" + // candor announcements: with " honest" pushed down, "If I'm being" shifts to a sibling (real, frank, totally, ...). + // That is still the same frame, so those are pushed down after "being" / "be" too. Earlier in the frame, "I'm" and + // "the short/real", most of the mass is on content, so the push is aimed there as well. + static const int32_t candor[] = {1865, 47045, 24969, 5507, 17290, 7435, 62179, 66842, 86775, // real blunt frank candid transparent straight upfront brutally truthful + 12116, 6379, 6999, 13542, 11040, 73018}; // totally completely fully perfectly entirely painfully + static const int32_t be_pre[] = {310, 951, 3172, 728, 1066, 16032}; // to, 'll, me, just, gonna + " be" + static const int32_t intens[] = {12116, 6379, 6999, 13542, 11040, 73018, 66842}; // "totally honest" + if (p0 == 1602 || p0 == 32985) for (int32_t id : candor) pen(id, k.hon); // being real + if ((p0 == 381 || p0 == 1323) && in(p1, be_pre)) { + for (int32_t id : candor) pen(id, k.hon); // to be blunt, i'll be straight + pen(2050, k.hon); // be direct + if (p1 == 310 || p1 == 951) { pen(6415, k.hon); pen(2708, k.hon); } // to be fair, to be clear + } + if (in(p0, intens) && (p1 == 1602 || p1 == 32985 || p1 == 381 || p1 == 1323)) { pen(10429, k.hon); for (int32_t id : candor) pen(id, k.hon); } + if (p0 == 2688 && (p1 == 353 || p1 == 585 || p1 == 40 || p1 == 72)) { pen(1602, 0.5f * k.hon); pen(10429, k.hon); } // I'm being / I'm honest + if (clause_start) pen(40285, k.hon); // " frankly" + if (p0 == 660) { pen(49034, k.hon); pen(71853, k.hon); } // "in all" fairness / seriousness + // "the honest / real / blunt / short answer", "short answer:", "my straight take": the frame noun after the frame word + static const int32_t frame_adj[] = {10429, 1865, 47045, 24969, 5507, 7435, 2716, 4145, 3841, 26860, 13983, + 8412, 12115, 10382, 7700, 12402, 8535, 4880, 32254}; // + short/Short/ Short/real/Real/ Real/ actual/ Actual + static const int32_t frame_det[] = {1719, 760, 279, 561, 2329, 821, 2921, 4888, 276, 449}; + static const int32_t frame[] = {4087, 1831, 56004, 2243, 7793, 1301, 11782, 14996, 3296}; // answer take framing version truth read summary assessment question + if (in(p0, frame_adj) && (in(p1, frame_det) || in(p1, sentence) || cls(p1) == 1 || p1 == ETHINK)) + for (int32_t id : frame) pen(id, k.hon); + // the branch points one token before the frame closes: a clause-initial "the" / "to" can still go to content + static const int32_t the_[] = {1719, 760, 279, 561}, to_[] = {1206, 1946, 951, 310}; + const bool p1_root = in(p1, sentence) || cls(p1) == 1 || p1 == ETHINK || p1 < 0; + if (in(p0, the_) && (p1_root || cls(p0) == 0 && (s.f & PBOL))) + for (int32_t id : {1865, 47045, 24969, 5507, 7435, 2716, 26860}) pen(id, 0.5f * k.hon); // the real/blunt/frank/candid/straight/short/brutal + if (in(p0, to_) && (p1_root || (s.f & PBOL))) pen(381, 0.5f * k.hon); // clause-initial "to be" + if (s.f & BOL) for (int32_t id : {8412, 12115, 7700, 12402}) pen(id, 0.5f * k.hon); // "short answer:" / "real talk" at a line start + // structure: bullets, bold and # headers, numbered lists, rules at a line start + if (s.f & BOL) { + const float v = k.md + ((s.f & COLON) ? k.colon : 0.0f); + static const int32_t md[] = {12, 9, 6458, 471, 348, 7061, 332, 2972, 2, 550, 13962, 794, 653, 7460, 16112, + 4277, 11862, 4006, 2218, 10, 49300}; // ... "---", "***", "–", "—", "+", "→" + for (int32_t id : md) pen(id, v); + for (int32_t id : kDigit) pen(id, (s.f & COLON) ? v : 0.25f * v); // light, unless the line above ends in ":" + } + if ((s.f & PBOL) && in(p0, kDigit)) { pen(13, k.md + k.colon); pen(8, k.md + k.colon); } // "1." "1)" + // a short label line ending in ":" that would open a list + if (in(p0, kColon) && s.len <= 6) { pen(NL, k.colon); pen(NL2, k.colon); } + return n; +} + +// The ends of a turn or a thought must not gain from a penalty: with a likely token pushed down, the rest of the row +// scales up by 1 / (1 - b), b the share taken away, and <|im_end|> is often next in line (a reply cut off mid-sentence). +// Their logits go down by log(1 / (1 - b)) so their probability stays what it was. lse is the row's log-sum-exp; a +// row whose penalized tokens all sit 12+ below its max loses under 6e-6 each and is skipped (no lse needed). +static const int32_t kHold[] = {IM_END, THINK, IM_START, 248044}; + +inline bool needs_hold(const Pen* p, int n, const float* lg, float row_max) { + for (int i = 0; i < n; ++i) if (lg[p[i].id] > row_max - 12.0f) return true; + return false; +} + +inline int hold(const Pen* p, int n, const float* lg, double lse, Pen* out) { + double b = 0.0; + for (int i = 0; i < n; ++i) b += std::exp((double) lg[p[i].id] - lse) * (1.0 - std::exp(-(double) p[i].v)); + if (b <= 0.0) return 0; + const float v = (float) -std::log1p(-std::min(b, 0.999999)); + int m = 0; + for (int32_t id : kHold) out[m++] = {id, v}; + return m; +} + +inline Knobs knobs() { + auto f = [](const char* e, float d) { const char* v = std::getenv(e); return v ? (float) std::atof(v) : d; }; + return {f("STRATA_ANTI_SLOP_PENALTY", 9.0f), f("STRATA_ANTI_SLOP_HONEST", 6.0f), f("STRATA_ANTI_SLOP_MD", 4.0f), + f("STRATA_ANTI_SLOP_COLON", 3.0f)}; +} + +} // namespace slop4 + class Window { public: bool init(const strata::core::ModelGeometry& g, strata::core::SessionState& ss, const std::string& shard1, int max_t, @@ -132,6 +349,11 @@ bool commit(int n_keep, std::string& err); bool finish(int Tn, const int32_t* tokens, int64_t pos0, bool head, int32_t* out, double t0); bool copy_logits(int t, float* host) const; + // anti-slop v4: the prompt paths do not pass finish, so the prompt's tokens advance the per-position states here + template void slop_scan(const I* tokens, int64_t n, int64_t pos0) { + slop4::State st = slop_at(pos0 - 1); + for (int64_t i = 0; i < n; ++i) slop_put(pos0 + i) = st = slop4::step(st, (int32_t) tokens[i]); + } int64_t vocab() const { return n_vocab_; } bool ready() const { return cx_ != nullptr; } bool streaming() const { return stream_; } @@ -371,6 +593,12 @@ double last_gpu_ms_ = 0; int last_n_disp_ = 0; int32_t last_tokens_[MAXT] = {}; + std::vector slop_; // anti-slop v4: the state after the token at each position (State{} = unknown) + slop4::State slop_at(int64_t p) const { return p >= 0 && (size_t) p < slop_.size() ? slop_[(size_t) p] : slop4::State{}; } + slop4::State& slop_put(int64_t p) { + if ((size_t) p >= slop_.size()) slop_.resize(((size_t) p + 4096) & ~(size_t) 4095); + return slop_[(size_t) p]; + } std::vector last_norm_; // the window's normalized PLE rows (T x hc_dim), for commit // STRATA_METAL_LOGPOS= (test tooling, one-token windows): per decode position the log-probability of the // token that came next (the next window's input: with --spec-follow, the forced text) and the top 64, in the @@ -1617,46 +1845,51 @@ ++x.n; x.pre += tp_pre_ - t0; x.rg += tp_rg_; x.gpu += last_gpu_ms_; x.fin += t1 - tp_pre_ - tp_rg_; x.all += t1 - t0; } - static const float slop_pen = std::getenv("STRATA_ANTI_SLOP_PENALTY") ? (float) std::atof(std::getenv("STRATA_ANTI_SLOP_PENALTY")) : 6.0f; - if (head && slop_pen > 0.0f) { + { // anti-slop v4 (slop4 above): every row's state is stored at its position; the head rows get the penalties + static const slop4::Knobs kn = slop4::knobs(); + const bool on = head && (kn.lex > 0.0f || kn.hon > 0.0f || kn.md > 0.0f || kn.colon > 0.0f); const int per = (n_vocab_ + 119) / 120; + slop4::State st = slop_at(pos0 - 1); for (int t = 0; t < Tn; ++t) { - const int32_t p0 = tokens[t]; - const int32_t p1 = t > 0 ? tokens[t - 1] : (last_T_ > 0 ? last_tokens_[last_T_ - 1] : -1); - int rej = -1; - if (p0 == 4598 || p0 == 24944 || p0 == 61745 || p0 == 70714 || - (p0 == 1792 && (p1 == 2515 || p1 == 2697 || p1 == 15656 || p1 == 43487))) rej = 1056; - else if (p0 == 1245 || p0 == 1246 || p0 == 5791 || p0 == 9785 || p0 == 26806 || p0 == 51140 || - (p0 == 481 && p1 == 2697)) rej = 3023; - else if (p0 == 310 && (p1 == 400 || p1 == 809 || p1 == 3601 || p1 == 3839 || p1 == 14637 || p1 == 55293)) rej = 809; - else if (p0 == 2716 || p0 == 8412 || p0 == 10382 || p0 == 12115 || p0 == 36639) rej = 2243; - else if (p0 == 6681 || p0 == 63151 || p0 == 66172 || (p0 == 31654 && (p1 == 325 || p1 == 5291))) rej = 1092; - else if (p0 == 579) { - if (p1 == 1532 || p1 == 5514 || p1 == 6527 || p1 == 8160 || p1 == 475) rej = 1092; - else if (p1 == 1092 || p1 == 3437 || p1 == 3710 || p1 == 12195 || p1 == 29348) rej = 3404; - } - auto apply_pen = [&](int tid_pen) { - if (tid_pen < 0 || tid_pen >= n_vocab_) return; - float* lg = lm_.as() + (size_t) t * n_vocab_; - lg[tid_pen] -= slop_pen; - if (tokout_.as()[t] == tid_pen) { - const int part = tid_pen / per, b0 = part * per, b1 = std::min(b0 + per, (int) n_vocab_); - float bv = -INFINITY; int bi = 0x7fffffff; - for (int i = b0; i < b1; ++i) if (!std::isnan(lg[i]) && (lg[i] > bv || (lg[i] == bv && i < bi))) { bv = lg[i]; bi = i; } - float* pv = apv_.as() + (size_t) t * 120; - int32_t* pi = api_.as() + (size_t) t * 120; - pv[part] = bv; pi[part] = bi; - bv = -INFINITY; bi = 0x7fffffff; - for (int k = 0; k < 120; ++k) if (pv[k] > bv || (pv[k] == bv && pi[k] < bi)) { bv = pv[k]; bi = pi[k]; } - tokout_.as()[t] = bi; - } - }; - if (rej >= 0) apply_pen(rej); - // Step-0 empty-turn guard right after \n\n (248069, 271/198): prevent immediate <|im_end|> (248046) or (248068) - if (p1 == 248069 && (p0 == 271 || p0 == 198)) { - apply_pen(248046); - apply_pen(248068); + st = slop4::step(st, tokens[t]); + slop_put(pos0 + t) = st; + if (!on) continue; + slop4::Pen pen[80]; + int np = slop4::penalties(st, kn, pen); + float* lg = lm_.as() + (size_t) t * n_vocab_; + float* pv = apv_.as() + (size_t) t * 120; + int32_t* pi = api_.as() + (size_t) t * 120; + if (np > 0) { + float mx = -INFINITY; + for (int k = 0; k < 120; ++k) mx = std::max(mx, pv[k]); + if (slop4::needs_hold(pen, np, lg, mx)) { + double s = 0.0; + for (int64_t i = 0; i < n_vocab_; ++i) s += std::exp((double) (lg[i] - mx)); + np += slop4::hold(pen, np, lg, (double) mx + std::log(s), pen + np); + } } + // no empty turn: <|im_end|> or right after \n\n + if (kn.lex > 0.0f && st.t2 == slop4::ETHINK && (st.t1 == slop4::NL2 || st.t1 == slop4::NL)) { + pen[np++] = {slop4::IM_END, kn.lex}; + pen[np++] = {slop4::THINK, kn.lex}; + } + if (np == 0) continue; + for (int i = 0; i < np; ++i) lg[pen[i].id] -= pen[i].v; + // the argmax (tokout_) from the GPU's 120 partition maxima: a partition whose max was pushed down is rescanned + bool moved = false; + for (int i = 0; i < np; ++i) { + const int part = pen[i].id / per; + if (pi[part] != pen[i].id) continue; + const int b0 = part * per, b1 = std::min(b0 + per, (int) n_vocab_); + float bv = -INFINITY; int bi = 0x7fffffff; + for (int j = b0; j < b1; ++j) if (!std::isnan(lg[j]) && (lg[j] > bv || (lg[j] == bv && j < bi))) { bv = lg[j]; bi = j; } + pv[part] = bv; pi[part] = bi; + moved = true; + } + if (!moved) continue; + float bv = -INFINITY; int bi = 0x7fffffff; + for (int k = 0; k < 120; ++k) if (pv[k] > bv || (pv[k] == bv && pi[k] < bi)) { bv = pv[k]; bi = pi[k]; } + tokout_.as()[t] = bi; } } logpos(Tn, tokens, pos0, head); @@ -1909,6 +2142,7 @@ pend_fold_ = 0; last_T_ = ids.empty() ? 0 : 1; if (!ids.empty()) last_tokens_[0] = ids.back(); + slop_scan(ids.data(), (int64_t) ids.size(), 0); ss_->ple_prev[0] = h.ple_prev[0]; ss_->ple_prev[1] = h.ple_prev[1]; n_cells = h.n_cells; @@ -1937,6 +2171,10 @@ }); for (auto& t : th) t.join(); if (!ok) err = "metal window: SCR relocation refused"; + if (ok && n > 0 && (size_t) (c_src + n) <= slop_.size()) { // the anti-slop states move with their cells + slop_put(c_dst + n - 1); + std::memmove(slop_.data() + c_dst, slop_.data() + c_src, (size_t) n * sizeof(slop4::State)); + } return ok; } @@ -3235,6 +3473,7 @@ bool prompt_run_rows(const int64_t* tokens, int64_t n, int64_t pos0, const std::function& on_chunk, std::string& err) { if (g_window == nullptr) { err = "metal prompt: the decode window is not up yet"; return false; } + g_window->slop_scan(tokens, n, pos0); static const bool windows = std::getenv("STRATA_METAL_PROMPT") && std::string(std::getenv("STRATA_METAL_PROMPT")) == "window"; static const int64_t min_rows = std::getenv("STRATA_METAL_PROMPT_MIN") ? std::atoll(std::getenv("STRATA_METAL_PROMPT_MIN")) : 32; if (!windows && !g_window->streaming() && n >= min_rows) return g_window->prompt(tokens, n, pos0, on_chunk, err); ``` `antislop_v4.inc`, which the diff inlines above `class Window`: ```cpp // Anti-slop v4: a sampler shaper over the head logits, outside , and ``` fences only. // Each position's state (the last two tokens, the region, where the line stands) follows from the one before it, so // it is kept per position: a draft row, a rejected suffix, a prompt read from a cached prefix all start from the // state of the position before them. Tables: the Qwen3.8-Flash-Next vocab (models/*/tokenizer/vocab.json). namespace slop4 { enum : uint8_t { PROSE = 1, TOOL = 2, FENCE = 4, BOL = 8, COLON = 16, PBOL = 32, KNOWN = 128 }; struct State { int32_t t1 = -1, t2 = -1; // the last token and the one before it uint8_t f = 0; // PROSE: after in this turn; BOL: at a line start; COLON: that line follows "...:" uint8_t len = 0; // content tokens on the current line (saturating) uint8_t pad[2] = {}; }; struct Knobs { float lex, hon, md, colon; }; struct Pen { int32_t id; float v; }; // token classes: 1 a newline run ("\n", "\n\n", " \n", ...), 2 blanks only (indentation) static const int32_t kNewline[] = { 198, 201, 271, 317, 695, 845, 987, 1031, 1358, 1517, 1726, 2228, 2317, 2449, 3265, 4326, 4408, 4558, 4969, 5690, 5744, 5773, 6010, 6159, 6219, 6710, 6987, 7388, 8301, 8675, 8726, 9122, 10109, 10367, 10628, 10797, 11463, 11953, 12213, 13488, 14200, 14305, 14379, 15216, 15334, 15398, 15602, 15684, 16391, 16750, 16957, 17116, 17694, 17817, 18015, 18050, 18438, 19075, 19721, 21996, 22616, 22727, 23574, 23745, 23840, 24630, 25263, 25401, 25445, 25698, 25952, 26467, 26480, 26771, 27330, 29644, 29773, 30858, 31472, 32077, 32527, 32811, 33006, 33019, 33339, 33438, 33496, 33954, 34127, 35369, 35412, 35624, 35973, 36444, 36826, 37050, 37528, 38336, 38544, 39000, 39598, 40260, 40315, 41210, 41585, 43622, 43916, 44058, 44281, 44879, 44902, 46813, 47458, 47947, 48305, 49404, 49420, 49746, 50946, 52245, 53585, 54701, 56137, 56580, 56955, 56963, 57103, 57633, 58943, 59366, 59776, 60121, 61332, 62386, 62591, 63063, 63111, 63454, 64137, 64576, 65117, 65382, 67511, 68830, 69229, 69340, 69896, 70273, 70707, 70866, 70929, 71356, 71848, 71990, 72666, 73714, 75128, 75321, 75668, 75981, 76294, 76374, 76421, 76507, 76909, 77670, 77933, 78035, 78254, 79519, 81018, 81927, 82471, 82946, 83774, 83829, 84075, 85709, 85748, 85809, 85844, 85935, 86092, 86182, 87199, 87251, 87328, 88065, 88574, 88927, 88985, 89803, 91421, 91668, 91804, 92699, 92766, 93957, 94820, 94861 }; static const int32_t kBlank[] = { 197, 220, 256, 257, 260, 262, 285, 297, 309, 336, 388, 406, 454, 493, 556, 646, 670, 761, 950, 967, 1025, 1107, 1335, 1604, 1638, 1734, 1779, 1855, 2118, 2215, 2467, 2475, 2596, 2672, 3235, 3388, 3551, 3696, 4152, 4420, 4448, 4946, 5015, 5071, 5233, 5780, 6173, 6247, 6323, 6447, 6800, 7216, 7237, 7322, 7536, 7599, 7879, 8074, 8424, 8830, 9081, 9411, 9876, 10190, 10275, 10747, 11449, 11530, 11611, 12473, 12670, 12687, 13076, 13152, 13300, 13857, 14221, 14826, 14843, 14978, 16203, 16754, 16851, 17025, 17122, 17468, 17780, 17895, 18186, 18625, 18688, 18690, 19679, 20200, 20334, 20839, 20851, 21519, 21641, 22539, 22568, 22691, 23420, 23581, 24952, 25232, 25870, 26058, 27729, 27860, 28959, 29034, 29374, 29427, 29708, 29915, 30026, 30631, 30927, 31593, 31632, 31726, 33368, 34158, 34381, 34450, 35315, 35696, 35853, 35914, 36909, 37181, 37207, 38177, 38448, 38772, 39700, 40595, 41293, 41785, 44606, 44714, 44914, 45043, 45207, 45550, 45964, 46330, 46572, 47256, 48546, 48742, 48835, 49349, 49373, 49642, 49741, 50328, 52872, 52989, 53928, 54652, 55036, 55133, 55720, 56288, 56612, 56683, 57926, 58096, 60270, 62037, 63023, 63067, 63287, 63288, 64278, 64571, 65897, 66227, 66639, 67204, 67685, 68190, 68977, 71032, 71400, 72030, 72352, 72738, 73753, 74019, 75424, 76010, 76332, 76785, 76937, 77120, 77968, 78052, 78420, 80396, 80905, 82130, 83375, 83770, 84703, 85012, 86978, 88011, 88074, 88531, 91507, 92130, 92492, 92770, 93340, 93457, 93647, 94057, 95323, 95459 }; inline uint8_t cls(int32_t t) { static const std::vector tab = [] { std::vector v(248320, 0); for (int32_t i : kNewline) v[(size_t) i] = 1; for (int32_t i : kBlank) v[(size_t) i] = 2; return v; }(); return t >= 0 && (size_t) t < tab.size() ? tab[(size_t) t] : 0; } template inline bool in(int32_t t, const int32_t (&s)[N]) { for (int32_t x : s) if (x == t) return true; return false; } enum : int32_t { IM_START = 248045, IM_END = 248046, THINK = 248068, ETHINK = 248069, TOOL_CALL = 248058, ETOOL_CALL = 248059, NL = 198, NL2 = 271, }; static const int32_t kFence[] = {71093, 52451}; // "```", " ```" static const int32_t kColon[] = {25, 64700, 92217}; // ":", ":**", "**:" static const int32_t kDigit[] = {16, 17, 18, 19, 20, 21, 22, 23, 24}; // "1".."9" inline State step(State s, int32_t tok) { s.t2 = s.t1; s.t1 = tok; s.f |= KNOWN; const uint8_t bol = s.f & BOL; s.f &= (uint8_t) ~PBOL; if (tok == IM_START || tok == IM_END || tok == THINK) { s.f = KNOWN | BOL; s.len = 0; } else if (tok == ETHINK) { s.f = KNOWN | PROSE | BOL; s.len = 0; } else if (tok == TOOL_CALL) { s.f |= TOOL; } else if (tok == ETOOL_CALL) { s.f = (uint8_t) ((s.f & ~(TOOL | COLON)) | BOL); s.len = 0; } else if (cls(tok) == 1) { if (!bol) s.f = (uint8_t) ((s.f & ~COLON) | (in(s.t2, kColon) ? COLON : 0)); s.f |= BOL; s.len = 0; } else if (cls(tok) == 2) { // indentation: the line has not started yet } else { if (in(tok, kFence) && (s.f & PROSE) && !(s.f & TOOL)) s.f ^= FENCE; s.f = (uint8_t) ((s.f & ~(BOL | COLON)) | (bol ? PBOL : 0)); if (s.len < 255) ++s.len; } return s; } // the penalties on the next token after state s; out holds at least 64 inline int penalties(const State& s, const Knobs& k, Pen* out) { if ((s.f & (KNOWN | PROSE | TOOL | FENCE)) != (KNOWN | PROSE)) return 0; int n = 0; auto pen = [&](int32_t id, float v) { if (v > 0.0f && n < 64) out[n++] = {id, v}; }; const int32_t p0 = s.t1, p1 = s.t2; // the 7 lexical cliches (results/banned_llm_isms.json), as in v3 static const int32_t rather[] = {4598, 24944, 61745, 70714}, ather_pre[] = {2515, 2697, 15656, 43487}; static const int32_t right[] = {1245, 1246, 5791, 9785, 26806, 51140}; static const int32_t end_pre[] = {400, 809, 3601, 3839, 14637, 55293}; static const int32_t shrt[] = {2716, 8412, 10382, 12115, 36639}; static const int32_t exactly[] = {6681, 63151, 66172}, ex_pre[] = {325, 5291}; static const int32_t here[] = {1532, 5514, 6527, 8160, 475}, what[] = {1092, 3437, 3710, 12195, 29348}; if (in(p0, rather) || (p0 == 1792 && in(p1, ather_pre))) pen(1056, k.lex); // " than" if (in(p0, right) || (p0 == 481 && p1 == 2697)) pen(3023, k.lex); // " away" if (p0 == 310 && in(p1, end_pre)) pen(809, k.lex); // "end to" -> " end" if (in(p0, shrt)) pen(2243, k.lex); // " version" if (in(p0, exactly) || (p0 == 31654 && in(p1, ex_pre))) pen(1092, k.lex); // " what" if (p0 == 579 && in(p1, here)) pen(1092, k.lex); // "here's" -> " what" if (p0 == 579 && in(p1, what)) pen(3404, k.lex); // "what's" -> " actually" // performative honesty static const int32_t det[] = {1719, 760, 279, 561, 276, 449, 1473, 2014, 2329, 821, 2921, 4888, 1323, 381, 1602, 32985, 1379, 3444, 7261, 6183, 777, 4254, 678}; // the/an/my/be/being/most/more/your static const int32_t clause[] = {13, 11, 0, 30, 25, 26, 1076, 2218, 1892, 12, 471, 8, 321, 694, 748, 429, 7834, 684, 1541, 1921, 1987, 2939, 3850, 4272}; static const int32_t h_[] = {71, 39, 3101, 75833}; // "h", "H", "(h", "\"h" static const int32_t sentence[] = {13, 0, 30, 25, 26, 1076, 2218, 1892, 8}; // . ! ? : ; ... — ) before " honest take" const bool line_start = cls(p0) == 1 || (s.f & BOL); const bool sentence_start = in(p0, sentence) || line_start; const bool clause_start = in(p0, clause) || line_start; pen(92115, k.hon); // "Honestly" pen(68540, k.hon); // " Honestly" pen(51397, k.hon); // " Honest" if (in(p0, det) || sentence_start) pen(10429, k.hon); // " honest" if (clause_start) pen(25211, k.hon); // " honestly" if (in(p0, h_)) { pen(33961, 1.5f * k.hon); pen(77696, 1.5f * k.hon); } // "h|onest", "h|onestly": committed if (p0 == 660) { pen(46253, k.hon); pen(25211, k.hon); } // "in all" -> " honesty" // candor announcements: with " honest" pushed down, "If I'm being" shifts to a sibling (real, frank, totally, ...). // That is still the same frame, so those are pushed down after "being" / "be" too. Earlier in the frame, "I'm" and // "the short/real", most of the mass is on content, so the push is aimed there as well. static const int32_t candor[] = {1865, 47045, 24969, 5507, 17290, 7435, 62179, 66842, 86775, // real blunt frank candid transparent straight upfront brutally truthful 12116, 6379, 6999, 13542, 11040, 73018}; // totally completely fully perfectly entirely painfully static const int32_t be_pre[] = {310, 951, 3172, 728, 1066, 16032}; // to, 'll, me, just, gonna + " be" static const int32_t intens[] = {12116, 6379, 6999, 13542, 11040, 73018, 66842}; // "totally honest" if (p0 == 1602 || p0 == 32985) for (int32_t id : candor) pen(id, k.hon); // being real if ((p0 == 381 || p0 == 1323) && in(p1, be_pre)) { for (int32_t id : candor) pen(id, k.hon); // to be blunt, i'll be straight pen(2050, k.hon); // be direct if (p1 == 310 || p1 == 951) { pen(6415, k.hon); pen(2708, k.hon); } // to be fair, to be clear } if (in(p0, intens) && (p1 == 1602 || p1 == 32985 || p1 == 381 || p1 == 1323)) { pen(10429, k.hon); for (int32_t id : candor) pen(id, k.hon); } if (p0 == 2688 && (p1 == 353 || p1 == 585 || p1 == 40 || p1 == 72)) { pen(1602, 0.5f * k.hon); pen(10429, k.hon); } // I'm being / I'm honest if (clause_start) pen(40285, k.hon); // " frankly" if (p0 == 660) { pen(49034, k.hon); pen(71853, k.hon); } // "in all" fairness / seriousness // "the honest / real / blunt / short answer", "short answer:", "my straight take": the frame noun after the frame word static const int32_t frame_adj[] = {10429, 1865, 47045, 24969, 5507, 7435, 2716, 4145, 3841, 26860, 13983, 8412, 12115, 10382, 7700, 12402, 8535, 4880, 32254}; // + short/Short/ Short/real/Real/ Real/ actual/ Actual static const int32_t frame_det[] = {1719, 760, 279, 561, 2329, 821, 2921, 4888, 276, 449}; static const int32_t frame[] = {4087, 1831, 56004, 2243, 7793, 1301, 11782, 14996, 3296}; // answer take framing version truth read summary assessment question if (in(p0, frame_adj) && (in(p1, frame_det) || in(p1, sentence) || cls(p1) == 1 || p1 == ETHINK)) for (int32_t id : frame) pen(id, k.hon); // the branch points one token before the frame closes: a clause-initial "the" / "to" can still go to content static const int32_t the_[] = {1719, 760, 279, 561}, to_[] = {1206, 1946, 951, 310}; const bool p1_root = in(p1, sentence) || cls(p1) == 1 || p1 == ETHINK || p1 < 0; if (in(p0, the_) && (p1_root || cls(p0) == 0 && (s.f & PBOL))) for (int32_t id : {1865, 47045, 24969, 5507, 7435, 2716, 26860}) pen(id, 0.5f * k.hon); // the real/blunt/frank/candid/straight/short/brutal if (in(p0, to_) && (p1_root || (s.f & PBOL))) pen(381, 0.5f * k.hon); // clause-initial "to be" if (s.f & BOL) for (int32_t id : {8412, 12115, 7700, 12402}) pen(id, 0.5f * k.hon); // "short answer:" / "real talk" at a line start // structure: bullets, bold and # headers, numbered lists, rules at a line start if (s.f & BOL) { const float v = k.md + ((s.f & COLON) ? k.colon : 0.0f); static const int32_t md[] = {12, 9, 6458, 471, 348, 7061, 332, 2972, 2, 550, 13962, 794, 653, 7460, 16112, 4277, 11862, 4006, 2218, 10, 49300}; // ... "---", "***", "–", "—", "+", "→" for (int32_t id : md) pen(id, v); for (int32_t id : kDigit) pen(id, (s.f & COLON) ? v : 0.25f * v); // light, unless the line above ends in ":" } if ((s.f & PBOL) && in(p0, kDigit)) { pen(13, k.md + k.colon); pen(8, k.md + k.colon); } // "1." "1)" // a short label line ending in ":" that would open a list if (in(p0, kColon) && s.len <= 6) { pen(NL, k.colon); pen(NL2, k.colon); } return n; } // The ends of a turn or a thought must not gain from a penalty: with a likely token pushed down, the rest of the row // scales up by 1 / (1 - b), b the share taken away, and <|im_end|> is often next in line (a reply cut off mid-sentence). // Their logits go down by log(1 / (1 - b)) so their probability stays what it was. lse is the row's log-sum-exp; a // row whose penalized tokens all sit 12+ below its max loses under 6e-6 each and is skipped (no lse needed). static const int32_t kHold[] = {IM_END, THINK, IM_START, 248044}; inline bool needs_hold(const Pen* p, int n, const float* lg, float row_max) { for (int i = 0; i < n; ++i) if (lg[p[i].id] > row_max - 12.0f) return true; return false; } inline int hold(const Pen* p, int n, const float* lg, double lse, Pen* out) { double b = 0.0; for (int i = 0; i < n; ++i) b += std::exp((double) lg[p[i].id] - lse) * (1.0 - std::exp(-(double) p[i].v)); if (b <= 0.0) return 0; const float v = (float) -std::log1p(-std::min(b, 0.999999)); int m = 0; for (int32_t id : kHold) out[m++] = {id, v}; return m; } inline Knobs knobs() { auto f = [](const char* e, float d) { const char* v = std::getenv(e); return v ? (float) std::atof(v) : d; }; return {f("STRATA_ANTI_SLOP_PENALTY", 9.0f), f("STRATA_ANTI_SLOP_HONEST", 6.0f), f("STRATA_ANTI_SLOP_MD", 4.0f), f("STRATA_ANTI_SLOP_COLON", 3.0f)}; } } // namespace slop4 ``` ## 7. files (`/tmp/opus_antislop_v4/`) - `antislop_v4.inc`: the shipped logic. `apply_patch.py`, `window_cpp_antislop_v4.patch`, `build/` (patched + base objects and CLIs) - `eval_l47.cpp`: the evaluator. `eval_k512_docs.tsv`, `eval_k256trunk_k512head.tsv`, `eval_k256trunk_k256head.tsv`: per-row results. `sw_*`: the lex/md sweep - `k512dump/docA.*`, `docB.*`: the new K512 L47 dumps (R_post47, inj47, hh47, sgv47). `docA_tokens.txt`, `docB_tokens.txt`, `docs.json` - `eval_k512_docs_top.tsv`, `v41_docs.tsv`, `v41b_docs.tsv`: per-row top-16 for the treadmill analysis. `gen_ids.json`, `gen_decoded.json`, `genfinal_text.json`: the A/B outputs. `replay_gen.cpp`: the generation replay check - `test_state.cpp`: the position-store check. `layerB_separability.py`. `mine_slop.py` → `mined_slop_stats.json`. `qwen38_tokenizer.json`, built from vocab.json + merges.txt with the qwen35 pre-tokenizer (reproduces v3's IDs exactly) - to reproduce a dump: `STRATA_METAL_DUMP_MOE= STRATA_METAL_ARENA_GB=3.4 STRATA_METAL_WIRE_ARENA=0 STRATA_METAL_PROMPT_CHUNK=1024 strata_window_cli --metal-gguf models/K512_iq2_xs/K512-iq2_xs-metal.gguf --ple-gguf --n-expert 512 --ctx 4096 --tokens-file --n-prompt --n-gen 1` (cwd `vendor/strata`, the rest of the env from `strata-k512-iq2xs.json`). then keep layer 47 of `.hh` and `.sgv` (`extract47.py`).