15 kB 104 lines
33f2c631
docs: add Oct 2026 arXiv preprints and traps #42-#43 (GDN state quant & DuoRate frequency cancellation)
dawn
⋯
04-sources-and-references.md
sourcerendered
download raw history
at main Markdown

04 — sources & references #

11. sources #

models / repos (Hugging Face) #

runtimes #

papers (arXiv) #

  • 2608.30320: Qwen3.8-Next architecture (tech report)
  • 2402.10962: instruction (in)stability / drift in dialogs
  • 2507.11538: IFScale, adherence vs instruction density
  • 2412.00099: MoCE, Mixture of Cache-Conditional Experts (Qualcomm)
  • 2609.37725: Context Language Models; SCR in §4.3 / App. B
  • 2608.26070: Prefix Sliding
  • 2605.00649: RCO, Riemannian Constrained Optimization
  • 2604.18556: GSQ, Gumbel-Softmax Quantization
  • 2510.15061: Antislop / FTPO
  • 2311.09677: R-Tuning
  • 2405.05904: Gekhman et al., fine-tuning on new knowledge → hallucination
  • 2403.05612: Kang et al., unfamiliar finetuning examples control hallucination
  • 2509.04664: Kalai et al., why language models hallucinate
  • 2312.17238: Eliseev & Mazur, MoE offloading (next-layer gate prefetch)
  • 2601.07372: Engram (input-side n-gram memory)
  • 2609.35750: KV-streams · 2606.17034: KVEraser · 2410.15332: EPIC (cited by the blueprint, not used in this plan)
  • 2602.02099: DDCA · 2505.14604: SBT · 2505.20501: Gatsby lipogram (misattributed by the blueprint)
  • 2407.11062: EfficientQAT (blueprint background)

october 2026 arXiv preprints — implications for swift-1.5-k512-iq2xs #

primary evidence: extracted LaTeX in /tmp/arxiv_src/<id>/; these are preprints, not independently reproduced Swift results. equations below retain the papers' conventions; deployment adaptations are marked explicitly.

  • 2610.22454 — Where Prefill Expert Reduction Hurts MoE Models. https://arxiv.org/abs/2610.22454 · https://arxiv.org/pdf/2610.22454

    • main.tex, background and §§“What Is Lost, and Where” / “Where the Full-Expert Budget Goes”: a reduced token retains its top $k'<K$ routed experts with unchanged weights; shared experts and generated tokens stay full. KeepEnds protects the first and last 32 prompt tokens; Interface-Full instead reduces only marked document spans and keeps system/instructions/questions/template full (about 4% of tokens on average in their tasks). boundary sensitivity can follow template/instruction content rather than absolute position; short-answer F1 repair is not equivalent to restoring all answer likelihood.
    • for Swift: compare our existing exact final tail + reduced prefill body with a protected global head and, eventually, marked-interface protection. do not reset head/tail protection at each chunk boundary or assume our MoCE weight handling matches their intervention. the paper motivates a quality/latency ablation, not a blanket “halve experts safely” default.

  • 2610.13328 — DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving. https://arxiv.org/abs/2610.13328 · https://arxiv.org/pdf/2610.13328

    • sections/03_method.tex, Eqs. two-rates, distortion, factorization, multidomain, Propositions 1–2: $$\mathcal R_{\rm store}=\frac1E\sum_eR(M_e),\qquad \mathcal R_{\rm read}=\frac{\sum_ef_eR(M_e)}{\sum_ef_e},\qquad d_e(M)=\sum_{ij}(\Delta W^{(M)}{ij})^2A{e,i}X_{e,j}=\sum_jS_{e,M,j}X_{e,j}.$$ with matched distortion/read weights, $M_e(\lambda)=\arg\min_M[d_e(M)+\lambda R(M)]$; frequency cancels from the per-expert choice, and promotions follow $[d_e(M)-d_e(M')]/[R(M')-R(M)]$. with an unweighted storage constraint the bit shadow price is $\lambda_s/(Ef_e)$; pooled domains use $\pi_{d\mid e}=w_df_{d,e}/F_e$, $F_e=\sum_dw_df_{d,e}$, and $\bar d_e=\sum_d\pi_{d\mid e}d_e^{(d)}$.
    • StagQ rate $R(M)=2.2704+(M-2)$ bpw; store uniform p4 (4.2704), calibrate workload input moments in one prefill, then choose p2/p3/p4 prefixes online. the same-format allocation gain is only about 0.07–0.11 read bpw; much of the headline quality gain is the format. sections/04_experiments.tex / 05_limitations.tex: one Qwen3-30B-A3B-Base, no end-to-end prefix-read kernel timing; quality evaluation also prefills at the read depth, not the proposed full stored depth.
    • for Swift: our verified 144 packed tensors (68 IQ2_S, 48 Q2_0, 24 IQ2_XXS, 4 IQ3_S; 2.3993 payload bpw) are fixed-type GGUF, not StagQ prefixes. the 36.2283 GB expert payload on SSD, 6.4990 GB page-rounded arena (4,129 slots / 2 classes), GPU reads, and SSD miss bytes are distinct budgets. deployment adaptation: expected demand SSD bytes are $\sum_ef_e(1-h_e),b_e(M_e)$ plus speculative/wasted read-ahead. cancellation does not generally survive this extra expert-/tier-dependent miss factor or MoCE-induced routing changes. retain the codec; profile real workload domains and bytes before considering a new tiered artifact.

  • 2610.20826 — Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors. https://arxiv.org/abs/2610.20826 · https://arxiv.org/pdf/2610.20826

    • manuscript_v2.tex, Eq. composite, tables flat, steep, control: $S_l^{(s)}=w_dz^{(s)}(1-H_l)+w_cz^{(s)}(\mathrm{CV}_l)+w_pz^{(s)}(1-\min(J_l^{rr},J_l^{pr}))+w_tz^{(s)}(\tau_l)$, max-aggregated across prompt/reasoning/response and rank-allocated under a byte budget. on Qwen-3.6-35B-A3B, flat-tier uniform KLD 0.014725 vs best proxy 0.015276 (neutral); steep uniform 0.034321 vs proxy 0.038142–0.038249, random 0.037728/0.037808; p99 rises from 0.437 to proxy 0.487–0.492. random is marginally better than proxy, within about 1 SE, not a decisive random-over-proxy win.
    • for Swift: expert-profile.bin popularity and phase routing can guide caching but cannot replace measured marginal quantization distortion. this study is layerwise, coding-only, with no direct Hessian/ablation ground truth and no BF16 routing-agreement control; it neither disproves Hessian-calibrated GSQ-RCO nor establishes a universal IQ2 floor cliff for our codecs.

  • 2610.20865 — Long-Context Drafting Does Not Require Long-Context Drafters. https://arxiv.org/abs/2610.20865 · https://arxiv.org/pdf/2610.20865

    • arxiv.tex, Eqs. alpha, rope, decomp: $\alpha_t=1-\mathrm{TV}(p_t,q_t)$ for speculative rejection sampling; $\mathrm{TV}(p_t,q_t)\le\mathrm{TV}(p_t,p_t^{(W)})+\mathrm{TV}(p_t^{(W)},q_t)$. a frozen target-conditioned drafter can attend to only its last $W$ tokens while the target verifies against full context. $\langle R(m)q,R(n)k\rangle=q^TR(n-m)k$ gives common-shift invariance; assignment-time $\tilde m=m\bmod P_{\max}$ is exact only when the dependency graph stays within one index band (not an unconditional wrap-boundary guarantee). measured greedy acceptance is exact-token agreement, not softmax TV.
    • for Swift: supports testing our already-windowed target-conditioned native MTP at smaller windows; no automatic reason to remove pinned sinks or add modulo positions to the target. our native MTP shares the target's position regime, unlike their external short-context drafters. keep full target verification, valid cached ranges and rollback exact; measure accepted tokens/round and wall time, not draft speed alone.

  • 2610.13778 — ReplaySSM: Cache SSM Inputs, Not State. https://arxiv.org/abs/2610.13778 · https://arxiv.org/pdf/2610.13778

    • sections/04_standard_decoding.tex, Eq. replay: $S_h=\gamma_hS_{\rm ckpt}+\sum_{j=1}^h\gamma_{j+1:h}v_jk_j^T$; 05_speculative_decoding.tex, Eq. output-only: $O=S_{\rm ckpt}QD+V(W\odot(K^TQ))$. cache rank-one inputs between checkpoint flushes (L=16); accept/reject by moving the committed input pointer instead of copying every intermediate full state. GDN stores correction $u_j$, not raw $v_j$; speculative GDN corrections require a triangular solve.
    • 06_low_precision.tex: retain a high-precision recurrent checkpoint plus a low-precision output-only copy; for GDN also cache raw $v$ and recompute corrections from the high-precision path at flush. this does not reduce total state storage to INT4. for Swift: a possible FP32-preserving traffic/rollback optimization for our 36 GDN layers; requires new kernels and session/SCR persistence semantics. H100/B300 large-batch results are not M5 batch-one speed evidence, and reordered FP arithmetic is not guaranteed byte-identical.

  • 2610.15139 — When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention LLMs. https://arxiv.org/abs/2610.15139 · https://arxiv.org/pdf/2610.15139

    • sec_mechanism.tex, sec_policy.tex, sec_whatworks.tex: $s_{t,c}=\max|\widehat S_{t,c}|/q_{\max}$ closes a feedback loop; yoking to $s_t=\max|S_t^{\rm FP}|/q_{\max}$ stops runaway without eliminating resolution loss. at H=2048, GDN-0.8B dynamic excess NLL 5.39, yoked INT4 3.57, yoked INT8 0.31. stochastic rounding adds 6.71 nats over yoked INT4 and raises RMS by $\sqrt2$, measured 1.42× (not $\sqrt{23}$).
    • the first recurrent layer alone supplies 63–78% of summed single-layer INT4 damage; protecting the first three layers is a separate allocation result. group-wise scaling is less damaging but still not lossless. for Swift: keep high-precision state; do not infer long-context safety from stable norms, short exposures, or perfect needle recall. neither their small GDN models nor their error fractions are measured on Swift.

  • 2610.15709 — Preserving Updates in Low-Bit Recurrent State Quantization. https://arxiv.org/abs/2610.15709 · https://arxiv.org/pdf/2610.15709

    • sec/2_bkg.tex, Eqs. acc_dev, dev_split: $E_T=\sum_{\tau\le T}A_{\tau:T}R_\tau$ and, for scalar decay, $|E_T|F^2=\sum\tau a_{\tau:T}^2|R_\tau|F^2+\sum{\tau\ne\tau'}a_{\tau:T}a_{\tau':T}\langle R_\tau,R_{\tau'}\rangle_F$. small lost updates drive coherent error; this frozen-update decomposition is diagnostic, not a full closed-loop GDN stability proof.
    • sec/3_lrb.tex, Eqs. lrb, merge, bdr: bypass $L_t=A_tL_{t-1}+k_t\bar v_t^T$; retain $L_t'=\mathcal T_k(L_t\odot\mathbf1[|L_t|<\epsilon_L\Delta_t])$ at merge and quantize the residual. BDR adds $\operatorname{sign}(d)\Delta_t$ only if effective update $\tilde d=Q(s+d)-s$ reverses $d$ and $|d|>\epsilon_B\Delta_t$, with an in-range-grid guard; overshoot is bounded by $|d|/\epsilon_B$.
    • GDN configuration: rank 1, merge interval 15, $\epsilon_L=0.50$, $\epsilon_B=0.30$, 5.25 average / 6.00 peak bits including bypass. WikiText-103 8K PPL 24.19 FP32 / 79.76 INT4 / 26.85 LRB+BDR. for Swift: investigate only if state traffic becomes material; published 1.3B, unquantized-weight, RTX 3060 batch 16/32/48 evidence does not validate this hybrid GSQ-RCO model. it complements the scale-feedback warning; it is not evidence for naive INT4 or stochastic rounding.

hardware #

local artifacts #

  • scripts/mine_llm_isms_from_transcripts.py: slop miner (all §9 + audits/opus_audit_slop_miner_and_1c_v3.md fixes applied; --self-test PASS; output in results/banned_llm_isms.json)
  • scripts/run_colab_a100_agent_persona.py: Phase 6 Colab A100 48-layer exact GGUF-backed 96-writer LoRA SFT + FTPO/DPO trainer & GGUF tensor transplanter
  • audits/r40_rocm14_metal_optimizations.md & scripts/strata_rocm14_metal_opts.patch: Round 40–43 Metal kernel & engine optimizations