[{"name":"ggml-blas","kind":"directory","size":0,"lastCommitHash":"153d324bcf86d220b235ca010eeb11213f32b5d1","lastCommitWhen":"2026-08-11T09:20:46+03:00","lastCommitMessage":"llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)\n\n* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used\n\n* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp\n\nCo-authored-by: Max Krasnyansky \n\n* set mmap_support to false on OpenCL backend\n\n* fix order of load modes\n\n* use -1 for auto\n\n* resolve load mode auto earlier to correctly pick gpu host or cpu memory\n\n* add load mode auto to llama-bench\n\n* bump virtgpu api version, regenerate docs\n\n---------\n\nCo-authored-by: Piotr Wilkin (ilintar) \nCo-authored-by: Max Krasnyansky \nCo-authored-by: Georgi Gerganov "},{"name":"ggml-cann","kind":"directory","size":0,"lastCommitHash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","lastCommitWhen":"2026-08-19T14:04:57+02:00","lastCommitMessage":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization"},{"name":"ggml-cpu","kind":"directory","size":0,"lastCommitHash":"adfffbe41b2cabcd51fff326ab045662265062bb","lastCommitWhen":"2026-09-25T11:42:36-07:00","lastCommitMessage":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds."},{"name":"ggml-cuda","kind":"directory","size":0,"lastCommitHash":"0163094fa579c4547035f7700b1f6ca009fc8331","lastCommitWhen":"2026-10-13T19:13:57-04:00","lastCommitMessage":"bonsai: q8_0 decode/verify flash attention raw K/V lead staging (fattn-verify-lead)\n\nq8_0 decode/verify flash attention (the 4x8 tile) stages raw K/V with cp.async and starts the next tile's K and V a phase early. Bit-identical; decode +2.9% at 64K depth.\n"},{"name":"ggml-et","kind":"directory","size":0,"lastCommitHash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","lastCommitWhen":"2026-08-19T14:04:57+02:00","lastCommitMessage":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization"},{"name":"ggml-hexagon","kind":"directory","size":0,"lastCommitHash":"00e8187618590ca52e6737f97d38173087458ebf","lastCommitWhen":"2026-09-08T15:47:22-07:00","lastCommitMessage":"graph: six fusions for the hybrid-attention decode graph (-25% ops/token; Metal 2B +7.7%, 27B +2.6% decode; CUDA 9B +2.5%) (#165)\n\n* metal: fuse the Hadamard sign flip into the FWHT kernel\n\nEvery folded weight's activation went through three encoded ops: an elementwise\nMUL by the +-1 sign vector, a RESHAPE and the FWHT-hint MUL_MAT. The sign factors\nare exact, so the transform now applies them while loading its row and the MUL\nis elided, matching what the CUDA backend already does. The encode list drops\nempty ops, so the MUL / RESHAPE / MUL_MAT pattern is matched on the graph's own\nindices and checked with ggml_can_fuse_subgraph_ext; the fused kernel writes the\nmatmul's output, so its ranges are checked for concurrency explicitly.\n\nMeasured on an M5 Pro, largest band PQ2_0 folded (block 1024), r=3 x2 interleaved:\nencoded nodes per decoded token 2359 -> 2103, tg128 27.05 -> 27.45 (+1.5%),\npp512 390.5 -> 397.2 (+1.7%). Greedy output byte-identical over 3 prompts;\ntest-backend-ops MUL_MAT_HADAMARD 24/24.\n\n* ggml, metal: fold the GDN gate activations into the fused op (raw gates)\n\nThe hybrid-attention layers spent four elementwise launches per layer per\ntoken on the gate scalars: sigmoid(beta), and softplus(alpha + dt_bias) * A\nfor the decay. ggml_gated_delta_net_set_raw_gates() marks a gated_delta_net\nop as taking beta and alpha pre-activation, with dt_bias and A as two extra\nsources, and the kernel applies the same formulas as the unary kernels while\nreading the scalars it already read. The qwen35 builder hands the fused op the\nraw projections (GGML_GDN_RAW_GATES_DISABLE=1 restores the separate ops); the\nnon-fused chunked path keeps the activated tensors. CPU reference and Metal\nkernel implemented here; CUDA follows in the next commit.\n\ntest-backend-ops -o GATED_DELTA_NET on MTL0: 39/39 incl. 4 raw-gate cases\n(decode, 2 seqs, prefill 64, rows mode). Encoded nodes per decoded token on the\nlargest band folded PQ2_0: 2103 -> 1910. Greedy output byte-identical to base.\nM5 Pro, r=3 x2 interleaved, both fusions vs base: 2B tg128 206.4 -> 214.8\n(+4.1%), pp512 +1.0%; largest band tg128 27.08 -> 27.55 (+1.7%), pp512 +1.3%.\n\n* cuda: raw gates for the fused gated_delta_net kernel\n\nSame folding as the Metal and CPU implementations: with the op's raw-gate flag\nset, beta goes through sigmoid and the decay is raw_a[h] * softplus(g +\nraw_dt_bias[h]) inside the kernel, using the formulas of the unary kernels.\nThe template gains a RAW parameter (scalar gate only; KDA asserts it off) and\nthe two tables are passed as extra pointers.\n\nCompile-checked only by inspection here; needs a CUDA build plus\ntest-backend-ops -o GATED_DELTA_NET before this branch is pushed.\n\n* ggml: decline raw-gate gated_delta_net on backends without it; gate the builder on device backend\n\nSYCL, OpenCL, WebGPU and Hexagon compute gated_delta_net but do not implement the\nraw-gate flag, so their supports_op now declines such ops (the CPU fallback is\ncorrect). The qwen35 builder only emits raw gates when every device is CPU,\nMetal, CUDA, ROCm or MUSA, so those backends keep their native op instead of a\nCPU fallback for the whole recurrence.\n\n* qwen35: gated norm as one swiglu_split op\n\nsilu(z) * rms_norm(o) was a unary and a mul; ggml_swiglu_split computes the same\nexpression in one kernel. Encoded nodes per decoded token on the largest folded band\nPQ2_0: 1910 -> 1862. Greedy output byte-identical. M5 Pro decode with the three\ngraph changes so far vs base: 2B 205.3 -> 219.9 tok/s (+7.1%), largest band 27.12 ->\n27.72 (+2.2%).\n\n* qwen35: one l2_norm over the adjacent q and k head groups\n\nq and k occupy adjacent, equally shaped head groups in the conv output, so a\nsingle l2_norm over the joint view replaces two launches per recurrent layer;\nq and k become views into its result. Encoded nodes per decoded token on the\nlargest band: 1862 -> 1814. Greedy output byte-identical.\n\n* metal: fuse the trailing silu into the ssm_conv kernels\n\nWhen the conv output is consumed only by a silu, the conv kernels apply it on\nstore (function constant) and write the silu's tensor directly, matching what\nthe CUDA backend already does. Encoded nodes per decoded token on the largest band:\n1814 -> 1766. test-backend-ops -o SSM_CONV 45/45; greedy output byte-identical.\nCumulative decode vs base on an M5 Pro: 2B 207.0 -> 223.0 tok/s (+7.7%),\nlargest band 27.15 -> 27.85 (+2.6%).\n\n* qwen35: scan the devices once per graph build for the GDN path choices\n\nThe fused-GDN state-rows check and the raw-gates check each looped over the\nmodel's devices on every recurrent layer, asking the backend for the device\ntype each time. Both flags are now computed once in the graph constructor.\n\n* tests: register the raw-gate gated_delta_net cases in the eval list\n\nThey had been appended to the perf-mode list, so test mode never ran them."},{"name":"ggml-hip","kind":"directory","size":0,"lastCommitHash":"e79e4bf660e19f2ad851e06c6913f7a8c5852621","lastCommitWhen":"2026-08-13T08:38:02+02:00","lastCommitMessage":"ggml-hip : remove -funsafe-math-optimizations (#26696)\n\nIt enables -fassociative-math, which reassociates FP reductions and can flip\ngreedy argmax on RDNA3.5 (e.g. MTP speculative decode diverging from the\nnon-speculative baseline). Drop it so HIP builds are IEEE-conformant.\n\nCo-authored-by: Jim Wu "},{"name":"ggml-metal","kind":"directory","size":0,"lastCommitHash":"df7c49e84f92f1c0d490d409fc3a4688b284bd30","lastCommitWhen":"2026-09-25T11:19:33-07:00","lastCommitMessage":"metal : add PTQ1_0 mat-vec for two to four columns (#262)\n\nAssisted-by: OpenAI Codex"},{"name":"ggml-musa","kind":"directory","size":0,"lastCommitHash":"db9d8aa428012cc5593e18635d4c3c54095f5138","lastCommitWhen":"2026-03-22T11:05:51+01:00","lastCommitMessage":"ggml-cuda: native bf16 flash attention for vec kernel (#20525)\n\n* ggml-cuda: native bf16 flash attention for vec and tile kernels\n\nmma kernel still converts bf16 to fp16 before launch, native mma bf16 todo\n\n* ggml-cuda: address code owner review feedback\n\nreverted tile kernel changes to avoid larger refactor\n\n* fix ci failures on turing and hip\n\n* fix bf16 vec kernel compile on hip v_dot2 platforms\n\n* add comments\n\n---------\n\nCo-authored-by: Johannes Gäßler "},{"name":"ggml-opencl","kind":"directory","size":0,"lastCommitHash":"00e8187618590ca52e6737f97d38173087458ebf","lastCommitWhen":"2026-09-08T15:47:22-07:00","lastCommitMessage":"graph: six fusions for the hybrid-attention decode graph (-25% ops/token; Metal 2B +7.7%, 27B +2.6% decode; CUDA 9B +2.5%) (#165)\n\n* metal: fuse the Hadamard sign flip into the FWHT kernel\n\nEvery folded weight's activation went through three encoded ops: an elementwise\nMUL by the +-1 sign vector, a RESHAPE and the FWHT-hint MUL_MAT. The sign factors\nare exact, so the transform now applies them while loading its row and the MUL\nis elided, matching what the CUDA backend already does. The encode list drops\nempty ops, so the MUL / RESHAPE / MUL_MAT pattern is matched on the graph's own\nindices and checked with ggml_can_fuse_subgraph_ext; the fused kernel writes the\nmatmul's output, so its ranges are checked for concurrency explicitly.\n\nMeasured on an M5 Pro, largest band PQ2_0 folded (block 1024), r=3 x2 interleaved:\nencoded nodes per decoded token 2359 -> 2103, tg128 27.05 -> 27.45 (+1.5%),\npp512 390.5 -> 397.2 (+1.7%). Greedy output byte-identical over 3 prompts;\ntest-backend-ops MUL_MAT_HADAMARD 24/24.\n\n* ggml, metal: fold the GDN gate activations into the fused op (raw gates)\n\nThe hybrid-attention layers spent four elementwise launches per layer per\ntoken on the gate scalars: sigmoid(beta), and softplus(alpha + dt_bias) * A\nfor the decay. ggml_gated_delta_net_set_raw_gates() marks a gated_delta_net\nop as taking beta and alpha pre-activation, with dt_bias and A as two extra\nsources, and the kernel applies the same formulas as the unary kernels while\nreading the scalars it already read. The qwen35 builder hands the fused op the\nraw projections (GGML_GDN_RAW_GATES_DISABLE=1 restores the separate ops); the\nnon-fused chunked path keeps the activated tensors. CPU reference and Metal\nkernel implemented here; CUDA follows in the next commit.\n\ntest-backend-ops -o GATED_DELTA_NET on MTL0: 39/39 incl. 4 raw-gate cases\n(decode, 2 seqs, prefill 64, rows mode). Encoded nodes per decoded token on the\nlargest band folded PQ2_0: 2103 -> 1910. Greedy output byte-identical to base.\nM5 Pro, r=3 x2 interleaved, both fusions vs base: 2B tg128 206.4 -> 214.8\n(+4.1%), pp512 +1.0%; largest band tg128 27.08 -> 27.55 (+1.7%), pp512 +1.3%.\n\n* cuda: raw gates for the fused gated_delta_net kernel\n\nSame folding as the Metal and CPU implementations: with the op's raw-gate flag\nset, beta goes through sigmoid and the decay is raw_a[h] * softplus(g +\nraw_dt_bias[h]) inside the kernel, using the formulas of the unary kernels.\nThe template gains a RAW parameter (scalar gate only; KDA asserts it off) and\nthe two tables are passed as extra pointers.\n\nCompile-checked only by inspection here; needs a CUDA build plus\ntest-backend-ops -o GATED_DELTA_NET before this branch is pushed.\n\n* ggml: decline raw-gate gated_delta_net on backends without it; gate the builder on device backend\n\nSYCL, OpenCL, WebGPU and Hexagon compute gated_delta_net but do not implement the\nraw-gate flag, so their supports_op now declines such ops (the CPU fallback is\ncorrect). The qwen35 builder only emits raw gates when every device is CPU,\nMetal, CUDA, ROCm or MUSA, so those backends keep their native op instead of a\nCPU fallback for the whole recurrence.\n\n* qwen35: gated norm as one swiglu_split op\n\nsilu(z) * rms_norm(o) was a unary and a mul; ggml_swiglu_split computes the same\nexpression in one kernel. Encoded nodes per decoded token on the largest folded band\nPQ2_0: 1910 -> 1862. Greedy output byte-identical. M5 Pro decode with the three\ngraph changes so far vs base: 2B 205.3 -> 219.9 tok/s (+7.1%), largest band 27.12 ->\n27.72 (+2.2%).\n\n* qwen35: one l2_norm over the adjacent q and k head groups\n\nq and k occupy adjacent, equally shaped head groups in the conv output, so a\nsingle l2_norm over the joint view replaces two launches per recurrent layer;\nq and k become views into its result. Encoded nodes per decoded token on the\nlargest band: 1862 -> 1814. Greedy output byte-identical.\n\n* metal: fuse the trailing silu into the ssm_conv kernels\n\nWhen the conv output is consumed only by a silu, the conv kernels apply it on\nstore (function constant) and write the silu's tensor directly, matching what\nthe CUDA backend already does. Encoded nodes per decoded token on the largest band:\n1814 -> 1766. test-backend-ops -o SSM_CONV 45/45; greedy output byte-identical.\nCumulative decode vs base on an M5 Pro: 2B 207.0 -> 223.0 tok/s (+7.7%),\nlargest band 27.15 -> 27.85 (+2.6%).\n\n* qwen35: scan the devices once per graph build for the GDN path choices\n\nThe fused-GDN state-rows check and the raw-gates check each looped over the\nmodel's devices on every recurrent layer, asking the backend for the device\ntype each time. Both flags are now computed once in the graph constructor.\n\n* tests: register the raw-gate gated_delta_net cases in the eval list\n\nThey had been appended to the perf-mode list, so test mode never ran them."},{"name":"ggml-openvino","kind":"directory","size":0,"lastCommitHash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","lastCommitWhen":"2026-08-19T14:04:57+02:00","lastCommitMessage":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization"},{"name":"ggml-rpc","kind":"directory","size":0,"lastCommitHash":"af5172627d3513a7efed526b206dca9cd6536452","lastCommitWhen":"2026-08-18T21:08:57+05:30","lastCommitMessage":"RPC: populate use_count to enable fusion inside backends (#27142)\n\n* RPC: populate use_count to enable fusion inside backends\n\n* bump minor version"},{"name":"ggml-sycl","kind":"directory","size":0,"lastCommitHash":"ccbf5a759d0e588ce92bf6902b2afd770e6669e0","lastCommitWhen":"2026-09-23T18:42:09-07:00","lastCommitMessage":"sycl: add PTQ1_0 and PQ2_0 support with MMVQ vector dot kernel (#235)\n\n* sycl: add PTQ1_0 and PQ2_0 support with MMVQ vector dot kernel\n\n- Implement base-3 trit unpacking and dequantization (dequantize_ptq1_0, dequantize_pq2_0) in dequantize.hpp and convert.cpp\n- Implement vec_dot_ptq1_0_q8_1 and mul_mat_vec_ptq1_0_q8_1_sycl in vecdotq.hpp and mmvq.cpp\n- Add get_rows and cpy dispatch for PTQ1_0 and PQ2_0\n- Enable PTQ1_0 in can_use_mul_mat_vec_q() for fast in-register Level-Zero vector execution\n- Verified on Intel Arc Pro B50 with oneAPI 2026.1 / Level-Zero (2.20 tok/s on 27B)\n\n* sycl: vectorize PTQ1_0 dot product with SIMD trit unpacking and DP4A\n\n* sycl: add native PQ2_0 MMVQ vector dot kernel with DP4A\n\n* sycl: spread PQ2_0 MMVQ across 16 lanes per block (~2x tg)\n\nThe MMVQ kernel derives its lane split from qi/vdr. PQ2_0 was passing\nQI_PQ2_0 (= QK_PQ2_0/32 = 4), which put only 4 lanes on each 128-element\nblock and left every lane an 8-deep serial dp4a chain over 8 bytes of qs.\n\nPass a dedicated MMVQ_PQ2_0_QI of 16 instead, so a block is covered by 16\nlanes reading 2 bytes / 8 elements each with 2 dp4a per call, and a\nsub-group keeps more blocks in flight. Measured on an Arc iGPU with\ntest-backend-ops perf, MUL_MAT m=4096 n=1 k=14336, alternating the two\nbuilds to cancel out background load: 827/846/837 us before, 397/395/384\nus after (~2.1x). On an idle machine the same case goes 247.6 -> 123.0 us.\n8 and 32 lanes per block were both tried and were slower; at 32 the\nper-call scale setup dominates the single remaining dp4a.\n\nAlso fold the 2-bit-to-byte-lane unpack shared with the q2_0 path into\nunpack_2bit_to_byte_lanes(), and switch the epilogue to the q4_0-style\nsumi*ds.x - zp*ds.y form. The previous PQ2_0 epilogue applied the zero\npoint with byte_sub_4 and then used only ds[0], dropping the ds.y term.\n\ntest-backend-ops test -o MUL_MAT -p type_a=pq2_0: 38/38 on both SYCL\ndevices, 3/3 backends passed. The unrelated q1_0 failures on SYCL1 are\npresent with and without this change.\n\n* sycl: gate mmvq on implemented types instead of aborting\n\ncan_use_mul_mat_vec_q() claimed every quantized type via\nggml_is_quantized(), so any type without a case in\nmul_mat_vec_q_switch_type() reached the GGML_ABORT in its default arm\nand killed the process. test-backend-ops -o MUL_MAT could not complete\non SYCL for this reason: it died at mmvq.cpp:2710 on tq2_0, which also\nmeant every case generated after that point never ran.\n\nGate on an explicit list of the types the switch actually implements.\nThe list matches the switch exactly, and is also identical to the set\nggml_get_to_fp16_sycl() handles, so a type this declines cannot be\nserved by the dequantize fallback either - hence the matching check in\nsupports_op(), which lets the scheduler place those nodes on CPU rather\nthan abort mid-graph.\n\nArc Pro B50, oneAPI 2026.1, Level Zero:\n before: test -o MUL_MAT aborts, exit 134\n after: 1098/1098 tests passed, exit 0\ntq2_0 goes from \"fatal error: unsupport data type\" to \"not supported\".\n\n* test-backend-ops: add Bonsai-shape MUL_MAT cases for PTQ1_0/PQ2_0\n\nThe stock generator only emits 16 x 256 cases for these types, all of\nwhich take the single-column mmvq path, so the multi-column dispatchers\n(ncols_dst 2..8) had no coverage at all.\n\nAdd the real Bonsai row lengths (k = 1024/5120/6144/17408) at odd row\ncounts (m = 67/70, landing a partial row group) with n sweeping 1..8.\n\nArc Pro B50, oneAPI 2026.1, Level Zero, per type:\n MUL_MAT 9 -> 102/102 passed\n MUL_MAT_ID 75/75 passed\n\n* sycl: decline float -> PTQ1_0/PQ2_0 copies instead of asserting\n\nAdding the two types to ggml_sycl_is_quantized_type() made supports_op\naccept f32/f16/bf16 -> PTQ1_0/PQ2_0 copies, but there is no quantize-rows\nkernel for either, so ggml_sycl_cpy() took the float -> quantized branch\nand died on GGML_ASSERT(ggml_sycl_can_quantize_rows_sycl(...)) at\ncpy.cpp:1298/1302/1308.\n\nNeither format can be produced by a row-at-a-time GPU kernel: the packing\nassumes the Hadamard rotation the offline converter applies, so the right\nanswer is to decline the pair and let the scheduler place it elsewhere,\nnot to add a kernel. Quantized -> same-quantized copies are unaffected and\nstill take the memcpy path.\n\nReported by @dwymark on an Arc B390 (Panther Lake Xe3); reproduced on an\nArc Pro B50.\n\n test -b SYCL0 -o CPY -p \"ptq1_0|pq2_0\"\n before: abort at cpy.cpp:1298\n after: 18/18 tests passed, float -> ternary reports \"not supported\"\n\nNo regression: MUL_MAT 204/204, MUL_MAT_ID 150/150, GET_ROWS 8/8 for the\ntwo types.\n\nThe unrelated tq2_0 -> tq2_0 abort at cpy.cpp:1455 and the SET_ROWS abort\nat set_rows.cpp:549 are both present on prism and untouched here."},{"name":"ggml-virtgpu","kind":"directory","size":0,"lastCommitHash":"153d324bcf86d220b235ca010eeb11213f32b5d1","lastCommitWhen":"2026-08-11T09:20:46+03:00","lastCommitMessage":"llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)\n\n* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used\n\n* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp\n\nCo-authored-by: Max Krasnyansky \n\n* set mmap_support to false on OpenCL backend\n\n* fix order of load modes\n\n* use -1 for auto\n\n* resolve load mode auto earlier to correctly pick gpu host or cpu memory\n\n* add load mode auto to llama-bench\n\n* bump virtgpu api version, regenerate docs\n\n---------\n\nCo-authored-by: Piotr Wilkin (ilintar) \nCo-authored-by: Max Krasnyansky \nCo-authored-by: Georgi Gerganov "},{"name":"ggml-vulkan","kind":"directory","size":0,"lastCommitHash":"a419d438a1e1932d37b8bf4d0b86312455aa9b3e","lastCommitWhen":"2026-09-21T20:53:58-07:00","lastCommitMessage":"vulkan: decline GATED_DELTA_NET raw gates instead of computing them wrong (#239)\n\nggml_gated_delta_net_set_raw_gates() delivers beta and g pre-activation,\nso the op must apply beta = sigmoid(beta) and\ng = a * softplus(g + dt_bias), with dt_bias in src[7] and a in src[8].\n\ngated_delta_net.comp has neither of those bindings nor that math - it\napplies exp(g) unconditionally - but supports_op never checked the flag,\nso Vulkan claimed these ops and returned results uncorrelated with the\nreference (ERR ~1.0 against a 1e-7 tolerance), silently and with no\nfallback.\n\nDecline raw gates in supports_op so they fall back to the CPU, which\nimplements them. This mirrors what the SYCL backend already does.\n\nOn Arc B390, test-backend-ops -b Vulkan0 goes from 3 failing\nGATED_DELTA_NET cases to a clean run: the full unfiltered suite reports\nFAIL before and OK after."},{"name":"ggml-webgpu","kind":"directory","size":0,"lastCommitHash":"00e8187618590ca52e6737f97d38173087458ebf","lastCommitWhen":"2026-09-08T15:47:22-07:00","lastCommitMessage":"graph: six fusions for the hybrid-attention decode graph (-25% ops/token; Metal 2B +7.7%, 27B +2.6% decode; CUDA 9B +2.5%) (#165)\n\n* metal: fuse the Hadamard sign flip into the FWHT kernel\n\nEvery folded weight's activation went through three encoded ops: an elementwise\nMUL by the +-1 sign vector, a RESHAPE and the FWHT-hint MUL_MAT. The sign factors\nare exact, so the transform now applies them while loading its row and the MUL\nis elided, matching what the CUDA backend already does. The encode list drops\nempty ops, so the MUL / RESHAPE / MUL_MAT pattern is matched on the graph's own\nindices and checked with ggml_can_fuse_subgraph_ext; the fused kernel writes the\nmatmul's output, so its ranges are checked for concurrency explicitly.\n\nMeasured on an M5 Pro, largest band PQ2_0 folded (block 1024), r=3 x2 interleaved:\nencoded nodes per decoded token 2359 -> 2103, tg128 27.05 -> 27.45 (+1.5%),\npp512 390.5 -> 397.2 (+1.7%). Greedy output byte-identical over 3 prompts;\ntest-backend-ops MUL_MAT_HADAMARD 24/24.\n\n* ggml, metal: fold the GDN gate activations into the fused op (raw gates)\n\nThe hybrid-attention layers spent four elementwise launches per layer per\ntoken on the gate scalars: sigmoid(beta), and softplus(alpha + dt_bias) * A\nfor the decay. ggml_gated_delta_net_set_raw_gates() marks a gated_delta_net\nop as taking beta and alpha pre-activation, with dt_bias and A as two extra\nsources, and the kernel applies the same formulas as the unary kernels while\nreading the scalars it already read. The qwen35 builder hands the fused op the\nraw projections (GGML_GDN_RAW_GATES_DISABLE=1 restores the separate ops); the\nnon-fused chunked path keeps the activated tensors. CPU reference and Metal\nkernel implemented here; CUDA follows in the next commit.\n\ntest-backend-ops -o GATED_DELTA_NET on MTL0: 39/39 incl. 4 raw-gate cases\n(decode, 2 seqs, prefill 64, rows mode). Encoded nodes per decoded token on the\nlargest band folded PQ2_0: 2103 -> 1910. Greedy output byte-identical to base.\nM5 Pro, r=3 x2 interleaved, both fusions vs base: 2B tg128 206.4 -> 214.8\n(+4.1%), pp512 +1.0%; largest band tg128 27.08 -> 27.55 (+1.7%), pp512 +1.3%.\n\n* cuda: raw gates for the fused gated_delta_net kernel\n\nSame folding as the Metal and CPU implementations: with the op's raw-gate flag\nset, beta goes through sigmoid and the decay is raw_a[h] * softplus(g +\nraw_dt_bias[h]) inside the kernel, using the formulas of the unary kernels.\nThe template gains a RAW parameter (scalar gate only; KDA asserts it off) and\nthe two tables are passed as extra pointers.\n\nCompile-checked only by inspection here; needs a CUDA build plus\ntest-backend-ops -o GATED_DELTA_NET before this branch is pushed.\n\n* ggml: decline raw-gate gated_delta_net on backends without it; gate the builder on device backend\n\nSYCL, OpenCL, WebGPU and Hexagon compute gated_delta_net but do not implement the\nraw-gate flag, so their supports_op now declines such ops (the CPU fallback is\ncorrect). The qwen35 builder only emits raw gates when every device is CPU,\nMetal, CUDA, ROCm or MUSA, so those backends keep their native op instead of a\nCPU fallback for the whole recurrence.\n\n* qwen35: gated norm as one swiglu_split op\n\nsilu(z) * rms_norm(o) was a unary and a mul; ggml_swiglu_split computes the same\nexpression in one kernel. Encoded nodes per decoded token on the largest folded band\nPQ2_0: 1910 -> 1862. Greedy output byte-identical. M5 Pro decode with the three\ngraph changes so far vs base: 2B 205.3 -> 219.9 tok/s (+7.1%), largest band 27.12 ->\n27.72 (+2.2%).\n\n* qwen35: one l2_norm over the adjacent q and k head groups\n\nq and k occupy adjacent, equally shaped head groups in the conv output, so a\nsingle l2_norm over the joint view replaces two launches per recurrent layer;\nq and k become views into its result. Encoded nodes per decoded token on the\nlargest band: 1862 -> 1814. Greedy output byte-identical.\n\n* metal: fuse the trailing silu into the ssm_conv kernels\n\nWhen the conv output is consumed only by a silu, the conv kernels apply it on\nstore (function constant) and write the silu's tensor directly, matching what\nthe CUDA backend already does. Encoded nodes per decoded token on the largest band:\n1814 -> 1766. test-backend-ops -o SSM_CONV 45/45; greedy output byte-identical.\nCumulative decode vs base on an M5 Pro: 2B 207.0 -> 223.0 tok/s (+7.7%),\nlargest band 27.15 -> 27.85 (+2.6%).\n\n* qwen35: scan the devices once per graph build for the GDN path choices\n\nThe fused-GDN state-rows check and the raw-gates check each looped over the\nmodel's devices on every recurrent layer, asking the backend for the device\ntype each time. Both flags are now computed once in the graph constructor.\n\n* tests: register the raw-gate gated_delta_net cases in the eval list\n\nThey had been appended to the perf-mode list, so test mode never ran them."},{"name":"ggml-zdnn","kind":"directory","size":0,"lastCommitHash":"153d324bcf86d220b235ca010eeb11213f32b5d1","lastCommitWhen":"2026-08-11T09:20:46+03:00","lastCommitMessage":"llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)\n\n* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used\n\n* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp\n\nCo-authored-by: Max Krasnyansky \n\n* set mmap_support to false on OpenCL backend\n\n* fix order of load modes\n\n* use -1 for auto\n\n* resolve load mode auto earlier to correctly pick gpu host or cpu memory\n\n* add load mode auto to llama-bench\n\n* bump virtgpu api version, regenerate docs\n\n---------\n\nCo-authored-by: Piotr Wilkin (ilintar) \nCo-authored-by: Max Krasnyansky \nCo-authored-by: Georgi Gerganov "},{"name":"ggml-zendnn","kind":"directory","size":0,"lastCommitHash":"bf0040e15fd5b716262658f4d652c9cee959cf91","lastCommitWhen":"2026-08-20T15:42:26+02:00","lastCommitMessage":"CI: Use LLVM's OpenMP over MSVC_DEBUG_non_redist on Windows (#26678)\n\n* CI: Use LLVM's OpenMP over MSFT_DEBUG_non_redist on Windows\n\nCurrently, we ship the non-redist debug version of microsoft's libomp.\nThis PR changes this to official LLVM's release, also packaging\nthe license as needed.\n\n* Remove LLVM SHA from job name to increase legibility\n\n* Add temp validations to CI\n\n* Revert \"Add temp validations to CI\"\n\nThis reverts commit eef97c88b5bac280803ebb3c3b7bb09f89b0fd88.\n\n* Build OpenMP in CI\n\n* Make OpenMP fetch self-contained in cmake and cache in CI\n\n* Robustify Licens-packaging\n\n1. Ship OpenMP license, not LLVM's.\n2. Invalidate cache also on checksum of the license\n\n* Remove stale reference in docs/build.md\n\n* No longer package base license in release\n\nThis was scope-creep\n\n* Add explanatory comment to OpenMP license\n\n* Remove arm64 smoke\n\nForgot this during conflict resolution during rebase of\nc54c0e9cf6030a5a54ce8bdd81b3e146d9787d42\n\n* Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC\n\n* whitespace changes"},{"name":"CMakeLists.txt","kind":"file","size":26946,"lastCommitHash":"adfffbe41b2cabcd51fff326ab045662265062bb","lastCommitWhen":"2026-09-25T11:42:36-07:00","lastCommitMessage":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds."},{"name":"ggml-alloc.c","kind":"file","size":47847,"lastCommitHash":"6036c635e2d1e9aeca11d01a84188c9db9ea1e2a","lastCommitWhen":"2026-08-24T10:43:04+03:00","lastCommitMessage":"ggml : fix ggml_clamp (#27644)\n\n* ggml : fix ggml_clamp\n\n* cont : update ggml-alloc"},{"name":"ggml-backend-dl.cpp","kind":"file","size":1051,"lastCommitHash":"ce38a4db478b90542874cd4af5cb48b3a0fcf311","lastCommitWhen":"2026-01-29T12:33:21-08:00","lastCommitMessage":"hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)\n\n* hexagon: updates to enable offloading to HTP on WoS\n\n* Update windows.md\n\n* Update windows.md\n\n* hexagon: enable -O3 optimizations\n\n* hexagon: move all _WINDOWS conditional compilation to _WIN32\n\n* hexagon: updates to enable offloading to HTP on WoS\n\n* hexagon: use run-time vs load-time dynamic linking for cdsp driver interface\n\n* refactor htp-drv\n\n* hexagon: add run-bench.ps1 script\n\n* hexagon: htdrv refactor\n\n* hexagon: unify Android and Windows build readmes\n\n* hexagon: update README.md\n\n* hexagon: refactor htpdrv\n\n* hexagon: drv refactor\n\n* hexagon: more drv refactor\n\n* hexagon: fixes for android builds\n\n* hexagon: factor out dl into ggml-backend-dl\n\n* hexagon: add run-tool.ps1 script\n\n* hexagon: merge htp-utils in htp-drv and remove unused code\n\n* wos: no need for getopt_custom.h\n\n* wos: add missing CR in htpdrv\n\n* hexagon: ndev enforecement applies only to the Android devices\n\n* hexagon: add support for generating and signing .cat file\n\n* hexagon: add .inf file\n\n* hexagon: working auto-signing and improved windows builds\n\n* hexagon: futher improve skel build\n\n* hexagon: add rough WoS guide\n\n* hexagon: updated windows guide\n\n* hexagon: improve cmake handling of certs and logging\n\n* hexagon: improve windows setup/build doc\n\n* hexagon: more windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* Update windows.md\n\n* Update windows.md\n\n* snapdragon: rename docs/backend/hexagon to docs/backends/snapdragon\n\nAlso added a power shell script to simplify build env setup.\n\n* hexagon: remove trailing whitespace and move cmake requirement to user-presets\n\n* hexagon: fix CMakeUserPresets path in workflow yaml\n\n* hexagon: introduce local version of libdl.h\n\n* hexagon: fix src1 reuse logic\n\ngpt-oss needs a bigger lookahead window.\nThe check for src[1] itself being quantized was wrong.\n\n---------\n\nCo-authored-by: Max Krasnyansky "},{"name":"ggml-backend-dl.h","kind":"file","size":791,"lastCommitHash":"ce38a4db478b90542874cd4af5cb48b3a0fcf311","lastCommitWhen":"2026-01-29T12:33:21-08:00","lastCommitMessage":"hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)\n\n* hexagon: updates to enable offloading to HTP on WoS\n\n* Update windows.md\n\n* Update windows.md\n\n* hexagon: enable -O3 optimizations\n\n* hexagon: move all _WINDOWS conditional compilation to _WIN32\n\n* hexagon: updates to enable offloading to HTP on WoS\n\n* hexagon: use run-time vs load-time dynamic linking for cdsp driver interface\n\n* refactor htp-drv\n\n* hexagon: add run-bench.ps1 script\n\n* hexagon: htdrv refactor\n\n* hexagon: unify Android and Windows build readmes\n\n* hexagon: update README.md\n\n* hexagon: refactor htpdrv\n\n* hexagon: drv refactor\n\n* hexagon: more drv refactor\n\n* hexagon: fixes for android builds\n\n* hexagon: factor out dl into ggml-backend-dl\n\n* hexagon: add run-tool.ps1 script\n\n* hexagon: merge htp-utils in htp-drv and remove unused code\n\n* wos: no need for getopt_custom.h\n\n* wos: add missing CR in htpdrv\n\n* hexagon: ndev enforecement applies only to the Android devices\n\n* hexagon: add support for generating and signing .cat file\n\n* hexagon: add .inf file\n\n* hexagon: working auto-signing and improved windows builds\n\n* hexagon: futher improve skel build\n\n* hexagon: add rough WoS guide\n\n* hexagon: updated windows guide\n\n* hexagon: improve cmake handling of certs and logging\n\n* hexagon: improve windows setup/build doc\n\n* hexagon: more windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* hexagon: windows readme updates\n\n* Update windows.md\n\n* Update windows.md\n\n* snapdragon: rename docs/backend/hexagon to docs/backends/snapdragon\n\nAlso added a power shell script to simplify build env setup.\n\n* hexagon: remove trailing whitespace and move cmake requirement to user-presets\n\n* hexagon: fix CMakeUserPresets path in workflow yaml\n\n* hexagon: introduce local version of libdl.h\n\n* hexagon: fix src1 reuse logic\n\ngpt-oss needs a bigger lookahead window.\nThe check for src[1] itself being quantized was wrong.\n\n---------\n\nCo-authored-by: Max Krasnyansky "},{"name":"ggml-backend-impl.h","kind":"file","size":13643,"lastCommitHash":"f20395dae59ba30ab0a10e0e0b0db6eeb8e8a282","lastCommitWhen":"2026-08-20T13:35:15+03:00","lastCommitMessage":"Revert \"tensor-split meta backend fixes (#26502)\" (#27433)\n\nThis reverts commit d59d455fd8ea09e5a2e87ce2a9d668267ffb5ccd."},{"name":"ggml-backend-meta.cpp","kind":"file","size":118081,"lastCommitHash":"bf0a29cc16066d313f6a8b52025620287a75cbc4","lastCommitWhen":"2026-08-24T09:20:25+03:00","lastCommitMessage":"Deepseek 4: `-sm tensor` (#26490)\n\n* DSV4: sm tensor\n\n* set coarser granularity for head splits\n\n* fix dspark\n\n* add model saving for dsv4 + allow dflash to return on specific device\n\n* add comment about dsv4 seq_rm\n\n* simplify\n\n* add shared expert delayed allreduce\n\n* remove special test for dsv4"},{"name":"ggml-backend-reg.cpp","kind":"file","size":18002,"lastCommitHash":"082b326fc76f6e9bbb835b3920a3022bfdb6691c","lastCommitWhen":"2026-07-10T12:38:34+08:00","lastCommitMessage":"ggml-et: Initial ET backend (#24179)\n\n* ggml-et: Add performance logging\n\n* ggml-et: Quants helpers\n\n* ggml-et: Add MUL_MAT kernel\n\n* ggml-et: Add ROPE kernel\n\n* ggml-et: Add RMS_NORM kernel\n\n* ggml-et: Add GLU kernel\n\n* ggml-et: Add SOFT_MAX kernel\n\n* ggml-et: Add GET_ROWS kernel\n\n* ggml-et: Add CONT kernel\n\n* ggml-et: Add SET_ROWS kernel\n\n* ggml-et: Add MUL_MAT_ID kernel\n\n* ggml-et: Build et kernels as part of ggml\n\n* ggml-et: Embed kernels with fs fallback\n\n* ggml-et: Build fixes\n\n* ggml-et: Add MUL_MAT F32xF32 op\n\n* ggml_et: Add MUL_MAT_ID op\n\n* ggml-et: Disable offloading for debug\n\n* ggml-et: Refactor out block ops\n\n* ggml-et: ggml backend API changes\n\n* ggml-et: Add RESHAPE/TRANSPOSE to supported\n\n* ggml-et: Add CONT_F16\n\n* ggml-et: Add supported ops doc\n\n* gglm-et: Initial doc\n\n* ggml-et: Remove runtime import hacks\n\nWe can now import the runtime by a simple find_package(), so we\ncan cleanup the CMakeLists.txt.\n\n* ggml-et: Fix GET_ROWS kernel\n\nFix lost batch dimension.\n\nAlso clean vibe-comments.\n\n* ggml-et: Fix SET_ROWS kernel\n\nRemove incorrect broadcasting guard.\n\n* ggml-et: Use custom instruction for fp32->fp16\n\n* ggml-et: Vectorize set_rows fp32->fp16\n\n* ggml-et: Fix ROPE kernel (yarn)\n\nggml-et: fix et_logf\n\nWIP: Fix ramp\n\nWIP: fix ROPE!\n\n* ggml-et: Better sinf\n\n* ggml-et: Fix SOFT_MAX\n\nAdd `max_bias` and `sink` support.\n\n* ggml-et: Fix CONT\n\nReorder from contiguous write to read with atomic stores.\n\n* ggml-et: Fix elmap kernel\n\nRemainder handlin\n\n* ggml-et: Fix MUL_MAT MUL_MAT_ID remainders\n\n* ggml-et: Fix ET-SOC reference\n\n* ggml-et: Fix embed kernels scripts for old python\n\nThis allows GGML-ET to build on pre-3.8 python.\n\n* Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (#6)\n\n* Example using ET-Soc-1 emulator configuration\n\nExample usage:\n```bash\ncmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON\ncmake --build build --config Release -j $(nproc)\n\ntime ./build/bin/test-backend-ops\n\n./build/bin/llama-server \\\n --model Qwen3-0.6B-Q8_0.gguf \\\n --alias Qwen3-0.6B-Q8_0 \\\n -fa 0 \\\n --ctx-size 1024 \\\n --no-warmup \\\n --host 127.0.0.1 \\\n --port 8080\n```\n\n* build: proper dep tracking for kernels\n\n* support host using MOLD linker\n\n* initial multi core GET_ROW F32 implementation\n\n* vectorized q8 dequant\n\n* wip: cland warning clenaups and initial logging refactor\n\n* wip: message default message cleanup\n\n* chore: message cleanups\n\n* cmake cleanup\n\n* migrate to use platform provided functions\n\n* cmake back into subdir\n\n* support et_print() in kernels\n\n* fix: repair kernel building\n\n* perf: operations run async by default\n\n* debug: proper kernel dep tracking and error detection on kenrel launch\n\n* fix: kernel binary dep tracking and fixing get_rows_f32 erroring\n\n* perf: back to doing async kernel runs by default\n\n* perf: vectorize and parallel device memset\n\n* merge matmul work\n\n* misc: align allocation and enable all offload\n\n* misc: delete deadcode and respect memory limits\n\n* fix: repair tensor debug print\n\n* fix: loosen RMS_NORM op percision\n\n* feat: Q4_0 GET_ROWS\n\n* perf: FP32 MUL_MAT using TensorFMA\n\n* update limitations\n\n* perf: redue L1 load in compute_block_dot_product_q8_0\n\n* feat: save kernel mapping (name to id) when profiling is enabled\n\n* chore: memops cleanup\n\n* perf: parallelize softmax by rows\n\n* perf: vectorize 2nd phase of softmax\n\n* perf: ban GET_ROWS from offloaded\n\n* perf: vectorize and non-atomic for eltwise ops and sub support\n\n* perf: vectorize normal rope\n\n* perf: glu runs in parallel\n\n* merge: manually merge saqib's work on kernel fixes\n\n* perf: more vectorized RoPE\n\n* perf: parallelize mul_mat_id\n\n* perf: parallelize set_rows_f32\n\n* perf: vectorize softmax\n\n* feat: support kernel fusion and fuse RMS_NORM + MUL\n\n* fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE\n\n* fix: bump max rope dims for gemma\n\n* feat: GeGLU and SCALE support to fully offload Gemma\n\n* perf: faster device memset\n\n* feat: get_rows supporting Q4_K and avoid cont cache coherent issues\n\n* better F32 MM\n\n* feat: NORM for ET backend\n\n* feat: SQR for ET backend\n\n* feat: UNARY on ET\n\n* feat: el_map support broadcasting for ET\n\n* feat: SUM_ROWS in ET backend\n\n* feat: more ops in ET backend\n\n* feat: WKV* operators in ET backend\n\n* perf: parallelize operators across cacheline instead of row\n\n* perf: parallelize get_rows on cacheline\n\n* wip: baseline FlashAttention for ET backend\n\n* wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on\n\n* feat: f16 x f16 -> f32 MM using matrix engine\n\n* wip: f16 FlashAttention using matrix engine\n\n* wip: clean up\n\n* feat: barriers\n\n* perf: optimize FA_F16 in ET\n\n* perf: vectorize pack_k_for_transpose16\n\n* perf: prefetch next loop matrix tile\n\n* perf: FlashAttention 2nd MM uses TensorFMA and optimizations\n\n* cleanup: flashattention reorg\n\n* perf: optimizations and fixes\n\n* feat: L2SCP API and make FlashAttention support DV = 256 for gemma\n\n* perf: parallelize norms beyond single row\n\n* feat: GATED_DELTA_NET support and relaxed L2_NORM requirment\n\n* feat: loosen RMS_NORM, NORM, ROPE contingous req too\n\n* feat: repeat supports brocasting on dim 0 and loosen cont check\n\n* feat: FILL and DIAG operator\n\n* feat: loosen UNARY support chcek\n\n* feat: TRI support\n\n* feat: SOLVE_TRI support\n\n* feat: basic SET support\n\n* feat: loosen CONT req\n\n* perf: fp16_to_fp32 use ASM\n\n* feat: IMROPE support\n\n* feat: PAD support\n\n* feat: global barrier\n\n* fix: view must live on the same backend as backing tensor\n\n* feat: relax CONCAT in ET backend\n\n* feat: dead simple CUMSUM implementation\n\n* feat: basic SSM_CONV support\n\n* feat: loosen CONCAT req\n\n* feat: relax GATED_DELTA_NET and add SET support proper\n\n* cleanup: cleanup LCM math\n\n* feat: SWIGLU single input\n\n* feat: SSM_SCAN support\n\n* feat: el_map supports non aligned tensors in best effort\n\n* feat: basic GROUP_NORM support\n\n* feat: loosen MUL_MAT capablities slightly\n\n* feat: loosen MUL_MAT and GET_ROWS and add IM2COL\n\n* feat: special case for softmax 1x1x1x1\n\n* feat: loosen SOFT_MAX req in ET backend\n\n* fix: el_map unaligned acse fixes\n\n* perf: optimize zero_acc_vec in flash_attn_ext_f16_me\n\n* perf: use hart 1 for packing in MM and FA for FP16\n\n* feat: kernel semaphore\n\n* perf: better instruction sequence in FlashAttention\n\n* fix: gated_delta_net with proper masking\n\n* perf: better parallelization for GATED_DELTA_NET\n\n* perf: parallelize SSM_CONV over nr\n\n* perf: vectorize SSM_CONV\n\n* perf: optimize MUL_MAT for q8\n\n* feat: support Gemma 4\n\n* fix: support multi-device\n\n* feat: broader GLU support\n\n* feat: unary ops supports view\n\n* fix: repair fp16 MM using matrix engine\n\n* perf: handle large N GEMV better\n\n* perf: better q8_0 MM\n\n* perf: better set_rows\n\n* add back deleted files\n\n* fix: repair after merge\n\n* feat: POC version of uberkernel\n\n* feat: RMS_NORM in uberkernel\n\n* feat: add more kernels into usage\n\n* chore: clean up uberkernel compilation\n\n* perf: faster flash attention\n\n* perf: opt flash attention for large seq length\n\n* feat: loosen op bounds. clamp and mean support\n\n* perf: vectorize ssm_scan\n\n* perf: slightly faster FA\n\n* perf: FlashAttention parallel MM and load\n\n* perf: fuse Q8 MM and ADD\n\n* feat: basic conv kernel for ET\n\n* softMAx_test\n\n* set_rows_f32\n\n* get_rows and cont\n\n* testing\n\n* set_rows_exp\n\n* Junk addition\n\n* Narrowing the issue\n\n* Update flash_attn_ext_f16_me.c\n\nFocusing FA_ext_f16_me\n\n* test\n\n* Eviction updated\n\n* Detailed cache eviction debug\n\n* mulmat\n\n* removeal of `BUILD_FOR_UBERKERNEL` flag\n\n* cleaning...\n\n* fix: balance FCC0 count\n\n* feat: implement mul_mat and mul_mat_id for Q4_0 type\n\n* optimize uberkernel plan upload\n\n* add mul_mat q4 into uberkernel\n\n* enable gating flush to just uberkernel\n\n* update docs for ET\n\n* update op support for ET\n\n* et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations\n\n* et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0\n\n* et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs\n\n* test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage\n\n* et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32\n\n* et-backend: vectorize Q4_0 matrix-engine dequantization\n\n* et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail\n\n* et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat\n\n* et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency\n\n* Update uberkernel.c\n\n* Update unary_f32.c\n\n* gemma 4\n\n* bisect gemma4: enable scale_f32 only\n\n* bisect gemma4: +rms_norm_f32\n\n* bisect gemma4: +rms_norm_mul_f32\n\n* bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT\n\n* bisect gemma4: +rope_f32 (skip rms_norm_mul)\n\n* bisect gemma4: +el_map_f32\n\n* bisect gemma4: +softmax_f32\n\n* bisect gemma4: +get_rows_f32\n\n* bisect gemma4: +glu_f32\n\n* bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine\n\n* bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine\n\n* bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0\n\n* bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me\n\n* bisect gemma4: +mul_mat_id_f32\n\n* bisect gemma4: +sum_rows_f32\n\n* bisect gemma4: +cont_f16\n\n* bisect gemma4: +fill_f32\n\n* bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul)\n\n* Update rms_norm_mul_f32.c\n\n* bisect2 gemma4 n64: +scale_f32 only\n\n* bisect2 gemma4 n64: +rms_norm_f32 +rope_f32\n\n* bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix)\n\n* bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul)\n\n* bisect2 gemma4 n64: all ops enabled except rms_norm_mul\n\n* bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn)\n\n* bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine\n\n* bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine\n\n* bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0\n\n* bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0)\n\n* bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks)\n\n* bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0)\n\n* run-3: matmul + rms_norm_mul\n\n* run-4\n\n* Revert \"run-4\"\n\n* run5\n\n* changes after cleanup\n\n* cleanup before upstream\n\n* restrict changes into ET backend\n\n* move kernel embedding from Python to CMake\n\n* move uberkernel gen into CMake\n\n* apply clang format\n\n* update CMake style\n\n* update to match C and C++ style\n\n* use source ggml and quant headers instead of ET's\n\n* MROPE support\n\n* absorb view ops into same branch as none\n\n* fix bad rebase\n\n* add marty1885 to codeowners\n\n* oops\n\n* remove redundant newline\n\n* fix CI editor warnings\n\n---------\n\nCo-authored-by: Vidas \nCo-authored-by: Gianluca Guida \nCo-authored-by: Gianluca Guida \nCo-authored-by: ubergarm \nCo-authored-by: SaqibAkram-10xE \nCo-authored-by: Rehan Qasim "},{"name":"ggml-backend.cpp","kind":"file","size":96594,"lastCommitHash":"f20395dae59ba30ab0a10e0e0b0db6eeb8e8a282","lastCommitWhen":"2026-08-20T13:35:15+03:00","lastCommitMessage":"Revert \"tensor-split meta backend fixes (#26502)\" (#27433)\n\nThis reverts commit d59d455fd8ea09e5a2e87ce2a9d668267ffb5ccd."},{"name":"ggml-common.h","kind":"file","size":136535,"lastCommitHash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","lastCommitWhen":"2026-09-04T13:12:38-07:00","lastCommitMessage":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex"},{"name":"ggml-feats.h","kind":"file","size":4885,"lastCommitHash":"1f368f354d9edcfea9fd6a1e0989b3e7335a050f","lastCommitWhen":"2026-08-13T07:57:24+03:00","lastCommitMessage":"ggml : fix arm builds, unused var (#26991)"},{"name":"ggml-impl.h","kind":"file","size":25604,"lastCommitHash":"3f7c29d318e317b63f54c558bc69803963d7d88c","lastCommitWhen":"2026-04-16T17:21:28+08:00","lastCommitMessage":"ggml: add graph_reused (#21764)\n\n* ggml: add graph_reused\n\n* use versioning instead of reuse flag\n\n* increment version with atomic\n\n* use top bits for split numbering\n\n* add assert\n\n* move counter to ggml.c\n\n* set uid in split_graph only\n\n* fix windows\n\n* address further review comments\n\n* get next_uid rather than doing bit manipulation\n\n* rename + add comment about uid"},{"name":"ggml-opt.cpp","kind":"file","size":42081,"lastCommitHash":"2dcb7f74edde532831c5ee9a9b6c403f2b68854f","lastCommitWhen":"2026-04-08T17:40:15+02:00","lastCommitMessage":"fix: free ctx_copy in ggml_opt_free to plug per-training-session leak (#21592)\n\n* fix: free ctx_copy in ggml_opt_free to plug per-training-session leak\n\nggml_opt_alloc populates opt_ctx->ctx_copy via a free+init pair every\ntime the allocated graph shape changes. The last ctx_copy from the\nfinal ggml_opt_alloc call survives until ggml_opt_free is invoked,\nbut ggml_opt_free was only freeing ctx_static and ctx_cpu, never\nctx_copy. Each opt_ctx lifetime therefore leaks the final per-batch\ncontext — ~900 KB for a typical GNN training session in\nsindarin-pkg-tensor, surfaced via AddressSanitizer.\n\nctx_copy is nullptr-initialized and ggml_free() handles NULL safely,\nso the new release is guard-free.\n\n* Update ggml/src/ggml-opt.cpp\n\nCo-authored-by: Johannes Gäßler \n\n---------\n\nCo-authored-by: realorko \nCo-authored-by: Johannes Gäßler "},{"name":"ggml-quants.c","kind":"file","size":234514,"lastCommitHash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","lastCommitWhen":"2026-09-04T13:12:38-07:00","lastCommitMessage":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex"},{"name":"ggml-quants.h","kind":"file","size":10544,"lastCommitHash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","lastCommitWhen":"2026-09-04T13:12:38-07:00","lastCommitMessage":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex"},{"name":"ggml-threading.cpp","kind":"file","size":250,"lastCommitHash":"ae8de6d50a09d49545e0afab2e50cc4acfb280e2","lastCommitWhen":"2024-11-14T18:04:35+01:00","lastCommitMessage":"ggml : build backends as libraries (#10256)\n\n* ggml : build backends as libraries\r\n\r\n---------\r\n\r\nSigned-off-by: Xiaodong Ye \r\nCo-authored-by: Georgi Gerganov \r\nCo-authored-by: R0CKSTAR "},{"name":"ggml-threading.h","kind":"file","size":198,"lastCommitHash":"cb13ef85a444eb52a3f1b82dce198ceb25606583","lastCommitWhen":"2024-12-12T19:02:49+01:00","lastCommitMessage":"remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS (#10797)\n\nother windows build fixes"},{"name":"ggml.c","kind":"file","size":261742,"lastCommitHash":"00e8187618590ca52e6737f97d38173087458ebf","lastCommitWhen":"2026-09-08T15:47:22-07:00","lastCommitMessage":"graph: six fusions for the hybrid-attention decode graph (-25% ops/token; Metal 2B +7.7%, 27B +2.6% decode; CUDA 9B +2.5%) (#165)\n\n* metal: fuse the Hadamard sign flip into the FWHT kernel\n\nEvery folded weight's activation went through three encoded ops: an elementwise\nMUL by the +-1 sign vector, a RESHAPE and the FWHT-hint MUL_MAT. The sign factors\nare exact, so the transform now applies them while loading its row and the MUL\nis elided, matching what the CUDA backend already does. The encode list drops\nempty ops, so the MUL / RESHAPE / MUL_MAT pattern is matched on the graph's own\nindices and checked with ggml_can_fuse_subgraph_ext; the fused kernel writes the\nmatmul's output, so its ranges are checked for concurrency explicitly.\n\nMeasured on an M5 Pro, largest band PQ2_0 folded (block 1024), r=3 x2 interleaved:\nencoded nodes per decoded token 2359 -> 2103, tg128 27.05 -> 27.45 (+1.5%),\npp512 390.5 -> 397.2 (+1.7%). Greedy output byte-identical over 3 prompts;\ntest-backend-ops MUL_MAT_HADAMARD 24/24.\n\n* ggml, metal: fold the GDN gate activations into the fused op (raw gates)\n\nThe hybrid-attention layers spent four elementwise launches per layer per\ntoken on the gate scalars: sigmoid(beta), and softplus(alpha + dt_bias) * A\nfor the decay. ggml_gated_delta_net_set_raw_gates() marks a gated_delta_net\nop as taking beta and alpha pre-activation, with dt_bias and A as two extra\nsources, and the kernel applies the same formulas as the unary kernels while\nreading the scalars it already read. The qwen35 builder hands the fused op the\nraw projections (GGML_GDN_RAW_GATES_DISABLE=1 restores the separate ops); the\nnon-fused chunked path keeps the activated tensors. CPU reference and Metal\nkernel implemented here; CUDA follows in the next commit.\n\ntest-backend-ops -o GATED_DELTA_NET on MTL0: 39/39 incl. 4 raw-gate cases\n(decode, 2 seqs, prefill 64, rows mode). Encoded nodes per decoded token on the\nlargest band folded PQ2_0: 2103 -> 1910. Greedy output byte-identical to base.\nM5 Pro, r=3 x2 interleaved, both fusions vs base: 2B tg128 206.4 -> 214.8\n(+4.1%), pp512 +1.0%; largest band tg128 27.08 -> 27.55 (+1.7%), pp512 +1.3%.\n\n* cuda: raw gates for the fused gated_delta_net kernel\n\nSame folding as the Metal and CPU implementations: with the op's raw-gate flag\nset, beta goes through sigmoid and the decay is raw_a[h] * softplus(g +\nraw_dt_bias[h]) inside the kernel, using the formulas of the unary kernels.\nThe template gains a RAW parameter (scalar gate only; KDA asserts it off) and\nthe two tables are passed as extra pointers.\n\nCompile-checked only by inspection here; needs a CUDA build plus\ntest-backend-ops -o GATED_DELTA_NET before this branch is pushed.\n\n* ggml: decline raw-gate gated_delta_net on backends without it; gate the builder on device backend\n\nSYCL, OpenCL, WebGPU and Hexagon compute gated_delta_net but do not implement the\nraw-gate flag, so their supports_op now declines such ops (the CPU fallback is\ncorrect). The qwen35 builder only emits raw gates when every device is CPU,\nMetal, CUDA, ROCm or MUSA, so those backends keep their native op instead of a\nCPU fallback for the whole recurrence.\n\n* qwen35: gated norm as one swiglu_split op\n\nsilu(z) * rms_norm(o) was a unary and a mul; ggml_swiglu_split computes the same\nexpression in one kernel. Encoded nodes per decoded token on the largest folded band\nPQ2_0: 1910 -> 1862. Greedy output byte-identical. M5 Pro decode with the three\ngraph changes so far vs base: 2B 205.3 -> 219.9 tok/s (+7.1%), largest band 27.12 ->\n27.72 (+2.2%).\n\n* qwen35: one l2_norm over the adjacent q and k head groups\n\nq and k occupy adjacent, equally shaped head groups in the conv output, so a\nsingle l2_norm over the joint view replaces two launches per recurrent layer;\nq and k become views into its result. Encoded nodes per decoded token on the\nlargest band: 1862 -> 1814. Greedy output byte-identical.\n\n* metal: fuse the trailing silu into the ssm_conv kernels\n\nWhen the conv output is consumed only by a silu, the conv kernels apply it on\nstore (function constant) and write the silu's tensor directly, matching what\nthe CUDA backend already does. Encoded nodes per decoded token on the largest band:\n1814 -> 1766. test-backend-ops -o SSM_CONV 45/45; greedy output byte-identical.\nCumulative decode vs base on an M5 Pro: 2B 207.0 -> 223.0 tok/s (+7.7%),\nlargest band 27.15 -> 27.85 (+2.6%).\n\n* qwen35: scan the devices once per graph build for the GDN path choices\n\nThe fused-GDN state-rows check and the raw-gates check each looped over the\nmodel's devices on every recurrent layer, asking the backend for the device\ntype each time. Both flags are now computed once in the graph constructor.\n\n* tests: register the raw-gate gated_delta_net cases in the eval list\n\nThey had been appended to the perf-mode list, so test mode never ran them."},{"name":"ggml.cpp","kind":"file","size":738,"lastCommitHash":"fedf034a98378059274e4243d2a370a243335e73","lastCommitWhen":"2025-06-01T13:43:57+03:00","lastCommitMessage":"ggml : Print backtrace on uncaught C++ exceptions (ggml/1232)\n\nThe goal is to have what users call \"full logs\" contain the backtrace.\n\nThis is registered upon ggml_init. Also fixes a minor fd leak on Linux.\n"},{"name":"gguf.cpp","kind":"file","size":59262,"lastCommitHash":"f2cf28855cd0a7f7d8518261877eb730760b74c9","lastCommitWhen":"2026-08-25T08:26:32Z","lastCommitMessage":"gguf: detect legacy Prism Q2_0 (group 128) files and point to PQ2_0 / group-64 downloads\n"}]