{"entries":[{"name":"CMakeLists.txt","mode":"0100644","size":37329,"lastCommit":{"hash":"adfffbe41b2cabcd51fff326ab045662265062bb","oid":"adfffbe41b2cabcd51fff326ab045662265062bb","parents":["985880c0b744e017fe62b61b396e7674812de323"],"author":{"name":"Scott Guyton","email":"kiljoy001@gmail.com","when":"2026-09-25T11:42:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:42:36-07:00"},"message":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds.","tree":"f701dae3247ff1ba55db24d1fd83dff0cd2edff8"}},{"name":"amx","mode":"0040000","size":0,"lastCommit":{"hash":"37a77fb0579be9d71e2c73da0553cfd42b7b103a","oid":"37a77fb0579be9d71e2c73da0553cfd42b7b103a","parents":["f4043fec0103872bf4339f6fa18d8b17824d5b6a"],"author":{"name":"Adrien Gallouët","email":"angt@huggingface.co","when":"2026-06-20T12:43:06+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-20T13:43:06+03:00"},"message":"ggml : optimize AMX (#24806)\n\nFlatten the partition over n_batch * M so every thread participates in\nthe quantization\n\n | CPU | Model | Test | t/s OLD | t/s NEW | Speedup |\n |:--------------------------------|:------------------------------|:-------|----------:|----------:|----------:|\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw | pp512 | 730.71 | 779.86 | 1.07 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw | tg128 | 87.88 | 86.79 | 0.99 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | pp512 | 725.09 | 1023.31 | 1.41 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | tg128 | 83.64 | 83.62 | 1.00 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0 | pp512 | 820.51 | 924.05 | 1.13 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0 | tg128 | 90.59 | 92.46 | 1.02 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1 | pp512 | 776.88 | 872.79 | 1.12 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1 | tg128 | 89.39 | 90.94 | 1.02 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M | pp512 | 719.28 | 1009.27 | 1.40 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M | tg128 | 80.62 | 80.86 | 1.00 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S | pp512 | 732.29 | 1077.29 | 1.47 |\n | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S | tg128 | 86.42 | 83.53 | 0.97 |\n\nSigned-off-by: Adrien Gallouët ","tree":"915af777ae13c0ad82e6c8ce5087f0f4af28f8ff"}},{"name":"arch-fallback.h","mode":"0100644","size":26740,"lastCommit":{"hash":"adfffbe41b2cabcd51fff326ab045662265062bb","oid":"adfffbe41b2cabcd51fff326ab045662265062bb","parents":["985880c0b744e017fe62b61b396e7674812de323"],"author":{"name":"Scott Guyton","email":"kiljoy001@gmail.com","when":"2026-09-25T11:42:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:42:36-07:00"},"message":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds.","tree":"f701dae3247ff1ba55db24d1fd83dff0cd2edff8"}},{"name":"arch","mode":"0040000","size":0,"lastCommit":{"hash":"adfffbe41b2cabcd51fff326ab045662265062bb","oid":"adfffbe41b2cabcd51fff326ab045662265062bb","parents":["985880c0b744e017fe62b61b396e7674812de323"],"author":{"name":"Scott Guyton","email":"kiljoy001@gmail.com","when":"2026-09-25T11:42:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:42:36-07:00"},"message":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds.","tree":"f701dae3247ff1ba55db24d1fd83dff0cd2edff8"}},{"name":"binary-ops.cpp","mode":"0100644","size":6709,"lastCommit":{"hash":"89181c0b6d6f57fdf82c33bf93ede10433088573","oid":"89181c0b6d6f57fdf82c33bf93ede10433088573","parents":["ceaa89b7867021c84f8806a6d8d2613213c90737"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2026-02-11T07:52:00+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-02-11T07:52:00+02:00"},"message":"ggml : extend bin bcast for permuted src1 (#19484)\n\n* tests : extend bin bcast for permuted src1\n\n* cont : extend bin support\n\n* cont : s0 is always 1\n\n* tests : simplify","tree":"453aa93af28dc42e8b523c33e2e2191742b4919b"}},{"name":"binary-ops.h","mode":"0100644","size":518,"lastCommit":{"hash":"a62d7fa7a979d21bcd760f1d7c19a66e55611f44","oid":"a62d7fa7a979d21bcd760f1d7c19a66e55611f44","parents":["e408d4351a4ad143656672479d8ae1479306ce31"],"author":{"name":"cmdr2","email":"secondary.cmdr2@gmail.com","when":"2025-03-29T11:37:13+05:30"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2025-03-30T08:33:31+03:00"},"message":"cpu: de-duplicate some of the operators and refactor (ggml/1144)\n\n* cpu: de-duplicate some of the operators and refactor\n\n* Fix PR comments\n\n* Fix PR comments\n","tree":"bb5517f81193a3541d5377e743193d64404f79c7"}},{"name":"cmake","mode":"0040000","size":0,"lastCommit":{"hash":"81b0d882ae7fcd3118920ff26c5213abecbc8c08","oid":"81b0d882ae7fcd3118920ff26c5213abecbc8c08","parents":["0f45f1a35cddd6e8fec2aa216cc11b2345205edd"],"author":{"name":"alex-spacemit","email":"jinghui.huang@spacemit.com","when":"2026-05-14T17:39:30+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-14T17:39:30+08:00"},"message":"ggml-cpu: Add IME2 Instruction Support for the SpacemiT Backend (#22863)","tree":"e0f7162c44bfc5d9e4ea4ba6fceac1e8addf581c"}},{"name":"common.h","mode":"0100644","size":2333,"lastCommit":{"hash":"684b36101c9eeb7e89c9e602f9ded05f1353a0c6","oid":"684b36101c9eeb7e89c9e602f9ded05f1353a0c6","parents":["3a00c98584e42a20675b6569d81beadb282b0952"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-02-15T11:09:24+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-02-15T11:09:24+05:30"},"message":"ggml-cpu: FA add GEMM microkernel (#19422)\n\n* ggml-cpu: FA add GEMM microkernel\n\n* add guard for sizeless vector types\n\n* fix case where DV % GGML_F32_EPR !=0\n\n* move memset out of the loop\n\n* move another memset out of the loop\n\n* use RM=4 for arm\n\n* simd_gemm: convert everything to int\n\n* convert everything to size_t to avoid warnings\n\n* fixup\n\n* add pragma for ignoring aggressive loop optimizations","tree":"0894b9e30fde28e53dfed9b8fc0d9ef35879e6f1"}},{"name":"ggml-cpu-impl.h","mode":"0100644","size":13684,"lastCommit":{"hash":"adfffbe41b2cabcd51fff326ab045662265062bb","oid":"adfffbe41b2cabcd51fff326ab045662265062bb","parents":["985880c0b744e017fe62b61b396e7674812de323"],"author":{"name":"Scott Guyton","email":"kiljoy001@gmail.com","when":"2026-09-25T11:42:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:42:36-07:00"},"message":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds.","tree":"f701dae3247ff1ba55db24d1fd83dff0cd2edff8"}},{"name":"ggml-cpu.c","mode":"0100644","size":132665,"lastCommit":{"hash":"ee8ad0ef6b03b34b8709ea9440a4b3927e8a6524","oid":"ee8ad0ef6b03b34b8709ea9440a4b3927e8a6524","parents":["0324c66521960d67aa7da8687fb1453a79a6565c"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-23T18:23:34-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-23T18:23:34-07:00"},"message":"ggml-cpu: fix unused type warnings in the mul_mat chunk helpers (#260)\n\nThe switch to ggml_cpu_vec_dot left type unused in both helpers, which breaks builds with LLAMA_FATAL_WARNINGS.","tree":"bece1d41c92f52995760fac2ad0df029b88c090d"}},{"name":"ggml-cpu.cpp","mode":"0100644","size":24798,"lastCommit":{"hash":"e0c828b90b499f3eb789435e558d8f9dd7c8bf75","oid":"e0c828b90b499f3eb789435e558d8f9dd7c8bf75","parents":["b8e0bf0b635f20d418a2d62b1abaf73ec131ce78"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-08-26T18:46:30-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-26T18:46:30-07:00"},"message":"llama/ggml: Hadamard weight-fold runtime, ported to prism-v7 (#121)\n\n* ggml: support F16 input for FWHT matmul hint\n\n(cherry picked from commit 9146baa453d955f66347b25ed70c1ffb1f6ed494)\n\n* ggml-cuda: advertise F16 Hadamard inputs\n\n(cherry picked from commit 14d8bb56adb2f7e93f2067a12d9ddce1ddfad21b)\n\n* ggml-cuda: extend FWHT matmul hint to 1024/2048 blocks\n\nRegister-resident warp kernel handles N up to 2048 for F32/F16 inputs.\ntest-backend-ops MUL_MAT_HADAMARD passes 14/14 on H200 (SM90) vs CPU\nreference, including H2048 F32/F16.\n\n(cherry picked from commit db4c6ee86a71900dda624d9fcad04e1727126ca7)\n\n* llama: prism.hadamard GGUF contract + activation-side rotation\n\nLoad prism.hadamard.* v1 metadata (block size, transform, axis, sign mode,\nfolded weight names), materialize one persistent rotation tensor per\n(block size, buffer type), and apply x' = Hx immediately before every\nfolded mul_mat / mul_mat_id via the FWHT hint. Identity signs only;\nunsupported contract values fail loading loudly.\n\n(cherry picked from commit 2f2ddf0f29e5c6e6b827d79db7f68920319c60f9)\n\n* convert: emit prism.hadamard metadata from packing manifest\n\nPick up hadamard_packing.json from the model dir, validate the fold\ncontract (schema v1, identity signs, last-axis), map folded tensor names\nto GGUF names, and write the prism.hadamard.* keys.\n\n(cherry picked from commit ddc5ed9afda4abffb711d06d3c8437c9279ac538)\n\n* ggml-metal: fix crash on F16 Hadamard input at unsupported FWHT widths\n\nMetal advertises every mul_mat type combo, but the Hadamard-hint fallback\nfor F16 inputs at widths without a dedicated FWHT kernel requests a\nnonexistent f32 x f16 mul_mv pipeline and crashes on the null pipeline.\nDecline the op in supports_op so it falls back to CPU instead.\n\ntest-backend-ops MUL_MAT_HADAMARD: MTL0 13/13 (F16 wide-block case now\ncorrectly unsupported), 3/3 backends pass.\n\n(cherry picked from commit fe88d0c716f0880081625c75a233206d05ac7e87)\n\n* llama: set up Hadamard rotations after tensor data load\n\nWith mmap-backed buffer types (Metal shared buffers), weight->buffer is\nnot assigned until load_all_data, so the rotation setup threw\n'weight has no buffer' on every folded model. CUDA allocates buffers\nup front, which masked the ordering bug.\n\n(cherry picked from commit f37f0474c904e2af1e6187c4c75fa0a3a77b9b16)\n\n* ggml-metal: add 1024/2048-wide FWHT kernels\n\nInstantiate the existing simdgroup-register FWHT template for 1024/2048\n(F32 and F16 inputs) and extend the supported-size gate, replacing the\ndense rotation-matmul fallback for wide Hadamard-hint blocks.\n\ntest-backend-ops MUL_MAT_HADAMARD: MTL0 14/14. Removes most of the\nHadamard-hint decode overhead on Apple silicon versus the fallback.\n\n(cherry picked from commit 610ddad4bf42f9d16c2521762cee6654fcedd6de)\n\n* llama, convert: restrict prism.hadamard contract to verified matmul paths\n\nThe activation-side transform is applied only where the graph goes through\nbuild_lora_mm/build_lora_mm_id, but the converter accepted any mappable\ntensor and the loader accepted any weight name. Architectures that multiply\nprojection weights with raw ggml_mul_mat (deepseek2 wq_a, kimi-linear,\nplm) would load a folded GGUF and silently skip the transform.\n\nGate the contract on both ends: the converter refuses archs outside a\nverified set (llama, qwen3, qwen3moe, qwen35, qwen35moe, qwen3next) and\ntensors outside an explicit projection-kind allowlist, and the loader\nenforces the same checks on prism.hadamard.weight_names so folded GGUFs\nfrom other tools refuse to load rather than run wrong math.\n\n(cherry picked from commit e76b3041636aee8f7c6956750b86ca6dcd9b8385)\n\n* ggml-vulkan: create F16 FWHT pipelines only when the device supports fp16\n\nThe fwht_f16 shaders declare float16_t, but VK_KHR_shader_float16_int8 is\nonly enabled when device->fp16 is set, so unconditional creation could\nfail pipeline setup on devices without shader-float16 (or under\nGGML_VK_DISABLE_F16) even when no F16 FWHT is ever dispatched. Guard both\nthe subgroup and shmem variants; ggml_vk_can_use_fwht already treats the\nnull pipeline as unsupported and falls back to the rotation-matrix matmul.\n\n(cherry picked from commit 4edee06d05e12b0a356374663444a1ffefb21f5d)\n\n* llama: explicit sign vectors for the Hadamard GGUF contract\n\nsign_mode 'explicit' carries per-width sign vectors in GGUF metadata\n(prism.hadamard.sign_widths + flattened sign_values). The loader\nmaterializes one F32 sign tensor per (width, buffer type) and the graph\napplies x' = H (s * x): elementwise sign flip, then the blockwise FWHT\nhint matmul. Identity mode is unchanged; unknown modes still refuse to\nload. Converter accepts schema v2 manifests with an explicit signs table.\n\n(cherry picked from commit c49590b0c0251ab8f6dc27ad71eb45d4a26d1cb7)\n\n* llama: inverse Hadamard transform for latent embedding tables\n\nTensors listed in prism.hadamard.inverse_weight_names store latent\n(rotated) rows; the graph restores the primal basis right after the\ntoken-embedding lookup: h = s * (H z). This lets the embedding table\nstay in the quantized latent format instead of a dequantized primal\ncopy. The rotation and sign tensors are shared with the matmul-side\ntransforms; the direct-embeddings input path is untouched.\n\n(cherry picked from commit 2e85ea475407322c25e45ae33aa36df69f16d572)\n\n* llama: grouped-order GDN out_proj support for Hadamard-folded models\n\nThe converter's tiled V-head reorder permutes out_proj's input axis,\nwhich is the rotation axis of a folded latent; a post-fold column\npermutation breaks the blockwise Hadamard correspondence and cannot be\nrefolded without destroying the ternary codes. For folded out_proj\ntensors the converter now keeps the training (grouped) V order and sets\nprism.hadamard.gdn_v_grouped; the runtime permutes the activation\ntiled->grouped (reshape+permute+cont) before the sign flip and rotation.\n\n(cherry picked from commit e71a350b219eeababfe3fe235cb112cbe2a38d7c)\n\n* convert: match Hadamard-folded out_proj by suffix\n\nWrapper prefixes are stripped from tensor names before modify_tensors,\nso the exact-name check against the manifest never fired and the tiled\nV reorder still permuted folded rotation axes.\n\n(cherry picked from commit e9e4ea30aee217c8713706c21063de1664bf6a57)\n\n* llama: verify folded weights receive their transform at graph build\n\nAn architecture whose matmul path bypasses the transform helpers would\nload a Hadamard-folded GGUF cleanly and silently compute wrong results.\nAfter reserving the worst-case graph, walk it once: every mul_mat\nconsuming a folded weight must take the hint matmul's output (through\nreshape/view) built from that weight's rotation, and every get_rows of\na latent table must feed an inverse transform. Violations abort context\ncreation with the offending tensor name.\n\n(cherry picked from commit 34c9b5cd9b1578abf42868b23543ba4bde0758b6)\n\n* llama: run the Hadamard coverage check before graph scheduling\n\nThe scheduler splits cross-backend paths with copy tensors (a CPU-mapped\nlatent embedding feeding a GPU FWHT), which breaks the producer chain\nthe check follows and produced a false positive. Verify the pristine\ngraph right after build_graph, once per context, and log the consumers\nof an unverified lookup before aborting.\n\n(cherry picked from commit 8362c899672279557d786644f3267c32d51caad4)\n\n* cuda: fuse the Hadamard sign flip into the FWHT load\n\nThe sign multiply was a separate full pass over the activation (4.3% of\ntraced GPU time, ~1250 launches per run). Detect the mul+reshape+FWHT-hint\nsubgraph in the CUDA graph evaluator and multiply the sign vector during\nthe transform kernel's load instead. Backend-internal: the graph and every\nfallback path are unchanged. test-backend-ops gains fused-pattern cases\nat widths 5120/6144/17408.\n\n(cherry picked from commit deb4dfd92a22faf588a89606bf5d53c0a3dd932b)\n\n* llama, convert: allowlist gaps + device-side inverse transforms\n\nTwo fixes surfaced by a real folded checkpoint against the new\nverified-path allowlist:\n\n- The allowlist rejected two kinds that are on the build_lora_mm path and\n covered by the graph-walk check: output.weight (the lm head) and\n blk.N.attn_gate (the linear-attention z projection). A checkpoint\n folding either failed to load or convert.\n- Inverse (lookup-side) transforms inherited the buffer type of a\n host-mapped embedding table, bouncing the per-token inverse across the\n PCIe boundary: measured 3x decode loss on an RTX 4090 (27.7 -> 80.9\n tok/s once device-side) while NVLink-class parts hid it. The inverse\n rotation and signs now use the buffer type the forward rotations chose.\n\n(cherry picked from commit 52ad701b544f7bbf4a9b0b8cc43f8b353981e740)\n\n* hadamard: fix dead F16 fusion gate, contract validation, and grouped-order scoping\n\nFive review findings, all real.\n\nThe signed F16 fusion was unreachable. ggml_mul builds its result with\nggml_dup_tensor, so mul->type follows x->type, and the gate required\nmul->type == F32 while allowing x to be F16. The condition collapsed to F32\nonly, so the templated signed half kernel never ran and the F16 signed test\ncovered the unfused path. The gate now requires mul->type == x->type with\neither supported input type.\n\nAn explicit sign contract with no widths passed validation, left the sign table\nempty, and was then read as identity mode, which silently changes the model\nfunction. Explicit mode now requires at least one width.\n\nThe converter accepted zero-width and non-block-aligned sign vectors that the\nruntime rejects, so a manifest could convert into a GGUF this same branch\ncannot load. It now applies the runtime's width rule.\n\nGrouped V order was selected by tensor-kind suffix, so one folded out_proj in\nthe manifest put every layer's out_proj in grouped order. A partially folded\ncheckpoint then computed the non-folded layers with mismatched ordering. The\nmatch is now per-tensor, comparing full names modulo wrapper prefixes.\n\nThe prepacked NVFP4 repack path bypassed that guard entirely: it column-permuted\nfolded out-projections on the rotation axis, which cannot be refolded, and never\nset the grouped flag. It now takes the same branch as modify_tensors.\n\ntest-backend-ops -o MUL_MAT_HADAMARD: 17/17 on Metal, 3/3 backends. The CUDA\ngate change is not covered by that run; it needs an NVIDIA host.","tree":"161a517cdf9795a9c3965abaa64158e3da464771"}},{"name":"hbm.cpp","mode":"0100644","size":2003,"lastCommit":{"hash":"f470bc36bed4d836b9de5a483fa0dfaee176d6f5","oid":"f470bc36bed4d836b9de5a483fa0dfaee176d6f5","parents":["8f47e25f56e9792093b7497c68e9f80bab82ed19"],"author":{"name":"xctan","email":"axunlei@gmail.com","when":"2025-06-09T22:47:13+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-09T16:47:13+02:00"},"message":"ggml-cpu : split arch-specific implementations (#13892)\n\n* move ggml-cpu-aarch64 to repack\n\n* split quantize_row_q8_0/1\n\n* split helper functions\n\n* split ggml_vec_dot_q4_0_q8_0\n\n* split ggml_vec_dot_q4_1_q8_1\n\n* split ggml_vec_dot_q5_0_q8_0\n\n* split ggml_vec_dot_q5_1_q8_1\n\n* split ggml_vec_dot_q8_0_q8_0\n\n* split ggml_vec_dot_tq1_0_q8_K\n\n* split ggml_vec_dot_tq2_0_q8_K\n\n* split ggml_vec_dot_q2_K_q8_K\n\n* split ggml_vec_dot_q3_K_q8_K\n\n* split ggml_vec_dot_q4_K_q8_K\n\n* split ggml_vec_dot_q5_K_q8_K\n\n* split ggml_vec_dot_q6_K_q8_K\n\n* split ggml_vec_dot_iq2_xxs_q8_K\n\n* split ggml_vec_dot_iq2_xs_q8_K\n\n* split ggml_vec_dot_iq2_s_q8_K\n\n* split ggml_vec_dot_iq3_xxs_q8_K\n\n* split ggml_vec_dot_iq3_s_q8_K\n\n* split ggml_vec_dot_iq1_s_q8_K\n\n* split ggml_vec_dot_iq1_m_q8_K\n\n* split ggml_vec_dot_iq4_nl_q8_0\n\n* split ggml_vec_dot_iq4_xs_q8_K\n\n* fix typos\n\n* fix missing prototypes\n\n* rename ggml-cpu-quants.c\n\n* rename ggml-cpu-traits\n\n* rename arm folder\n\n* move cpu-feats-x86.cpp\n\n* rename ggml-cpu-hbm\n\n* update arm detection macro in quants.c\n\n* move iq quant tables\n\n* split ggml_quantize_mat_q8_0/K\n\n* split ggml_gemv_*\n\n* split ggml_gemm_*\n\n* rename namespace aarch64 to repack\n\n* use weak aliases to replace test macros\n\n* rename GGML_CPU_AARCH64 to GGML_CPU_REPACK\n\n* rename more aarch64 to repack\n\n* clean up rebase leftover\n\n* fix compilation errors\n\n* remove trailing spaces\n\n* try to fix clang compilation errors\n\n* try to fix clang compilation errors again\n\n* try to fix clang compilation errors, 3rd attempt\n\n* try to fix clang compilation errors, 4th attempt\n\n* try to fix clang compilation errors, 5th attempt\n\n* try to fix clang compilation errors, 6th attempt\n\n* try to fix clang compilation errors, 7th attempt\n\n* try to fix clang compilation errors, 8th attempt\n\n* try to fix clang compilation errors, 9th attempt\n\n* more cleanup\n\n* fix compilation errors\n\n* fix apple targets\n\n* fix a typo in arm version of ggml_vec_dot_q4_K_q8_K\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov ","tree":"3a82619a28973086e6d1067a760b0ed915515136"}},{"name":"hbm.h","mode":"0100644","size":155,"lastCommit":{"hash":"f470bc36bed4d836b9de5a483fa0dfaee176d6f5","oid":"f470bc36bed4d836b9de5a483fa0dfaee176d6f5","parents":["8f47e25f56e9792093b7497c68e9f80bab82ed19"],"author":{"name":"xctan","email":"axunlei@gmail.com","when":"2025-06-09T22:47:13+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-09T16:47:13+02:00"},"message":"ggml-cpu : split arch-specific implementations (#13892)\n\n* move ggml-cpu-aarch64 to repack\n\n* split quantize_row_q8_0/1\n\n* split helper functions\n\n* split ggml_vec_dot_q4_0_q8_0\n\n* split ggml_vec_dot_q4_1_q8_1\n\n* split ggml_vec_dot_q5_0_q8_0\n\n* split ggml_vec_dot_q5_1_q8_1\n\n* split ggml_vec_dot_q8_0_q8_0\n\n* split ggml_vec_dot_tq1_0_q8_K\n\n* split ggml_vec_dot_tq2_0_q8_K\n\n* split ggml_vec_dot_q2_K_q8_K\n\n* split ggml_vec_dot_q3_K_q8_K\n\n* split ggml_vec_dot_q4_K_q8_K\n\n* split ggml_vec_dot_q5_K_q8_K\n\n* split ggml_vec_dot_q6_K_q8_K\n\n* split ggml_vec_dot_iq2_xxs_q8_K\n\n* split ggml_vec_dot_iq2_xs_q8_K\n\n* split ggml_vec_dot_iq2_s_q8_K\n\n* split ggml_vec_dot_iq3_xxs_q8_K\n\n* split ggml_vec_dot_iq3_s_q8_K\n\n* split ggml_vec_dot_iq1_s_q8_K\n\n* split ggml_vec_dot_iq1_m_q8_K\n\n* split ggml_vec_dot_iq4_nl_q8_0\n\n* split ggml_vec_dot_iq4_xs_q8_K\n\n* fix typos\n\n* fix missing prototypes\n\n* rename ggml-cpu-quants.c\n\n* rename ggml-cpu-traits\n\n* rename arm folder\n\n* move cpu-feats-x86.cpp\n\n* rename ggml-cpu-hbm\n\n* update arm detection macro in quants.c\n\n* move iq quant tables\n\n* split ggml_quantize_mat_q8_0/K\n\n* split ggml_gemv_*\n\n* split ggml_gemm_*\n\n* rename namespace aarch64 to repack\n\n* use weak aliases to replace test macros\n\n* rename GGML_CPU_AARCH64 to GGML_CPU_REPACK\n\n* rename more aarch64 to repack\n\n* clean up rebase leftover\n\n* fix compilation errors\n\n* remove trailing spaces\n\n* try to fix clang compilation errors\n\n* try to fix clang compilation errors again\n\n* try to fix clang compilation errors, 3rd attempt\n\n* try to fix clang compilation errors, 4th attempt\n\n* try to fix clang compilation errors, 5th attempt\n\n* try to fix clang compilation errors, 6th attempt\n\n* try to fix clang compilation errors, 7th attempt\n\n* try to fix clang compilation errors, 8th attempt\n\n* try to fix clang compilation errors, 9th attempt\n\n* more cleanup\n\n* fix compilation errors\n\n* fix apple targets\n\n* fix a typo in arm version of ggml_vec_dot_q4_K_q8_K\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov ","tree":"3a82619a28973086e6d1067a760b0ed915515136"}},{"name":"kleidiai","mode":"0040000","size":0,"lastCommit":{"hash":"62b22690602665a0f34ce5f722915839f4e4913d","oid":"62b22690602665a0f34ce5f722915839f4e4913d","parents":["ff14356e0caf6988f61f1f15f9dfe7d5ab398271"],"author":{"name":"Charles Xu","email":"charles.xu@arm.com","when":"2026-08-21T10:33:30+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-21T11:33:30+03:00"},"message":"kleidiai : add SME2 F32 GEMV kernel support (#26891)","tree":"ba1c72669e3478939c2fd869d72a7cde306aed08"}},{"name":"llamafile","mode":"0040000","size":0,"lastCommit":{"hash":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","oid":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","parents":["1a07bfa5f4144274c8f1c9963821dd9d9a51854b"],"author":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"committer":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"message":"ggml-cpu: AVX2/AVX-VNNI kernels for PQ2_0 + Q8_K activations + tiled GEMM (x86 without AVX-512)\n\nThe fork's fast PQ2_0 paths (gemv/gemm_pq2_0_4x8) are gated on AVX-512, so 12th-14th gen Intel and other AVX2/AVX-VNNI CPUs fell back to\nthe generic scalar dot. This adds:\n\n- ggml_vec_dot_pq2_0_q8_0 / ggml_vec_dot_pq2_0_q8_K (AVX2, AVX-VNNI, AVX-512VNNI-VL): 2-bit codes unpacked with srlv+and, dpbusd\n against the permuted activation bytes, unsigned-code correction via a lane-wise sum(y) so per-lane partials stay exact.\n- PQ2_0 vec_dot_type switched to Q8_K (one activation scale per 256): 4 dpbusd + 1 sub + 1 cvt + 1 fmadd per 128 weights instead of\n 4x(2 dpbusd + sub + cvt + fmadd). Perplexity unchanged within 0.004 (Bonsai-2 27B: 7.0542 -> 7.0583 +/- 0.58).\n- llamafile/sgemm.cpp: tinyBLAS_PQ2K_AVX / tinyBLAS_PQ2_AVX tiled GEMMs. A 128-weight block is unpacked once per row tile and reused\n across RN token columns; the activation permute and lane sums are staged once per matmul call per thread; jobs are row tiles\n (each weight row streams from DRAM exactly once) claimed dynamically from the threadpool counter in runs of 8 so P- and E-cores\n share the compute-bound loop. Float accumulation order and reduction are identical to the vec_dot path, so batched and\n single-token results are bit-identical (PPL identical with PQ2_SGEMM=0/1).\n\nMeasured on an i7-13620H (6P+4E, AVX2+AVX-VNNI, single-channel DDR5-5200 ~28 GB/s effective), Ternary-Bonsai-2-27B PQ2_0:\n decode (N=1) 1.3 -> 3.9-4.2 tok/s (generic -> vec_dot)\n batch cost, ms/token N=4: 123 -> 78, N=8: 99 -> 68, N=16: 89 -> 54 (t=12, 65 W), N=64: 89 -> 69\n prompt processing 6.7 -> 16 tok/s\n MTP speculative verify (N=3) becomes ~1 DRAM sweep, giving 6.8-8.3 tok/s greedy decode with the grafted nextn head.\nEnv: PQ2_SGEMM=0 disables the tiled path (A/B).\n\nCo-Authored-By: Claude Fable 5.1 \nClaude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R\n","tree":"723f6ad65b3e660085318bf516cd002fcd5c86ba"}},{"name":"ops.cpp","mode":"0100644","size":422237,"lastCommit":{"hash":"78e16408f29465f1058f6cdf7d66a24ab8be65da","oid":"78e16408f29465f1058f6cdf7d66a24ab8be65da","parents":["bda59eaa03ae75c99f2f9bd7b968f957f8ead29a"],"author":{"name":"lenny76","email":"lenny76@lenny76.com","when":"2026-09-25T20:12:35+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:12:35-07:00"},"message":"ggml-cpu : AVX-512 tiled kernel for single-token flash attention (#254)\n\nThe split-KV decode path called one_chunk once per q head, so each\nf16 KV row was loaded and converted once per q head sharing it. The\nnew kernel handles all q heads of a KV head together, 16 KV rows at a\ntime: K/V converted to f32 once per tile, scores and online softmax\non the whole tile with a vector exp, and each head's VKQ accumulator\nupdated 16 floats at a time in a register. It writes the same\n[M, S, VKQ] partials, so the reduction is unchanged. Used only for\nF16 K/V, F32 Q, no softcap, D % 16 == 0, D <= 512, G <= 16.\n\nCo-authored-by: Claude Opus 5.5 ","tree":"4d4f127f1336d4c3bb8c536f4612df1e7b45791e"}},{"name":"ops.h","mode":"0100644","size":10103,"lastCommit":{"hash":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","oid":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","parents":["b2dd28a3b601417b8ea345c4db72df3048e1114f"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-07-17T00:33:33+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-17T00:33:33+08:00"},"message":"DeepseekV4: Add fused hyper-connection ops (#25585)\n\n* dsv4 hc-ops\n\n* add missing files;\n\n* add cparams\n\n* update rpc version\n\n* address review comments\n\n* address review comments","tree":"ca61063d563441412e69a532868a6918b1721dec"}},{"name":"quants.c","mode":"0100644","size":50586,"lastCommit":{"hash":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","oid":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","parents":["1a07bfa5f4144274c8f1c9963821dd9d9a51854b"],"author":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"committer":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"message":"ggml-cpu: AVX2/AVX-VNNI kernels for PQ2_0 + Q8_K activations + tiled GEMM (x86 without AVX-512)\n\nThe fork's fast PQ2_0 paths (gemv/gemm_pq2_0_4x8) are gated on AVX-512, so 12th-14th gen Intel and other AVX2/AVX-VNNI CPUs fell back to\nthe generic scalar dot. This adds:\n\n- ggml_vec_dot_pq2_0_q8_0 / ggml_vec_dot_pq2_0_q8_K (AVX2, AVX-VNNI, AVX-512VNNI-VL): 2-bit codes unpacked with srlv+and, dpbusd\n against the permuted activation bytes, unsigned-code correction via a lane-wise sum(y) so per-lane partials stay exact.\n- PQ2_0 vec_dot_type switched to Q8_K (one activation scale per 256): 4 dpbusd + 1 sub + 1 cvt + 1 fmadd per 128 weights instead of\n 4x(2 dpbusd + sub + cvt + fmadd). Perplexity unchanged within 0.004 (Bonsai-2 27B: 7.0542 -> 7.0583 +/- 0.58).\n- llamafile/sgemm.cpp: tinyBLAS_PQ2K_AVX / tinyBLAS_PQ2_AVX tiled GEMMs. A 128-weight block is unpacked once per row tile and reused\n across RN token columns; the activation permute and lane sums are staged once per matmul call per thread; jobs are row tiles\n (each weight row streams from DRAM exactly once) claimed dynamically from the threadpool counter in runs of 8 so P- and E-cores\n share the compute-bound loop. Float accumulation order and reduction are identical to the vec_dot path, so batched and\n single-token results are bit-identical (PPL identical with PQ2_SGEMM=0/1).\n\nMeasured on an i7-13620H (6P+4E, AVX2+AVX-VNNI, single-channel DDR5-5200 ~28 GB/s effective), Ternary-Bonsai-2-27B PQ2_0:\n decode (N=1) 1.3 -> 3.9-4.2 tok/s (generic -> vec_dot)\n batch cost, ms/token N=4: 123 -> 78, N=8: 99 -> 68, N=16: 89 -> 54 (t=12, 65 W), N=64: 89 -> 69\n prompt processing 6.7 -> 16 tok/s\n MTP speculative verify (N=3) becomes ~1 DRAM sweep, giving 6.8-8.3 tok/s greedy decode with the grafted nextn head.\nEnv: PQ2_SGEMM=0 disables the tiled path (A/B).\n\nCo-Authored-By: Claude Fable 5.1 \nClaude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R\n","tree":"723f6ad65b3e660085318bf516cd002fcd5c86ba"}},{"name":"quants.h","mode":"0100644","size":12067,"lastCommit":{"hash":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","oid":"164c33700e48de0189a83f2c7b8fed549c8d5dfe","parents":["1a07bfa5f4144274c8f1c9963821dd9d9a51854b"],"author":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"committer":{"name":"abdulaziz","email":"alanziabdulaziz4@gmail.com","when":"2026-09-19T10:28:24+03:00"},"message":"ggml-cpu: AVX2/AVX-VNNI kernels for PQ2_0 + Q8_K activations + tiled GEMM (x86 without AVX-512)\n\nThe fork's fast PQ2_0 paths (gemv/gemm_pq2_0_4x8) are gated on AVX-512, so 12th-14th gen Intel and other AVX2/AVX-VNNI CPUs fell back to\nthe generic scalar dot. This adds:\n\n- ggml_vec_dot_pq2_0_q8_0 / ggml_vec_dot_pq2_0_q8_K (AVX2, AVX-VNNI, AVX-512VNNI-VL): 2-bit codes unpacked with srlv+and, dpbusd\n against the permuted activation bytes, unsigned-code correction via a lane-wise sum(y) so per-lane partials stay exact.\n- PQ2_0 vec_dot_type switched to Q8_K (one activation scale per 256): 4 dpbusd + 1 sub + 1 cvt + 1 fmadd per 128 weights instead of\n 4x(2 dpbusd + sub + cvt + fmadd). Perplexity unchanged within 0.004 (Bonsai-2 27B: 7.0542 -> 7.0583 +/- 0.58).\n- llamafile/sgemm.cpp: tinyBLAS_PQ2K_AVX / tinyBLAS_PQ2_AVX tiled GEMMs. A 128-weight block is unpacked once per row tile and reused\n across RN token columns; the activation permute and lane sums are staged once per matmul call per thread; jobs are row tiles\n (each weight row streams from DRAM exactly once) claimed dynamically from the threadpool counter in runs of 8 so P- and E-cores\n share the compute-bound loop. Float accumulation order and reduction are identical to the vec_dot path, so batched and\n single-token results are bit-identical (PPL identical with PQ2_SGEMM=0/1).\n\nMeasured on an i7-13620H (6P+4E, AVX2+AVX-VNNI, single-channel DDR5-5200 ~28 GB/s effective), Ternary-Bonsai-2-27B PQ2_0:\n decode (N=1) 1.3 -> 3.9-4.2 tok/s (generic -> vec_dot)\n batch cost, ms/token N=4: 123 -> 78, N=8: 99 -> 68, N=16: 89 -> 54 (t=12, 65 W), N=64: 89 -> 69\n prompt processing 6.7 -> 16 tok/s\n MTP speculative verify (N=3) becomes ~1 DRAM sweep, giving 6.8-8.3 tok/s greedy decode with the grafted nextn head.\nEnv: PQ2_SGEMM=0 disables the tiled path (A/B).\n\nCo-Authored-By: Claude Fable 5.1 \nClaude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R\n","tree":"723f6ad65b3e660085318bf516cd002fcd5c86ba"}},{"name":"repack.cpp","mode":"0100644","size":226410,"lastCommit":{"hash":"ea50aba8cd7ea261168346172e38f3d562b3d317","oid":"ea50aba8cd7ea261168346172e38f3d562b3d317","parents":["a419d438a1e1932d37b8bf4d0b86312455aa9b3e"],"author":{"name":"Strike3D","email":"a.spaziani@gmail.com","when":"2026-09-22T06:19:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-21T21:19:57-07:00"},"message":"# Pull request — testo pronto da incollare su GitHub (#233)\n\n## Titolo\n\n ggml-cpu: AVX2 / AVX-VNNI GEMV+GEMM kernels for Q1_0 (4x8 repack) + shared-scale Q8 activations\n\n## Messaggio di commit (prima riga = titolo, poi corpo)\n\n ggml-cpu: AVX2/AVX-VNNI kernels for Q1_0 4x8 repack path\n\n On x86 the fast Q1_0 (4x8 repack) path was only enabled with AVX-512 VNNI.\n CPUs without AVX-512 (Intel Core Ultra / Alder..Arrow Lake, AMD Zen 2/3,\n older Intel) fell back to the per-row vec_dot, which is compute-bound\n (~11 ops per 32 weights) and leaves most of the memory bandwidth unused.\n\n This adds:\n - AVX2 GEMV and GEMM kernels for block_q1_0x4 (8-byte interleave), using\n AVX-VNNI vpdpbusd when available and pmaddubsw otherwise. Sign bits are\n kept in place as byte factors 2^q and multiplied directly; lanes are\n arranged so each dword sums equal factors, and the 2^q normalization is\n a single arithmetic shift per 128-weight block instead of per 32.\n - Q8_0 activation quantizers with one scale per 128 values (aligned to\n QK1_0): ggml_quantize_row_q8_0_g128 / ggml_quantize_mat_q8_0_4x8_g128.\n Layout is unchanged (block_q8_0 / block_q8_0x4), so the existing\n AVX-512 and NEON kernels keep working; the new kernels accumulate a\n whole block in int32 and convert to float once. Precision matches Q8_K.\n - Repack selection: Q1_0 now uses the 4x8 path on any AVX2 CPU.\n - GGML_Q1_0_DISABLE_AVX512: test-only switch to exercise the AVX2 path on\n AVX-512 machines.\n\n Measured (see PR description): kernel microbench 1.75-2.6x GEMV,\n 2.0-2.9x GEMM; end-to-end prompt processing 1.4-1.9x, generation\n 1.0-1.27x (memory-bandwidth bound after the change) on Intel Core\n Ultra 7 255H and AMD Ryzen 9 3900X, Clang and MSVC.\n\n## Descrizione della PR\n\n### Problem\n\nThe Q1_0 (Bonsai) repack path (`q1_0_4x8_q8_0`) is selected on x86 only when\n`ggml_cpu_has_avx512() && ggml_cpu_has_avx512_vnni()`. Every other x86 CPU\n— including all recent Intel consumer parts (Alder Lake → Arrow Lake / Core\nUltra, which have no AVX-512) and AMD Zen 2/3 — goes through\n`ggml_vec_dot_q1_0_q8_0`. That kernel is compute-bound (~11 instructions per\n32 weights: bit expansion into byte masks, sign xor, sum, int→float, FMA),\nso on these CPUs Bonsai-27B runs at ~5.5 tok/s while using roughly a quarter of\nthe available memory bandwidth.\n\n### Change\n\n1. **AVX2 GEMV/GEMM for `block_q1_0x4`** (`ggml/src/ggml-cpu/arch/x86/repack.cpp`).\n AVX2 has no mask registers, so expanding 32 sign bits into 32 byte masks\n costs 4 ops. Instead, each bit is kept where it is as the byte factor\n `2^q` and multiplied directly with `vpdpbusd` (AVX-VNNI / AVX512-VL VNNI)\n or `pmaddubsw`+`pmaddwd` (plain AVX2). Lanes are arranged as\n `L = 4*q + c` so that the four bytes summed by one dword all carry the\n same factor; dividing by `2^q` is one `vpsravd` per 128-weight block.\n With `P` = sum of activations over positive weights and `S` = total,\n `dot = 2P - S`. Activations are permuted once per call to match.\n GEMM shares the weight expansion across 4 activation rows.\n\n2. **Shared-scale Q8 activations** (`repack.cpp`, `repack.h`):\n `ggml_quantize_row_q8_0_g128` and `ggml_quantize_mat_q8_0_4x8_g128`\n quantize with one scale per 128 values (= `QK1_0`). The block layout is\n the plain `block_q8_0` / `block_q8_0x4`, so every existing kernel still\n works unchanged; the new kernels rely on the four scales of a group being\n equal and convert int→float once per block instead of once per 32. This\n is the precision of Q8_K (256-group scales), used throughout llama.cpp.\n Dispatched via `ggml_repack_quantize_row/mat`, specialized\n for `block_q1_0` only.\n\n3. **Selection**: `Q1_0` → `q1_0_4x8_q8_0` on any CPU with AVX2.\n\n4. `GGML_Q1_0_DISABLE_AVX512`: compile-time switch (off by default) to force\n the AVX2 path on AVX-512 machines, for testing/benchmarking only.\n\n### Correctness\n\nStandalone harness (`q1test.cpp`, not included in the PR; happy to add it as a\n`tests/` target if wanted) compares the new kernels against the generic\n`*_generic` kernels and against a double-precision reference computed from\nthe quantized integers, for n ∈ {128 … 16640}, 1–16 rows, normal and\nheavy-tailed scales, both the VNNI and the pmaddubsw variants.\nMax relative error: new kernels 2.1e-5, upstream generic kernels 3.4e-4.\nGreedy (temperature 0) outputs of Bonsai-27B are unchanged in practice.\n\n### Performance\n\nKernel microbenchmark, single thread (Xeon, Skylake-SP; ratios vs the current\n`vec_dot` path, n=5120, nc=5120):\n\n| | pmaddubsw (AVX2) | vpdpbusd (VNNI) |\n|---|---|---|\n| GEMV | 1.75x | 2.6x |\n| GEMM (16 rows) | 2.0x | 2.9x |\n\nEnd-to-end, `llama-bench -p 512 -n 128` on Bonsai-27B-Q1_0 (tok/s). Every row\nis the same build with and without this patch.\n\n**Intel Core Ultra 7 255H** (Arrow Lake-H, 6P+8E+2LPE, no AVX-512, AVX-VNNI;\n32 GB DDR5-5600 single channel), Clang 19, `-march=native`:\n\n| threads | pp512 before | pp512 after | tg128 before | tg128 after |\n|---|---|---|---|---|\n| 6 | 8.22 | 14.28 (1.74x) | 5.34 | 6.47 (1.21x) |\n| 8 | 8.85 | 16.07 (1.82x) | 5.50 | 6.81 (1.24x) |\n| 16 | 10.34 | 18.53 (1.79x) | 5.19 | 6.38 (1.23x) |\n\n**AMD Ryzen 9 3900X** (Zen 2, 12C/24T, AVX2 without VNNI; 64 GB DDR4-3200\ndual channel), three build recipes:\n\n| build | threads | pp512 before → after | tg128 before → after |\n|---|---|---|---|\n| Clang 19, `-march=native`, no OpenMP | 6 | 5.43 → 10.29 (1.90x) | 4.37 → 5.55 (1.27x) |\n| | 12 | 8.62 → 15.70 (1.82x) | 6.00 → 6.72 (1.12x) |\n| MSVC 19.44 + OpenMP, `GGML_CPU_ALL_VARIANTS` | 6 | 6.74 → 9.83 (1.46x) | 5.17 → 5.65 (1.09x) |\n| | 12 | 10.54 → 14.33 (1.36x) | 6.06 → 6.48 (1.07x) |\n| **CI recipe**: Clang 19 + libomp, `GGML_CPU_ALL_VARIANTS` (haswell variant) | 6 | 5.91 → 10.48 (1.77x) | 4.66 → 5.52 (1.18x) |\n| | 12 | 9.99 → 15.83 (1.58x) | 6.50 → 6.63 (1.02x) |\n\nObservations:\n- Prompt processing gains 1.4–1.9x everywhere.\n- Generation gains 1.1–1.27x and then flattens: after the patch, tg is\n memory-bandwidth bound (thread scaling saturates at 3–4 threads on the\n laptop, and at 12 threads on the 3900X the gain vanishes). This is the\n expected regime for a 3.5 GiB weight stream; the kernel is no longer the\n bottleneck.\n- MSVC produces noticeably slower code for these kernels than Clang (1.4x vs\n 1.9x on pp). Not a regression, but there is room to help MSVC's codegen.\n\n### Notes for reviewers\n\n- The g128 activation quantizer also feeds the existing AVX-512 and NEON\n Q1_0 kernels, which stay correct but see slightly coarser activation\n scales (one per 128 instead of per 32). If you prefer to keep those paths\n numerically untouched, the g128 quantizer can be limited to the AVX2 path\n by giving it a dedicated traits type; I chose the minimal change.\n- MSVC does not define `__F16C__`; the fp16 column-scale conversion falls\n back to scalar there (4 conversions per 128-weight block, negligible).\n- GEMM allocates a per-call scratch buffer (`4*n` bytes + scales) with\n `malloc`; GEMV uses a 16 KiB stack buffer with heap fallback.\n\nTested on Windows 11 with Clang 19.1.5 (`-march=native` and the CI\n`GGML_CPU_ALL_VARIANTS` recipe) and MSVC 19.44, and on Linux with GCC 13\n(AVX2 and AVX512-VL+VNNI builds).","tree":"fbf09b30c6171195f3b402c1071a837fcb9c2fa0"}},{"name":"repack.h","mode":"0100644","size":22370,"lastCommit":{"hash":"ea50aba8cd7ea261168346172e38f3d562b3d317","oid":"ea50aba8cd7ea261168346172e38f3d562b3d317","parents":["a419d438a1e1932d37b8bf4d0b86312455aa9b3e"],"author":{"name":"Strike3D","email":"a.spaziani@gmail.com","when":"2026-09-22T06:19:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-21T21:19:57-07:00"},"message":"# Pull request — testo pronto da incollare su GitHub (#233)\n\n## Titolo\n\n ggml-cpu: AVX2 / AVX-VNNI GEMV+GEMM kernels for Q1_0 (4x8 repack) + shared-scale Q8 activations\n\n## Messaggio di commit (prima riga = titolo, poi corpo)\n\n ggml-cpu: AVX2/AVX-VNNI kernels for Q1_0 4x8 repack path\n\n On x86 the fast Q1_0 (4x8 repack) path was only enabled with AVX-512 VNNI.\n CPUs without AVX-512 (Intel Core Ultra / Alder..Arrow Lake, AMD Zen 2/3,\n older Intel) fell back to the per-row vec_dot, which is compute-bound\n (~11 ops per 32 weights) and leaves most of the memory bandwidth unused.\n\n This adds:\n - AVX2 GEMV and GEMM kernels for block_q1_0x4 (8-byte interleave), using\n AVX-VNNI vpdpbusd when available and pmaddubsw otherwise. Sign bits are\n kept in place as byte factors 2^q and multiplied directly; lanes are\n arranged so each dword sums equal factors, and the 2^q normalization is\n a single arithmetic shift per 128-weight block instead of per 32.\n - Q8_0 activation quantizers with one scale per 128 values (aligned to\n QK1_0): ggml_quantize_row_q8_0_g128 / ggml_quantize_mat_q8_0_4x8_g128.\n Layout is unchanged (block_q8_0 / block_q8_0x4), so the existing\n AVX-512 and NEON kernels keep working; the new kernels accumulate a\n whole block in int32 and convert to float once. Precision matches Q8_K.\n - Repack selection: Q1_0 now uses the 4x8 path on any AVX2 CPU.\n - GGML_Q1_0_DISABLE_AVX512: test-only switch to exercise the AVX2 path on\n AVX-512 machines.\n\n Measured (see PR description): kernel microbench 1.75-2.6x GEMV,\n 2.0-2.9x GEMM; end-to-end prompt processing 1.4-1.9x, generation\n 1.0-1.27x (memory-bandwidth bound after the change) on Intel Core\n Ultra 7 255H and AMD Ryzen 9 3900X, Clang and MSVC.\n\n## Descrizione della PR\n\n### Problem\n\nThe Q1_0 (Bonsai) repack path (`q1_0_4x8_q8_0`) is selected on x86 only when\n`ggml_cpu_has_avx512() && ggml_cpu_has_avx512_vnni()`. Every other x86 CPU\n— including all recent Intel consumer parts (Alder Lake → Arrow Lake / Core\nUltra, which have no AVX-512) and AMD Zen 2/3 — goes through\n`ggml_vec_dot_q1_0_q8_0`. That kernel is compute-bound (~11 instructions per\n32 weights: bit expansion into byte masks, sign xor, sum, int→float, FMA),\nso on these CPUs Bonsai-27B runs at ~5.5 tok/s while using roughly a quarter of\nthe available memory bandwidth.\n\n### Change\n\n1. **AVX2 GEMV/GEMM for `block_q1_0x4`** (`ggml/src/ggml-cpu/arch/x86/repack.cpp`).\n AVX2 has no mask registers, so expanding 32 sign bits into 32 byte masks\n costs 4 ops. Instead, each bit is kept where it is as the byte factor\n `2^q` and multiplied directly with `vpdpbusd` (AVX-VNNI / AVX512-VL VNNI)\n or `pmaddubsw`+`pmaddwd` (plain AVX2). Lanes are arranged as\n `L = 4*q + c` so that the four bytes summed by one dword all carry the\n same factor; dividing by `2^q` is one `vpsravd` per 128-weight block.\n With `P` = sum of activations over positive weights and `S` = total,\n `dot = 2P - S`. Activations are permuted once per call to match.\n GEMM shares the weight expansion across 4 activation rows.\n\n2. **Shared-scale Q8 activations** (`repack.cpp`, `repack.h`):\n `ggml_quantize_row_q8_0_g128` and `ggml_quantize_mat_q8_0_4x8_g128`\n quantize with one scale per 128 values (= `QK1_0`). The block layout is\n the plain `block_q8_0` / `block_q8_0x4`, so every existing kernel still\n works unchanged; the new kernels rely on the four scales of a group being\n equal and convert int→float once per block instead of once per 32. This\n is the precision of Q8_K (256-group scales), used throughout llama.cpp.\n Dispatched via `ggml_repack_quantize_row/mat`, specialized\n for `block_q1_0` only.\n\n3. **Selection**: `Q1_0` → `q1_0_4x8_q8_0` on any CPU with AVX2.\n\n4. `GGML_Q1_0_DISABLE_AVX512`: compile-time switch (off by default) to force\n the AVX2 path on AVX-512 machines, for testing/benchmarking only.\n\n### Correctness\n\nStandalone harness (`q1test.cpp`, not included in the PR; happy to add it as a\n`tests/` target if wanted) compares the new kernels against the generic\n`*_generic` kernels and against a double-precision reference computed from\nthe quantized integers, for n ∈ {128 … 16640}, 1–16 rows, normal and\nheavy-tailed scales, both the VNNI and the pmaddubsw variants.\nMax relative error: new kernels 2.1e-5, upstream generic kernels 3.4e-4.\nGreedy (temperature 0) outputs of Bonsai-27B are unchanged in practice.\n\n### Performance\n\nKernel microbenchmark, single thread (Xeon, Skylake-SP; ratios vs the current\n`vec_dot` path, n=5120, nc=5120):\n\n| | pmaddubsw (AVX2) | vpdpbusd (VNNI) |\n|---|---|---|\n| GEMV | 1.75x | 2.6x |\n| GEMM (16 rows) | 2.0x | 2.9x |\n\nEnd-to-end, `llama-bench -p 512 -n 128` on Bonsai-27B-Q1_0 (tok/s). Every row\nis the same build with and without this patch.\n\n**Intel Core Ultra 7 255H** (Arrow Lake-H, 6P+8E+2LPE, no AVX-512, AVX-VNNI;\n32 GB DDR5-5600 single channel), Clang 19, `-march=native`:\n\n| threads | pp512 before | pp512 after | tg128 before | tg128 after |\n|---|---|---|---|---|\n| 6 | 8.22 | 14.28 (1.74x) | 5.34 | 6.47 (1.21x) |\n| 8 | 8.85 | 16.07 (1.82x) | 5.50 | 6.81 (1.24x) |\n| 16 | 10.34 | 18.53 (1.79x) | 5.19 | 6.38 (1.23x) |\n\n**AMD Ryzen 9 3900X** (Zen 2, 12C/24T, AVX2 without VNNI; 64 GB DDR4-3200\ndual channel), three build recipes:\n\n| build | threads | pp512 before → after | tg128 before → after |\n|---|---|---|---|\n| Clang 19, `-march=native`, no OpenMP | 6 | 5.43 → 10.29 (1.90x) | 4.37 → 5.55 (1.27x) |\n| | 12 | 8.62 → 15.70 (1.82x) | 6.00 → 6.72 (1.12x) |\n| MSVC 19.44 + OpenMP, `GGML_CPU_ALL_VARIANTS` | 6 | 6.74 → 9.83 (1.46x) | 5.17 → 5.65 (1.09x) |\n| | 12 | 10.54 → 14.33 (1.36x) | 6.06 → 6.48 (1.07x) |\n| **CI recipe**: Clang 19 + libomp, `GGML_CPU_ALL_VARIANTS` (haswell variant) | 6 | 5.91 → 10.48 (1.77x) | 4.66 → 5.52 (1.18x) |\n| | 12 | 9.99 → 15.83 (1.58x) | 6.50 → 6.63 (1.02x) |\n\nObservations:\n- Prompt processing gains 1.4–1.9x everywhere.\n- Generation gains 1.1–1.27x and then flattens: after the patch, tg is\n memory-bandwidth bound (thread scaling saturates at 3–4 threads on the\n laptop, and at 12 threads on the 3900X the gain vanishes). This is the\n expected regime for a 3.5 GiB weight stream; the kernel is no longer the\n bottleneck.\n- MSVC produces noticeably slower code for these kernels than Clang (1.4x vs\n 1.9x on pp). Not a regression, but there is room to help MSVC's codegen.\n\n### Notes for reviewers\n\n- The g128 activation quantizer also feeds the existing AVX-512 and NEON\n Q1_0 kernels, which stay correct but see slightly coarser activation\n scales (one per 128 instead of per 32). If you prefer to keep those paths\n numerically untouched, the g128 quantizer can be limited to the AVX2 path\n by giving it a dedicated traits type; I chose the minimal change.\n- MSVC does not define `__F16C__`; the fp16 column-scale conversion falls\n back to scalar there (4 conversions per 128-weight block, negligible).\n- GEMM allocates a per-call scratch buffer (`4*n` bytes + scales) with\n `malloc`; GEMV uses a 16 KiB stack buffer with heap fallback.\n\nTested on Windows 11 with Clang 19.1.5 (`-march=native` and the CI\n`GGML_CPU_ALL_VARIANTS` recipe) and MSVC 19.44, and on Linux with GCC 13\n(AVX2 and AVX512-VL+VNNI builds).","tree":"fbf09b30c6171195f3b402c1071a837fcb9c2fa0"}},{"name":"simd-gemm.h","mode":"0100644","size":7471,"lastCommit":{"hash":"57b50e1f6b50e01eb14c43fc1253602af74c1870","oid":"57b50e1f6b50e01eb14c43fc1253602af74c1870","parents":["68a521b591edd2f36a456809230d63aa81003dfc"],"author":{"name":"tyronecai","email":"tyronecai@163.com","when":"2026-07-08T13:00:05+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-08T08:00:05+03:00"},"message":"ggml : fix A indexing in simd_gemm scalar tail-column path (#25390)\n\n`simd_gemm()` has an incorrect A-matrix index in the scalar tail-column path for full row blocks.","tree":"76da12bb7064bcc400831738a40661df4ff9c0c4"}},{"name":"simd-mappings.h","mode":"0100644","size":54751,"lastCommit":{"hash":"cd644c39545aac3dca63261f99a9bfc35956cb25","oid":"cd644c39545aac3dca63261f99a9bfc35956cb25","parents":["947fd9bb2bdeaa72e9dd74b6aa3b5d68f03f3d6a"],"author":{"name":"Pascal","email":"admin@serveurperso.com","when":"2026-08-19T22:03:13+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-19T22:03:13+02:00"},"message":"ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE (#26860)\n\n* ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE\n\n__ARM_NEON only signals NEON availability. The __fp16 type also needs\nthe IEEE half format, implied on AArch64 but selected with\n-mfp16-format=ieee on 32 bit Arm, where the compiler otherwise rejects\nthe type.\n\nThe guard keeps every toolchain that provides the type on the same code\nand sends that one configuration to the generic lookup path.\n\n* ggml-cpu: gate the NEON+FMA block on __ARM_FP16_FORMAT_IEEE\n\nBoth halves of the F16 section dereference __fp16, so armv7 with\nneon-vfpv4 hits the same unknown type error. Without the IEEE\nformat the configuration now falls back to the scalar path.\n\nAddress review from @JonathanC-ARM","tree":"2cf59aafb7838d6ee7e8ca807ffc4519493f73ef"}},{"name":"spacemit","mode":"0040000","size":0,"lastCommit":{"hash":"08659901c43b51de735740f1cf61bb82fbe0c4e4","oid":"08659901c43b51de735740f1cf61bb82fbe0c4e4","parents":["61141f1487e63d9b22aec193131253bb2ea0800c"],"author":{"name":"Hao-Chen2337","email":"2113996104@qq.com","when":"2026-08-09T18:16:53+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-09T18:16:53+08:00"},"message":"ggml-cpu : fix missing Q5_0 dispatch in SpaceMiT backend (#26792)","tree":"9ae780f13650ac3d45e4e345f208163ad744dd6d"}},{"name":"traits.cpp","mode":"0100644","size":1231,"lastCommit":{"hash":"0d8831543cdc368fb248bae6f1b4aa5516684edc","oid":"0d8831543cdc368fb248bae6f1b4aa5516684edc","parents":["65c797c4fad4d9966695ac4b4a1560be44109267"],"author":{"name":"Diego Devesa","email":"slarengh@gmail.com","when":"2025-08-06T05:37:35-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-06T14:37:35+02:00"},"message":"ggml : fix fallback to CPU for ununsupported ops (#15118)\n\n","tree":"5adcc4f029a453e7e004916d9e8710df294831f1"}},{"name":"traits.h","mode":"0100644","size":1157,"lastCommit":{"hash":"0d8831543cdc368fb248bae6f1b4aa5516684edc","oid":"0d8831543cdc368fb248bae6f1b4aa5516684edc","parents":["65c797c4fad4d9966695ac4b4a1560be44109267"],"author":{"name":"Diego Devesa","email":"slarengh@gmail.com","when":"2025-08-06T05:37:35-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-06T14:37:35+02:00"},"message":"ggml : fix fallback to CPU for ununsupported ops (#15118)\n\n","tree":"5adcc4f029a453e7e004916d9e8710df294831f1"}},{"name":"unary-ops.cpp","mode":"0100644","size":11555,"lastCommit":{"hash":"914dde72babffc8aa2e04c97dafcec029a671b2e","oid":"914dde72babffc8aa2e04c97dafcec029a671b2e","parents":["3136a849dba799490d863ba03ef56b92ff052caf"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2026-02-11T18:58:43+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-02-11T18:58:43+02:00"},"message":"ggml : unary ops support non-cont src0 + metal F16 unary ops (#19511)\n\n* ggml : unary ops support non-cont src0\n\n* metal : support F16 unary ops + fix ELU","tree":"11cd0b6815a69d1f16e90befee94e1230326aabb"}},{"name":"unary-ops.h","mode":"0100644","size":2442,"lastCommit":{"hash":"389ac78b2675ffe5054134f3037b11642a31a09f","oid":"389ac78b2675ffe5054134f3037b11642a31a09f","parents":["a19bd6f7ce7e7b76e0dd7a614668a3042f5c1bfe"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-11-13T19:54:47+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-11-13T20:54:47+02:00"},"message":"ggml : add ops SOFTPLUS, EXPM1, TRI, SOLVE_TRI, CUMSUM (#17063)\n\n* Add ops needed for new hybrid models: SOFTPLUS, EXPM1, TRI, SOLVE_TRI, CUMSUM\n\n* Update ggml/include/ggml.h\n\nCo-authored-by: Georgi Gerganov \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* Code review\n\n* Whitespace\n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Diego Devesa \n\n* This is actually sigmoid, duh.\n\n* Add CONST, remove TRI_KEEP, other changes from review\n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* Update ggml/src/ggml.c\n\nCo-authored-by: Georgi Gerganov \n\n* Update ggml/src/ggml.c\n\nCo-authored-by: Georgi Gerganov \n\n* Update ggml/src/ggml-cuda/unary.cu\n\nCo-authored-by: Aman Gupta \n\n* Remove extra script\n\n* Update ggml/src/ggml.c\n\nCo-authored-by: Diego Devesa \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Diego Devesa \n\n* moving changes from laptop [no ci]\n\n* pre-rebase\n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Refactor tests\n\n* ggml : cleanup\n\n* cont : fix ggml_fill srcs\n\n* tests : add note\n\n* ggml : add ggml_fill_inplace\n\n* ggml : add asserts\n\n* ggml : fix ggml_fill constant cast\n\n* cont : ggml_tri minor\n\n* Use TENSOR_LOCALS\n\n* Fix regression from #14596, regenerate\n\n* Don't make commits at night...\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Diego Devesa \nCo-authored-by: Aman Gupta \nCo-authored-by: Sigbjørn Skjæret ","tree":"3583523fb8ec5af8f7f554495d11a8cf00d1639e"}},{"name":"vec.cpp","mode":"0100644","size":25309,"lastCommit":{"hash":"c16c35b8142ac34b4559a5e8584e8daaf3958416","oid":"c16c35b8142ac34b4559a5e8584e8daaf3958416","parents":["1a87dcdc452de056a66796b679e4ba7071488cf7"],"author":{"name":"Tarek Dakhran","email":"tarek@liquid.ai","when":"2026-06-26T09:41:56+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-26T10:41:56+03:00"},"message":"ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 (#24699)\n\n* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32\n\n2D convolutions with kernel size 9 produced different results on SVE\nenabled ARM devices. After debugging it turned out that ggml_vec_dot_f32\nwas using data from inactive lanes.\n\nUse svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1.\n\n* cont : clean-up\n\n---------\n\nCo-authored-by: Georgi Gerganov ","tree":"253cf542a0577beb8ad033a193c264ba2ef05d8c"}},{"name":"vec.h","mode":"0100644","size":66060,"lastCommit":{"hash":"e31cdaa0ebc21b06e4e5267ab4cbacf7a4ec7c57","oid":"e31cdaa0ebc21b06e4e5267ab4cbacf7a4ec7c57","parents":["491c4d7d2e2378c81fa0eb2bf916b437bbee4b0f"],"author":{"name":"Martin Klacer","email":"martin.klacer@arm.com","when":"2026-05-28T08:04:21+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-28T10:04:21+03:00"},"message":"ggml: fixed Arm SVE usage bug in vec.h, vec.cpp (#22841)\n\n* Updated vec.h/vec.cpp code to accumulate to F32 rather than F16\n\n\n\nChange-Id: I0cb789347f2bf60ffaf9047319f727e788c825f8\n\nSigned-off-by: Martin Klacer \nCo-authored-by: Milos Puzovic ","tree":"f797f6654c0bcd07d687b45cfb0818950006404e"}}],"lastCommit":{"hash":"adfffbe41b2cabcd51fff326ab045662265062bb","oid":"adfffbe41b2cabcd51fff326ab045662265062bb","parents":["985880c0b744e017fe62b61b396e7674812de323"],"author":{"name":"Scott Guyton","email":"kiljoy001@gmail.com","when":"2026-09-25T11:42:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-25T11:42:36-07:00"},"message":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds.","tree":"f701dae3247ff1ba55db24d1fd83dff0cd2edff8"}}