[{"name":"peg-parser","kind":"directory","size":0,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"snapshots","kind":"directory","size":0,"lastCommitHash":"47f686f53f2a20fe82d87d79fa5c835714ee8f46","lastCommitWhen":"2026-07-30T19:34:04+03:00","lastCommitMessage":"tests : avoid building get-model.cpp many times (#26317)\n\n* tests : remove get-model.cpp\n\n* tests : fix quant type selection"},{"name":".gitignore","kind":"file","size":58,"lastCommitHash":"7992aa7c8e21ea2eb7a5e4802da56eec7b376036","lastCommitWhen":"2026-04-02T22:53:58+02:00","lastCommitMessage":"tests : add unit test coverage for llama_tensor_get_type (#20112)\n\n* Add unit test coverage for llama_tensor_get_type\n\n* Fix merge conflicts, add more schemas\n\n* clang formatter changes\n\n* Trailing whitespace\n\n* Update name\n\n* Start rebase\n\n* Updating files with upstream changes prior to rebase\n\n* Changes needed from rebase\n\n* Update attn_qkv schema, change throw behaviour\n\n* Fix merge conflicts\n\n* White space\n\n* Update with latest changes to state counters\n\n* Revert accidental personal CLAUDE.md changes\n\n* Change quotation mark\n\n* Reuse metadata.name since we have it\n\n* Move test-only stuff out of llama-quant.cpp\n\n* Hide the regex functionality back in llama-quant.cpp, use a unique pointer to a new struct 'compiled_tensor_type_patterns' which contains the patterns\n\n* cont : inital deslop guidelines\n\n* Cleanup based on review comments\n\n* Continue cleanup\n\n* Small cleanup\n\n* Manually set proper ordering of tensors, mostly applies to gemma\n\n* Formatting\n\n* Update tests/test-quant-type-selection.cpp\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Fix merge conflicts\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Sigbjørn Skjæret "},{"name":"CMakeLists.txt","kind":"file","size":17022,"lastCommitHash":"8906fd8c9fd9d2dcbffad469432b8d2edf0925fd","lastCommitWhen":"2026-10-18T08:02:45-04:00","lastCommitMessage":"cache: conversations are stored against their learned prefix\n"},{"name":"gen-tiny-dfly.py","kind":"file","size":4815,"lastCommitHash":"857e5f205e3d5a5b9068325f90c1e0cd2aa24fef","lastCommitWhen":"2026-09-13T21:04:40-07:00","lastCommitMessage":"dflash: fix DFly conversion + loading against the reference checkpoint\n\nRunning the real AngelSlim/Qwen3-8B-DFly-Block8 (revision 5712926) end to end\nturned up three things the synthetic tests could not:\n\n * conversion/__init__.py holds an architecture -> module map used to lazily\n import converters, separate from @ModelBase.register. Without an entry there\n the DFly classes never load and conversion fails with \"Model Qwen3DFlyModel\n is not supported\".\n\n * The fusion parameter is `layer_fusion_weights` in the checkpoint, with no\n `.weight` suffix, so the writer emits a bare `layer_fusion` tensor. The\n loader looked for `layer_fusion.weight` and would not have detected DFly at\n all. Load it suffix-less, like d2t. The tiny-model generator matched the\n loader rather than the converter, so it agreed with the bug.\n\n * The draft context also decodes ordinary batches (context staging before the\n K/V injection), not just noise blocks. Those have no anchor to condition on:\n slot 0 is the caller's id_last, which is LLAMA_TOKEN_NULL until the first\n token is committed. Chaining one of those indexes the embedding table with\n row -1. Check the block shape before building the chain, and take the anchor\n embedding from the rows the decoder already gathered instead of re-fetching\n it by id -- a build-time check alone cannot hold once graphs are reused.\n\nMeasured with the real pair on an M5 Pro (Qwen3-8B bf16 target + DFly bf16\ndrafter, Metal): the drafter converts, loads, auto-detects as 'draft-dflash',\nand runs the fusion and layer-varying injection end to end, producing coherent\ntarget output at 12.9 tok/s with draft_n=637.\n\nAcceptance is 0, because the predecessor-correction chain is NOT yet enabled.\nRunning it aborts nondeterministically on an uninitialised index (-1) that is\nnot produced by any argmax, in a get_rows outside the chain's own nodes. The\nchain calls the target's lm_head through ctx_other once per block position and\nfeeds each argmax back into a get_rows on the draft's embeddings; the existing\nDSpark head never builds such a cross-context dependency, it only biases\nalready-computed logits. That is the suspect and the reason the chain is now\nopt-in behind LLAMA_DFLY_CHAIN=1, with a load-time warning so a near-zero\nacceptance run cannot be mistaken for a working DFly.\n\nThe likely fix is to move the correction out of the model graph into the\nsampler chain, which is where the reference runtime puts it.\n"},{"name":"gen-tiny-dspark.py","kind":"file","size":8388,"lastCommitHash":"6067dbea986ed1359db98c42ab06eb49426bbf16","lastCommitWhen":"2026-09-17T21:25:55-07:00","lastCommitMessage":"dspark: add corrected draft runtime and shared packed target head\n"},{"name":"gguf-model-data.cpp","kind":"file","size":23245,"lastCommitHash":"35a74c8fb94b45cae4eec7696b0ed5b7a3ca16a1","lastCommitWhen":"2026-05-26T19:05:48+03:00","lastCommitMessage":"ci : add `[no release]` keyword + fix sanitizer builds (#23728)\n\n* ci : skip release workflow on master when commit message contains [no release]\n\nAssisted-by: llama.cpp:local pi\n\n* ci : restrict sanitizer builds to x86_64 + fix build type\n\nthe spark is apparently too slow for some reason\n\n* tests : fix undefined warning\n\n[no ci]"},{"name":"gguf-model-data.h","kind":"file","size":1986,"lastCommitHash":"7992aa7c8e21ea2eb7a5e4802da56eec7b376036","lastCommitWhen":"2026-04-02T22:53:58+02:00","lastCommitMessage":"tests : add unit test coverage for llama_tensor_get_type (#20112)\n\n* Add unit test coverage for llama_tensor_get_type\n\n* Fix merge conflicts, add more schemas\n\n* clang formatter changes\n\n* Trailing whitespace\n\n* Update name\n\n* Start rebase\n\n* Updating files with upstream changes prior to rebase\n\n* Changes needed from rebase\n\n* Update attn_qkv schema, change throw behaviour\n\n* Fix merge conflicts\n\n* White space\n\n* Update with latest changes to state counters\n\n* Revert accidental personal CLAUDE.md changes\n\n* Change quotation mark\n\n* Reuse metadata.name since we have it\n\n* Move test-only stuff out of llama-quant.cpp\n\n* Hide the regex functionality back in llama-quant.cpp, use a unique pointer to a new struct 'compiled_tensor_type_patterns' which contains the patterns\n\n* cont : inital deslop guidelines\n\n* Cleanup based on review comments\n\n* Continue cleanup\n\n* Small cleanup\n\n* Manually set proper ordering of tensors, mostly applies to gemma\n\n* Formatting\n\n* Update tests/test-quant-type-selection.cpp\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Fix merge conflicts\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Sigbjørn Skjæret "},{"name":"test-alloc.cpp","kind":"file","size":22644,"lastCommitHash":"f4253ef965c59e902fac7b8193f96bcc89f66e55","lastCommitWhen":"2026-07-13T15:36:51+03:00","lastCommitMessage":"tests: Harmonize header use (#25616)\n\n* tests: Harmonize the use of private ggml includes\n\n* tests: In test-backend-ops, use quoted includes\n\nAs with all other tests. This is to ensure that the build uses shipped\nheaders over possibly system-installed ones."},{"name":"test-arg-parser.cpp","kind":"file","size":14447,"lastCommitHash":"dd1ea524333b1e697489067d7a4c39c60d32beee","lastCommitWhen":"2026-08-10T16:58:56+03:00","lastCommitMessage":"llama : support multi-output backend sampling (#25532)\n\n* Enable backend sampling with token speculation\n\n* Clamp the mask sum before converting it into the sampled index\n\n* Add a numeric context parameter declaring the maximum outputs one sequence\n\n* More fixes\n\n* Don't reuse memory for output views.\n\n* Match dist between CPU and GPU\n\n* Fix CPU and backend sampling mismatches\n\n* Simpify some of the changes\n\n* Fix tests on Vulkan\n\n* More test fixes\n\n* Rebase changes\n\n* Rebase and address review comments\n\n* Address review comments\n\n* Address review comments\n\n* Update src/llama-sampler.cpp\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-autorelease.cpp","kind":"file","size":679,"lastCommitHash":"47f686f53f2a20fe82d87d79fa5c835714ee8f46","lastCommitWhen":"2026-07-30T19:34:04+03:00","lastCommitMessage":"tests : avoid building get-model.cpp many times (#26317)\n\n* tests : remove get-model.cpp\n\n* tests : fix quant type selection"},{"name":"test-backend-ops.cpp","kind":"file","size":491474,"lastCommitHash":"c0209b63a39312975f3cbff7670afda41f653cd9","lastCommitWhen":"2026-10-13T19:13:57-04:00","lastCommitMessage":"bonsai: in-place GDN recurrent state reading from cache rows (gdn-rows)\n\nThe DeltaNet layers read each seq's recurrent state straight from its cache row instead of gathering it first (only while every seq's row is its own cell; otherwise the old gather). Bit-identical; MTP decode +1.6-2.3%. GGML_GDN_STATE_GATHER=1 restores the gather.\n"},{"name":"test-backend-sampler.cpp","kind":"file","size":88646,"lastCommitHash":"4801e3c567d5131dd41b387df5f2d4b1370d92be","lastCommitWhen":"2026-08-11T07:21:32+03:00","lastCommitMessage":"tests : disable backend sampler hip multi output (#26878)\n\n* test-backend-sampler: skip multi_output_sampling_chain on HIP\n\nThe new multi_output_sampling_chain test uses top_k, whose backend probs\npath needs CUB (unavailable on HIP), so sampled_probs is null and the test\naborts. Add it to the existing HIP skip list alongside the other TOP_K tests.\n\n* ci: keep gpu-rocm logs in a per-run dir keyed by GitHub run id\n\nThe self-hosted gpu-rocm runner can't upload logs to Azure blob (egress\nfirewalled), so a run's logs were wiped by the next run. Write each run's\nlogs to $OUT/run--/ so an Actions run URL maps to its logs.\n\n* test-backend-sampler: also skip multi_output_cpu on HIP\n\nLike the other TOP_K-based subtests, multi_output_cpu's backend sampler\nnever initializes on HIP (no CUB TOP_K), so it aborts. Add it to the skip list.\n\n---------\n\nCo-authored-by: Jim Wu "},{"name":"test-barrier.cpp","kind":"file","size":7671,"lastCommitHash":"e1f4921980444b30d3b4fa1c6e18cdecd85b0690","lastCommitWhen":"2025-12-10T12:32:23-08:00","lastCommitMessage":"Fix race conditions in threadpool when dealing with dynamic/frequent n_threads changes (#17748)\n\n* tests: update barrier test to check for race condition in active threads\n\n* cpu: combine n_graph and n_threads into a single atomic update\n\n* tests: add multi-graph test for test_barrier"},{"name":"test-batch-alloc.cpp","kind":"file","size":23212,"lastCommitHash":"961e4b26a7dd0e01e20599b27d709a74788ecb55","lastCommitWhen":"2026-07-10T11:04:31+08:00","lastCommitMessage":"llama-batch: add unit test (#25471)\n\n* llama-batch: add unit test\n\n* fix win32 builds\n\n* add not implemented assertion in unused methods\n\n* remove unreachable code"},{"name":"test-c.c","kind":"file","size":38,"lastCommitHash":"d4cdd9c1c3cebdafca735958597de4ff7b7c0f54","lastCommitWhen":"2025-07-03T07:48:32+03:00","lastCommitMessage":"ggml : remove kompute backend (#14501)\n\nggml-ci"},{"name":"test-chat-analysis.cpp","kind":"file","size":25832,"lastCommitHash":"4a08fa29705b8177e332b134306566c2c4b95902","lastCommitWhen":"2026-08-23T18:38:51+02:00","lastCommitMessage":"test: move tools/parser to tests (#27548)"},{"name":"test-chat-auto-parser.cpp","kind":"file","size":124695,"lastCommitHash":"4a08fa29705b8177e332b134306566c2c4b95902","lastCommitWhen":"2026-08-23T18:38:51+02:00","lastCommitMessage":"test: move tools/parser to tests (#27548)"},{"name":"test-chat-peg-parser.cpp","kind":"file","size":48317,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-chat-template.cpp","kind":"file","size":69061,"lastCommitHash":"4a08fa29705b8177e332b134306566c2c4b95902","lastCommitWhen":"2026-08-23T18:38:51+02:00","lastCommitMessage":"test: move tools/parser to tests (#27548)"},{"name":"test-chat.cpp","kind":"file","size":307154,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-col2im-1d.cpp","kind":"file","size":6706,"lastCommitHash":"26021699bc31abfa5458326d4e0e4e964d4a48f5","lastCommitWhen":"2026-06-09T12:01:37+03:00","lastCommitMessage":"ggml : add GGML_OP_COL2IM_1D (#24206)\n\n* cpu: add GGML_OP_COL2IM_1D\n\nAdd the overlap-add (scatter-add) step of a 1D transposed convolution.\nA ConvTranspose1d factorizes as a GEMM followed by col2im: a weight\npre-permuted to [IC, K*OC] is contracted against the [IC, T_in] input\nwith mul_mat to produce a column matrix [K*OC, T_in], and col2im_1d\nscatters those columns back into the [T_out, OC] signal, with\nT_out = (T_in - 1)*s0 + K - 2*p0.\n\nKeeping the contraction as a plain mul_mat leaves the heavy work on the\noptimized (and quantizable) matmul kernels, so col2im_1d only does the\ncheap overlap-add.\n\nCPU uses a gather formulation parallelized over output channels,\nsupporting F32, F16 and BF16 with an F32 accumulator.\n\n* tests: add backend coverage for GGML_OP_COL2IM_1D\n\nAdd test_col2im_1d next to the conv_transpose_1d cases, covering F32,\nF16 and BF16 across eight geometries: the canonical kernel = 2*stride\nDAC upsampling shape, overlap, no overlap, cropping (p0 = 1 and\np0 = stride/2), kernel < stride with zeroed gaps, kernel not a\nmultiple of stride, and a single column unfold.\n\nPerf mode gets three real vocoder stage shapes reporting memory\nbandwidth. max_nmse_err relaxes to 5e-4 for F16 and BF16.\n\n* cpu: harden GGML_OP_COL2IM_1D\n\nggml_col2im_1d validates s0, oc, p0 and input contiguity at graph\nbuild time, before the oc division, protecting every backend at once.\nThe kernel asserts the contiguity its flat indexing assumes and its\ndoc states the full output length including the crop term.\n\nThe kernel parallelizes over the time axis: the split stays balanced\ndown to OC = 1, where the previous channel split was single threaded.\nValues are bit identical on the three real vocoder chains, two out of\nthree improve.\n\n* tests: extend the GGML_OP_COL2IM_1D grid\n\nThe eval grid grows to eleven geometries: OC = 1 (mono output stage),\nK = 1 with stride > 1 (sparse scatter, every gap position zeroed) and\na crop down to T_out = 2 where all the gather bounds act at once.\n\n* tests: add col2im_1d equivalence test\n\ntests/test-col2im-1d.cpp proves mul_mat + col2im_1d matches the\nnative ggml_conv_transpose_1d on the CPU backend, F32 bit exact, F16\nand BF16 through casts of the column matrix. test-backend-ops cannot\ncover this for a CPU only op since the CPU backend is its own\nreference there.\n\n* rpc: bump protocol patch version for GGML_OP_COL2IM_1D\n\nGGML_OP_COUNT goes from 96 to 97 with the new op, which trips the\nstatic_assert in ggml-rpc.h. Bump RPC_PROTO_PATCH_VERSION since the\nop is appended and no existing op code shifts."},{"name":"test-dfly-fusion.cpp","kind":"file","size":7458,"lastCommitHash":"2863b1c7d7352090f6e1ebef382ee9c6fbe65243","lastCommitWhen":"2026-09-13T21:04:40-07:00","lastCommitMessage":"dflash: AngelSpec DFly drafter support\n\nDFly extends the DFlash draft path with two things, and this wires both onto the\nexisting multi-target-layer capture that DFlash already had:\n\n 1. Per-DRAFT-layer context fusion. The shared projection (context_proj -> fc)\n now only produces a base context; each draft layer adds its own\n softmax-weighted mix of the raw per-target-layer features, so the encoder\n emits one context per draft layer instead of one shared context. The\n decoder's embd batch slices its own layer's context back out for the K/V\n injection. DFly drops DFlash's encoder hidden_norm in favour of a\n post-fusion context_norm, so exactly one of the two is present.\n\n 2. A TreeFlash predecessor correction chained across the block: position i's\n logits come from the draft hidden state at row i corrected by the embedding\n of the token drafted at row i-1, then projected through the target head.\n Slot 0 is the committed anchor, not a prediction slot, matching the row\n convention the DFlash reader in common/speculative.cpp already uses.\n\nThe whole host-side contract change is n_embd_out(): both the nextn output\nbuffer and the embd batch that feeds it back are already sized from it, so\nwidening it for DFly needs no plumbing changes. Nothing is needed in\ncommon/speculative.cpp either -- DFly reports no Markov head, so it takes the\nDFlash reader (rows 1..n-1) and the plain-probability draft cut, which is the\nthreshold form of D-cut that path already implements.\n\nDFly is detected from layer_fusion rather than a KV so a DFly export cannot load\nas plain DFlash, and a checkpoint declaring both a DFly fusion and a DSpark\nMarkov head is rejected with that reason rather than failing later on a missing\ntensor. Reduced draft vocabularies (d2t) are rejected: the reference chain runs\nthe full target head.\n\nConversion covers Qwen3DFlyModel and the Qwen3DSparkDFlareV2Model alias, and\nrefuses a config whose target_layer_ids exceed its stated target depth -- the\nreference checkpoint's published metadata does exactly that, and it loads clean\nand then mis-projects.\n\ntests/test-dfly-fusion.cpp pins the fusion math and the resulting flat layout\nagainst an independent scalar reference; the axis handling needs two\nreshape/permute round trips and would otherwise be easy to get wrong in a way\nthat still loads and still drafts plausibly. tests/gen-tiny-dfly.py builds a\ntiny random DFly GGUF for loader smoke tests.\n\nNot yet measured: acceptance rate or speedup against a real target/drafter pair.\n"},{"name":"test-double-float.cpp","kind":"file","size":1837,"lastCommitHash":"370b1f7e7a7514d9a63daf15f7b0f00319b7f908","lastCommitWhen":"2024-07-12T10:46:02+03:00","lastCommitMessage":"ggml : minor naming changes (#8433)\n\n* ggml : minor naming changes\r\n\r\nggml-ci\r\n\r\n* ggml : use PRId64 [no ci]\r\n\r\n* ggml : revert FA K/Q names"},{"name":"test-dspark-forward.cpp","kind":"file","size":55411,"lastCommitHash":"985880c0b744e017fe62b61b396e7674812de323","lastCommitWhen":"2026-09-25T11:35:42-07:00","lastCommitMessage":"tests : build test-dspark-forward with MSVC (#273)\n\nThe Windows CRT has no POSIX setenv/unsetenv, so MSVC failed to compile\ntest-dspark-forward.cpp and broke any Windows build with\nLLAMA_BUILD_TESTS=ON. Add _putenv_s-based shims under _WIN32.\n\nThe test also calls internal functions that are not exported with\nLLAMA_API (llama_hparams::n_layer, llama_model::get_tensor), so like the\nother internal-symbol tests it is skipped on Windows when building with\nshared libraries."},{"name":"test-dspark-logsnr-meta.cpp","kind":"file","size":5735,"lastCommitHash":"6067dbea986ed1359db98c42ab06eb49426bbf16","lastCommitWhen":"2026-09-17T21:25:55-07:00","lastCommitMessage":"dspark: add corrected draft runtime and shared packed target head\n"},{"name":"test-dspark-loop.cpp","kind":"file","size":20362,"lastCommitHash":"08ac1fd17ef176abaab6287981b515fbc5582210","lastCommitWhen":"2026-09-17T22:59:12-07:00","lastCommitMessage":"dspark: validate mask selection and graph reuse contracts\n"},{"name":"test-dspark-real-eval.cpp","kind":"file","size":48395,"lastCommitHash":"08ac1fd17ef176abaab6287981b515fbc5582210","lastCommitWhen":"2026-09-17T22:59:12-07:00","lastCommitMessage":"dspark: validate mask selection and graph reuse contracts\n"},{"name":"test-export-graph-ops.cpp","kind":"file","size":6454,"lastCommitHash":"657e01125aa49577a62a5531fde24cbcc007006d","lastCommitWhen":"2026-07-14T13:15:41+03:00","lastCommitMessage":"tests: export-graph-ops: exit gracefully when called w/o arguments (#25619)\n\nFixes a segfault when `test-export-graph-ops` is called without any\narguments."},{"name":"test-gbnf-validator.cpp","kind":"file","size":3402,"lastCommitHash":"13b4548877326fdabee3e831b8cfd65d9844383c","lastCommitWhen":"2025-04-24T16:00:10+03:00","lastCommitMessage":"cmake : do not include ./src as public for libllama (#13062)\n\n* cmake : do not include ./src as public for libllama\n\nggml-ci\n\n* cmake : rework tests\n\nggml-ci\n\n* llguidance : remove unicode include\n\nggml-ci\n\n* cmake : make c++17 private\n\nggml-ci"},{"name":"test-gguf-model-data.cpp","kind":"file","size":6951,"lastCommitHash":"7992aa7c8e21ea2eb7a5e4802da56eec7b376036","lastCommitWhen":"2026-04-02T22:53:58+02:00","lastCommitMessage":"tests : add unit test coverage for llama_tensor_get_type (#20112)\n\n* Add unit test coverage for llama_tensor_get_type\n\n* Fix merge conflicts, add more schemas\n\n* clang formatter changes\n\n* Trailing whitespace\n\n* Update name\n\n* Start rebase\n\n* Updating files with upstream changes prior to rebase\n\n* Changes needed from rebase\n\n* Update attn_qkv schema, change throw behaviour\n\n* Fix merge conflicts\n\n* White space\n\n* Update with latest changes to state counters\n\n* Revert accidental personal CLAUDE.md changes\n\n* Change quotation mark\n\n* Reuse metadata.name since we have it\n\n* Move test-only stuff out of llama-quant.cpp\n\n* Hide the regex functionality back in llama-quant.cpp, use a unique pointer to a new struct 'compiled_tensor_type_patterns' which contains the patterns\n\n* cont : inital deslop guidelines\n\n* Cleanup based on review comments\n\n* Continue cleanup\n\n* Small cleanup\n\n* Manually set proper ordering of tensors, mostly applies to gemma\n\n* Formatting\n\n* Update tests/test-quant-type-selection.cpp\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Fix merge conflicts\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Sigbjørn Skjæret "},{"name":"test-gguf.cpp","kind":"file","size":51909,"lastCommitHash":"d8a8beac22d450ebadf175a8ce7b6bf49b66db14","lastCommitWhen":"2026-08-12T15:07:48+03:00","lastCommitMessage":"gguf : harden loader against malformed tensor dims and metadata types (#25596)\n\n* gguf : harden loader against malformed tensor dims and metadata types\n\n* gguf: address review on malformed-metadata hardening\n\n- report the expected vs. actual type when general.alignment is not u32\n- use ggml_nelements() > 0 for the zero-element guard and keep the\n representability checks visually aligned\n- add test-gguf cases for a wrong-typed alignment key and a zero-dim\n tensor (both used to crash: assert-abort and SIGFPE respectively)\n\nRan tests/test-gguf: 164/164 pass. Used an AI assistant to help draft\nthese edits; reviewed and verified by me.\n\n* cont : less comments\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-grammar-integration.cpp","kind":"file","size":42408,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-grammar-llguidance.cpp","kind":"file","size":39565,"lastCommitHash":"147a5216363c5b2edcdeb17f01ef3b28372ec872","lastCommitWhen":"2025-12-22T11:00:37+01:00","lastCommitMessage":"tool/ex/tests: consistently free ctx, then model (#18168)\n\n"},{"name":"test-grammar-parser.cpp","kind":"file","size":18408,"lastCommitHash":"cd0fa6051abd04fd7f82ccb42ad378ec43885455","lastCommitWhen":"2026-08-05T07:39:10-05:00","lastCommitMessage":"grammar : degrade max repetition >= 2000 to unbounded (#26613)"},{"name":"test-jinja.cpp","kind":"file","size":79151,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-json-schema-to-grammar.cpp","kind":"file","size":52604,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-kv-mean-center.cpp","kind":"file","size":14252,"lastCommitHash":"36d1deef278725a43aedcdd8d68a9be7815cbbe6","lastCommitWhen":"2026-08-25T05:55:11Z","lastCommitMessage":"kv-cache: mean-centering on hybrid models + calibration-basis guard (fork #53)\n"},{"name":"test-kv-share.cpp","kind":"file","size":8288,"lastCommitHash":"8906fd8c9fd9d2dcbffad469432b8d2edf0925fd","lastCommitWhen":"2026-10-18T08:02:45-04:00","lastCommitMessage":"cache: conversations are stored against their learned prefix\n"},{"name":"test-llama-archs.cpp","kind":"file","size":34863,"lastCommitHash":"7584430716ee229751771ed0d6bbcb780d105eeb","lastCommitWhen":"2026-08-24T13:39:31+03:00","lastCommitMessage":"tests : disable DOTS3NOTE arch test for WebGPU (#27654)\n\nCo-authored-by: Stanisław Szymczyk "},{"name":"test-llama-grammar.cpp","kind":"file","size":11302,"lastCommitHash":"990e4d96980d0b016a2b07049cc9031642fb9903","lastCommitWhen":"2026-03-21T18:43:35+01:00","lastCommitMessage":"common/grammar: fix grammar parsing issues to prevent stack overflow and hangs (#18604)\n\n* grammar: add test case for nullable symbol loop\n\nReproduce stack overflow (or OOM) with ( [x]* )* found while adding\nGBNF support to ripgrep-edit.\n\nllama-server reproducer:\n\ncurl \\\n -X POST \\\n -d '{\n \"messages\": [{ \"role\": \"user\", \"content\": \"write yes\" }],\n \"grammar\": \"root ::= ( [x]* )*\"\n }' \\\n -H \"Content-Type: application/json\" \\\n http://localhost:8811/v1/chat/completions\n\n* grammar: prevent stack overflow with nullable symbol loop\n\nFix a potential stack overflow in llama_grammar_advance_stack that\ncould occur when processing grammars with nullable symbols that lead\nto infinite derivations of empty strings. The fix introduces cycle\ndetection by tracking visited stacks to prevent infinite recursion.\n\nrg-edit regexp: llama_grammar_advance_stack\nrg-edit extra-args: -A20\nrg-edit directive: \"\"\"Rewrite: fix the following segfault:\n\n[..]\n⚫ Testing segfault. Grammar:\n root ::= ( [x]* )*\n\n root ::= ( [x]* )*\n\nSegmentation fault build/bin/test-grammar-integration\"\"\"\n\ngptel-context:\n((\"~/llama.cpp/src/llama-grammar.cpp\")\n (\"~/llama.cpp/tests/test-grammar-integration.cpp\")\n (\"~/llama.cpp/grammars/./list.gbnf\")\n (\"~/llama.cpp/grammars/./json_arr.gbnf\")\n (\"~/llama.cpp/grammars/./json.gbnf\")\n (\"~/llama.cpp/grammars/./japanese.gbnf\")\n (\"~/llama.cpp/grammars/./english.gbnf\")\n (\"~/llama.cpp/grammars/./chess.gbnf\")\n (\"~/llama.cpp/grammars/./c.gbnf\")\n (\"~/llama.cpp/grammars/./arithmetic.gbnf\")\n (\"~/llama.cpp/grammars/./README.md\"))\n\n* grammar: convert recursive llama_grammar_advance_stack to iterative\n\nThis change converts the function to an iterative approach using\nexplicit stacks, which prevents deep recursion and eliminates the risk\nof stack overflow.\n\nrg-edit regexp: llama_grammar_advance_stack\nrg-edit extra-args: -A30\nrg-edit directive: \"\"\"Rewrite: fix the following segfault:\n\n[..]\n⚫ Testing segfault. Grammar:\n root ::= ( [x]* )*\n\n root ::= ( [x]* )*\n\nSegmentation fault build/bin/test-grammar-integration\n\nconvert from recursive to interactive\"\"\"\n\ngptel-context:\n((\"~/llama.cpp/src/llama-grammar.cpp\")\n (\"~/llama.cpp/tests/test-grammar-integration.cpp\")\n (\"~/llama.cpp/grammars/./list.gbnf\")\n (\"~/llama.cpp/grammars/./json_arr.gbnf\")\n (\"~/llama.cpp/grammars/./json.gbnf\")\n (\"~/llama.cpp/grammars/./japanese.gbnf\")\n (\"~/llama.cpp/grammars/./english.gbnf\")\n (\"~/llama.cpp/grammars/./chess.gbnf\")\n (\"~/llama.cpp/grammars/./c.gbnf\")\n (\"~/llama.cpp/grammars/./arithmetic.gbnf\")\n (\"~/llama.cpp/grammars/./README.md\"))\n\nv2: Added a `std::set` to perform tree-based lookups with O(N log N)\ncomplexity. Testing with a parallel run of `test-grammar-integration`\nshows a double-digit percentage increase in runtime. An\n`unordered_set` with O(1) hashing was also evaluated, but the overhead\nof constructing hash keys from pointers made it significantly slower\nthan the rbtree implementation that only requires an ordering\noperator. The performance regression in the test suite appears\njustified by the overall reduction in algorithmic complexity.\n\nCo-developed-by: Piotr Wilkin (ilintar) \n\n* grammar: add test case for hang in repetition grammar processing\n\nThis commit adds a new test case to the grammar integration tests that\nspecifically targets a hang scenario in the repetition grammar parser\nfound while adding GBNF support to ripgrep-edit.\n\nllama-server reproducer:\n\ncurl \\\n -X POST \\\n -d '{\n \"messages\": [{ \"role\": \"user\", \"content\": \"write yes\" }],\n \"grammar\": \"root ::= (([^x]*){0,99}){0,99}\"\n }' \\\n -H \"Content-Type: application/json\" \\\n http://localhost:8811/v1/chat/completions\n\n* grammar: add repetition threshold check\n\nThe change introduces a maximum repetition threshold to avoid\nexcessive rule expansion during grammar parsing. When parsing\nrepetition patterns like {m,n}, the parser now calculates the\npotential number of rules that would be generated and throws an error\nif the product of previous rules and new rules exceeds the threshold.\n\nA test case was added to verify the threshold is properly enforced for\ndeeply nested repetition patterns that would otherwise cause hangs."},{"name":"test-log.cpp","kind":"file","size":1305,"lastCommitHash":"7b95ea5d110241cf8b0fb6936927387f63f18a32","lastCommitWhen":"2026-04-29T10:58:43+03:00","lastCommitMessage":"common: Intentionally leak logger instance to fix hanging on Windows (#22273)\n\n* Changed to leak logger singleton to prevent hanging on Windows\n\n* Fix comment\n\n* Stopped using static vector\n\nUsing std::vector will cause g_col to be released before the logger thread exits, causing the logger thread to touch freed memory causing a crash\n\n* Change so all logs are output before exit\n\n* Added debug logging\n\n* added more logging\n\n* Added logging\n\n* Explicitly free logger to avoid hanging on Win\n\n* Reverted to leak logger instance again\n\n* Removed debug log and fixed comment\n\n* Fixed comment\n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-lora-conversion-inference.sh","kind":"file","size":5648,"lastCommitHash":"6c2131773c5975c719863a21a8195a66dea4a4d0","lastCommitWhen":"2025-12-10T15:28:59+01:00","lastCommitMessage":"cli: new CLI experience (#17824)\n\n* wip\n\n* wip\n\n* fix logging, add display info\n\n* handle commands\n\n* add args\n\n* wip\n\n* move old cli to llama-completion\n\n* rm deprecation notice\n\n* move server to a shared library\n\n* move ci to llama-completion\n\n* add loading animation\n\n* add --show-timings arg\n\n* add /read command, improve LOG_ERR\n\n* add args for speculative decoding, enable show timings by default\n\n* add arg --image and --audio\n\n* fix windows build\n\n* support reasoning_content\n\n* fix llama2c workflow\n\n* color default is auto\n\n* fix merge conflicts\n\n* properly fix color problem\n\nCo-authored-by: bandoti \n\n* better loading spinner\n\n* make sure to clean color on force-exit\n\n* also clear input files on \"/clear\"\n\n* simplify common_log_flush\n\n* add warning in mtmd-cli\n\n* implement console writter\n\n* fix data race\n\n* add attribute\n\n* fix llama-completion and mtmd-cli\n\n* add some notes about console::log\n\n* fix compilation\n\n---------\n\nCo-authored-by: bandoti "},{"name":"test-model-load-cancel.cpp","kind":"file","size":783,"lastCommitHash":"47f686f53f2a20fe82d87d79fa5c835714ee8f46","lastCommitWhen":"2026-07-30T19:34:04+03:00","lastCommitMessage":"tests : avoid building get-model.cpp many times (#26317)\n\n* tests : remove get-model.cpp\n\n* tests : fix quant type selection"},{"name":"test-model-resolution.cpp","kind":"file","size":20106,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"test-mtmd-c-api.c","kind":"file","size":5735,"lastCommitHash":"15586e2d7165570fb3aa7c26e0d442e289ef69de","lastCommitWhen":"2026-08-06T19:46:40+02:00","lastCommitMessage":"mtmd: add chunk save/load function (#26645)\n\n* mtmd: add chunk save/load function\n\n* nits\n\n* add tests\n\n* rn _MAX --> _COUNT"},{"name":"test-mtmd-impl.cpp","kind":"file","size":4854,"lastCommitHash":"95c409c13625a23da2aa37270339ce9179215a18","lastCommitWhen":"2026-08-19T13:48:22+02:00","lastCommitMessage":"mtmd: add mtmd_bitmap_set_mergeable (#27348)"},{"name":"test-opt.cpp","kind":"file","size":40317,"lastCommitHash":"bcbddcd54f0d5c22eab180831fdea6484107112f","lastCommitWhen":"2025-08-26T22:14:38+02:00","lastCommitMessage":"tests : fix test-opt with GGML_BACKEND_DL (#15599)\n\n"},{"name":"test-peg-parser.cpp","kind":"file","size":624,"lastCommitHash":"566059a26b0ce8faec4ea053605719d399c64cc5","lastCommitWhen":"2026-03-06T21:01:00+01:00","lastCommitMessage":"Autoparser - complete refactoring of parser architecture (#18675)\n\n* Autoparser - full single commit squish\n\n* Final pre-merge changes: minor fixes, Kimi 2.5 model parser"},{"name":"test-pq2-row-shapes.cpp","kind":"file","size":4349,"lastCommitHash":"d2c9ddd053aa97ed19f9705cfccd487378d11950","lastCommitWhen":"2026-09-21T20:39:12+03:00","lastCommitMessage":"ggml-cpu: keep PQ2_0 rows that are not multiples of 256 on the Q8_0 activation path\n\nP2 from the review of 10df29881. The Q8_K activation switch was applied to every\nPQ2_0 tensor via the static trait table, but PQ2_0 stores 128 weights per block,\nso 128, 384, 640, ... are all legal row widths and appear in existing models.\nQ8_K only exists in 256-element blocks, so those rows cannot be expressed in it.\n\nGuarding the quantizer, as the previous commit did, does not help: tensors inside\nan already published model never pass through tensor_type_fallback. That guard is\nreverted here - it only cost PQ2_0 coverage on shapes that are now handled.\n\nSelection is now shape-aware. ggml_cpu_vec_dot_type / ggml_cpu_vec_dot return\nQ8_0 + ggml_vec_dot_pq2_0_q8_0 when ne[0] % QK_K != 0 and the trait-table entry\notherwise, and are used at all six sites that previously read the table directly,\nincluding both ggml_graph_plan work-size computations - a mismatch there would\nsize the activation buffer for a different type than the compute path consumes.\nllamafile_sgemm already dispatches on Btype, so those rows also pick up\ntinyBLAS_PQ2_AVX rather than falling back to the scalar dot.\n\nRows that are multiples of 256 - which is every PQ2_0 tensor in a typical model -\ntake an unchanged code path and keep the faster Q8_K kernels.\n\nNew test tests/test-pq2-row-shapes.cpp exercises pre-existing PQ2_0 tensors\nrather than the quantizer: it reference-encodes a PQ2_0 weight tensor the way a\n.gguf holds one, runs ggml_mul_mat through ggml_graph_compute_with_ctx for\nK in {128, 256, 384, 512, 640} x N in {1, 4, 16}, and checks the result against\ndequantized weights times exact float activations.\n\nBefore (Release, x86-64 AVX2 + AVX-VNNI) - the assert the report hit is not\nenabled in this configuration, so the failure is silently wrong output and reads\npast the activation buffer:\n\n K= 128 N= 1 FAIL rel_rms=1.0000 K= 384 N= 1 FAIL rel_rms=0.4793\n K= 128 N= 4 FAIL rel_rms=1.0000 K= 384 N= 4 FAIL rel_rms=0.9874\n K= 128 N=16 FAIL rel_rms=1.0000 K= 384 N=16 FAIL rel_rms=4.3e36\n K= 256 all OK rel_rms<=0.0053 K= 640 all FAIL up to rel_rms=3.3e36\n K= 512 all OK rel_rms<=0.0068\n\nAfter: 15/15 OK, rel_rms <= 0.0068. The K=256 and K=512 figures are unchanged to\nall digits, so the fast path is untouched.\n\nRegression checks:\n test-backend-ops -o MUL_MAT -b CPU 1294/1294\n test-backend-ops -o MUL_MAT_ID -b CPU 1022/1022\n test-quantize-fns pass\n llama-bench Bonsai-2-27B-PQ2_0 tg32 3.97 +/- 0.09 t/s (PR range 3.9-4.2)\n\nThat model's 402 PQ2_0 tensors are all 256-multiples, so its graph provably does\nnot change; the branch is evaluated once per matmul node, not per row.\n\naarch64 cross-build (gcc-aarch64-linux-gnu), all defined, none undefined:\n T ggml_vec_dot_pq2_0_q8_0\n T ggml_vec_dot_pq2_0_q8_0_generic\n T ggml_vec_dot_pq2_0_q8_K\n\nCo-Authored-By: Claude Opus 5 \nClaude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R\n"},{"name":"test-ptq1_0-cuda-dot.cpp","kind":"file","size":11480,"lastCommitHash":"3b19c377d18157bfec39ec71bad193e9ef000cf2","lastCommitWhen":"2026-09-23T09:04:45-07:00","lastCommitMessage":"rocm: vectorized HIP PTQ1_0 vec_dot (#211)\n\n* vectorized HIP PTQ1_0 vec_dot\n\nThe model file stores weights as densely packed base-3 digits (trits). To use\nthem, the GPU must unpack each trit and multiply-accumulate it with the\nactivation. The old AMD code did this one trit at a time in a long serial\nchain (128 slow steps per block). The new code unpacks 4 trits at once using\nthe GPU's byte-shuffle and 4-wide dot-product instructions. Same math, ~3x\nless instruction work, which is why decode got ~88% faster. A wrong guess\nabout a shuffle instruction's argument order was caught by on-device probes\nand unit tests before it could do harm; the corrected convention is now\ndocumented in a code comment.\n\n* tests : cover PTQ1_0 HIP vector unpack in host dot test\n\nTranscribe the vector idiom (unsigned digits, bias fold, swapped-operand\nperm, qh tail) and assert it against the scalar reference over all 256\nbyte values at every word position. Unwrap the vecdotq comment to one\nsentence per line.\n\nAssisted-by: Opencode with Muse Spark\n\n* ggml-cuda : point PTQ1_0 HIP branch at its host test mirror"},{"name":"test-ptq1_0-element-map.cpp","kind":"file","size":2671,"lastCommitHash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","lastCommitWhen":"2026-09-04T13:12:38-07:00","lastCommitMessage":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex"},{"name":"test-quant-type-selection.cpp","kind":"file","size":19113,"lastCommitHash":"47f686f53f2a20fe82d87d79fa5c835714ee8f46","lastCommitWhen":"2026-07-30T19:34:04+03:00","lastCommitMessage":"tests : avoid building get-model.cpp many times (#26317)\n\n* tests : remove get-model.cpp\n\n* tests : fix quant type selection"},{"name":"test-quantize-fns.cpp","kind":"file","size":13160,"lastCommitHash":"adfffbe41b2cabcd51fff326ab045662265062bb","lastCommitWhen":"2026-09-25T11:42:36-07:00","lastCommitMessage":"x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)\n\n* x86: add SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0\n\nPTQ1_0 had no x86 SIMD path at all - arch-fallback.h aliased it to the\ngeneric C implementation, so every x86 CPU decoded base-3 trits one\nweight at a time. PQ2_0 had an x86 kernel, but it is gated on VNNI\n(dpbusd), so everything below Ice Lake / Zen 4 fell through to the\nscalar loop.\n\nAdd a 128-bit path for both, in two tiers:\n\n SSE2 - trit decode by multiply-and-shift, sign-extended _mm_madd_epi16\n for the dot. Works on any x86-64.\n SSSE3 - _mm_maddubs_epi16 for the unsigned x signed multiply-add, which\n halves the ALU work in the inner loop.\n\nThe ternary offset (codes are 0..2, values -1..1) is folded into the\naccumulator as a separate maddubs against ones rather than subtracted\nper element, which avoids negating -128 activations.\n\nAlso add GGML_SSSE3 as a build variant so the dispatcher can score it,\nmatching how the other x86 feature tiers are handled.\n\nMeasured on an i5-12600K with AVX disabled at compile time, 8 threads.\nIsolated A/B: same binary, only libggml-cpu.so swapped, runs alternated\nto cancel background load, two pairs per configuration.\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.33/15.39 9.49/9.62\n SSE2 42.91/43.23 25.20/25.35\n SSSE3 59.49/58.98 32.07/31.42\n\n Ternary Bonsai 2 27B PTQ1_0 pp64 tg16\n scalar 0.71/0.72 0.60/0.62\n SSSE3 2.66/2.66 2.11/2.10\n\n3.3x tg on PQ2_0 and 3.5x on PTQ1_0 over scalar.\n\nAlso verified on a dual Xeon E5645 (Westmere, SSE4.2 ceiling, no AVX):\nbuilds clean and runs both formats, which is the hardware tier this is\nmainly for.\n\n* test-quantize-fns: exact vec_dot check for PTQ1_0 and PQ2_0\n\nThe existing test_vec_dot_q() compares against a tolerance, which is the\nright call for lossy formats but too loose to catch an unpack bug in a\nternary kernel: a wrong trit ordering still lands inside the error bound.\n\nAdd a check that is exact instead. Weights and activations are driven\nover 256 bit patterns per format, and every scale is a power of two, so\ndequantize-then-dot and the packed vec_dot must agree bit for bit. Any\nmismatch in trit order, plane split, or the -1 offset shows up as a hard\nfailure rather than a slightly larger error.\n\nCovers both block counts (1 and 3) so the multi-block loop is exercised.\n\nPasses on SSE2, SSSE3 and AVX2 builds, and on a dual Xeon E5645 where\nneither AVX path is available.\n\n* x86: follow PQ2_0 to the Q8_K activation path\n\nPQ2_0 moved to Q8_K activations, so ggml_vec_dot_pq2_0_q8_0 is no longer\nwhat the traits table dispatches and the SSE tier added earlier in this\nbranch was dead code for normal inference. Add the same 128-bit tier to\nggml_vec_dot_pq2_0_q8_K instead, below the AVX2 path.\n\nStructure mirrors the AVX2 kernel at half the width: the four sub-block\ndots of a 128-weight block accumulate in int32 and the block scale is\napplied once, rather than per sub-block. The existing unpack already\nemits byte b's bit-pair j at element 4b+j, which is the order the Q8_K\nactivations are in, so no repermutation is needed.\n\nCPUs with AVX2 keep the upstream kernel; this only changes what runs\nbelow it, where PQ2_0 was falling back to the scalar loop.\n\nAlso fix the ternary test to build the right activation type per format\n(Q8_K for PQ2_0, Q8_0 for PTQ1_0) and expand Q8_K by hand, since it is\nan activation-only type with no to_float. Verified the exact comparison\nstill catches errors: injecting an off-by-one in the accumulator fails\nall 512 PQ2_0 cases.\n\ni5-12600K, AVX disabled at compile time, 8 threads, isolated A/B\n(same binary, only libggml-cpu.so swapped, alternated, two pairs):\n\n Ternary Bonsai 1.7B PQ2_0 pp128 tg32\n scalar 15.04/15.12 9.97/9.83\n SSSE3 83.81/82.98 44.91/45.21\n\n4.5x tg. Higher than the 3.3x the Q8_0 path gave, because one activation\nscale per 256 removes the per-sub-block float chain.\n\ntest-quantize-fns passes on SSE2, SSSE3 and AVX2 builds."},{"name":"test-quantize-perf.cpp","kind":"file","size":13880,"lastCommitHash":"bee378e0988b44bbe93b0768208080951b312363","lastCommitWhen":"2025-09-25T08:06:06+03:00","lastCommitMessage":"ci: run the x64 and arm ci on the github machines instead (#16183)\n\n* run the x64 ci on regular machines\n\n* set up the same thing for arm\n\nfix test-quantize-perf just like #12306\n\n* try to disable sve\n\n* add another sve run"},{"name":"test-quantize-stats.cpp","kind":"file","size":16241,"lastCommitHash":"680a9ae63d60d35c21a0dcd7d3fabdb9c6bfc963","lastCommitWhen":"2026-08-12T14:15:03+02:00","lastCommitMessage":"cmake : introduce semantic versioning (#26839)\n\n* cmake : introduce semantic versioning (wip)\n\nThis commit introduces semantic versioning to llama.cpp.\n\n* squash! cmake : introduce semantic versioning (wip)\n\n* cmake : update test-cmake README notes [no ci]\n\n* include libmtmd in output so show its semversioned\n\n* ci : add make-release workflow\n\n* ci : fix build number check in build-cmake-pkg.yml\n\n* examples : remove trailing whitespace\n\n* ci : abort if upstream ggml version does not exist\n\n* ci : extract step contents into scripts\n\n* ci : add GGML_NATIVE=OFF to ubuntu job\n\n* examples : remove CI build information from test-cmake [no ci]\n\nThis commit removes the nightly/release information that I added\npreviously to keep this focused only on using building and installing\nllama.cpp with cmake and being able to quickly verify changes or\ntroubleshoot issues.\n\n* ci : merge scripts into single script\n\n* remove -dev-build_number support\n\nThis commit removes the incremental build number (versioning) support\nthat I added. This was incorrect and we should only use the semver for\nthe version. Releases will be tag a nightly build and package\nmaintainers/managers that build from source can use the tag and it is\ntherefor important that the correct version is reported. So a\nnightly-build will report the semver without the build number. The build\nnumber and commit as availble via cmake and test-cmake has been updated\nto include an example of using them:\n```console\n$ ./build.sh\n[test-cmake] version: 0.1.0, build: 10360 (08c69e381)\n...\n```\n\nRefs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969\n\n* docs: add initial release.md documentation\n\n* cmake : clean-up and add LLAMA_BUILD_IS_DEV option\n\n* ci : remove version input from make-release job\n\n* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml\n\nRefs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145\n\n* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]\n\n* ci : add TODO to winget workflow [no ci]\n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-reasoning-budget.cpp","kind":"file","size":22529,"lastCommitHash":"910196f6b3dfc6aca88fa732e2b02f270ff9b56b","lastCommitWhen":"2026-07-25T11:58:09+02:00","lastCommitMessage":"common : add support for multiple end sequences in the reasoning budget sampler (#25544)\n\n* common : extract trie/ac to a separate file\n\n* common : support multiple token sequences in the reasoning budget sampler\n\n* common/trie : return matched word index\n\n* common/trie : rename \"word\" to \"pattern\"\n\n* common/reasoning-budget : expose matched end sequence\n\n* common/sampling : replay end sequence when reasoning budget is done\n\n* cont : update to use multiple end sequences\n\n* cont : clean up"},{"name":"test-recurrent-state-rollback.cpp","kind":"file","size":15047,"lastCommitHash":"b0539c43ed13b16bf0d8a0840646faea65469702","lastCommitWhen":"2026-08-23T13:57:49+03:00","lastCommitMessage":"DeepseekV4: fix rollback with multi-seq (#26756)\n\n* DeepseekV4: fix rollback with multi-seq\n\n* fix model loading\n\n* make pending rollback single use\n\n* only clear cache for seq_id for full load\n\n* add assert for compress ratio\n\n* make graph topology static\n\n* pass true instead of flags in clear_compressed\n\n* cont : clean-up + TODOs\n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-rope.cpp","kind":"file","size":8147,"lastCommitHash":"73460f62789a3ed02109d601a2653c3842f2074e","lastCommitWhen":"2025-11-11T13:33:24+02:00","lastCommitMessage":"ggml-cpu: templateify ggml_compute_forward_rope_f32 and _f16 (#16805)\n\n* extract rotate_pairs logic from ggml_compute_forward_rope_f32\n\n* templateify ggml_compute_forward_rope_f32 and _f16\n\n* abort when rope type not supported, remove GLM from test-rope\n\n* add imrope branch to switch\n\n* add rope tests for perf\n\n* Update ggml/src/ggml-cpu/ops.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* Update ggml/src/ggml-cpu/ops.cpp\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-rset-release.cpp","kind":"file","size":1794,"lastCommitHash":"47f686f53f2a20fe82d87d79fa5c835714ee8f46","lastCommitWhen":"2026-07-30T19:34:04+03:00","lastCommitMessage":"tests : avoid building get-model.cpp many times (#26317)\n\n* tests : remove get-model.cpp\n\n* tests : fix quant type selection"},{"name":"test-sampling.cpp","kind":"file","size":17634,"lastCommitHash":"dd1ea524333b1e697489067d7a4c39c60d32beee","lastCommitWhen":"2026-08-10T16:58:56+03:00","lastCommitMessage":"llama : support multi-output backend sampling (#25532)\n\n* Enable backend sampling with token speculation\n\n* Clamp the mask sum before converting it into the sampled index\n\n* Add a numeric context parameter declaring the maximum outputs one sequence\n\n* More fixes\n\n* Don't reuse memory for output views.\n\n* Match dist between CPU and GPU\n\n* Fix CPU and backend sampling mismatches\n\n* Simpify some of the changes\n\n* Fix tests on Vulkan\n\n* More test fixes\n\n* Rebase changes\n\n* Rebase and address review comments\n\n* Address review comments\n\n* Address review comments\n\n* Update src/llama-sampler.cpp\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: Georgi Gerganov "},{"name":"test-save-load-state.cpp","kind":"file","size":14895,"lastCommitHash":"7ef790f90a4772eadd6966815f6d3ae7d93c7e03","lastCommitWhen":"2026-07-27T13:11:20+03:00","lastCommitMessage":"tests : remove unnecessary sync in test-save-load-state (#26166)"},{"name":"test-scr.cpp","kind":"file","size":4249,"lastCommitHash":"a33d3e21afd4bd4952a0b56fcd6dfee92e3f9ad3","lastCommitWhen":"2026-10-14T02:26:30-04:00","lastCommitMessage":"scr: retain up to 64 matching spans for fragmented transcripts\n"},{"name":"test-state-restore-fragmented.cpp","kind":"file","size":4528,"lastCommitHash":"bf76ac77bed3f53a6d35968019ee34078bebf701","lastCommitWhen":"2026-05-05T09:23:50+02:00","lastCommitMessage":"common : only load backends when required (#22290)\n\n* common : only load backends when required\n\nSigned-off-by: Adrien Gallouët \n\n* llama : call ggml_backend_load_all() directly from llama_backend_init()\n\nSigned-off-by: Adrien Gallouët \n\n* Add ggml_backend_load_all() where llama_backend_init() is not used\n\nSigned-off-by: Adrien Gallouët \n\n---------\n\nSigned-off-by: Adrien Gallouët "},{"name":"test-thread-safety.cpp","kind":"file","size":5730,"lastCommitHash":"fdbd6abee20e408de21e90ca77a24cd50a6ea073","lastCommitWhen":"2026-06-25T09:22:51+03:00","lastCommitMessage":"tests : synchronize contexts at end of test-thread-safety (#24935)\n\nAssisted-by: Claude"},{"name":"test-tokenizer-0.cpp","kind":"file","size":11579,"lastCommitHash":"147a5216363c5b2edcdeb17f01ef3b28372ec872","lastCommitWhen":"2025-12-22T11:00:37+01:00","lastCommitMessage":"tool/ex/tests: consistently free ctx, then model (#18168)\n\n"},{"name":"test-tokenizer-0.py","kind":"file","size":2004,"lastCommitHash":"c8ac02fa1b9f8a154e110b655f7f7e4907796b0c","lastCommitWhen":"2026-04-09T12:36:29+02:00","lastCommitMessage":"requirements : update transformers to 5.5.1 (#21617)\n\n* requirements : update transformers to 5.5.0\n\nThis commit updates the transformers dependency to version 5.5.0.\n\nThe motivation for this is that transformers 5.5.0 includes support for\nGemma4 and is required to be able to convert Gemma4 models. This is also\ncausing issues for user of gguf-my-repo.\n\nRefs: https://huggingface.co/spaces/ggml-org/gguf-my-repo/discussions/202\n\n* fix huggingface_hub version\n\n* set version of transformers to 5.5.0\n\n* convert : add ty ignore directives to convert_hf_to_gguf.py\n\nThis commit adds `ty: ignore` directives to transformers tokenizers\nfield/methods to avoid type check errors. There might be better ways to\nhandle this and perhaps this can be done in a follow up commit.\n\nThe motivation for this is that it looks like in transformers 5.5.0\nAutoTokenizer.from_pretrained can return generic tokenizer types or None\nand the type checker now produces an error when the conversion script\naccesses field like tokenizer.vocab.\n\n* convert : add ty ignore to suppress type check errors\n\n* convert : remove incorrect type ignores\n\n* convert : fix remaining python checks\n\nI was running a newer version of ty locally but I've switched to\nversion 0.0.26 which is what CI uses and I was then able to reproduce\nthe errors. Sorry about the noise.\n\n* update transformers version to 5.5.1"},{"name":"test-tokenizer-0.sh","kind":"file","size":1146,"lastCommitHash":"66287bdaaca3042bd4df5fc8f7bd4d58bfcd208c","lastCommitWhen":"2026-02-26T12:14:09+01:00","lastCommitMessage":"model : add Jina Embeddings v5 Nano (partial EuroBERT) support (#19826)\n\n* WIP: Add EuroBERT support with autoformatting changes\n\nThis commit includes:\n- EuroBERT model implementation for GGUF conversion\n- C++ backend support for EuroBERT architecture\n- Unintended autoformatting changes to Python files\n\nSaving before reverting formatting-only changes.\n\n* feat: add back eos assert when not last token pooling\n\n* feat: removed duplicated code and cleanup\n\n* feat: removed not working architectures and unnecessary check\n\n* fix: typo\n\n* fix: dynamic pooling config\n\n* feat: added an example model for eurobert\n\n* feat: proper llama-vocab implementation for jina-v5\n\n* fix: removed unnecessary comments"},{"name":"test-tokenizer-1-bpe.cpp","kind":"file","size":4867,"lastCommitHash":"147a5216363c5b2edcdeb17f01ef3b28372ec872","lastCommitWhen":"2025-12-22T11:00:37+01:00","lastCommitMessage":"tool/ex/tests: consistently free ctx, then model (#18168)\n\n"},{"name":"test-tokenizer-1-spm.cpp","kind":"file","size":3644,"lastCommitHash":"147a5216363c5b2edcdeb17f01ef3b28372ec872","lastCommitWhen":"2025-12-22T11:00:37+01:00","lastCommitMessage":"tool/ex/tests: consistently free ctx, then model (#18168)\n\n"},{"name":"test-tokenizer-random.py","kind":"file","size":22192,"lastCommitHash":"c8ac02fa1b9f8a154e110b655f7f7e4907796b0c","lastCommitWhen":"2026-04-09T12:36:29+02:00","lastCommitMessage":"requirements : update transformers to 5.5.1 (#21617)\n\n* requirements : update transformers to 5.5.0\n\nThis commit updates the transformers dependency to version 5.5.0.\n\nThe motivation for this is that transformers 5.5.0 includes support for\nGemma4 and is required to be able to convert Gemma4 models. This is also\ncausing issues for user of gguf-my-repo.\n\nRefs: https://huggingface.co/spaces/ggml-org/gguf-my-repo/discussions/202\n\n* fix huggingface_hub version\n\n* set version of transformers to 5.5.0\n\n* convert : add ty ignore directives to convert_hf_to_gguf.py\n\nThis commit adds `ty: ignore` directives to transformers tokenizers\nfield/methods to avoid type check errors. There might be better ways to\nhandle this and perhaps this can be done in a follow up commit.\n\nThe motivation for this is that it looks like in transformers 5.5.0\nAutoTokenizer.from_pretrained can return generic tokenizer types or None\nand the type checker now produces an error when the conversion script\naccesses field like tokenizer.vocab.\n\n* convert : add ty ignore to suppress type check errors\n\n* convert : remove incorrect type ignores\n\n* convert : fix remaining python checks\n\nI was running a newer version of ty locally but I've switched to\nversion 0.0.26 which is what CI uses and I was then able to reproduce\nthe errors. Sorry about the noise.\n\n* update transformers version to 5.5.1"},{"name":"test-tokenizers-repo.sh","kind":"file","size":893,"lastCommitHash":"624207e676ab5eb3ce7af631902bb45fb73a8359","lastCommitWhen":"2025-09-27T02:03:33+08:00","lastCommitMessage":"devops: add s390x & ppc64le CI (#15925)\n\n* devops: move s390x and ppc64le ci build\n\nwe have access to ubuntu-24.04-s390x and ppc64le images now\n\nSigned-off-by: Aaron Teo \n\n* devops: disable ppc64le for now since they have compiler errors\n\nSigned-off-by: Aaron Teo \n\n* devops: stop warnings as errors\n\nSigned-off-by: Aaron Teo \n\n* devops: switch to non-macro flag\n\nSigned-off-by: Aaron Teo \n\n* devops: going the llama macro route\n\nSigned-off-by: Aaron Teo \n\n* devops: add big-endian gguf test models\n\nSigned-off-by: Aaron Teo \n\n* devops: disable ppc64le to test s390x, check test build\n\nSigned-off-by: Aaron Teo \n\n* devops: dup .gguf.inp files for big-endian tests\n\nSigned-off-by: Aaron Teo \n\n* devops: dup .gguf.out files for big-endian too\n\nSigned-off-by: Aaron Teo \n\n* devops: add python setup and endian byteswap\n\nSigned-off-by: Aaron Teo \n\n* devops: pooring thing does not have s390x python3\n\nSigned-off-by: Aaron Teo \n\n* devops: add missing rust compiler for s390x\n\nSigned-off-by: Aaron Teo \n\n* devops: try rust actions runner\n\nSigned-off-by: Aaron Teo \n\n* Revert \"devops: try rust actions runner\"\n\nThis reverts commit 3f8db04356033d6c1d7eccc75ca396bc5298250c.\n\nSigned-off-by: Aaron Teo \n\n* devops: try a different path for rust\n\nSigned-off-by: Aaron Teo \n\n* devops: dump home directory and user info\n\nSigned-off-by: Aaron Teo \n\n* devops: install gguf-py only\n\nSigned-off-by: Aaron Teo \n\n* devops: missed relative path\n\nSigned-off-by: Aaron Teo \n\n* devops: remove big-endian files since local swapping is working\n\nSigned-off-by: Aaron Teo \n\n* devops: revert test-tokenizer-0 cmakelists\n\nSigned-off-by: Aaron Teo \n\n* Fix unicode flags conversion from and to uint16_t\n\nBitfields are allocated in different order on s390x\n\nSigned-off-by: Aaron Teo \n\n* Simplify byteswap command\n\nSigned-off-by: Aaron Teo \n\n* Add byteswapping and git-lfs for test-tokenizers-ggml-vocabs\n\nSigned-off-by: Aaron Teo \n\n* Fix endianness detection in vocab loader\n\nSigned-off-by: Aaron Teo \n\n* Disable test-thread-safety on s390x\n\nIn this test a model is downloaded,\nthen immediately loaded to check if more downloads are needed,\nand then used for test.\n\nThere is no clean way to separate all those steps\n to add byteswapping between them, so just skip this test.\n\nSigned-off-by: Aaron Teo \n\n* Fix q8_0 test in test-quantize-fns\n\nvec_signed uses unexpected rounding mode.\nExplicitly use different rounding function.\n\nSigned-off-by: Aaron Teo \n\n* devops: add big-endian stories260K\n\nSigned-off-by: Aaron Teo \n\n* devops: add s390x test-eval-callback\n\nSigned-off-by: Aaron Teo \n\n* devops: fix test does not exist\n\nSigned-off-by: Aaron Teo \n\n* devops: fix model not found llama-eval-callback\n\nSigned-off-by: Aaron Teo \n\n* Fix q3_K dot product error in test-quantize-fns on s390x\n\nArray q8bytes had only 4 elements allocated, but 8 elements accessed.\nThis lead to write out of bounds and later read of overwritten values out of bounds\nand incorrect result.\n\nSigned-off-by: Aaron Teo \n\n* devops: re-enable ppc64le for testing\n\nSigned-off-by: Aaron Teo \n\n* devops: activate test-thread-safety for s390x\n\nSigned-off-by: Aaron Teo \n\n* devops: disable ppc64le tests\n\nfor some reason it keeps failing test-thread-safety tests and I do not\n have a machine that is able to replicate the tests.\n\nSigned-off-by: Aaron Teo \n\n* devops: LLAMA_FATAL_WARNINGS=ON\n\nSigned-off-by: Aaron Teo \n\n* Correct repository URL for s390x for test-thread-safety model\n\nSigned-off-by: Aaron Teo \n\n* Fix fs_get_cache_directory\n\nEnsure it works even if both XDG_CACHE_HOME and HOME are unset.\nThis might happen in containers.\n\nSigned-off-by: Aaron Teo \n\n* Re-enable CI for ppc64le\n\nSigned-off-by: Aaron Teo \n\n* Fortify ggml_rope_impl\n\nOnly memcpy data from sections argument if it's non-NULL.\n\nSigned-off-by: Aaron Teo \n\n* Add TODO in struct unicode_cpt_flags to reimplement it in endian-independent way\n\n* Update URL for big-endian model\n\n* Update .github/workflows/build.yml\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Update remaining mentions of BE models to ggml-org/models repo\n\n---------\n\nSigned-off-by: Aaron Teo \nCo-authored-by: Aleksei Nikiforov \nCo-authored-by: Aleksei Nikiforov <103434461+AlekseiNikiforovIBM@users.noreply.github.com>\nCo-authored-by: Sigbjørn Skjæret "},{"name":"test-unicode.cpp","kind":"file","size":589,"lastCommitHash":"afd439df1f081ce0f6bc76965aa0a0ed762ecc91","lastCommitWhen":"2026-08-18T15:15:22+02:00","lastCommitMessage":"unicode : include '~' in collapsed symbol class (#26972)\n\nThe collapsed \\p{S} class was missing '~', which split \" ~\" into\nseparate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.\nThis caused re-tokenized prompts to diverge from sampled tokens and\nbroke KV cache reuse.\n\nAssisted-by: Codex"},{"name":"testing.h","kind":"file","size":7600,"lastCommitHash":"b94041a98ec86e29e63b02decd29175d6820b2a1","lastCommitWhen":"2026-08-16T12:45:33+02:00","lastCommitMessage":"chat: refactor handling supports_string_content / supports_typed_content (#27130)\n\n* better supports_string_content cap detect\n\n* test: add \"skip\"\n\n* messages_inp_normalizer"}]