[{"name":"batched-bench","kind":"directory","size":0,"lastCommitHash":"bbce619adb409880fb6db850a1c5a5f36a4dc7b1","lastCommitWhen":"2026-05-22T11:46:26+03:00","lastCommitMessage":"cmake : add install() for impl libraries + fix apple builds (#23511)\n\n* pi : update\n\n* ci : fix ios build\n\n* ci : fix andoroid\n\n* ci : fix apple builds\n\n* cmake : add install() for impl libraries\n\nAdd install(TARGETS <target> LIBRARY) for all -impl libraries that were\nchanged from STATIC to shared (controlled by BUILD_SHARED_LIBS) in\ncommit bb28c1fe2. Without this, cmake --install fails to copy the shared\nlibraries, causing runtime errors like:\n\n  llama-server: error while loading shared libraries: libllama-server-impl.so\n\nRef: https://github.com/ggml-org/llama.cpp/issues/23494#issuecomment-4512912515\n\nAssisted-by: llama.cpp:local pi\n\n* ci : fix xcframework build"},{"name":"cli","kind":"directory","size":0,"lastCommitHash":"d9f918d2d06079b4336688e819eee821c8a9cd9e","lastCommitWhen":"2026-08-22T16:28:28+02:00","lastCommitMessage":"common: add json.h abstraction (#27511)\n\n* add common/json\n\n* migrate common\n\n* adapt jinja\n\n* migrate server\n\n* big wip\n\n* migrate tests\n\n* wip\n\n* revert some excessive changes\n\n* wip\n\n* wip 2\n\n* revert redundant changes\n\n* fix server crash\n\n* various fixes\n\n* fix ci\n\n* harden a bit\n\n* clean up\n\n* rm json-shim\n\n* add some comments\n\n* rm redundant decl"},{"name":"completion","kind":"directory","size":0,"lastCommitHash":"0177dcc7300bad8914bb838baabce87899812491","lastCommitWhen":"2026-08-15T16:35:53+08:00","lastCommitMessage":"common: migrate the deprecated --mmap/--no-mmap to --load-mode (#26934)\n\nReplace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with\nthe unified --load-mode argument across scripts, examples, and documentation.\nInternal warning message and env var docs updated accordingly.\n\nSigned-off-by: Fathi Boudra <fathi.boudra@linaro.org>"},{"name":"cvector-generator","kind":"directory","size":0,"lastCommitHash":"680a9ae63d60d35c21a0dcd7d3fabdb9c6bfc963","lastCommitWhen":"2026-08-12T14:15:03+02:00","lastCommitMessage":"cmake :  introduce semantic versioning  (#26839)\n\n* cmake : introduce semantic versioning (wip)\n\nThis commit introduces semantic versioning to llama.cpp.\n\n* squash! cmake : introduce semantic versioning (wip)\n\n* cmake : update test-cmake README notes [no ci]\n\n* include libmtmd in output so show its semversioned\n\n* ci : add make-release workflow\n\n* ci : fix build number check in build-cmake-pkg.yml\n\n* examples : remove trailing whitespace\n\n* ci : abort if upstream ggml version does not exist\n\n* ci : extract step contents into scripts\n\n* ci : add GGML_NATIVE=OFF to ubuntu job\n\n* examples : remove CI build information from test-cmake [no ci]\n\nThis commit removes the nightly/release information that I added\npreviously to keep this focused only on using building and installing\nllama.cpp with cmake and being able to quickly verify changes or\ntroubleshoot issues.\n\n* ci : merge scripts into single script\n\n* remove -dev-build_number support\n\nThis commit removes the incremental build number (versioning) support\nthat I added. This was incorrect and we should only use the semver for\nthe version. Releases will be tag a nightly build and package\nmaintainers/managers that build from source can use the tag and it is\ntherefor important that the correct version is reported. So a\nnightly-build will report the semver without the build number. The build\nnumber and commit as availble via cmake and test-cmake has been updated\nto include an example of using them:\n```console\n$ ./build.sh\n[test-cmake] version: 0.1.0, build: 10360 (08c69e381)\n...\n```\n\nRefs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969\n\n* docs: add initial release.md documentation\n\n* cmake : clean-up and add LLAMA_BUILD_IS_DEV option\n\n* ci : remove version input from make-release job\n\n* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml\n\nRefs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145\n\n* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]\n\n* ci : add TODO to winget workflow [no ci]\n\n---------\n\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>"},{"name":"export-lora","kind":"directory","size":0,"lastCommitHash":"7b6c5a2aeda694dc3ea0479380ab27d0e63ce3d7","lastCommitWhen":"2026-06-18T16:46:17+02:00","lastCommitMessage":"docs: fix export-lora --lora-scaled syntax [no release] (#24703)\n\nAssisted-by: Codex"},{"name":"fit-params","kind":"directory","size":0,"lastCommitHash":"2fb989b9e79bf4da8159855e24892c8f4c20300f","lastCommitWhen":"2026-08-22T16:16:06+02:00","lastCommitMessage":"fit: also take into account n_streams (#27496)\n\n* fit: also take into account n_streams\n\n* server: make the draft context follow the target context\n\nWith a non-unified KV cache the target context now holds n_ctx_train\ntokens per sequence, while the draft context was still created with\nn_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot\nfilled beyond that point makes the draft batch fail to decode, and the\nserver answers 500 on the request.\n\nThe draft context now takes its size from the target context, so both\nhold the same number of tokens per sequence. Contexts that share their\ncells with the target no longer need the kv_size override.\n\nThe memory reserved for the draft model before fitting is measured at\nthe largest context the target can take, since the draft context grows\nwith the target and a fixed byte margin cannot express that.\n\n* fit: take an optional second model into account\n\nIllustrates the alternative discussed on the draft context fix. The\nmemory of a draft or MTP context is currently handed to the fit as a\nfixed byte margin, which cannot express a memory that grows with the\ncontext the fit is still deciding on.\n\ncommon_fit_params now takes an optional second model that shares the\ndevices of the main one. Its context follows the main context and its\nmemory is measured again whenever that context changes, so the reduce\npath stays exact instead of conservative. A model that cannot be\nmeasured on its own, such as a shared cell MTP context, is skipped with\na warning and the main model is fitted alone.\n\nThis drops the reservation block in the server, which no longer has to\nprobe the trained context size of the target to guess an upper bound.\n\n---------\n\nCo-authored-by: Pascal <admin@serveurperso.com>"},{"name":"gguf-split","kind":"directory","size":0,"lastCommitHash":"680a9ae63d60d35c21a0dcd7d3fabdb9c6bfc963","lastCommitWhen":"2026-08-12T14:15:03+02:00","lastCommitMessage":"cmake :  introduce semantic versioning  (#26839)\n\n* cmake : introduce semantic versioning (wip)\n\nThis commit introduces semantic versioning to llama.cpp.\n\n* squash! cmake : introduce semantic versioning (wip)\n\n* cmake : update test-cmake README notes [no ci]\n\n* include libmtmd in output so show its semversioned\n\n* ci : add make-release workflow\n\n* ci : fix build number check in build-cmake-pkg.yml\n\n* examples : remove trailing whitespace\n\n* ci : abort if upstream ggml version does not exist\n\n* ci : extract step contents into scripts\n\n* ci : add GGML_NATIVE=OFF to ubuntu job\n\n* examples : remove CI build information from test-cmake [no ci]\n\nThis commit removes the nightly/release information that I added\npreviously to keep this focused only on using building and installing\nllama.cpp with cmake and being able to quickly verify changes or\ntroubleshoot issues.\n\n* ci : merge scripts into single script\n\n* remove -dev-build_number support\n\nThis commit removes the incremental build number (versioning) support\nthat I added. This was incorrect and we should only use the semver for\nthe version. Releases will be tag a nightly build and package\nmaintainers/managers that build from source can use the tag and it is\ntherefor important that the correct version is reported. So a\nnightly-build will report the semver without the build number. The build\nnumber and commit as availble via cmake and test-cmake has been updated\nto include an example of using them:\n```console\n$ ./build.sh\n[test-cmake] version: 0.1.0, build: 10360 (08c69e381)\n...\n```\n\nRefs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969\n\n* docs: add initial release.md documentation\n\n* cmake : clean-up and add LLAMA_BUILD_IS_DEV option\n\n* ci : remove version input from make-release job\n\n* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml\n\nRefs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145\n\n* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]\n\n* ci : add TODO to winget workflow [no ci]\n\n---------\n\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>"},{"name":"imatrix","kind":"directory","size":0,"lastCommitHash":"38406d597fe651aafc7d843d270cb25b5ec2617a","lastCommitWhen":"2026-08-11T11:18:19-04:00","lastCommitMessage":"imatrix.cpp: Move finite check and only check touched experts (#26861)"},{"name":"kv-mean-center","kind":"directory","size":0,"lastCommitHash":"36d1deef278725a43aedcdd8d68a9be7815cbbe6","lastCommitWhen":"2026-08-25T05:55:11Z","lastCommitMessage":"kv-cache: mean-centering on hybrid models + calibration-basis guard (fork #53)\n"},{"name":"llama-bench","kind":"directory","size":0,"lastCommitHash":"2fb989b9e79bf4da8159855e24892c8f4c20300f","lastCommitWhen":"2026-08-22T16:16:06+02:00","lastCommitMessage":"fit: also take into account n_streams (#27496)\n\n* fit: also take into account n_streams\n\n* server: make the draft context follow the target context\n\nWith a non-unified KV cache the target context now holds n_ctx_train\ntokens per sequence, while the draft context was still created with\nn_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot\nfilled beyond that point makes the draft batch fail to decode, and the\nserver answers 500 on the request.\n\nThe draft context now takes its size from the target context, so both\nhold the same number of tokens per sequence. Contexts that share their\ncells with the target no longer need the kv_size override.\n\nThe memory reserved for the draft model before fitting is measured at\nthe largest context the target can take, since the draft context grows\nwith the target and a fixed byte margin cannot express that.\n\n* fit: take an optional second model into account\n\nIllustrates the alternative discussed on the draft context fix. The\nmemory of a draft or MTP context is currently handed to the fit as a\nfixed byte margin, which cannot express a memory that grows with the\ncontext the fit is still deciding on.\n\ncommon_fit_params now takes an optional second model that shares the\ndevices of the main one. Its context follows the main context and its\nmemory is measured again whenever that context changes, so the reduce\npath stays exact instead of conservative. A model that cannot be\nmeasured on its own, such as a shared cell MTP context, is skipped with\na warning and the main model is fitted alone.\n\nThis drops the reservation block in the server, which no longer has to\nprobe the trained context size of the target to guess an upper bound.\n\n---------\n\nCo-authored-by: Pascal <admin@serveurperso.com>"},{"name":"mtmd","kind":"directory","size":0,"lastCommitHash":"160c6b0bdd9739d6c8c8539e7067f64e8ea57067","lastCommitWhen":"2026-08-24T09:59:04+02:00","lastCommitMessage":"mtmd: video: fix moov atom at the end of file (#27596)\n\n* mtmd: video: fix moov at the end of file\n\nCo-authored-by: rkfg <rkfg@rkfg.me>\n\n* fix SIGPIPE\n\n* windows: handle broken pipe case\n\n---------\n\nCo-authored-by: rkfg <rkfg@rkfg.me>"},{"name":"perplexity","kind":"directory","size":0,"lastCommitHash":"0596704284bffe8350eef4a9510316258ed6b87a","lastCommitWhen":"2026-08-18T16:22:32+02:00","lastCommitMessage":"quant : Optimise memory usage by evicting weights after processing each layer (#22877)\n\n* Evict weights from memory after processing each layer\n\n* Revert changes\n\n* Move unmap to libllama\n\n* Unmap weights offloaded to backend\n\n* Change member's constness\n\n* Remove unmap weights offloaded to backend"},{"name":"quantize","kind":"directory","size":0,"lastCommitHash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","lastCommitWhen":"2026-09-04T13:12:38-07:00","lastCommitMessage":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n  qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n  ggml_quantize_chunk   - without a dispatch case the returned byte count did not\n                          match the row size and GGML_ASSERT(result == nrows *\n                          row_size) aborted a worker thread mid-tensor, leaving a\n                          truncated output file behind.\n  ggml_ftype -> wtype   - the ftype could not resolve to a ggml type.\n  validate_row_data     - \"invalid type 143\", so quantization refused the result\n                          it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n  quantized tensors  PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n                     (1.75/2.125 bpw predicts 17.65%)\n  perplexity, 4 chunks at ctx 512, identical to every printed digit:\n                     per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n                     final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n                    PQ2_0            PTQ1_0\n  pp128        4505 tok/s       2590 tok/s   0.57x\n  tg32          182.5 tok/s      43.6 tok/s   0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n  1. byte-owning threads         29.8 -> 136.4   a thread now consumes all five\n     trits of a byte it loads once, instead of eight threads each reloading the\n     same qs bytes for one trit index; the block's 26 bytes are read once total\n  2. y staged in registers      136.4 -> 143.5   y was being re-read from device\n     memory once per row; it is now gathered once per block and reused, with the\n     ternary offset folded into a single sumy subtraction\n  3. N_R0 8 -> 4                143.5 -> 163.8   rows-per-thread was the real\n     limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n  4. balanced qh                163.8 -> 165.3   qh's 8 elements went to threads\n     0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n     simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n  pp128   4541 -> 3998 tok/s   0.88x\n  tg32     186 ->  165 tok/s   0.89x\n  size    357 ->  294 MiB      17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n  256-entry trit lookup table       142.1  vs 143.5 recurrence   no decode gain\n                                    (helps mul_mm, which keeps it)\n  independent pow3 extraction       159.8  vs 166.0 recurrence   WORSE\n  contiguous byte ownership         162.0  vs 165.3 scattered    WORSE\n  N_R0 = 3 / 5 / 6                  129.8 / 156.7 / 149.6        all worse than 4\n  N_SG = 8                          161.1                        worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n  PQ2_0   128 x (shift, mask, cvt, fma)                = 512 ops, 34 B read\n  PTQ1_0  128 x (mul, shift, cvt, fma, mask-for-next)  = 640 ops, 28 B read\n          ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n  exact integer-accumulator checks   20000, mismatches 0\n  error / sum of term magnitudes     2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n  vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n  test-backend-ops -o MUL_MAT   1283/1283 passed, Backend CUDA0: OK\n                                45 ptq1_0 cases OK, 0 failures\n  host element-map test         2,560,000 positions, 0 mismatches\n  host MMVQ dot test            20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n        n=1     n=2     n=3     n=8    n=512\n  q1_0   8.88    9.82   13.05   30.14   225.4\n  pq2_0  9.75   11.05   14.92   31.27   232.4\n  ptq1_0 46.8    64.6    99.1   287.3   525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n   vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n   dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n   the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n   falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n   dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n   and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n   earlier, same helper, different backend**: defined in the dequantize chain, called\n   from the matmul chain. Fixed the same way, with a shared header\n   vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n   duplicated: two copies would have to stay in step with the CPU codec in\n   ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n   build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n   pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n   comment recording that the absence is deliberate. (1) and (3) had to be fixed\n   together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n  -DGGML_VULKAN=ON            builds clean\n  SPIR-V generated            1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n  test-backend-ops MUL_MAT    1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex"},{"name":"results","kind":"directory","size":0,"lastCommitHash":"6990e2f1f7581d332a6a1f34d6c567be70138184","lastCommitWhen":"2026-04-17T11:11:46+03:00","lastCommitMessage":"libs : rename libcommon -> libllama-common (#21936)\n\n* cmake : allow libcommon to be shared\n\n* cmake : rename libcommon to libllama-common\n\n* cont : set -fPIC for httplib\n\n* cont : export all symbols\n\n* cont : fix build_info exports\n\n* libs : add libllama-common-base\n\n* log : add common_log_get_verbosity_thold()"},{"name":"rpc","kind":"directory","size":0,"lastCommitHash":"c299a92c38b6de6a1139617652b66081828648db","lastCommitWhen":"2026-06-27T10:31:29+03:00","lastCommitMessage":"binaries : Improve rpc-server and export-graph-ops names. (#25045)\n\nTests are generally prefixed with -test, so rename export-graph-ops\naccordingly.\n\nrpc-server is probably too generic a name for /usr/bin. Because it\nshould work with any ggml application, it is renamed to ggml-rpc-server."},{"name":"scr-eval","kind":"directory","size":0,"lastCommitHash":"904cb4ae461799612c62c826737bb23ddc3ec7fb","lastCommitWhen":"2026-10-13T19:13:57-04:00","lastCommitMessage":"bonsai: suffix cache reuse (SCR) for linear attention hybrids (scr)\n\nSuffix cache reuse: after an edit in the middle of a cached prompt (compaction, a trimmed tool output, a system prompt change), the attention KV of the text after the edit is moved to its new positions (q8_0 keys re-rotated) and the DeltaNet layers continue from the pre-edit state, so only the new text is prefilled. Approximate (KL ~0.017 nats/token after a compaction); per request \"scr\", server default --scr.\n"},{"name":"server","kind":"directory","size":0,"lastCommitHash":"8906fd8c9fd9d2dcbffad469432b8d2edf0925fd","lastCommitWhen":"2026-10-18T08:02:45-04:00","lastCommitMessage":"cache: conversations are stored against their learned prefix\n"},{"name":"tokenize","kind":"directory","size":0,"lastCommitHash":"aff6eb6e7503538fec1532dec2f584bc7a4a4e4d","lastCommitWhen":"2026-07-15T18:41:51+02:00","lastCommitMessage":"tokenize : drop --stdin mutual-exclusion check (#25672)\n\nmatch cli and completion, which don't enforce it"},{"name":"tts","kind":"directory","size":0,"lastCommitHash":"681c29d36a13be54d317ee147b272da9163dbef3","lastCommitWhen":"2026-08-20T18:45:37+02:00","lastCommitMessage":"mtmd: add --mmproj-device argument (#23255)\n\n* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var\n\n* feat: load mmproj device backend immediately, add -mmdev shortflag\n\n* fix: its a pointer now get the name\n\n* clean up\n\n* gen docs\n\n* nits\n\n---------\n\nCo-authored-by: Xuan Son Nguyen <son@huggingface.co>"},{"name":"tuning","kind":"directory","size":0,"lastCommitHash":"f280b26983ad0fdb705a0d9ebf0503e76f2899b0","lastCommitWhen":"2026-08-24T19:22:27+03:00","lastCommitMessage":"metal : per-device tuned (Q, NE) for flash-attn vec (#26570)\n\n* metal : per-device tuned (Q, NE) for flash-attn vec (#25750)\n\n* rebase Q-generic FA vec body from 01dc93607 (#23114)\n\n* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)\n\n* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback\n\n* add  FA vec (Q,NE) perf sweep\n\n* fill tuning result\n\n* fold family table into a per-family representative SKU\n\n* refactor tuning result format\n\n* extend FA vec tuning to quantized KV caches\n\n* sync fa vec tuner bucketing with runtime, use pointwise tuning regret\n\n* update tuned table\n\n* format and cleanup\n\n* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active\n\n* add device id -> token lookup for the offline tuning tool\n\n* add ggml-metal-tuning skeleton\n\n* add op-agnostic perf cell + median timing for the tuner\n\n* add FA-vec graph build + tensor init to the tuner\n\n* tools : add FA-vec (Q,NE) sweep, compression and table emit\n\n* cool down and re-measure the dirty window on thermal drift\n\n* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice\n\n* tools : document the Metal tuner, point the table comment at it\n\n* abort on unknown KV type, single-source fa_vec_legal_ne\n\n* cleanup\n\n* honor -o in the FA vec (Q,NE) slice\n\n* retune FA-vec (Q, NE) under a pointwise no-harm gate\n\n* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max\n\n---------\n\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>"},{"name":"ui","kind":"directory","size":0,"lastCommitHash":"d05f89562d50e16e8c025081aa03bf3e449efd82","lastCommitWhen":"2026-08-23T19:37:19+02:00","lastCommitMessage":"fix: Change chat tabs nav shortcuts (#27609)"},{"name":"CMakeLists.txt","kind":"file","size":1083,"lastCommitHash":"904cb4ae461799612c62c826737bb23ddc3ec7fb","lastCommitWhen":"2026-10-13T19:13:57-04:00","lastCommitMessage":"bonsai: suffix cache reuse (SCR) for linear attention hybrids (scr)\n\nSuffix cache reuse: after an edit in the middle of a cached prompt (compaction, a trimmed tool output, a system prompt change), the attention KV of the text after the edit is moved to its new positions (q8_0 keys re-rotated) and the DeltaNet layers continue from the pre-edit state, so only the new text is prefilled. Approximate (KL ~0.017 nats/token after a compaction); per request \"scr\", server default --scr.\n"}]