[{"name":"templates","kind":"directory","size":0,"lastCommitHash":"ad1de39e0708e3ced9c71bb3c82d93a2c046a73f","lastCommitWhen":"2026-08-15T17:11:05+02:00","lastCommitMessage":"model: add Kimi-K3 text model (#26185)\n\n* model: add Kimi-K3 text model\n\nHybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five\nthings that architecture does not have:\n\n  1. cross-layer residual attention  (attn_res_block_size)\n  2. latent MoE                      (routed experts run at n_expert_latent)\n  3. situ activation                 (replaces SwiGLU everywhere)\n  4. MLA output gate                 (sigmoid gate before o_proj)\n  5. full-rank KDA gate              (single ssm_g instead of ssm_g_a/ssm_g_b)\n\nK3's text_config reports KimiLinearForCausalLM - the older 48B architecture -\nso get_model_architecture routes on the top-level name instead.\n\nThe KDA decay gate has two forms, selected by linear_attn_config's\ngate_lower_bound. It is not a clamp: when set it swaps the activation entirely\n(fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to\nlower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it\nunset, so that path is unchanged.\n\nCross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is\nCPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels\nexist.\n\nThe routed experts ship as compressed-tensors \"mxfp4-pack-quantized\". That is\nbit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale\nbyte, only the nibble positions within a block differ - so they are repacked\nrather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip.\nThe repack is built lazily because gguf_writer holds every added tensor until\nthe final write. DeepSeek-V4 was already doing the identical bit-shuffling, so\nit now shares the helper.\n\nVerified against Moonshot's own code path (transformers + fla's Triton KDA\nkernels) on a tiny model exercising every K3-specific feature. Final-position\nlogits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the\nchunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source\nweights with 0.0e+00 error.\n\nAssisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>\n\n* model: fix ty errors in the Kimi-K3 converter\n\n- `_res_parts` buffers (kind, tensor) pairs, not bare tensors\n- `get_tensors` must return an Iterator, matching ModelBase\n- LazyBase's `func` takes one argument, so pass the expert loaders through\n  `args` instead of the closure\n- borrowing KimiLinearModel.set_vocab from an unrelated TextModel is\n  deliberate and safe, but not expressible in the signature\n\nNo behaviour change: the MXFP4 repack still dequantizes to the source weights\nwith 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel,\ncorr 0.99996630).\n\nAssisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>\n\n* Update conversion/kimi_k3.py\n\nCo-authored-by: Boris Dvorkin  <b_dvorkin@niuitmo.ru>\n\n* Increase LLAMA_MAX_EXPERTS from 512 to 1024\n\n* tests : support for Kimi K3 in archs test\n\n* chat : add Kimi K3 chat format (reasoning, content, typed tool calls)\n\nK3's assistant output is an XTML-ish tagged format built by the template's\nopen_tag/close_tag macros. Two properties break generic parsing:\n\n1. The generation prompt ends with open_tag('think'), so the completion\n   starts inside the think section with no opening marker in the output\n   (thinking_forced_open).\n2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag\n   names (\"think\", \"response\", \"message\") are ordinary text tokens.\n\nAdds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the\nmarker trio, reasoning extraction, response unwrapping, and tool-call\nparsing of the tools/call/argument tag structure with argument types\ntaken from the tool schema. Includes the K3 chat template fixture and 9\ntest-chat cases derived from real generations of the full 2.8T model.\n\nVerified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:\ncontent, reasoning_content, streaming deltas, and tool_calls all correct;\nfinish_reason stop/tool_calls as appropriate.\n\nCo-Authored-By: Claude Fable 5 <noreply@anthropic.com>\n\n* chat : add message_delimiters for Kimi K3\n\nPer-role message-start markers for token-level span splitting. User and\nassistant messages carry only the role attribute, so their full opener\n(through <|sep|>) is used; system and tool messages continue with more\nattributes (type=/tool=/index=), so those delimiters stop after the\nrole's closing quote. Verified against the K3 tiktoken vocabulary that\nthe closing quote is always a standalone token across all attribute\nvariants, so the token-level prefix match stays exact.\n\nCo-Authored-By: Claude Fable 5 <noreply@anthropic.com>\n\n* fix: apply nits from @ngxson and text fixes from @danielhanchen\n\n* tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs\n\n* chore : move overly verbose header file comments to Kimi K3 source file\n\n* tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend\n\n* model-saver : emit kda_gate_lower_bound for Kimi K3\n\nQuick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model\nsaver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0.\n\nI propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line).\n\n* Refactor conditional for model architecture check\n\n* tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend\n\n* fix code comments\n\n* add template on conversion\n\n* move repack_mxfp4_blocks to model base\n\n* nits\n\n* add_value_length\n\n* optimize res_stack construction\n\n* nits\n\n---------\n\nCo-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>\nCo-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>\nCo-authored-by: Deepankar Singh <singh.deepankar39@gmail.com>\nCo-authored-by: Claude Fable 5 <noreply@anthropic.com>\nCo-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com>\nCo-authored-by: Xuan Son Nguyen <son@huggingface.co>"},{"name":".editorconfig","kind":"file","size":12},{"name":"ggml-vocab-aquila.gguf","kind":"file","size":4825676},{"name":"ggml-vocab-baichuan.gguf","kind":"file","size":1340998},{"name":"ggml-vocab-bert-bge.gguf","kind":"file","size":627549,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-bert-bge.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-bert-bge.gguf.out","kind":"file","size":1636,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-command-r.gguf","kind":"file","size":10874545,"lastCommitHash":"889bdd76866ea31a7625ec2dcea63ff469f3e981","lastCommitWhen":"2024-05-05T08:19:30+03:00","lastCommitMessage":"command-r : add BPE pre-tokenization (#7063)\n\n* Add BPE pre-tokenization for Command-R/R+.\r\n\r\n* Bump transformers convert requirement.\r\n\r\n* command-r : add individual digits regex\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>"},{"name":"ggml-vocab-command-r.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-command-r.gguf.out","kind":"file","size":1905,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-deepseek-coder.gguf","kind":"file","size":1156067,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-deepseek-coder.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-deepseek-coder.gguf.out","kind":"file","size":2096,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-deepseek-llm.gguf","kind":"file","size":3970167,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-deepseek-llm.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-deepseek-llm.gguf.out","kind":"file","size":1920,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-falcon.gguf","kind":"file","size":2287728,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-falcon.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-falcon.gguf.out","kind":"file","size":2002,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-gemma-4.gguf","kind":"file","size":15776467,"lastCommitHash":"0ec191e1d761a4daec873dfd73278ee356944fe5","lastCommitWhen":"2026-04-09T11:41:14+02:00","lastCommitMessage":"vocab: add gemma4 tokenizer tests, fix edge case (#21534)\n\n* YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!\n* Remove unnecessary hash  from update script.\n* minor: move constant"},{"name":"ggml-vocab-gemma-4.gguf.inp","kind":"file","size":1922,"lastCommitHash":"0ec191e1d761a4daec873dfd73278ee356944fe5","lastCommitWhen":"2026-04-09T11:41:14+02:00","lastCommitMessage":"vocab: add gemma4 tokenizer tests, fix edge case (#21534)\n\n* YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!\n* Remove unnecessary hash  from update script.\n* minor: move constant"},{"name":"ggml-vocab-gemma-4.gguf.out","kind":"file","size":2185,"lastCommitHash":"0ec191e1d761a4daec873dfd73278ee356944fe5","lastCommitWhen":"2026-04-09T11:41:14+02:00","lastCommitMessage":"vocab: add gemma4 tokenizer tests, fix edge case (#21534)\n\n* YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!\n* Remove unnecessary hash  from update script.\n* minor: move constant"},{"name":"ggml-vocab-gpt-2.gguf","kind":"file","size":1766807,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-gpt-2.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-gpt-2.gguf.out","kind":"file","size":2147,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-gpt-neox.gguf","kind":"file","size":1771431},{"name":"ggml-vocab-llama-bpe.gguf","kind":"file","size":7818140,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-llama-bpe.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-llama-bpe.gguf.out","kind":"file","size":1739,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-llama-spm.gguf","kind":"file","size":723869,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-llama-spm.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-llama-spm.gguf.out","kind":"file","size":2685,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-mpt.gguf","kind":"file","size":1771393,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-mpt.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-mpt.gguf.out","kind":"file","size":1895,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-nomic-bert-moe.gguf","kind":"file","size":6821877,"lastCommitHash":"aa50ba462f63759e1731227f20f5009cfc2a0f16","lastCommitWhen":"2025-05-25T16:22:29+02:00","lastCommitMessage":"tests : improve UGM tokenizer test coverage (#13773)\n\n"},{"name":"ggml-vocab-phi-3.gguf","kind":"file","size":726019,"lastCommitHash":"3b38d48609280aa5f8ab7ea135a4351b2a5ee240","lastCommitWhen":"2024-06-04T09:17:17+02:00","lastCommitMessage":"Per token attributes (#7685)\n\n* Add per token attributes enum\r\n* Using phi-3 for testing 'rstrip'\r\n* Using jina-v2 for testing 'lstrip'\r\n* Brute force test for 'lstrip' and 'rstrip'\r\n* Implement 'rstrip' and 'lstrip'\r\n* Update phi-3 GGUF file (obsolete since 917dc8c)\r\n* Replace llama_token_type with llama_token_attribs\r\n"},{"name":"ggml-vocab-phi-3.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-phi-3.gguf.out","kind":"file","size":2685,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-qwen2.gguf","kind":"file","size":5928681,"lastCommitHash":"229ffff872f8ad0d21c997d18ee7a23692ae60a0","lastCommitWhen":"2024-05-08T15:06:43+03:00","lastCommitMessage":"llama : add BPE pre-tokenization for Qwen2 (#7114)\n\n* Add BPE pre-tokenization for Qwen2.\r\n\r\n* minor : fixes\r\n\r\n---------\r\n\r\nCo-authored-by: Ren Xuancheng <17811943+jklj077@users.noreply.github.com>\r\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>"},{"name":"ggml-vocab-qwen2.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-qwen2.gguf.out","kind":"file","size":1765,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-qwen35.gguf","kind":"file","size":5928682,"lastCommitHash":"42532afff43910e619a650c1704525b3acbbec5a","lastCommitWhen":"2026-05-14T11:03:40+02:00","lastCommitMessage":"unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)\n\n* unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests\n\n- Add unicode_regex_split_custom_qwen35() to [src/unicode.cpp](src/unicode.cpp), a non-backtracking handler for Qwen3.5's [\\p{L}\\p{M}]+ regex (letters + combining marks).\n- Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes #21919).\n- Add [models/ggml-vocab-qwen35.gguf](models/ggml-vocab-qwen35.gguf) (test vocab), [models/ggml-vocab-qwen35.gguf.inp](models/ggml-vocab-qwen35.gguf.inp) (test cases), and [models/ggml-vocab-qwen35.gguf.out](models/ggml-vocab-qwen35.gguf.out) (expected output) for regression testing.\n- Update [tests/CMakeLists.txt](tests/CMakeLists.txt) to include the new test entry.\n\nThis mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.\n\nCloses #21919.\n\n* fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks\n\n* cont : remove trailing whitespace\n\n---------\n\nCo-authored-by: Kabir <kabir@example.com>\nCo-authored-by: Alde Rojas <hello@alde.dev>"},{"name":"ggml-vocab-qwen35.gguf.inp","kind":"file","size":2651,"lastCommitHash":"42532afff43910e619a650c1704525b3acbbec5a","lastCommitWhen":"2026-05-14T11:03:40+02:00","lastCommitMessage":"unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)\n\n* unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests\n\n- Add unicode_regex_split_custom_qwen35() to [src/unicode.cpp](src/unicode.cpp), a non-backtracking handler for Qwen3.5's [\\p{L}\\p{M}]+ regex (letters + combining marks).\n- Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes #21919).\n- Add [models/ggml-vocab-qwen35.gguf](models/ggml-vocab-qwen35.gguf) (test vocab), [models/ggml-vocab-qwen35.gguf.inp](models/ggml-vocab-qwen35.gguf.inp) (test cases), and [models/ggml-vocab-qwen35.gguf.out](models/ggml-vocab-qwen35.gguf.out) (expected output) for regression testing.\n- Update [tests/CMakeLists.txt](tests/CMakeLists.txt) to include the new test entry.\n\nThis mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.\n\nCloses #21919.\n\n* fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks\n\n* cont : remove trailing whitespace\n\n---------\n\nCo-authored-by: Kabir <kabir@example.com>\nCo-authored-by: Alde Rojas <hello@alde.dev>"},{"name":"ggml-vocab-qwen35.gguf.out","kind":"file","size":3570,"lastCommitHash":"42532afff43910e619a650c1704525b3acbbec5a","lastCommitWhen":"2026-05-14T11:03:40+02:00","lastCommitMessage":"unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)\n\n* unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests\n\n- Add unicode_regex_split_custom_qwen35() to [src/unicode.cpp](src/unicode.cpp), a non-backtracking handler for Qwen3.5's [\\p{L}\\p{M}]+ regex (letters + combining marks).\n- Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes #21919).\n- Add [models/ggml-vocab-qwen35.gguf](models/ggml-vocab-qwen35.gguf) (test vocab), [models/ggml-vocab-qwen35.gguf.inp](models/ggml-vocab-qwen35.gguf.inp) (test cases), and [models/ggml-vocab-qwen35.gguf.out](models/ggml-vocab-qwen35.gguf.out) (expected output) for regression testing.\n- Update [tests/CMakeLists.txt](tests/CMakeLists.txt) to include the new test entry.\n\nThis mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.\n\nCloses #21919.\n\n* fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks\n\n* cont : remove trailing whitespace\n\n---------\n\nCo-authored-by: Kabir <kabir@example.com>\nCo-authored-by: Alde Rojas <hello@alde.dev>"},{"name":"ggml-vocab-refact.gguf","kind":"file","size":1720710,"lastCommitHash":"92139b90af4841d7fd060b526bdd443b621770ff","lastCommitWhen":"2024-05-04T08:32:32+03:00","lastCommitMessage":"tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)\n\n* tests : add test-tokenizer-0.sh\r\n\r\n* unicode : add all unicode number ranges\r\n\r\n* starcoder : fix pre-tokenizer\r\n\r\n* tests : add test that fails with DeepSeek tokenizers\r\n\r\n* falcon : fix regex\r\n\r\n* unicode : regenerate unicode tables\r\n\r\n* refact : add tokenizer model\r\n\r\n* lint : fix\r\n\r\n* tests : disable failing tests\r\n\r\nggml-ci\r\n\r\n* refact : add tests files\r\n\r\nggml-ci\r\n\r\n* convert : print -> logging\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* unicode : digit -> number\r\n\r\n* phi-3 : update"},{"name":"ggml-vocab-refact.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-refact.gguf.out","kind":"file","size":1918,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-starcoder.gguf","kind":"file","size":1719346,"lastCommitHash":"f4ab2a41476600a98067a9474ea8f9e6db41bcfa","lastCommitWhen":"2024-04-29T16:58:41+03:00","lastCommitMessage":"llama : fix BPE pre-tokenization (#6920)\n\n* merged the changes from deepseeker models to main branch\r\n\r\n* Moved regex patterns to unicode.cpp and updated unicode.h\r\n\r\n* Moved header files\r\n\r\n* Resolved issues\r\n\r\n* added and refactored unicode_regex_split and related functions\r\n\r\n* Updated/merged the deepseek coder pr\r\n\r\n* Refactored code\r\n\r\n* Adding unicode regex mappings\r\n\r\n* Adding unicode regex function\r\n\r\n* Added needed functionality, testing remains\r\n\r\n* Fixed issues\r\n\r\n* Fixed issue with gpt2 regex custom preprocessor\r\n\r\n* unicode : fix? unicode_wstring_to_utf8\r\n\r\n* lint : fix whitespaces\r\n\r\n* tests : add tokenizer tests for numbers\r\n\r\n* unicode : remove redundant headers\r\n\r\n* tests : remove and rename tokenizer test scripts\r\n\r\n* tests : add sample usage\r\n\r\n* gguf-py : reader prints warnings on duplicate keys\r\n\r\n* llama : towards llama3 tokenization support (wip)\r\n\r\n* unicode : shot in the dark to fix tests on Windows\r\n\r\n* unicode : first try custom implementations\r\n\r\n* convert : add \"tokenizer.ggml.pre\" GGUF KV (wip)\r\n\r\n* llama : use new pre-tokenizer type\r\n\r\n* convert : fix pre-tokenizer type writing\r\n\r\n* lint : fix\r\n\r\n* make : add test-tokenizer-0-llama-v3\r\n\r\n* wip\r\n\r\n* models : add llama v3 vocab file\r\n\r\n* llama : adapt punctuation regex + add llama 3 regex\r\n\r\n* minor\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : set bomb\r\n\r\n* unicode : always use std::wregex\r\n\r\n* unicode : support \\p{N}, \\p{L} and \\p{P} natively\r\n\r\n* unicode : try fix windows\r\n\r\n* unicode : category support via std::regex\r\n\r\n* unicode : clean-up\r\n\r\n* unicode : simplify\r\n\r\n* convert : add convert-hf-to-gguf-update.py\r\n\r\nggml-ci\r\n\r\n* lint : update\r\n\r\n* convert : add falcon\r\n\r\nggml-ci\r\n\r\n* unicode : normalize signatures\r\n\r\n* lint : fix\r\n\r\n* lint : fix\r\n\r\n* convert : remove unused functions\r\n\r\n* convert : add comments\r\n\r\n* convert : exercise contractions\r\n\r\nggml-ci\r\n\r\n* lint : fix\r\n\r\n* cmake : refactor test targets\r\n\r\n* tests : refactor vocab tests\r\n\r\nggml-ci\r\n\r\n* tests : add more vocabs and tests\r\n\r\nggml-ci\r\n\r\n* unicode : cleanup\r\n\r\n* scripts : ignore new update script in check-requirements.sh\r\n\r\n* models : add phi-3, mpt, gpt-2, starcoder\r\n\r\n* tests : disable obsolete\r\n\r\nggml-ci\r\n\r\n* tests : use faster bpe test\r\n\r\nggml-ci\r\n\r\n* llama : more prominent warning for old BPE models\r\n\r\n* tests : disable test-tokenizer-1-bpe due to slowness\r\n\r\nggml-ci\r\n\r\n---------\r\n\r\nCo-authored-by: Jaggzh <jaggz.h@gmail.com>\r\nCo-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>"},{"name":"ggml-vocab-starcoder.gguf.inp","kind":"file","size":1943,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"},{"name":"ggml-vocab-starcoder.gguf.out","kind":"file","size":1920,"lastCommitHash":"07e4351ce663a7802b31f92fd7bc0e555c2044b6","lastCommitWhen":"2025-05-30T12:24:37+02:00","lastCommitMessage":"convert : allow partial update to the chkhsh pre-tokenizer list (#13847)\n\n* convert : allow partial update to the chkhsh pre-tokenizer list\n\n* code style\n\n* update tokenizer out\n\n* rm inp/out files for models not having gguf\n\n* fixed hash for glm\n\n* skip nomic-bert-moe test\n\n* Update convert_hf_to_gguf_update.py\n\n* fix minerva-7b hash\n\n* rm redundant import"}]