{"entries":[{"name":"CMakeLists.txt","mode":"0100644","size":11879,"lastCommit":{"hash":"084596253b66e2566d3ac498545502748f6dc276","oid":"084596253b66e2566d3ac498545502748f6dc276","parents":["c572efdcdcdfeba0779a5d4c49fabe7512fa71cb"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T09:11:00Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T09:11:00Z"},"message":"cuda: opt-in Hopper wgmma prefill path for Q1_0/PQ2_0 (sm_90a) [UNVERIFIED - needs Hopper HW]\n","tree":"780f88f27148db9e57ffe1f6d39f13d32531ec40"}},{"name":"acc.cu","mode":"0100644","size":2426,"lastCommit":{"hash":"10d2af0eaa0aafd7c6577b279dfa5221ff44a63f","oid":"10d2af0eaa0aafd7c6577b279dfa5221ff44a63f","parents":["064cc596ac44308dc326a17c9e3163c34a6f29d1"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2025-05-12T14:44:49+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-05-12T14:44:49+02:00"},"message":"llama/ggml: add LLM training support (#10544)\n\n* llama/ggml: add LLM training support\n\nmore compact progress bar\n\nllama_save_model_to_file\n\nllama_opt_param_filter\n\nggml_graph_dup force_grads\n\nrefactor ggml_opt, fix test-opt\n\n* remove logits_all\n\n* refactor CUDA implementation for ACC\n\n* reset graph at beginning of opt period","tree":"cbeacf669e12d14e4b44fff1f5836a149744fa85"}},{"name":"acc.cuh","mode":"0100644","size":131,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"add-id.cu","mode":"0100644","size":1901,"lastCommit":{"hash":"a094f381432d92c4bf92d2d6167284316ba73a62","oid":"a094f381432d92c4bf92d2d6167284316ba73a62","parents":["fb22dd07a639e81c7415e30b146f545f1a2f2caf"],"author":{"name":"R0CKSTAR","email":"yeahdongcn@gmail.com","when":"2025-08-20T10:17:37+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-20T10:17:37+08:00"},"message":"musa: fix build warnings (#15258)\n\n* musa: fix build warnings\n\nSigned-off-by: Xiaodong Ye \n\n* fix warning: comparison of integers of different signs: 'const int' and 'unsigned int' [-Wsign-compare]\n\nSigned-off-by: Xiaodong Ye \n\n---------\n\nSigned-off-by: Xiaodong Ye ","tree":"8e8494758f054abd26695fce418c2cd78a0050f2"}},{"name":"add-id.cuh","mode":"0100644","size":101,"lastCommit":{"hash":"fd1234cb468935ea087d6929b2487926c3afff4b","oid":"fd1234cb468935ea087d6929b2487926c3afff4b","parents":["f324a3b715d5c1081c110ce459f8a8486fb1ee89"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2025-08-05T22:10:36+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-05T22:10:36+03:00"},"message":"llama : add gpt-oss (#15091)\n\n* oai moe\n\n* compat with new checkpoint\n\n* add attn sink impl\n\n* add rope scaling yarn\n\n* logits match with latest transformers code\n\n* wip chat template\n\n* rm trailing space\n\n* use ggml_scale_bias\n\n* rm redundant is_swa_all\n\n* convert interleaved gate_up\n\n* graph : fix activation function to match reference (#7)\n\n* vocab : handle o200k_harmony special tokens\n\n* ggml : add attention sinks support (#1)\n\n* llama : add attn sinks\n\n* ggml : add attn sinks\n\n* cuda : add attn sinks\n\n* vulkan : add support for sinks in softmax\n\nremove unnecessary return\n\n* ggml : add fused swiglu_oai op (#11)\n\n* ggml : add fused swiglu_oai op\n\n* Update ggml/src/ggml-cpu/ops.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* update CUDA impl\n\n* cont : metal impl\n\n* add vulkan impl\n\n* test-backend-ops : more test cases, clean up\n\n* llama : remove unfused impl\n\n* remove extra lines\n\n---------\n\nCo-authored-by: Georgi Gerganov \n\n---------\n\nCo-authored-by: slaren \n\n* repack mxfp4 upon conversion\n\n* clean up a bit\n\n* enable thinking\n\n* add quick hack to render only some special tokens\n\n* fix bf16 conversion\n\n* remove vocab hack\n\n* webui ok\n\n* support chat parsing for gpt-oss\n\n* fix webui\n\n* direct mapping mxfp4, FINALLY\n\n* force using mxfp4\n\n* properly use lazy tensor\n\n* ggml : add mxfp4\n\nggml : use e8m0 conversion instead of powf\n\nCo-authored-by: Diego Devesa \n\nchange kvalues_mxfp4 table to match e2m1 (#6)\n\nmetal : remove quantization for now (not used)\n\ncuda : fix disabled CUDA graphs due to ffn moe bias\n\nvulkan : add support for mxfp4\n\ncont : add cm2 dequant\n\n* ggml : add ggml_add_id (#13)\n\n* ggml : add ggml_add_id\n\n* add cuda impl\n\n* llama : add weight support check for add_id\n\n* perf opt\n\n* add vulkan impl\n\n* rename cuda files\n\n* add metal impl\n\n* allow in-place ggml_add_id\n\n* llama : keep biases on CPU with --cpu-moe\n\n* llama : fix compile error\n\nggml-ci\n\n* cuda : add fallback for __nv_cvt_e8m0_to_bf16raw\n\nggml-ci\n\n* cleanup\n\nggml-ci\n\n* sycl : fix supports_op for MXFP4\n\nggml-ci\n\n* fix Unknown reasoning format\n\n* ggml-cpu : fix AVX build\n\nggml-ci\n\n* fix hip build\n\nggml-ci\n\n* cuda : add mxfp4 dequantization support for cuBLAS\n\nggml-ci\n\n* ggml-cpu : fix mxfp4 fallback definitions for some architectures\n\nggml-ci\n\n* cuda : fix version required for __nv_cvt_e8m0_to_bf16raw\n\n---------\n\nCo-authored-by: Xuan Son Nguyen \nCo-authored-by: slaren ","tree":"cc60fd4a798fdc49947173bfec4de98b13d202ca"}},{"name":"allreduce.cu","mode":"0100644","size":45274,"lastCommit":{"hash":"7f3f843c31cd32dc4adc10b393342dfee071c332","oid":"7f3f843c31cd32dc4adc10b393342dfee071c332","parents":["ec562eb673ab6fef464d0cf976bcb57925b5eff0"],"author":{"name":"scutler-nv","email":"scutler@nvidia.com","when":"2026-05-13T13:36:14-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-13T22:36:14+02:00"},"message":"Fix for issue #22974. Cast intermediate results to float before adding and casting the result to the destination type. Avoids half+half operator ambiguity. (#22994)","tree":"9c29c6a952e16d6e72028f2b162d750075171caa"}},{"name":"allreduce.cuh","mode":"0100644","size":1014,"lastCommit":{"hash":"f3c3e0e9a087835639733485b8900b195ba4ca47","oid":"f3c3e0e9a087835639733485b8900b195ba4ca47","parents":["5755a100cde732c2fedfeb68256a45f8ac14d13a"],"author":{"name":"scutler-nv","email":"scutler@nvidia.com","when":"2026-05-10T02:05:22-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-10T11:05:22+02:00"},"message":"internal AllReduce kernel for CUDA provider (#22299)\n\n* ggml-cuda: add internal AllReduce provider for tensor parallelism\n\nIntroduces a NCCL-free AllReduce implementation for LLAMA_SPLIT_MODE_TENSOR\nusing a single-phase CUDA kernel that pipelines D2H copy, cross-GPU\nhandshake via pinned-memory volatile flags, and the reduction in one\nkernel launch per GPU.\n\nNew files:\n- ggml/src/ggml-cuda/comm.cuh — ggml_cuda_allreduce_provider enum\n- ggml/src/ggml-cuda/allreduce.cuh — pipeline API declarations\n- ggml/src/ggml-cuda/allreduce.cu — kernel + pipeline init/dispatch\n\nggml-cuda.cu changes:\n- ggml_backend_cuda_comm_context gains ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* llama-bench: add --allreduce flag to select AllReduce provider\n\nAdds --allreduce to llama-bench (and via the shared\nfield pattern, consistent with other multi-value flags). Useful for\nisolating hangs or regressions in tensor-parallel mode: pass --allreduce nccl\nto force NCCL and bypass the internal provider.\n\nAlso fixes ggml_cuda_select_allreduce_provider() to treat an empty\nGGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when\nllama-bench sets it to \"\" for the \"auto\" case).\n\nCo-Authored-By: Claude Sonnet 4.6 \nxt gains ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* llama-bench: rename --allreduce to --reduction-provider / -rp\n\nCo-Authored-By: Claude Sonnet 4.6 \n via the shared\nfield pattern, consistent with other multi-value flags). Useful for\nisolating hangs or regressions in tensor-parallel mode: pass --allreduce nccl\nto force NCCL and bypass the internal provider.\n\nAlso fixes ggml_cuda_select_allreduce_provider() to treat an empty\nGGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when\nllama-bench sets it to \"\" for the \"auto\" case).\n\nCo-Authored-By: Claude Sonnet 4.6 \nxt gains ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* llama-bench: pass WARN/ERROR log messages through in non-verbose mode\n\nThe null log callback was silently dropping all messages. WARN and ERROR\nshould always be visible since they indicate legitimate issues (e.g. a\nrequested reduction provider not being available).\n\nCo-Authored-By: Claude Sonnet 4.6 \nvider.\n\nAlso fixes ggml_cuda_select_allreduce_provider() to treat an empty\nGGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when\nllama-bench sets it to \"\" for the \"auto\" case).\n\nCo-Authored-By: Claude Sonnet 4.6 \nxt gains ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* cmake: improve NCCL detection for source-tree builds, add static/dynamic switch\n\nFindNCCL.cmake now searches the cmake source-build layout used by the Windows\nNCCL port (cmake/lib/Release for static, cmake/src/Release for dynamic import\nlib) and also checks src/include for the generated nccl.h header.\n\nNew option GGML_CUDA_NCCL_STATIC (default OFF) selects static vs dynamic\nlinking and controls which paths and library names are searched.\n\nCo-Authored-By: Claude Sonnet 4.6 \n for the \"auto\" case).\n\nCo-Authored-By: Claude Sonnet 4.6 \nxt gains ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* ggml-cuda: add AllReduce hang watchdog (GGML_CUDA_AR_WATCHDOG)\n\nWhen compiled with -DGGML_CUDA_AR_WATCHDOG=ON, uses a debug kernel\nvariant that writes per-GPU spin diagnostics to pinned host memory.\nA host-side blocking poll (cudaEventQuery + volatile reads) detects\nhangs and logs WARN with the last observed arrival counters and spin\ncounts, controlled by GGML_CUDA_AR_WATCHDOG (ms timeout) and\nGGML_CUDA_AR_MAX_SPIN (kernel bailout) env vars at runtime.\n\nZero overhead on the production path — all debug code is behind #ifdef.\n\nCo-Authored-By: Claude Sonnet 4.6 \n ar_pipeline field\n- Provider selection via GGML_CUDA_ALLREDUCE env var (\"nccl\" / \"internal\")\n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* ggml-cuda: fix intermittent AllReduce hang on Blackwell PCIe\n\nAdd __threadfence_system() before the arrival signal write in\nsignal_set to ensure D2H data is globally visible before the peer\nobserves the arrival flag. Without this fence, the peer could enter\nPhase 3 host reads before the data had fully landed, causing an\nintermittent deadlock on RTX 5090 (Blackwell, PCIe-only).\n\nAlso redesign the watchdog from a blocking dispatch-thread poll to a\nnon-blocking background thread, eliminating the ~20ms per-slot\nlatency the old design added.\n\nVerified: 30/30 soak test runs clean at ~50 t/s (previously ~1-in-15\nhang rate).\n\nCo-Authored-By: Claude Sonnet 4.6 \n- INTERNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* ggml-cuda: fix watchdog shutdown ordering and pipeline_free drain\n\n- Stop watchdog thread BEFORE destroying GPU resources (events, streams)\n to prevent polling destroyed handles → spurious \"busy\" readings\n- Add cudaStreamSynchronize in pipeline_free to drain in-flight kernels\n before freeing pinned host buffers they may still be reading\n- Sleep-first watchdog polling: no +0ms noise, only logs when a kernel\n is genuinely stuck past the poll interval\n- Check wdog_stop in both outer and inner loops so join() returns\n promptly instead of draining the entire queue\n- Add Phase 3 breadcrumbs to debug[3] for hang localization\n\nCo-Authored-By: Claude Sonnet 4.6 \nRNAL provider initialises the pipeline at comm_init time\n- Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend\n CPU reduce for unsupported sizes or GPU counts (> 2)\n\nCurrent scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* ggml-cuda: replace event-based watchdog with per-GPU ring buffer\n\nCompletely rework the GGML_CUDA_AR_WATCHDOG system:\n\n- Replace the shared debug_buf + event-polling + queue design with\n per-GPU ring buffers in pinned host memory\n- Kernel writes a debug record only on spin-limit bailout: claims a\n ring slot via atomicAdd (single-GPU host atomics work on RTX 5090),\n writes fields, fences, sets completion flag, then all threads exit\n- Watchdog thread simply polls ring head counters every 1ms and prints\n any new complete records — no CUDA event queries, no mutex, no queue\n- Zero overhead on the dispatch path (no queue posting, no memset)\n- Watchdog shutdown returns within ~1ms (atomic bool, no drain)\n- On bailout the kernel skips Phase 3 entirely and exits cleanly\n\nVerified: 20/20 prefill soak test clean at ~1112 t/s, no hangs.\n\nCo-Authored-By: Claude Sonnet 4.6 \nP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* fix: normalize line endings to LF (undo Windows CRLF conversion)\n\nFive files were inadvertently converted to CRLF by the Windows\ndevelopment environment, causing every line to show as changed in\ndiffs against master.\n\nCo-Authored-By: Claude Sonnet 4.6 \nimit bailout: claims a\n ring slot via atomicAdd (single-GPU host atomics work on RTX 5090),\n writes fields, fences, sets completion flag, then all threads exit\n- Watchdog thread simply polls ring head counters every 1ms and prints\n any new complete records — no CUDA event queries, no mutex, no queue\n- Zero overhead on the dispatch path (no queue posting, no memset)\n- Watchdog shutdown returns within ~1ms (atomic bool, no drain)\n- On bailout the kernel skips Phase 3 entirely and exits cleanly\n\nVerified: 20/20 prefill soak test clean at ~1112 t/s, no hangs.\n\nCo-Authored-By: Claude Sonnet 4.6 \nP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* .gitattributes: force LF line endings to prevent Windows CRLF conversion\n\nCo-Authored-By: Claude Sonnet 4.6 \nelopment environment, causing every line to show as changed in\ndiffs against master.\n\nCo-Authored-By: Claude Sonnet 4.6 \nimit bailout: claims a\n ring slot via atomicAdd (single-GPU host atomics work on RTX 5090),\n writes fields, fences, sets completion flag, then all threads exit\n- Watchdog thread simply polls ring head counters every 1ms and prints\n any new complete records — no CUDA event queries, no mutex, no queue\n- Zero overhead on the dispatch path (no queue posting, no memset)\n- Watchdog shutdown returns within ~1ms (atomic bool, no drain)\n- On bailout the kernel skips Phase 3 entirely and exits cleanly\n\nVerified: 20/20 prefill soak test clean at ~1112 t/s, no hangs.\n\nCo-Authored-By: Claude Sonnet 4.6 \nP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* ggml-cuda: move GGML_CUDA_AR_WATCHDOG from CMake option to local define\n\nThe watchdog is development-only; a global CMake option is overkill.\nMove the toggle to a #define at the top of allreduce.cu (set to 0 by\ndefault) and remove the option from ggml/CMakeLists.txt and the CUDA\nCMakeLists.txt add_compile_definitions block.\n\nCo-Authored-By: Claude Sonnet 4.6 \n fences, sets completion flag, then all threads exit\n- Watchdog thread simply polls ring head counters every 1ms and prints\n any new complete records — no CUDA event queries, no mutex, no queue\n- Zero overhead on the dispatch path (no queue posting, no memset)\n- Watchdog shutdown returns within ~1ms (atomic bool, no drain)\n- On bailout the kernel skips Phase 3 entirely and exits cleanly\n\nVerified: 20/20 prefill soak test clean at ~1112 t/s, no hangs.\n\nCo-Authored-By: Claude Sonnet 4.6 \nP32, tensors <= 256 KB. Notes in NOTES-allreduce.md.\n\nCo-Authored-By: Claude Sonnet 4.6 \n\n* unify kernel debug paths\n\n* use __threadfence_system explicitly (not in ggml_cuda_ar_signal_set)\n\n* preferentially use internal reduction for <=2 GPUs\n\n* templatize the main kernel to support fp16/bf16\n\n* restore llama-bench.cpp changes\n\n* revert CMakeLists changes\n\n* remove notes from repo\n\n* remove dead warmup code\n\n* fix comments\n\n* improve reduction provider fallback code\n\n* add messages for allreduce fallback\n\n* rework reduction provider init to not call ncclCommInitAll if using the internal provider\n\n* fix case where a given tensor has not been computed\n\n* add chunked mode to the kernel for unlimited vector size\n\n* rework a few checks/fallbacks\n\n* various small cleanups\n\n* allow disabling CUDA reductions completely (falling back to the non-CUDA butterfly mode)\n\n* simplify reduction provider selection\n\n* minor simplifications\n\n* more cleanups/fixes\n\n* prototype alternate path for large reductions\n\n* chunked version of large reduction path\n\n* use bf16 for large reductions\n\n* experimental reduction using cudaMemcpyPeerAsync (slightly slower)\n\n* revert experimental change\n\n* add combined conversion/reduction kernel\n\n* add bf16 wire format for single kernel mode\n\n* experimental on-stream small reduction kernel\n\n* double buffer arrival slots, use token (incrementing) method\n\n* double buffer host_buf for small reductions\n\n* put in waits for use of host_mem in large reduction case (prevents stomping on in-use memory\n\n* remove watchdog code\n\n* various cleanups / dead code removal\n\n* fix fp16 mode\n\n* fix some comments/logging statements\n\n* use increasing token scheme for arrival signals\n\n* add top-level comment to allreduce.cu\n\n* improve top-level comment in allreduce.cu\n\n* fix comments in ggml_cuda_ar_kernel\n\n* improve event handling for hostmem buffer usage tracking\n\n* change ev_pool to fixed 2D array\n\n* add chunked memcpy fallback for extra-large reductions (>32 MB)\n\n* change thresholds for copy-engine path and bf16 demotion\n\n* multi-block kernel test\n\n* more fine-tuning for chukn-size, etc.\n\n* various fixes for PR review\n\n* more PR fixes\n\n* fix semantics of all host mappings\n\n* require ampere+\n\n* small cleanups\n\n* properly use host pointer for src/dst in cudaMemcpy calls\n\n* allreduce: lazy-init the internal pipeline on first use\n\nA config that lives entirely on NCCL never needs the chunked-kernel\npipeline (host_buf, host_large, dev_tmp, streams, events, arrival ring).\nDefer pipeline creation to the first try_allreduce_internal call using the\nsame std::call_once pattern as ensure_nccl, so those resources stay\nunallocated when only NCCL is in use.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: assert n_backends == 2 instead of soft-fallback\n\nar_pipeline_init already requires n_devices == 2 and bails before any AR can\nget here, so by the time we reach try_allreduce_internal we know we have\nexactly two backends. Replace the runtime-debug-log fallback with a hard\nassert.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n NCCL is in use.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* rework reduction provider selection. internal/nccl is OS dependent; most fallbacks are removed\n\n* remove unneeded Turing arch check (llama.cpp doesn't even compile pre-Turing anyway)\n\n* allreduce: ASCII-only comments and ggml_cuda_cast for value conversions\n\nReplace non-ASCII characters in comments (em dashes, right arrows) with\nASCII equivalents (--, ->) so the source stays in the ggml/upstream norm.\n\nIn the kernel-side code, replace static_cast/static_cast\nwith ggml_cuda_cast<...> so the BF16 conversions go through the fast\n__float2bfloat16 / __bfloat162float intrinsics from convert.cuh. Pure\npointer and integer casts stay as static_cast.\n\nAlso drops two stray garbage tokens that snuck in from earlier merges\n(a duplicated 'return ok; }' tail in allreduce.cu and a leftover '_reg)'\nfragment in ggml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: use ggml_cuda_memcpy_1 for the chunked-kernel vector copies\n\nThe chunked kernel's two 16-byte register<->host transfers (Phase 1 store\nand Phase 3 load) used reinterpret_cast on both sides. Replace\nwith ggml_cuda_memcpy_1, which is the canonical helper for\nthis pattern and emits the same int4 LD/ST under the hood.\n\nConformance passes; 5x reruns of 70b internal pp512 show 1832-1836 t/s,\nmatching the prior matrix value of 1831 t/s -- no perf change as expected.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nok; }' tail in allreduce.cu and a leftover '_reg)'\nfragment in ggml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: assert cuda_ctx->device matches the pipeline's device\n\nBoth ggml_cuda_ar_pipeline and ggml_backend_cuda_context carry the device\nthey were created for; if they ever disagree, every cuda call that follows\nruns on the wrong device. Add GGML_ASSERT at each cuda_ctx retrieval site\nin the AR path so the misuse fails fast rather than silently corrupting.\n\nAlso: rename __nv_bfloat16 -> nv_bfloat16 (typedef alias) for consistency\nwith the rest of the file, and tighten one cudaGetLastError check to fire\nonly after the to_bf16 call that can actually fail.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \ngml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: expand one-liner for loops to braced bodies\n\nCode-style preference -- match the rest of the file by writing every for\nloop with the body on its own braced line. Three sites in the copy-engine\ntyped dispatch.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nin the AR path so the misuse fails fast rather than silently corrupting.\n\nAlso: rename __nv_bfloat16 -> nv_bfloat16 (typedef alias) for consistency\nwith the rest of the file, and tighten one cudaGetLastError check to fire\nonly after the to_bf16 call that can actually fail.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \ngml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: rename template parameters Tdst/Twire/Tsrc -> T_dst/T_wire/T_src\n\nCode-style preference per PR review -- T_dst/T_wire/T_src is more\nconsistent with surrounding code. Whole-word rename across all 58 sites\nin allreduce.cu (kernel definitions, internal uses, and comment text).\n\nRealigned the parameter columns in three function signatures whose\nT_src/T_dst lines shifted by 1 char relative to their non-templated\nneighbors.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nto fire\nonly after the to_bf16 call that can actually fail.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \ngml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: drop hyphen in 'chunked-kernel' across comments\n\nPer PR review feedback -- 'chunked kernel' (no hyphen) reads more naturally\nin running prose, especially for ESL readers. Pure comment-only change;\nall 10 occurrences in allreduce.cu updated.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nthree function signatures whose\nT_src/T_dst lines shifted by 1 char relative to their non-templated\nneighbors.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nto fire\nonly after the to_bf16 call that can actually fail.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \ngml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: use ggml_cuda_get_max_cpy_bytes() instead of hardcoded 16\n\nThe chunked kernel hardcoded a 16-byte vector unit; replace with the\nggml_cuda_get_max_cpy_bytes() helper that fattn-common.cuh uses for the\nsame purpose, so ELEMS_PER_VEC self-adjusts to the arch's widest\nsingle-instruction copy.\n\nPerf-neutral on supported targets (Volta+ returns 16).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nhbors.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nto fire\nonly after the to_bf16 call that can actually fail.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \ngml-cuda.cu).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* ggml-cuda: PR review fixes -- annotate #endif, fix stale comment, assert nbytes alignment\n\nThree separate but minor changes from PR #22299 review feedback:\n\n1. Annotate the five GGML_USE_NCCL #endif lines with the matching condition\n so the pairing is visible without scrolling back.\n\n2. The comment block on ggml_backend_cuda_comm_context claimed NCCL is\n lazy-initialised; that was true at one point but the dispatch refactor\n (727b141c0) made both NCCL and the internal pipeline eager. Rewrite\n the comment to match current behaviour.\n\n3. Assert in ggml_backend_cuda_comm_allreduce_internal that the tensor's\n byte size is a 16-byte multiple. The chunked-kernel issues full-width\n vector loads/stores, so this is a precondition; tensor-parallel splits\n of hidden-dim-multiples satisfy it trivially, but a hard assert turns\n any caller-side bug into a clear failure rather than UB.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n device's new AR\nrecords its ev.ker -- otherwise the second device's wait sees the first\ndevice's just-recorded event (the in-flight new AR) and creates a circular\ndependency with the in-kernel peer signal. Two-pass dispatch (all waits,\nthen all launches) avoids this.\n\nBump POOL_SIZE 2 -> 8 (small memory cost, more breathing room for the\nGPU's view of the event chain) and add a runtime env override for the\nhybrid kernel chunk size (GGML_CUDA_AR_HYBRID_CHUNK_BYTES) for tuning.\nOne-shot stderr diagnostic at first AR prints the chosen path + sizing.\n\nResult on 2x RTX 5090 Linux, 70b ub_sweep:\n\n ub=64 (1 MB AR): 913 -> 1036 t/s (+13.5% vs old, +1.8% vs NCCL)\n ub=128 (2 MB AR): 1056 -> 1181 (+11.9%, +3.7% vs NCCL)\n ub=256 (4 MB AR): 1212 -> 1424 (+17.5%, +3.5% vs NCCL)\n\nInternal now beats NCCL at every size (+1.8% to +15.6%), recovering all\nground in the 1-4 MB regime that was previously a 10-12% loss.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* simplify the init logic\n\n* address some other PR requests\n\n* ggml-cuda: stub internal AllReduce on HIP/MUSA, drop pre-Ampere mention, gate NCCL fallback warning on !HIP\n\nThe internal AllReduce relies on cudaHostAllocPortable/Mapped,\ncudaHostGetDevicePointer, and __nanosleep -- none of which the HIP or\nMUSA shims expose -- so wrap the implementation in\n!defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) and provide\nnullptr/no-op/false stubs in the #else branch. The dispatcher already\ntreats a null pipeline as init failure and silently falls back to the\nmeta backend's generic AllReduce, so HIP/MUSA builds compile clean and\nbehave correctly without further call-site changes.\n\nPR review follow-ups:\n - drop \"or pre-Ampere?\" from the internal-init failure warning -- the\n kernel doesn't require Ampere or newer.\n - guard the \"NCCL not compiled in\" fallback warning behind\n !defined(GGML_USE_HIP); the suggestion to install NCCL only makes\n sense on NVIDIA builds.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nhind, now +6-8% ahead at ub=1024-4096.\nPerplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: guard __nanosleep on Volta+ and reject pre-Volta devices at init\n\n__nanosleep is the only Volta-specific intrinsic in the kernel; wrap it\nin #if __CUDA_ARCH__ >= GGML_CUDA_CC_VOLTA / NO_DEVICE_CODE so the file\nstill compiles cleanly when targeting older arches (the dispatcher's\ninit check below ensures the kernel is never actually launched on\npre-Volta).\n\nAdd a per-device compute-capability check in pipeline_init that returns\nnullptr if any device is below sm70. The dispatcher already treats\nnullptr as init failure and silently falls back to the meta backend's\ngeneric AllReduce.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nrom the internal-init failure warning -- the\n kernel doesn't require Ampere or newer.\n - guard the \"NCCL not compiled in\" fallback warning behind\n !defined(GGML_USE_HIP); the suggestion to install NCCL only makes\n sense on NVIDIA builds.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nhind, now +6-8% ahead at ub=1024-4096.\nPerplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* allreduce: fix CI -Werror warnings (sign-compare, format, restrict alias, maybe-uninitialized)\n\nThe CUDA CI builds with -Werror -Wsign-compare -Wformat -Wrestrict\n-Wmaybe-uninitialized. Address each:\n\n - n_devices is size_t; change `int i; i < n_devices` to size_t in the\n three init loops, and the matching GGML_LOG_INFO format from %d to %zu.\n - ggml_cuda_ar_kernel was launched with sendbuf == recvbuf (in-place\n reduction), so the __restrict__ qualifiers on those parameters were\n technically UB. Drop __restrict__ from sendbuf and recvbuf; an A/B\n sweep showed <0.6% perf delta (within noise) on Linux.\n - The buf/src/dst pointer arrays in ggml_cuda_ar_allreduce and the\n per-iteration arrays in ggml_cuda_ar_allreduce_copy_outer were\n declared with size GGML_CUDA_MAX_DEVICES but the loop only writes\n indices [0, n_devices); zero-initialise so the compiler sees the\n tail elements as defined.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nnow +6-8% ahead at ub=1024-4096.\nPerplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n* ggml-cuda: drop unused-function warning by guarding try_allreduce_nccl behind GGML_USE_NCCL\n\nThe only call site (in init_nccl) is already inside #ifdef GGML_USE_NCCL,\nso the function is unreferenced in non-NCCL builds and trips\nnvcc's -Werror=unused-function check. Move the guard from inside the\nfunction body to around the entire definition.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nce\n reduction), so the __restrict__ qualifiers on those parameters were\n technically UB. Drop __restrict__ from sendbuf and recvbuf; an A/B\n sweep showed <0.6% perf delta (within noise) on Linux.\n - The buf/src/dst pointer arrays in ggml_cuda_ar_allreduce and the\n per-iteration arrays in ggml_cuda_ar_allreduce_copy_outer were\n declared with size GGML_CUDA_MAX_DEVICES but the loop only writes\n indices [0, n_devices); zero-initialise so the compiler sees the\n tail elements as defined.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \nnow +6-8% ahead at ub=1024-4096.\nPerplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \n\n---------\n\nCo-authored-by: Claude Sonnet 4.6 ","tree":"9941b277920fd1d9132016416ab1842cc4bf9a4b"}},{"name":"arange.cu","mode":"0100644","size":1215,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"arange.cuh","mode":"0100644","size":137,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"argmax.cu","mode":"0100644","size":2854,"lastCommit":{"hash":"58062860afb88e555857c1266d3a17e1b65b5eb9","oid":"58062860afb88e555857c1266d3a17e1b65b5eb9","parents":["2973a65ecb6c884ca609de6eb5f1b6dc08631aaf"],"author":{"name":"Aadeshveer Singh","email":"aadeshveer07@gmail.com","when":"2025-12-17T09:17:01+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-17T11:47:01+08:00"},"message":"ggml : use WARP_SIZE/2 for argmax reduction offset (#18092)\n\n","tree":"5512b2fb692f0ad4b4a23da9f758f22facea39f0"}},{"name":"argmax.cuh","mode":"0100644","size":98,"lastCommit":{"hash":"fabdc3bda396307565c4f3f4ecbc3a751a2eb6d3","oid":"fabdc3bda396307565c4f3f4ecbc3a751a2eb6d3","parents":["eee39bdc96065b69242877fe8f1be05c885fc2aa"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-10-03T17:29:59+02:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-10-03T21:17:26+03:00"},"message":"ggml/ex: calculate accuracy in graph, adapt MNIST (ggml/980)\n","tree":"219d3588467c6690d02c975a1bda2b2fa792eeef"}},{"name":"argsort.cu","mode":"0100644","size":13110,"lastCommit":{"hash":"074944998d3f25e7001ede30d152b59dff741c8c","oid":"074944998d3f25e7001ede30d152b59dff741c8c","parents":["3de7dd4c8f5d9806279249310b6c3db24a1a67ab"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-09T20:07:12+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-09T20:07:12+02:00"},"message":"ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (#24776)\n\n* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage\n\n* ggml : allocate tmp_dst only only once before the loop\n\n* chore : whitespaces\n\nCo-authored-by: Georgi Gerganov \n\n* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations\n\n* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement\n\nCo-authored-by: Johannes Gäßler \n\n* chore : replace ternary operators with min/max\n\n---------\n\nCo-authored-by: Stanisław Szymczyk \nCo-authored-by: Georgi Gerganov \nCo-authored-by: Johannes Gäßler ","tree":"3099742364a1901d11ed17e56650db687d59b2a7"}},{"name":"argsort.cuh","mode":"0100644","size":950,"lastCommit":{"hash":"074944998d3f25e7001ede30d152b59dff741c8c","oid":"074944998d3f25e7001ede30d152b59dff741c8c","parents":["3de7dd4c8f5d9806279249310b6c3db24a1a67ab"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-09T20:07:12+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-09T20:07:12+02:00"},"message":"ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (#24776)\n\n* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage\n\n* ggml : allocate tmp_dst only only once before the loop\n\n* chore : whitespaces\n\nCo-authored-by: Georgi Gerganov \n\n* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations\n\n* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement\n\nCo-authored-by: Johannes Gäßler \n\n* chore : replace ternary operators with min/max\n\n---------\n\nCo-authored-by: Stanisław Szymczyk \nCo-authored-by: Georgi Gerganov \nCo-authored-by: Johannes Gäßler ","tree":"3099742364a1901d11ed17e56650db687d59b2a7"}},{"name":"binbcast.cu","mode":"0100644","size":24790,"lastCommit":{"hash":"f728adab683387b2bdb085b3854a0f3487a13d0a","oid":"f728adab683387b2bdb085b3854a0f3487a13d0a","parents":["3e61ea0e2f67fd9c76e82aa2aef6f794520f4a75"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-06-25T10:06:44+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-25T10:06:44+02:00"},"message":"ggml : address integer overflows in binary ops CUDA implementation (#24706)\n\n* ggml : address integer overflows in binary ops CUDA implementation\n\n* ggml : add size_t casts to avoid integer overflows\n\n* ggml : add more asserts checking integer overflows in binary ops CUDA implementation\n\n---------\n\nCo-authored-by: Stanisław Szymczyk ","tree":"ca0f16fde1f8f870f3ea21e077dc4e29b2789469"}},{"name":"binbcast.cuh","mode":"0100644","size":672,"lastCommit":{"hash":"e34f0421544b42cae1a03bab528f571cf49814b0","oid":"e34f0421544b42cae1a03bab528f571cf49814b0","parents":["d132f22fc92f36848f7ccf2fc9987cd0b0120825"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-04-10T10:24:09+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-04-10T10:24:09+08:00"},"message":"CUDA: fuse muls (#21665)","tree":"430be1d1303bbc018aa8abdb1afce2c9326d600a"}},{"name":"clamp.cu","mode":"0100644","size":1588,"lastCommit":{"hash":"b64d7cc2720a90c03480d9408fae40d936c110e2","oid":"b64d7cc2720a90c03480d9408fae40d936c110e2","parents":["3d1cf3cf3394f2288dca3e1fb1fbb467d366e4bb"],"author":{"name":"cmdr2","email":"secondary.cmdr2@gmail.com","when":"2025-03-03T20:51:31+05:30"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2025-03-03T18:18:11+02:00"},"message":"cuda: unary ops as float + de-duplicate (ggml/1130)\n","tree":"9e8b3b9f387b3e25c34879336b37aa6bdd3e7a1e"}},{"name":"clamp.cuh","mode":"0100644","size":135,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"col2im-1d.cu","mode":"0100644","size":3057,"lastCommit":{"hash":"3a3edc9ac65cca79584ca497be41d70c75a58ba8","oid":"3a3edc9ac65cca79584ca497be41d70c75a58ba8","parents":["40f3aafc45990c1646397c67d2bd3e8eff3f543e"],"author":{"name":"Pascal","email":"admin@serveurperso.com","when":"2026-06-18T22:23:01+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-18T22:23:01+02:00"},"message":"Ggml/cuda col2im 1d (#24417)\n\n* cuda: add GGML_OP_COL2IM_1D, follow-up to the CPU op\n\n* cuda: col2im_1d use fast_div_modulo for the index decomposition\n\n* cuda: col2im_1d tighten supports_op, type match and contiguous dst","tree":"bcf53677e53ca56b83b25cdebd037a876204aba4"}},{"name":"col2im-1d.cuh","mode":"0100644","size":104,"lastCommit":{"hash":"3a3edc9ac65cca79584ca497be41d70c75a58ba8","oid":"3a3edc9ac65cca79584ca497be41d70c75a58ba8","parents":["40f3aafc45990c1646397c67d2bd3e8eff3f543e"],"author":{"name":"Pascal","email":"admin@serveurperso.com","when":"2026-06-18T22:23:01+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-18T22:23:01+02:00"},"message":"Ggml/cuda col2im 1d (#24417)\n\n* cuda: add GGML_OP_COL2IM_1D, follow-up to the CPU op\n\n* cuda: col2im_1d use fast_div_modulo for the index decomposition\n\n* cuda: col2im_1d tighten supports_op, type match and contiguous dst","tree":"bcf53677e53ca56b83b25cdebd037a876204aba4"}},{"name":"common.cuh","mode":"0100644","size":66126,"lastCommit":{"hash":"113d61c842e3889406777388f7bb929f288e12b0","oid":"113d61c842e3889406777388f7bb929f288e12b0","parents":["8ea08e9c7b8a8bfd72305b28efd7019c33dfa969"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: Hadamard transform writes per-128 q8 tiles directly (fwht-q8)\n\nThe Hadamard transform in front of a PTQ1_0 matmul that takes the MMQ path also writes the per-128 q8 tiles MMQ reads, so the matmul skips its own quantize pass (and the f32 copy when nothing else reads it). Bit-identical; +3% prefill.\n","tree":"c293c37ef5b6d25c91b4d4c27cea63a33e4b9d54"}},{"name":"concat.cu","mode":"0100644","size":12378,"lastCommit":{"hash":"81a5c121f5ddfda40c037d117b5b0c685a7fa924","oid":"81a5c121f5ddfda40c037d117b5b0c685a7fa924","parents":["d15f521847cd02b33ce1284c4fef0db0a5d63835"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: MTP catch-up rows ride in next draft batch (mtp-decode)\n\nDecode: MTP catch-up rows of a verify batch ride in the next draft batch (one MTP pass per round instead of two, rows computed identically), and the conv-state concat takes the transposing kernel. Greedy and sampled output identical; ~+6% decode on the 3060.\n","tree":"336374a5256a3f1f3bb29388b1927b5c8f8c196e"}},{"name":"concat.cuh","mode":"0100644","size":137,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"conv-transpose-1d.cu","mode":"0100644","size":3391,"lastCommit":{"hash":"72874f559c598b8f89fbb24864868337cf5afb4c","oid":"72874f559c598b8f89fbb24864868337cf5afb4c","parents":["2da668617612d2df773f966e3b0ee22dc2beef7b"],"author":{"name":"adavyas","email":"121313528+adavyas@users.noreply.github.com","when":"2026-07-05T20:49:06-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-06T11:49:06+08:00"},"message":"ggml-cuda: optimize conv_transpose_1d indexing (#25310)","tree":"64e6920eb17420347e2695c2b37cc3083ddf5665"}},{"name":"conv-transpose-1d.cuh","mode":"0100644","size":158,"lastCommit":{"hash":"fde13b3bb9f569f07fd8af74696ee48a43d05131","oid":"fde13b3bb9f569f07fd8af74696ee48a43d05131","parents":["470939d483d1c89b7292f78bac1fd27c42c171ce"],"author":{"name":"John Balis","email":"phobossystems@gmail.com","when":"2024-07-02T11:09:52-05:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-07-08T12:23:00+03:00"},"message":"feat: cuda implementation for `ggml_conv_transpose_1d` (ggml/854)\n\n* conv transpose 1d passing test for 1d input and kernel\n\n* working for different input and output channel counts, added test for variable stride\n\n* initial draft appears to work with stride other than 1\n\n* working with all old and new conv1d tests\n\n* added a test for large tensors\n\n* removed use cuda hardcoding\n\n* restored test-conv-transpose.c\n\n* removed unused arugments, and fixed bug where test failure would cause subsequent tests to fail\n\n* fixed accumulator bug\n\n* added test to test-backend-ops\n\n* fixed mistake\n\n* addressed review\n\n* fixed includes\n\n* removed blank lines\n\n* style and warning fixes\n\n* return failure when test fails\n\n* fix supports_op\n\n---------\n\nCo-authored-by: slaren \n","tree":"9cb214f47d31271b6e1946787947aec30756a3ba"}},{"name":"conv2d-dw.cu","mode":"0100644","size":7390,"lastCommit":{"hash":"9eaa51e7f08593f123f00136591179a8f5956ecd","oid":"9eaa51e7f08593f123f00136591179a8f5956ecd","parents":["8f71d0f3e86ccbba059350058af8758cafed73e6"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-06-20T09:50:24+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-20T09:50:24+08:00"},"message":"CUDA: add conv_2d_dw (#14265)\n\n* CUDA: add conv_2d_dw\n\n* better naming\n\n* simplify using template\n\n* Review: fix operation ordering in ggml-cuda, use __forceinline__, use more const","tree":"02f294dab005eedb878603c8aba3d3a44eb166f8"}},{"name":"conv2d-dw.cuh","mode":"0100644","size":155,"lastCommit":{"hash":"9eaa51e7f08593f123f00136591179a8f5956ecd","oid":"9eaa51e7f08593f123f00136591179a8f5956ecd","parents":["8f71d0f3e86ccbba059350058af8758cafed73e6"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-06-20T09:50:24+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-20T09:50:24+08:00"},"message":"CUDA: add conv_2d_dw (#14265)\n\n* CUDA: add conv_2d_dw\n\n* better naming\n\n* simplify using template\n\n* Review: fix operation ordering in ggml-cuda, use __forceinline__, use more const","tree":"02f294dab005eedb878603c8aba3d3a44eb166f8"}},{"name":"conv2d-transpose.cu","mode":"0100644","size":4818,"lastCommit":{"hash":"0a524f240456d7727570043f97757ea2c249003b","oid":"0a524f240456d7727570043f97757ea2c249003b","parents":["c0159f9c1f874da15e94f371d136f5920b4b5335"],"author":{"name":"Yihao Wang","email":"42559837+AgainstEntropy@users.noreply.github.com","when":"2026-03-25T19:19:14-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-03-26T10:19:14+08:00"},"message":"CUDA & CPU: support F32 kernel type for `CONV_TRANSPOSE_2D` (#17094)\n\n* Refactor CUDA 2D transpose implementation to support multiple kernel types and improve parameter handling\n\n- Introduced a `conv2d_transpose_params` struct for better parameter management.\n- Updated `conv2d_transpose_kernel` to be templated for different kernel types (float and half).\n- Modified `ggml_cuda_conv_2d_transpose_p0` to handle both F16 and F32 kernel types.\n- Enhanced test cases to validate functionality for both kernel types.\n\n* Refactor test cases for 2D convolution transpose to support dynamic kernel types\n\n- Updated `test_conv_transpose_2d` structure to improve parameter handling by reordering constructor arguments.\n- Enhanced test case generation to iterate over kernel types, allowing for flexible testing of different configurations.\n- Removed hardcoded kernel type instances in favor of a loop for better maintainability and scalability.\n\n* Refactor ggml_compute_forward_conv_transpose_2d to support both F16 and F32 tensor types.\n\n* Refactor conv2d transpose kernel to use a template for kernel type, enhancing flexibility for different data types.\nUpdate test cases to include both F16 and F32 tensor types for comprehensive coverage.\n\n* Update ggml/src/ggml-cuda/conv2d-transpose.cu\n\nCo-authored-by: Aman Gupta \n\n* Update ggml/src/ggml-cpu/ggml-cpu.c\n\nCo-authored-by: Aman Gupta \n\n* Refactor conv2d transpose implementation by removing the conv2d_transpose_params struct and dispatching with direct kernel launch.\n\n* Enhance cpu conv2d transpose implementation by introducing a templated kernel type for improved flexibility with F16 and F32 data types.\n\n---------\n\nCo-authored-by: Aman Gupta ","tree":"f9a174b1fec65809e43a537d771cbe9ec0209aff"}},{"name":"conv2d-transpose.cuh","mode":"0100644","size":158,"lastCommit":{"hash":"0a524f240456d7727570043f97757ea2c249003b","oid":"0a524f240456d7727570043f97757ea2c249003b","parents":["c0159f9c1f874da15e94f371d136f5920b4b5335"],"author":{"name":"Yihao Wang","email":"42559837+AgainstEntropy@users.noreply.github.com","when":"2026-03-25T19:19:14-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-03-26T10:19:14+08:00"},"message":"CUDA & CPU: support F32 kernel type for `CONV_TRANSPOSE_2D` (#17094)\n\n* Refactor CUDA 2D transpose implementation to support multiple kernel types and improve parameter handling\n\n- Introduced a `conv2d_transpose_params` struct for better parameter management.\n- Updated `conv2d_transpose_kernel` to be templated for different kernel types (float and half).\n- Modified `ggml_cuda_conv_2d_transpose_p0` to handle both F16 and F32 kernel types.\n- Enhanced test cases to validate functionality for both kernel types.\n\n* Refactor test cases for 2D convolution transpose to support dynamic kernel types\n\n- Updated `test_conv_transpose_2d` structure to improve parameter handling by reordering constructor arguments.\n- Enhanced test case generation to iterate over kernel types, allowing for flexible testing of different configurations.\n- Removed hardcoded kernel type instances in favor of a loop for better maintainability and scalability.\n\n* Refactor ggml_compute_forward_conv_transpose_2d to support both F16 and F32 tensor types.\n\n* Refactor conv2d transpose kernel to use a template for kernel type, enhancing flexibility for different data types.\nUpdate test cases to include both F16 and F32 tensor types for comprehensive coverage.\n\n* Update ggml/src/ggml-cuda/conv2d-transpose.cu\n\nCo-authored-by: Aman Gupta \n\n* Update ggml/src/ggml-cpu/ggml-cpu.c\n\nCo-authored-by: Aman Gupta \n\n* Refactor conv2d transpose implementation by removing the conv2d_transpose_params struct and dispatching with direct kernel launch.\n\n* Enhance cpu conv2d transpose implementation by introducing a templated kernel type for improved flexibility with F16 and F32 data types.\n\n---------\n\nCo-authored-by: Aman Gupta ","tree":"f9a174b1fec65809e43a537d771cbe9ec0209aff"}},{"name":"conv2d.cu","mode":"0100644","size":6565,"lastCommit":{"hash":"958d9c0b61d0025ca8e2fff32d66a1576173fde9","oid":"958d9c0b61d0025ca8e2fff32d66a1576173fde9","parents":["432d7ffe2c3b4e539f3d0d4ae0a4893090a018d6"],"author":{"name":"pmaybank","email":"113125070+pmaybank@users.noreply.github.com","when":"2026-07-30T18:14:16+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-31T01:14:16+08:00"},"message":"Test support for alternative conv layout (#25617)\n\n* add bool cwhn = true to conv_2d test cases\n\n* add layout check at graph building time\n\n* extend layout checks for conv2d.cu kernel\n\n* in CPU back-end kernel needs to be stored contiguously to prevent test failures with cwhn=1\n\n* trim white space\n\n* do op support check in vulkan backend\n\n* fix CI failure and vulkan run-time assert failure by introducing new graph build-time check in ggml_backend_vk_device_supports_op\n\n* add additional check in support_op function for Vulkan to fix run-time assert failure","tree":"c3a53dcdb2328f1f179aacbc3ccdd1bf9fcc8ee3"}},{"name":"conv2d.cuh","mode":"0100644","size":149,"lastCommit":{"hash":"c97dc093912ad014f6d22743ede0d4d7fd82365a","oid":"c97dc093912ad014f6d22743ede0d4d7fd82365a","parents":["6c442f42ff25564a0cd6b1435d9abc1b0178eac5"],"author":{"name":"mnehete32","email":"33429707+mnehete32@users.noreply.github.com","when":"2025-08-29T00:03:03+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-28T20:33:03+02:00"},"message":"CUDA: add conv2d (#15635)\n\n* CUDA: add conv2d\n\n* CUDA: conv2d - correct formatting and added const","tree":"c58cf3361d547230ff97ab99825311cd7517ee96"}},{"name":"convert.cu","mode":"0100644","size":31391,"lastCommit":{"hash":"9294043b25f7aad1f34a933559ea1213d28ae543","oid":"9294043b25f7aad1f34a933559ea1213d28ae543","parents":["3dd767559a1b35fbad6cc8921ff639797dd07755"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-08T00:33:31-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T00:33:31-07:00"},"message":"cuda: native PTQ1_0 MMQ tile loader and packed dequantize (#160)\n\nLarge-batch PTQ1_0 matmul had no MMQ path, so it fell back to a full\ndequantize to fp16 followed by cuBLAS: measured 2.26x slower than PQ2_0 at\nn=512 while its decode already led. Add the tile loader, the MMQ type\ndispatch and the DECL_MMQ_CASE instance so PTQ1_0 takes the same quantized\nGEMM path as the other low-bit types, plus a packed, coalesced dequantize for\nthe remaining conversion sites.\n\nPorted from a handoff worktree whose gates recorded 45/45 PTQ1_0 MUL_MAT on\nCUDA and an H100 perf pass at the pre-#152 base; its Metal and FWHT parts are\nsuperseded by #152 and #157 and are not included. This branch has not yet\nbeen compiled or run on CUDA hardware by the committer; that verification is\nthe gate before it is pushed.","tree":"92c9b373f2802e6d81aabe141d6a98a01bba1b7f"}},{"name":"convert.cuh","mode":"0100644","size":2386,"lastCommit":{"hash":"bd3f1d9d65113737e795cd6d62eb66f90628663c","oid":"bd3f1d9d65113737e795cd6d62eb66f90628663c","parents":["23c9182ce872666aff6bd0fc571c3e3b6ae5fd89"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2026-03-22T17:53:33+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-03-22T17:53:33+01:00"},"message":"CUDA: fix BF16 FA compilation (#20865)","tree":"e09acd824bb48dcb313891985e733fa00c606c95"}},{"name":"count-equal.cu","mode":"0100644","size":2061,"lastCommit":{"hash":"5b359bb1e3585de45bec79fd6c18934897662cdf","oid":"5b359bb1e3585de45bec79fd6c18934897662cdf","parents":["e89213492d3e01705739789733f0f2d250b4c449"],"author":{"name":"SXX","email":"sxx1136965276@gmail.com","when":"2024-11-09T15:35:46+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-11-09T08:35:46+01:00"},"message":"ggml: fix zero division in ‘dne’ calculation in CUDA COUNT_EQUAL operator when ‘ne’ is small (#10213)\n\n","tree":"d55a5fdcaf0eedf69d78eee7ea25f3ef93ecce3c"}},{"name":"count-equal.cuh","mode":"0100644","size":144,"lastCommit":{"hash":"fabdc3bda396307565c4f3f4ecbc3a751a2eb6d3","oid":"fabdc3bda396307565c4f3f4ecbc3a751a2eb6d3","parents":["eee39bdc96065b69242877fe8f1be05c885fc2aa"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-10-03T17:29:59+02:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-10-03T21:17:26+03:00"},"message":"ggml/ex: calculate accuracy in graph, adapt MNIST (ggml/980)\n","tree":"219d3588467c6690d02c975a1bda2b2fa792eeef"}},{"name":"cp-async.cuh","mode":"0100644","size":2769,"lastCommit":{"hash":"70bf64bab3f1db9e955e0143ea2ca94a0da85874","oid":"70bf64bab3f1db9e955e0143ea2ca94a0da85874","parents":["a3a22122eb800d4d7ffa104f3ad6547e692afa3e"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ cp.async double-buffering (mmq-pipeline)\n\nPTQ1_0 MMQ: activations double-buffered with cp.async and the next weight tile fetched into registers during compute. Bit-identical, +3.3% on the matmuls.\n","tree":"189acb482fee8c114a339d712c7e8f40c3522638"}},{"name":"cpy-utils.cuh","mode":"0100644","size":6332,"lastCommit":{"hash":"96ac5a2329029dfc35c9cbbb24c09fd91ae9416b","oid":"96ac5a2329029dfc35c9cbbb24c09fd91ae9416b","parents":["bc809e9c53ee34fe059d7b820072e2a22c51b9e2"],"author":{"name":"Sigbjørn Skjæret","email":"sigbjorn.skjaeret@scala.com","when":"2025-11-23T11:13:34+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-11-23T11:13:34+01:00"},"message":"cuda : support non-contiguous i32 to i32 copy (#17326)\n\n* support non-contiguous i32 to i32 copy\n\n* add tests\n\n* rename cpy_flt to cpy_scalar and reindent params","tree":"3cd57c3c05be5a70fcb8e089af57f533f99da02e"}},{"name":"cpy.cu","mode":"0100644","size":31513,"lastCommit":{"hash":"69bf6437914596fbbc4caf09a7ac16f2acdd1a94","oid":"69bf6437914596fbbc4caf09a7ac16f2acdd1a94","parents":["3653e6d6d547ec763317d9ecd0ace334a7e21359"],"author":{"name":"Rafail Giavrimis","email":"47496212+grafail@users.noreply.github.com","when":"2026-08-08T05:40:04+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-08T07:40:04+03:00"},"message":"CUDA: fix thread/block count in quantized cpy kernel launches (#26731)\n\n* CUDA: fix thread/block count in quantized cpy kernel launches\n\n* tests: add uneven block count cpy case","tree":"b0ce255996a1bc91c6a64ea2d301b3a7a635fbcb"}},{"name":"cpy.cuh","mode":"0100644","size":227,"lastCommit":{"hash":"5b6913c47b6bc71a6f927805a45387d5657d8b89","oid":"5b6913c47b6bc71a6f927805a45387d5657d8b89","parents":["bc07349a7f87ba6eb31ed4b0ea9d9a7352185213"],"author":{"name":"Anav Prasad","email":"anavp@nvidia.com","when":"2025-10-14T09:53:49Z"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-10-14T11:53:49+02:00"},"message":"cuda : remove legacy copy-op pointer indirection code (#16485)\n\n* remove legacy copy-op pointer indirection code\n\n* further removal of copy-op indirection code\n\n* renamed check_node_graph_compatibility_and_refresh_copy_ops function","tree":"8589729d0db1fd5671c02c386f1fe8f33d8309ab"}},{"name":"cross-entropy-loss.cu","mode":"0100644","size":6076,"lastCommit":{"hash":"55c2646b452419be7552157b9b11373ccc1bf2f2","oid":"55c2646b452419be7552157b9b11373ccc1bf2f2","parents":["e75ba4c0434eb759eb7ff74e034ebe729053e575"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-07-03T07:45:11+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-07-03T07:45:11+08:00"},"message":"CUDA: add dynamic shared mem to softmax, refactor general usage (#14497)\n\n","tree":"6a4bd96f503e7e5fac00c3e6586e334cf60c1c34"}},{"name":"cross-entropy-loss.cuh","mode":"0100644","size":251,"lastCommit":{"hash":"424c5d00a9b97dd5559635872db9b57f87c23b02","oid":"424c5d00a9b97dd5559635872db9b57f87c23b02","parents":["a6809c6a2e8163cba296d4cc0c5cd4bbc8073638"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-09-20T19:04:44+03:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-09-20T21:15:05+03:00"},"message":"ggml/examples: add backend support for numerical optimization (ggml/949)\n\n* CUDA eval works\n\n* stochastic gradient descent op\n\n* Adam except decay\n\n* CUDA CROSS_ENTROPY_LOSS_BACK\n\n* CUDA mnist-fc training works\n\n* backend CLI arg\n\n* refactor gguf load\n\n* remove sched from opt_step_adam\n\n* implement l1 regularization (weight decay)\n\n* extra call to add optimizer\n\n* initialize gradients with ggml_graph_reset\n\n* gradient accumulation\n\n* increment iter per eval instead of epoch\n\n* adjust backend interfaces\n\n* fix ggml_graph_reset without backend\n\n* fix ggml graph export/import\n\n* fixup\n\n* rename\n\n* revert ggml_opt changes\n\n* more general CUDA repeat_back\n\n* update documentation, fix CNN\n\n* validation split\n\n* add clarifying comment\n\n* optimize PyTorch training\n\n* adjust buffer size, thread count\n\n* fix 0.0f validation split\n\n* Update examples/mnist/mnist-common.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* fix gradient accumulation\n\n* tensor flag for accumulators -> tensor hash set\n\n* Update include/ggml.h\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* fix test prints\n\n* Update src/ggml-backend.c\n\nCo-authored-by: Georgi Gerganov \n\n* better CUDA support for noncontiguous out_prod\n\n* add comment\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: slaren \n","tree":"6f3a1a31269e26c2ac6c650aeecd8d713357317d"}},{"name":"cumsum.cu","mode":"0100644","size":10987,"lastCommit":{"hash":"d3dce4e0a5f2fd497a312a3152f2eab939f0d519","oid":"d3dce4e0a5f2fd497a312a3152f2eab939f0d519","parents":["4974bf53cf14073c7b66e1151348156aabd42cb8"],"author":{"name":"Daniel Bevenius","email":"daniel.bevenius@gmail.com","when":"2026-01-04T21:22:16+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-01-04T22:22:16+02:00"},"message":"sampling : add support for backend sampling (#17004)\n\n* sampling : add support for backend sampling\n\nThis commit adds support for performing sampling operations on the\nbackend (e.g. GPU) as part of the model computation graph.\n\nThe motivation for this feature is to enable sampling to be performed\ndirectly on the backend as part of the computation graph being executed,\nallowing for some or all of the sampling to be done on the backend.\n\nFor example, the backend sampler chain might select/sample a token\ndirectly in which case only the sampled token needs to be transferred\nfrom device memory to host memory.\n\nIt is also possible for the backend samplers to perform filtering of\nthe logits, or compute and filter the probability distribution, in\nwhich case only the filtered logits or probabilites need to be\ntransferred back to system memory for further processing by CPU\nsamplers.\n\nCurrently the backend sampling works in a similar manner to how\npooling works, it is a function that is called by build_graph and the\nsampler operations become part of the models computation graph.\n\n* llama-cli : add backend sampler configuration\n\n* server : add backend sampling options/configuration\n\n* webui : add backend sampling options\n\n* ggml : add initial cumsum implementation for CUDA\n\n* sampling : enable all backend sampler tests\n\nThis commit enables all exisiting backend sampler tests in the\ntest-backend-sampler. Previously, some tests were disabled because\nthere were missing ggml operation implementations.\n\n* graph : do not include llama-model.h\n\n* sampling : always expose sampled_ids\n\nThis commit precomputes and caches the full-vocab token id list in\nllama_context's constructor, so llama_get_backend_sampled_token_ids_ith\nalways returns a valid pointer.\n\nThe motivation for this is that this enables both common/sampling.cpp\nand src/llama-sampling.cpp can simplify their logic.\n\nNot all backends samplers that process logits need to set the\nsampled_tokens_id as they may not change the order of the logits, for\nexample the temperature sampler only scales the logits but does not\nchange their order. Simliar the logit bias sampler only adds bias to\nspecific token ids but does not change the order of the logits. In\nthese cases there will not be a device to host copy of the sampled\ntoken ids, and this is the use case where having this precomputed\nlist is useful.\n\n* sampling : ensure at most one output token per seq\n\nThis commit adds a check in the batch allocator to ensure that when\nbackend sampling is enabled, at most one output token is specified per\nsequence.\n\n* CUDA: Optimize argsort for gpu-based token sampling\n\nArgsort is used for top-k currently. WE optimize argsort by 2 things:\n\n1. Use `DeviceRadixSort` for single-row/sequence to parallelize it\n across our SMs\n2. Use `DeviceSegmentedSort` for multi-row/sequence as this is the\n correct entrypoint (the function chooses different execution paths,\n it contains `DeviceSegmentedRadixSort` as one of the paths and will\n choose the best one according to heuristics.\n https://nvidia.github.io/cccl/cub/api/structcub_1_1DeviceSegmentedSort.html#overview\n\nSome perf numbers for a RTX PRO 6000:\n\nOn the kernel level, tested with\n`GGML_CUDA_DISABLE_GRAPHS=1 ./test-backend-ops -o ARGSORT perf`\nBefore:\n```\n ARGSORT(type=f32,ne=[65000,16,1,1],order=0): 4130 runs - 359.24 us/run\n ARGSORT(type=f32,ne=[200000,1,1,1],order=0): 8192 runs - 861.34 us/run\n ARGSORT(type=f32,ne=[200000,16,1,1],order=0): 1343 runs - 1020.01 us/run\n```\n\nAfter:\n```\n ARGSORT(type=f32,ne=[65000,16,1,1],order=0): 4130 runs - 312.41 us/run\n ARGSORT(type=f32,ne=[200000,1,1,1],order=0): 16384 runs - 63.48 us/run\n ARGSORT(type=f32,ne=[200000,16,1,1],order=0): 1343 runs - 874.36 us/run\n```\n\n---\nOn the model level, tested with\n`llama-cli -m gpt-oss-20b-mxfp4.gguf -n 200 -p \"What is\nthe Capital of Sweden?\" -no-cnv -fa 1 --backend-sampling`\n\nBefore:\n```\nllama_perf_sampler_print: sampling time = 0.25 ms / 207 runs ( 0.00 ms per token, 824701.20 tokens per second)\nllama_perf_context_print: load time = 18215.58 ms\nllama_perf_context_print: prompt eval time = 28.20 ms / 7 tokens ( 4.03 ms per token, 248.19 tokens per second)\nllama_perf_context_print: eval time = 714.79 ms / 199 runs ( 3.59 ms per token, 278.40 tokens per second)\nllama_perf_context_print: total time = 857.62 ms / 206 tokens\n```\n\nAfter\n```\nllama_perf_sampler_print: sampling time = 0.25 ms / 207 runs ( 0.00 ms per token, 828000.00 tokens per second)\nllama_perf_context_print: load time = 18366.92 ms\nllama_perf_context_print: prompt eval time = 35.92 ms / 7 tokens ( 5.13 ms per token, 194.87 tokens per second)\nllama_perf_context_print: eval time = 532.79 ms / 199 runs ( 2.68 ms per token, 373.50 tokens per second)\nllama_perf_context_print: total time = 683.65 ms / 206 tokens\n```\n\n* sampling : remove version from sampler chain\n\nThis commit removes the version field from the sampler chain and instead\nused the sampler pointer itself for change detection.\n\n* sampling : always populate logits for sampled probs\n\nThis commit updates common/sampler.cpp set_logits and\nsrc/llama-sampling.cpp llama_sampler_sample to always populate the\nlogits field when backend sampled probabilities are available.\n\nThe motivation for this is that this ensure that CPU sampler always have\naccess to the logits values even when probabilites have been produced by\nbackend samplers.\n\n* sampling : simplify backend sampling logic decode\n\nThis commit tries to simplify the backend sampling logic in\nllama_context::decode.\n\n* squash! sampling : simplify backend sampling logic decode\n\nFix condition to check if backend actually sampled tokens, not just that\nbackend samplers are available.\n\n* common : fix regression caused by extra memory allocations during sampling\n\n* squash! sampling : simplify backend sampling logic decode\n\nThe commit fixes a variable shadowing issue in the\n`llama_context::decode` function which was introduced in a previous\nrefactoring.\n\n* squash! common : fix regression caused by extra memory allocations during sampling\n\nApply the same changes to llama-sampling.cpp, llama_sampler_sample as\nwere applied in commit 38f408c25.\n\n* sampling : introduce sampling_info struct\n\nThis commit introduces a sampling_info struct to encapsulate all\nbackend sampling related data within the llama_context class.\n\nIt also updates to use more descriptive names for sampled tokens and\ncandidates in the backend sampler ggml data structure.\n\n* sampling : return early if backend sampling is disabled\n\n* sampling : use pinned memory for backend sampling buffers\n\n* common, tools : refactor model loading to support backend samplers\n\nThis commit refactors the model loading process in common/common.cpp\nto enable backend sampler to be configure prior to the llama_context\ncreation.\n\nThe motivation for this change is that just being able to set/reset the\nbackend samplers after the llama_context has been created will cause a\nresize to occur in llama_context::output_reserve which we want to avoid.\n\n* sampling : add stride variable for clarity\n\n* sampling: clarify candidate ids usage in comments\n\n* sampling : fix copying both sampled tokens and logits/probs from backend\n\nThis commit fixes the issue where both sampled tokens and logits/probs\nwere not being copied correctly from the backend to the host when\nmultiple backend samplers were used.\n\nA test for this scenario has also been added to ensure that both types\nof data are copied correctly when different backend samplers are\nemployed.\n\n* tests : cleanup test-backend-sampler.cpp\n\n* common : remove build-info.cpp from commit [no ci]\n\nThis file was generated during the build process and should not be\nincluded in previous commits.\n\n* sampling : cleanup and clarify output_reserve\n\n* sampling : remove redundant checks for stride and size [no ci]\n\n* sampling : add debug log when backend sampler selects token\n\nThis commit adds a debug log statement in the llama_sampler_sample\nto indicate when a backend sampler has selected a token for a given\nindex.\n\nThe modification helps in tracing the sampling process and understanding\nthe flow of control when backend samplers are used.\n\n* examples : update batched to use backend sampling\n\nThis commit updates the batched example to demonstrate how to use\nbackend samplers.\n\n* llama-cli : fix dangling reference to sampler config\n\n* common : initialize backend samplers\n\n* samplers : add missing cont\n\n* sampling : add assertions for contiguous tensors in async copy functions\n\n* examples : add info about hybrid sampling in batched [no ci]\n\n* sampling : remove backend-dist option (wip)\n\nThis commit removes the `--backend-dist` option and instead uses the\nconfigured --samplers chain to determine which samplers run on the\nbackend.\n\nBackend sampling is still enabled using With `--backend_sampling`, and\nthe sampler chain, either explictly specified using `--samplers` or the\ndefault, is automatically analyzed to determine which samplers can run\non the backend. The system finds the longest contiguous chain of\nbackend supported samplers from the start of the sampler sequence.\nFor example:\n\n* If the chain is `top-k -> temperature -> top-p`, and both `top-k` and\n `temperature` are backend-supported but `top-p` is not, then `top-k`\n and `temperature` will run on the backend, while `top-p` and\n subsequent samplers run on the CPU.\n\n* If all configured samplers are supported, the final distribution\n sampling will also happen on the backend, transferring only the\n sampled token IDs back to the host.\n\n* If the sampler chain starts with an unsupported sampler (e.g.,\n `penalties`), all sampling runs on the CPU. Note that this is\n currently the case with the default sampler so to use backend sampling\n it is required to specify a sampler chain. See below for an example.\n\nThe following shows how llama-cli can be run with backend sampling:\n```console\n$ llama-cli -m models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf \\\n --prompt 'What is the capital of Sweden?' \\\n -n 20 \\\n -no-cnv \\\n --verbose-prompt \\\n -ngl 40 \\\n --backend-sampling \\\n --samplers 'top_k;temperature'\n```\nIn this case the all sampling will happen on the backend since both\n`top_k` and `temperature` are supported backend samplers.\n\nTo enable a partial backend sampling (hybrid sampling), for example\nrunning `top_k` and `temperature` on the backend and `typ_p` on the CPU\nthe following sampler chain could be specified:\n```console\n$ llama-cli -m models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf \\\n --prompt 'What is the capital of Sweden?' \\\n -n 20 \\\n -no-cnv \\\n --verbose-prompt \\\n -ngl 40 \\\n --backend-sampling \\\n --samplers 'top_k;temperature;top_p'\n```\n\nIf this looks good then I'll follow up with updates the llama-cli and\nllama-server documentation to reflect these changes.\n\n* CUDA: Add top-k implementation\n\n* sampling : add min-p backend sampler\n\n* Use `FetchContent` over CPM as it's bundled with CMake\n\nThanks @ggerganov for the suggestion\n\n* common : add get_active_samplers function to check enabled samplers\n\nThis commit adds a function to check if a sampler is actually enabled,\nmeaning that it does not have values that disables its effect. This is\nthen used by the backend samplers initialization to avoid considering\nsamplers that are not enabled when determining the split point between\nthem.\n\nThe motivation for this is that this allows the default sampler chain\nfor `--samplers` to be used and any sampler that is not enabled will not\ncause the backend samplers to be skipped.\nFor example, before this change if the penalties sampler was included in\nthe samplers list but had default values that disable it, it would cause\nthe backend samplers to be skipped entirely.\n\nThis commit also contains some refactoring to remove some code\nduplication.\n\n* cuda : fix editorconfig-checker warning\n\n* sampling : use argmax for min-p sampling\n\n* sampling : fix temperature check to allow zero temperature\n\nThis commit modifies the temperature sampling check to allow a\ntemperature value of zero. Previously, the check only allowed\npositive temperature values, which excluded the valid case of\nzero temperature.\n\nThe motivation for this is to enable a zero temperature setting which is\nalso currently causing the following test to fail:\n```console\n(venv) $ cd tools/server/tests\n(venv) $ ./tests.sh unit/test_basic.py::test_load_split_model\n```\n\n* cuda : fix top-k compilation when CUB is unavailable\n\nThis commit adds a macro guard around argsort_f32_i32_cuda_cub usage\nin the top-k fallback path, falling back to bitonic sort when\nGGML_CUDA_USE_CUB is not defined.\n\nThe motivation for this is that some environments like AMD HIP\ndo not have CUB available, causing compilation failure.\n\nRefs: https://github.com/ggml-org/llama.cpp/actions/runs/19728226426/job/56523606840#step:6:208\n\n* sampling : add comments about backend sampler [no ci]\n\nThis commit adds a comment to llama_context's constructor explaining why\nbackend samplers are initialized early in the process.\n\n* sampling : remove backend sampling chain from common_sampler\n\nThis commit removes the backend sampling chain from the common_sampler\nstructure and related functions.\n\nThe motivation for this change is that the backend samplers are not\ncurrently set on the context, and if they are they would cause the\na graph reallocation to occur. Instead, the intialization is handled\nlike it currently is by llama_context's constructor.\n\n* Fix top-k comp & behavior for non-CUB path\n\nSome changes were made in 5ea3be265ba6f8916daf52e19e3fb8efe9a03637\nwhich were incomplete. In the case of non-CUB, bitonic sort and its\nlimitations of ncols < 1024 have to apply, similar to argsort.cu\n\n* sampling : support intermixed backend/cpu samplers\n\nThis commit updates the backend sampling implementation to support\nintermixed usage of backend and CPU samplers within the same batch.\n\nThe initial implementation was developed as an all-or-nothing solution:\neither perform backend sampling for the entire batch, or perform CPU\nsampling for the entire batch.\n\nThe motivation for this change is to support batches with mixed\nsequences. For example, we may have a backend sampler configured for\nsequence 0, while sequence 1 in the same batch uses CPU sampling. This\nwas not supported in the initial implementation.\n\nThis issue manifested in llama-server with the webui: decoding with\nbackend samplers would work initially, but after changing to CPU\nsampling, a slot (sequence) could still be using a backend sampler.\nThis meant that logits in output_reserve would not be allocated,\nresulting in an error.\n\nThe solution in this commit inspects the batch to determine which\nsampling modes are needed and allocates buffers accordingly. However,\nthere is a known inefficiency: when we have intermixed backend/CPU\nsamplers in the same batch, we currently copy all logits to the host,\neven for sequences using backend samplers.\n\nAdded test_backend_cpu_mixed_batch to verify correct behavior with\nmixed backend/CPU samplers in a single batch, including dynamic\nsampler switching between decode calls.\n\n* squash! sampling : support intermixed backend/cpu samplers\n\nAdd check that logits is not null which is can happen for embeddings.\n\n* squash! sampling : support intermixed backend/cpu samplers\n\nFix llama-save-load-state which currently fails by handling the case\nwhen batch.logits is nullptr (like when loading state) by allocating\nspace for all outputs as CPU logits.\n\n* refactor : simplify and improve memory management\n\n* Add initial version for top-p sampling\n\nAs we only support static graphs for the time and we don't know the size\nof the output of top-p, we have to do value-scaling same as for min-p\noperator.\n\nFurther improvements can be applied to the unit-test (i.e. check for\nequivalence of top_p happening on backend with top_p happening on cpu)\nand also by constructing candidates and sorting those as opposed to\nreversing the sort of the logits (this would be arange +\nget_rows instead of argsort + get_rows)\n\n* sampling : use logits directly for min-p filtering\n\n* sampling : simplify\n\n* llama : simplify\n\n* llama : cleanup + naming\n\n* llama : call backend_init once\n\n* llama : reserve graphs with samplers\n\n* llama : naming\n\n* cont : naming\n\n* sampling : lower log level for output buffer reallocations [no ci]\n\nThis commit changes the logging level for output buffer reallocations\nin the llama_context::output_reserve function from INFO to DEBUG.\n\nThe motivation for this is that it currently logs to info and when\nenabling verbose logging for llama-cli this will get mixed with the\noutput, for example:\n\n```console\nWhat is the capital of Sweden?output_reserve: reallocating output buffer from size 0.58 MiB to 1.74 MiB\n 1. Stockholm\n2\\. Helsinki\nBased are the options\n1. Stockholm\nExplanation: Stockholm is the capital of\n...\n```\n\n* Fix backend_top_p_sampler\n\nsoftmax(softmax) will return uniform distribution, so we should not\nreturn the softmax but the logits instead.\n\n* Factor out `ggml_sort` into its own function\n\n* Make backend's top_p sampler inclusive\n\nIn addition to match the algorithm proposed in the original\n[paper](https://arxiv.org/abs/1904.09751), this resolves the edge-case\nwhere `max_p is > top_p` for a single logit, where the mask would\notherwise be empty (and we thus sample from the whole vocabulary with\nequal likelihood)\n\n* common : simplify sampler chain initialization\n\n* sampling : do not create empty samplers\n\n* sampling : fix top_p empty condition\n\n* examples : remove outdated backend sampling section\n\nThis commit removes the outdated section about using backend samplers\nfrom the README.md file in the examples/batched.\n\n* sampling : fix backend temp sampler for zero temperature\n\nThis commit fixes the implementation of the temperature-based sampler\nfor the case when the temperature is set to zero. This now correctly\nselects the most probable token by masking out all other tokens in the\nlogits.\n\n* CUDA: Move cccl fetch to after cuda has been enabled in CMakeLists.txt\n\nThis will allow cccl to set build flags for the CUDA compiler, required\ne.g. for MSVC compat, see also\nhttps://github.com/NVIDIA/cccl/pull/6791\n\n* CUDA: Use standard-compliant preprocessor for MSVC builds\n\nWorkarounds of https://github.com/NVIDIA/cccl/pull/6791 will not be\nbackported to CCCL 3.2, only the diagnostics/error messages will:\nhttps://github.com/NVIDIA/cccl/pull/6827\n\n* CUDA: Update CCCL's rc candidate\n\n* squash! sampling : fix backend temp sampler for zero temperature\n\nThis modifies the parent commit to simply return the most probably token\ninstead of masking the logits.\n\n* sampling : implement temp_ext_backend sampling\n\nThis commit implements the apply function for the extended temperature\nsampling.\n\n* sampling : minor cleanup\n\n* sampling : stop short if backend sampler sampled a token\n\nThis commit modifies the graph building logic to immediately continue\nwhen a token has already been sampled by the backend sampler.\n\nIt also updates the test for backend temporary sampling to include\ntop-k and distribution samplers in the chain to verify that they are not\nproducing any logits (they are not run).\n\n* Revert \"sampling : stop short if backend sampler sampled a token\"\n\nThis reverts commit 87b2719eca55b30afff600fc7f61c6cce9452cbf.\n\n* sampling : fix backend temp sampling to use logits masking\n\n* sampling : simplify temp sampling\n\n* sampling : remove redundant calls to ggml_build_forward_expand\n\n* sampling : check backend support during init\n\n* cont : keep backend sampling disabled for now\n\n* sampling : fix outputs and device checks\n\n* sampling : fix candidates logic\n\n* Add perf-tests for CUMSUM\n\n* Readd `cub::DeviceScan::InclusiveSum`-based CumSum\n\nFor single rows and large columns doing a for-loop over the function\n`cub::DeviceScan::InclusiveSum` offered by CUB outperforms the\n`cumsum_cub_kernel` where `cub::BlockScan` is used.\n\nNumbers before this change\n\n Backend 1/3: CUDA0\n Device description: NVIDIA RTX 6000 Ada Generation\n Device memory: 48510 MB (48039 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 311258 runs - 3.26 us/run - 2048 kB/run - 599.76 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 229390 runs - 4.40 us/run - 5120 kB/run - 1110.23 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 37583 runs - 29.63 us/run - 6250 kB/run - 201.18 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 892819 runs - 1.12 us/run - 1 kB/run - 0.85 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 450505 runs - 2.25 us/run - 8 kB/run - 3.39 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 155629 runs - 6.61 us/run - 32 kB/run - 4.62 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 81910 runs - 12.60 us/run - 64 kB/run - 4.85 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 49146 runs - 23.99 us/run - 128 kB/run - 5.09 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 24573 runs - 47.10 us/run - 256 kB/run - 5.18 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 16382 runs - 93.57 us/run - 512 kB/run - 5.22 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 8191 runs - 184.79 us/run - 1024 kB/run - 5.29 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 8191 runs - 280.43 us/run - 1562 kB/run - 5.31 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 2148 runs - 2771.23 us/run - 15625 kB/run - 5.38 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 458696 runs - 2.21 us/run - 4 kB/run - 1.73 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 360404 runs - 2.82 us/run - 32 kB/run - 10.83 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 147438 runs - 7.12 us/run - 128 kB/run - 17.15 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 81910 runs - 12.90 us/run - 256 kB/run - 18.92 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 49146 runs - 24.32 us/run - 512 kB/run - 20.08 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 24573 runs - 47.28 us/run - 1024 kB/run - 20.66 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 16382 runs - 93.21 us/run - 2048 kB/run - 20.96 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 8191 runs - 185.04 us/run - 4096 kB/run - 21.11 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 5369 runs - 282.08 us/run - 6250 kB/run - 21.13 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 537 runs - 2806.46 us/run - 62500 kB/run - 21.26 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 458696 runs - 2.20 us/run - 8 kB/run - 3.47 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 360404 runs - 2.82 us/run - 64 kB/run - 21.66 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 147438 runs - 7.12 us/run - 256 kB/run - 34.28 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 81910 runs - 12.90 us/run - 512 kB/run - 37.84 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 49146 runs - 24.32 us/run - 1024 kB/run - 40.15 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 47.28 us/run - 2048 kB/run - 41.31 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 93.20 us/run - 4096 kB/run - 41.92 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 8194 runs - 185.05 us/run - 8192 kB/run - 42.22 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 5370 runs - 282.15 us/run - 12500 kB/run - 42.26 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 269 runs - 4067.61 us/run - 125000 kB/run - 29.36 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 303067 runs - 3.32 us/run - 16 kB/run - 4.60 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 303067 runs - 3.32 us/run - 128 kB/run - 36.76 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 147438 runs - 7.17 us/run - 512 kB/run - 68.13 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 81910 runs - 12.90 us/run - 1024 kB/run - 75.68 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 49146 runs - 24.33 us/run - 2048 kB/run - 80.28 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 47.30 us/run - 4096 kB/run - 82.59 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 93.24 us/run - 8192 kB/run - 83.80 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 6147 runs - 185.07 us/run - 16384 kB/run - 84.45 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 4029 runs - 282.40 us/run - 25000 kB/run - 84.46 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 270 runs - 4118.40 us/run - 250000 kB/run - 58.11 GB/s\n Backend CUDA0: OK\nBackend 2/3: CUDA1\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96677 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 368595 runs - 2.73 us/run - 2048 kB/run - 715.83 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 216282 runs - 4.72 us/run - 5120 kB/run - 1035.32 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 32214 runs - 34.33 us/run - 6250 kB/run - 173.64 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 810909 runs - 1.24 us/run - 1 kB/run - 0.77 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 401359 runs - 2.52 us/run - 8 kB/run - 3.03 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 139247 runs - 7.44 us/run - 32 kB/run - 4.10 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 73719 runs - 14.27 us/run - 64 kB/run - 4.28 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 40955 runs - 27.24 us/run - 128 kB/run - 4.48 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 24573 runs - 53.46 us/run - 256 kB/run - 4.57 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 16382 runs - 105.29 us/run - 512 kB/run - 4.64 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 8191 runs - 210.15 us/run - 1024 kB/run - 4.65 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 8191 runs - 318.22 us/run - 1562 kB/run - 4.68 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 2148 runs - 3142.23 us/run - 15625 kB/run - 4.74 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 303067 runs - 3.34 us/run - 4 kB/run - 1.14 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 253921 runs - 4.03 us/run - 32 kB/run - 7.58 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 73719 runs - 14.96 us/run - 256 kB/run - 16.32 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 40955 runs - 28.66 us/run - 512 kB/run - 17.04 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 24573 runs - 54.21 us/run - 1024 kB/run - 18.01 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 16382 runs - 106.49 us/run - 2048 kB/run - 18.34 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 8191 runs - 210.88 us/run - 4096 kB/run - 18.52 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 5369 runs - 321.77 us/run - 6250 kB/run - 18.53 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 537 runs - 3191.79 us/run - 62500 kB/run - 18.69 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 376786 runs - 2.67 us/run - 8 kB/run - 2.86 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 245730 runs - 4.10 us/run - 64 kB/run - 14.90 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 122865 runs - 8.20 us/run - 256 kB/run - 29.79 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 65528 runs - 16.38 us/run - 512 kB/run - 29.82 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 40955 runs - 28.69 us/run - 1024 kB/run - 34.04 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 55.28 us/run - 2048 kB/run - 35.33 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 108.50 us/run - 4096 kB/run - 36.00 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 8194 runs - 213.75 us/run - 8192 kB/run - 36.55 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 5370 runs - 326.31 us/run - 12500 kB/run - 36.54 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 538 runs - 3252.68 us/run - 125000 kB/run - 36.72 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 303067 runs - 3.32 us/run - 16 kB/run - 4.60 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 253921 runs - 4.06 us/run - 128 kB/run - 30.09 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 122865 runs - 8.20 us/run - 512 kB/run - 59.57 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 65528 runs - 16.38 us/run - 1024 kB/run - 59.63 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 28.69 us/run - 2048 kB/run - 68.09 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 55.28 us/run - 4096 kB/run - 70.67 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 108.50 us/run - 8192 kB/run - 72.02 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 6147 runs - 213.60 us/run - 16384 kB/run - 73.17 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 4029 runs - 326.04 us/run - 25000 kB/run - 73.15 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 270 runs - 5458.69 us/run - 250000 kB/run - 43.84 GB/s\n\n----\nNumbers after:\n\nBackend 1/3: CUDA0\n Device description: NVIDIA RTX 6000 Ada Generation\n Device memory: 48510 MB (48039 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 311258 runs - 3.25 us/run - 2048 kB/run - 601.62 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 229390 runs - 4.40 us/run - 5120 kB/run - 1110.14 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 37583 runs - 29.67 us/run - 6250 kB/run - 200.89 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 892819 runs - 1.12 us/run - 1 kB/run - 0.85 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 458696 runs - 2.21 us/run - 8 kB/run - 3.45 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 376786 runs - 2.66 us/run - 32 kB/run - 11.46 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 393168 runs - 2.59 us/run - 64 kB/run - 23.57 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 393168 runs - 2.59 us/run - 128 kB/run - 47.15 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 376786 runs - 2.69 us/run - 256 kB/run - 90.69 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 327640 runs - 3.06 us/run - 512 kB/run - 159.65 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 311258 runs - 3.28 us/run - 1024 kB/run - 297.77 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 270303 runs - 3.74 us/run - 1562 kB/run - 398.14 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 137472 runs - 7.35 us/run - 15625 kB/run - 2026.94 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 876437 runs - 1.14 us/run - 4 kB/run - 3.33 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 442314 runs - 2.28 us/run - 32 kB/run - 13.39 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 155629 runs - 6.69 us/run - 128 kB/run - 18.24 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 81910 runs - 12.53 us/run - 256 kB/run - 19.49 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 49146 runs - 24.18 us/run - 512 kB/run - 20.20 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 65528 runs - 15.34 us/run - 1024 kB/run - 63.66 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 73719 runs - 14.76 us/run - 2048 kB/run - 132.35 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 65528 runs - 16.01 us/run - 4096 kB/run - 244.07 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 64428 runs - 16.51 us/run - 6250 kB/run - 360.97 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 33831 runs - 29.59 us/run - 62500 kB/run - 2016.08 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 868246 runs - 1.16 us/run - 8 kB/run - 6.59 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 442314 runs - 2.28 us/run - 64 kB/run - 26.76 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 155629 runs - 6.69 us/run - 256 kB/run - 36.48 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 81910 runs - 12.53 us/run - 512 kB/run - 38.97 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 49146 runs - 24.17 us/run - 1024 kB/run - 40.41 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 47.53 us/run - 2048 kB/run - 41.10 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 61.25 us/run - 4096 kB/run - 63.77 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 32776 runs - 31.79 us/run - 8192 kB/run - 245.82 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 32220 runs - 32.90 us/run - 12500 kB/run - 362.35 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 6725 runs - 151.99 us/run - 125000 kB/run - 785.77 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 851864 runs - 1.18 us/run - 16 kB/run - 12.97 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 442314 runs - 2.30 us/run - 128 kB/run - 53.13 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 155629 runs - 6.68 us/run - 512 kB/run - 73.13 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 81910 runs - 12.68 us/run - 1024 kB/run - 77.00 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 24.56 us/run - 2048 kB/run - 79.53 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 47.52 us/run - 4096 kB/run - 82.21 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 93.44 us/run - 8192 kB/run - 83.62 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 16392 runs - 63.36 us/run - 16384 kB/run - 246.68 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 16116 runs - 65.25 us/run - 25000 kB/run - 365.53 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 3375 runs - 304.46 us/run - 250000 kB/run - 785.98 GB/s\n Backend CUDA0: OK\nBackend 2/3: CUDA1\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96677 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 376786 runs - 2.69 us/run - 2048 kB/run - 727.04 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 216282 runs - 4.64 us/run - 5120 kB/run - 1053.30 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 32214 runs - 34.21 us/run - 6250 kB/run - 174.27 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 819100 runs - 1.22 us/run - 1 kB/run - 0.78 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 409550 runs - 2.47 us/run - 8 kB/run - 3.09 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 303067 runs - 3.31 us/run - 32 kB/run - 9.21 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 237539 runs - 4.33 us/run - 64 kB/run - 14.08 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 237539 runs - 4.33 us/run - 128 kB/run - 28.17 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 188393 runs - 5.37 us/run - 256 kB/run - 45.47 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 188393 runs - 5.41 us/run - 512 kB/run - 90.20 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 188393 runs - 5.41 us/run - 1024 kB/run - 180.41 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 188393 runs - 5.41 us/run - 1562 kB/run - 275.27 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 128880 runs - 7.76 us/run - 15625 kB/run - 1920.33 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 802718 runs - 1.26 us/run - 4 kB/run - 3.03 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 401359 runs - 2.51 us/run - 32 kB/run - 12.18 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 139247 runs - 7.51 us/run - 128 kB/run - 16.26 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 73719 runs - 14.17 us/run - 256 kB/run - 17.23 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 40955 runs - 27.37 us/run - 512 kB/run - 17.84 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 40955 runs - 26.33 us/run - 1024 kB/run - 37.10 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 40955 runs - 26.19 us/run - 2048 kB/run - 74.59 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 40955 runs - 26.35 us/run - 4096 kB/run - 148.26 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 42952 runs - 24.18 us/run - 6250 kB/run - 246.51 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 32757 runs - 31.01 us/run - 62500 kB/run - 1923.68 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 786336 runs - 1.28 us/run - 8 kB/run - 5.95 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 393168 runs - 2.57 us/run - 64 kB/run - 23.73 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 131056 runs - 7.67 us/run - 256 kB/run - 31.82 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 73719 runs - 14.43 us/run - 512 kB/run - 33.84 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 40955 runs - 27.90 us/run - 1024 kB/run - 35.01 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 54.63 us/run - 2048 kB/run - 35.75 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 72.24 us/run - 4096 kB/run - 54.08 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 20485 runs - 52.66 us/run - 8192 kB/run - 148.37 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 21480 runs - 48.00 us/run - 12500 kB/run - 248.42 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 16140 runs - 61.99 us/run - 125000 kB/run - 1926.51 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 786336 runs - 1.28 us/run - 16 kB/run - 11.90 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 393168 runs - 2.57 us/run - 128 kB/run - 47.57 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 131056 runs - 7.65 us/run - 512 kB/run - 63.83 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 73719 runs - 14.42 us/run - 1024 kB/run - 67.74 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 27.87 us/run - 2048 kB/run - 70.09 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 54.54 us/run - 4096 kB/run - 71.63 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 107.53 us/run - 8192 kB/run - 72.66 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 10245 runs - 105.10 us/run - 16384 kB/run - 148.70 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 10744 runs - 95.36 us/run - 25000 kB/run - 250.11 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 5400 runs - 186.97 us/run - 250000 kB/run - 1279.90 GB/s\n\n* sampling : expand support (wip)\n\n* tests : fix memory leaks\n\n* cont : fixes\n\n* tests : check temp back to 0.0\n\n* sampling : fix top-p\n\n* sampling : handle n_probs case\n\n* server : handle unsupported cases\n\n* metal : print node names for debugging\n\n* ggml : remove redundant src in ggml_cast\n\n* ggml-alloc : fix reuse-parent logic for misaligned sizes\n\n* Revert \"ggml : remove redundant src in ggml_cast\"\n\nThis reverts commit 62d1b0082dbad699fbeea85a096bc334e3c1c0e6.\n\n* CUDA: Add Cooperative-Groups-based parallelization of ncols in softmax\n\nOld implementation parallelizes rows across SMs, which does not fit the\nneeds of backend-sampling (where we have ncols >> nrows and thus want to\nparallelize ncols across SMs)\n\n* Add TODOs to and adjust heuristics of row-wise soft_max in CUDA\n\nHeuristics were selected based on the following numbers:\n\n```\n-- Before\nBackend 1/2: CUDA0\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96691 MB free)\n\n SOFT_MAX(type=f32,ne=[4096,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 2236 runs - 450.34 us/run - 655360 kB/run - 1401.20 GB/s\n SOFT_MAX(type=f32,ne=[12888,256,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 17748 runs - 56.80 us/run - 128880 kB/run - 2168.19 GB/s\n SOFT_MAX(type=f32,ne=[77,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 57204 runs - 18.35 us/run - 12320 kB/run - 640.57 GB/s\n SOFT_MAX(type=f32,ne=[1024,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9840 runs - 102.46 us/run - 81920 kB/run - 763.45 GB/s\n SOFT_MAX(type=f32,ne=[77,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98064 runs - 10.25 us/run - 6160 kB/run - 573.43 GB/s\n SOFT_MAX(type=f32,ne=[256,256,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98310 runs - 10.25 us/run - 10240 kB/run - 953.20 GB/s\n SOFT_MAX(type=f32,ne=[64,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 5.99 us/run - 640 kB/run - 101.84 GB/s\n SOFT_MAX(type=f32,ne=[77,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 5.97 us/run - 770 kB/run - 123.02 GB/s\n SOFT_MAX(type=f32,ne=[8192,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 6.00 us/run - 64 kB/run - 10.16 GB/s\n SOFT_MAX(type=f32,ne=[8192,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 163820 runs - 6.12 us/run - 256 kB/run - 39.91 GB/s\n SOFT_MAX(type=f32,ne=[8192,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.88 us/run - 1024 kB/run - 141.92 GB/s\n SOFT_MAX(type=f32,ne=[16384,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n SOFT_MAX(type=f32,ne=[16384,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 114674 runs - 8.87 us/run - 512 kB/run - 55.06 GB/s\n SOFT_MAX(type=f32,ne=[16384,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 2048 kB/run - 190.82 GB/s\n SOFT_MAX(type=f32,ne=[32768,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 21.37 us/run - 256 kB/run - 11.43 GB/s\n SOFT_MAX(type=f32,ne=[32768,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 22.54 us/run - 1024 kB/run - 43.33 GB/s\n SOFT_MAX(type=f32,ne=[32768,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 23.92 us/run - 4096 kB/run - 163.32 GB/s\n SOFT_MAX(type=f32,ne=[65536,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 32764 runs - 38.94 us/run - 512 kB/run - 12.54 GB/s\n SOFT_MAX(type=f32,ne=[65536,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24573 runs - 41.94 us/run - 2048 kB/run - 46.57 GB/s\n SOFT_MAX(type=f32,ne=[65536,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24582 runs - 43.09 us/run - 8192 kB/run - 181.32 GB/s\n SOFT_MAX(type=f32,ne=[131072,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 16382 runs - 74.56 us/run - 1024 kB/run - 13.10 GB/s\n SOFT_MAX(type=f32,ne=[131072,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 16382 runs - 79.85 us/run - 4096 kB/run - 48.92 GB/s\n SOFT_MAX(type=f32,ne=[131072,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 12294 runs - 82.41 us/run - 16384 kB/run - 189.64 GB/s\n SOFT_MAX(type=f32,ne=[262144,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8191 runs - 145.16 us/run - 2048 kB/run - 13.46 GB/s\n SOFT_MAX(type=f32,ne=[262144,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8194 runs - 155.46 us/run - 8192 kB/run - 50.26 GB/s\n SOFT_MAX(type=f32,ne=[262144,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 7175 runs - 160.70 us/run - 32768 kB/run - 194.56 GB/s\n SOFT_MAX(type=f32,ne=[524288,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8191 runs - 285.81 us/run - 4096 kB/run - 13.67 GB/s\n SOFT_MAX(type=f32,ne=[524288,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 4098 runs - 306.91 us/run - 16384 kB/run - 50.92 GB/s\n SOFT_MAX(type=f32,ne=[524288,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 3591 runs - 317.06 us/run - 65536 kB/run - 197.32 GB/s\n\n-- After\nBackend 1/2: CUDA0\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96691 MB free)\n\n SOFT_MAX(type=f32,ne=[4096,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 2236 runs - 450.67 us/run - 655360 kB/run - 1400.15 GB/s\n SOFT_MAX(type=f32,ne=[12888,256,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 17748 runs - 56.97 us/run - 128880 kB/run - 2161.50 GB/s\n SOFT_MAX(type=f32,ne=[77,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 57204 runs - 18.35 us/run - 12320 kB/run - 640.36 GB/s\n SOFT_MAX(type=f32,ne=[1024,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9840 runs - 102.46 us/run - 81920 kB/run - 763.42 GB/s\n SOFT_MAX(type=f32,ne=[77,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98064 runs - 10.25 us/run - 6160 kB/run - 573.43 GB/s\n SOFT_MAX(type=f32,ne=[256,256,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98310 runs - 10.25 us/run - 10240 kB/run - 953.21 GB/s\n SOFT_MAX(type=f32,ne=[64,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 7.00 us/run - 640 kB/run - 87.26 GB/s\n SOFT_MAX(type=f32,ne=[77,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.99 us/run - 770 kB/run - 105.05 GB/s\n SOFT_MAX(type=f32,ne=[8192,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 6.02 us/run - 64 kB/run - 10.13 GB/s\n SOFT_MAX(type=f32,ne=[8192,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 163820 runs - 6.12 us/run - 256 kB/run - 39.87 GB/s\n SOFT_MAX(type=f32,ne=[8192,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.91 us/run - 1024 kB/run - 141.40 GB/s\n SOFT_MAX(type=f32,ne=[16384,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n SOFT_MAX(type=f32,ne=[16384,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 114674 runs - 8.79 us/run - 512 kB/run - 55.54 GB/s\n SOFT_MAX(type=f32,ne=[16384,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 2048 kB/run - 190.82 GB/s\n SOFT_MAX(type=f32,ne=[32768,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 131056 runs - 8.11 us/run - 256 kB/run - 30.12 GB/s\n SOFT_MAX(type=f32,ne=[32768,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 22.54 us/run - 1024 kB/run - 43.33 GB/s\n SOFT_MAX(type=f32,ne=[32768,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 23.32 us/run - 4096 kB/run - 167.50 GB/s\n SOFT_MAX(type=f32,ne=[65536,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.19 us/run - 512 kB/run - 59.63 GB/s\n SOFT_MAX(type=f32,ne=[65536,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 40955 runs - 24.59 us/run - 2048 kB/run - 79.43 GB/s\n SOFT_MAX(type=f32,ne=[65536,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24582 runs - 43.21 us/run - 8192 kB/run - 180.84 GB/s\n SOFT_MAX(type=f32,ne=[131072,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.19 us/run - 1024 kB/run - 119.25 GB/s\n SOFT_MAX(type=f32,ne=[131072,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 40955 runs - 24.59 us/run - 4096 kB/run - 158.87 GB/s\n SOFT_MAX(type=f32,ne=[131072,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 12294 runs - 82.37 us/run - 16384 kB/run - 189.74 GB/s\n SOFT_MAX(type=f32,ne=[262144,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 2048 kB/run - 238.28 GB/s\n SOFT_MAX(type=f32,ne=[262144,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 36873 runs - 28.66 us/run - 8192 kB/run - 272.61 GB/s\n SOFT_MAX(type=f32,ne=[262144,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9225 runs - 108.51 us/run - 32768 kB/run - 288.13 GB/s\n SOFT_MAX(type=f32,ne=[524288,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 4096 kB/run - 381.65 GB/s\n SOFT_MAX(type=f32,ne=[524288,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 32784 runs - 31.74 us/run - 16384 kB/run - 492.43 GB/s\n SOFT_MAX(type=f32,ne=[524288,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8721 runs - 121.20 us/run - 65536 kB/run - 516.19 GB/s\n```\n\n* Fix compiler warnings by casting `const` away\n\n* llama : require backend samplers to be of type llama_sampler_chain\n\n* sampling : use host buffer type for inputs\n\n* Try fixing HIP build errors by adding corresponding #defines\n\nWill likely have to disable for MUSA as I didn't find any docs online\n\n* Fix launch logic when supports_cooperative_launch=false\n\n* Disable cooperative groups for musa\n\nDidn't find any doc online, so I don't even know if they support this\n\n* server : reconnect the backend_sampling setting in the WebUI\n\n* graph : make the compute graph constant with respect to active samplers\n\n* batch : fix sequence id ownage\n\n* graph : respect sampler order for graph reuse\n\n* HIP/MUSA: fix build for backend sampling\n\n* sampling : optimize logit_bias sampler\n\n* cont : fix build\n\n* sampling : generic ggml op support detection\n\n* sampling : fix greedy\n\n* tests : run backend sampler tests always on the CPU\n\n* Apply suggestions from code review\n\nCo-authored-by: Johannes Gäßler \n\n* webui : fix lint\n\n* Fix data-race in `soft_max_f32_parallelize_cols_single_row`\n\nBy using `tmp_vals` to store both max values and exponential\naccumulator there was a potential data-race, where the exponential accumulator\nfor a given CTA may have written to `tmp_vals` before all others CTAs have\nread the max value from it.\n\nTo avoid a third g.sync(), an additional temporary data-storage was\nadded. Given that there are syncs in place after writing to gmem, it is\nguaranteed that the previous values for sums/max were read by all CTAs now.\n\n* Apply automated code-formating to softmax.cu\n\n* llama : clarify backend_accept/backend_set_input comments [no ci]\n\n* llama : fix typo in comment [no ci]\n\n* tests : use smart pointers for backend samplers\n\n* tests : use smart pointers for model and context\n\n* tests : remove vocab member from test_model_context\n\nAlso includes some minor cleanups related to nullptr checks.\n\n* tests : extract batch info update to separate method\n\n* tests : fix batch token position tracking in test_backend_sampler.cpp\n\n* tests : add --device option support to backend sampler tests\n\nThis commit adds support for specifying a device to run the test on.\n\n* common : disable backend sampling when grammar is involved\n\n* Fix different RNG-states between backend-sampling and llama-sampling\n\nBy default, we perform a warm-up step where the ggml_cgraph is computed\nonce. For backend-sampling, this graph contains the sampler, and thus\nthe RNG state of the backend's dist sampler is advanced once.\n\nSolution to this is to reset the samplers after the warmup has finished\n\n* Make backend dist sampler use same rnd's as dist sampler\n\nWe sample in double precision and cast to float to match rnd numbers of\nllama_dampler_dist which uses double precision (sampling from\nstd::uniform_real_distribution and\nstd::uniform_real_distribution with same rng will produce\ndifferent sequences).\n\n* Update CCCL version to v3.2.0-rc2\n\n* Build with CCCL 3.2 for CUDA backends\n\nGives best perf for backend-sampling on CUDA. Flag can be removed once\nCCCL 3.2 is bundled within CTK and that CTK version is used in llama.cpp\n\n* tests : revert server test changes (no longer needed)\n\n* ggml : include cub/cub.cuh instead of block_scan.cuh\n\nThis commit updates the include directive in cumsum.cu to use\ncub/cub.cuh instead of cub/block/block_scan.cuh.\n\nThe motivation of this change is that without it compilation fails\nwith the following error:\n```console\n/llama.cpp/ggml/src/ggml-cuda/cumsum.cu(196): error: name followed by \"::\" must be a class or namespace name\n cub::DeviceScan::InclusiveSum(nullptr,\n ^\n\n/llama.cpp/ggml/src/ggml-cuda/cumsum.cu(207): error: name followed by \"::\" must be a class or namespace name\n cub::DeviceScan::InclusiveSum((void *) tmp_alloc.get(), tmp_size, src, dst, ne, stream);\n ^\n\n2 errors detected in the compilation of \"/llama.cpp/ggml/src/ggml-cuda/cumsum.cu\".\ngmake[2]: *** [ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/build.make:317: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/cumsum.cu.o] Error 2\n```\nCommit 83b3b1c271c78bd77664120431aa8c354d68daac (\"cuda: optimize\ncumsum cub path (#18362)\") updated the include directive replacing\ndevice_scan.cuh which is causing this issue.\n\nThis commit uses cub/cub.cuh umbrella header which is consistent with\nother files in the ggml-cuda directory like mean.cu, sum.cu, etc.\n\n* arg : add shorthand for --backend-sampling\n\n* ci : add server workflow with backend sampling\n\n* sampling : fix reshapes\n\n* server : remove printfs\n\n* sampling : zero-initialize input buffers\n\n* minor : add comments + some cleanup\n\n* llama : assert at most one output token per sequence\n\n* tests : add more top_k tests\n\n* CUDA: Fix non-determinism of CUB-based Top-K\n\nDeviceTopK::MaxPairs is an iterative algorithm, where `d_keys_out` is\nwritten after every iteration. As a consequence, it must not overlap\nwith `d_keys_in`, or otherwise undefined behavior occurs (keys are no\nlonger unique in d_keys_in and may map to different values between\niterations)\n\n* CUDA: Optimize index of top_k_cub\n\nBy using the fancy\n[`counting_iterator`](https://nvidia.github.io/cccl/thrust/api/classthrust_1_1counting__iterator.html#classthrust_1_1counting__iterator)\nexposed by CCCL, we can avoid materializing the index to GPU memory,\nsaving VRAM + 1 kernel invocation\n\n* Apply code-formatting to top-k.cu\n\n* CUDA: Remove obsolete temp_keys from CUB\n\nSince we use cuda::discard_iterator to avoid writing out the keys, we\ncan directly pass in src instead of copying it to `temp_keys`\n\n* minor : cleanup, TODOs, etc.\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Oliver Simons \nCo-authored-by: Johannes Gäßler ","tree":"e9f980f624ccdc6f2afe7c6ea75cd3f403d54b35"}},{"name":"cumsum.cuh","mode":"0100644","size":137,"lastCommit":{"hash":"96fe9badfc5235ff0a049aca647bff8c448055aa","oid":"96fe9badfc5235ff0a049aca647bff8c448055aa","parents":["bde188d60f58012ada0725c6dd5ba7c69fe4dd87"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-12-04T22:19:51+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-04T22:19:51+01:00"},"message":"Add support for CUMSUM and TRI for CUDA. (#17584)\n\n* Add support for CUMSUM and TRI for CUDA.\n\n* Minor optimizations.\n\n* Correct warp_prefix_inclusive_sum in float2 variant to return float2\n\n* Optimize TRI\n\n* Whitespace\n\n* Fix strides.\n\n* Implement double loop\n\n* Whitespace\n\n* Fix HIP compilation bugs\n\n* Optimizations + big case performance tests\n\n* Implement using CUB with fallback to custom kernel\n\n* Remove error message.\n\n* Fixes from code review\n\n* Comment out CPU-unsupported F16/BF16 cases to fix CI\n\n* Fine, you win :P\n\n* Fix last cast, use NO_DEVICE_CODE and GGML_UNUSED_VARS\n\n* Vary warp-size based on physical warp size\n\n* Add GGML_UNUSED_VARS in tri as well\n\n* Use constexpr and call prefix_inclusive with warp_size template param\n\n* Update ggml/src/ggml-cuda/cumsum.cu\n\nCo-authored-by: Johannes Gäßler \n\n* Apply suggestions from code review\n\nCo-authored-by: Johannes Gäßler \n\n* Change to tid % warp_size\n\n* Fix strides; hardcode mask; add ggml_lane_mask_t\n\n* Missing renames, remove unused get_warp_mask(), explicit calls to ggml_cuda_info()\n\n* Too hasty...\n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"21a2f9aca78c491c0ee1df33efbcc51d4d2e03f3"}},{"name":"dequantize.cuh","mode":"0100644","size":18574,"lastCommit":{"hash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","oid":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","parents":["633168fb66919408e7b649bcf7a651d065ebed3e"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-04T12:56:48-07:00"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@users.noreply.github.com","when":"2026-09-04T13:12:38-07:00"},"message":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex","tree":"8db3c0f9c4945a0216b85c2a7166448174b43af0"}},{"name":"diag.cu","mode":"0100644","size":2668,"lastCommit":{"hash":"b63509262ac88db248fa70a1364c8a76e339e93f","oid":"b63509262ac88db248fa70a1364c8a76e339e93f","parents":["48f47565a7e6555c92cc506ecfec8d71bdb1838f"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-12-09T20:28:57+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-09T20:28:57+01:00"},"message":"Add DIAG for CUDA (#17873)\n\n* Add DIAG for CUDA\n\n* Refactor parameters","tree":"8aad49c295e57879052947ed7057d5181b11bcd7"}},{"name":"diag.cuh","mode":"0100644","size":133,"lastCommit":{"hash":"b63509262ac88db248fa70a1364c8a76e339e93f","oid":"b63509262ac88db248fa70a1364c8a76e339e93f","parents":["48f47565a7e6555c92cc506ecfec8d71bdb1838f"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-12-09T20:28:57+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-09T20:28:57+01:00"},"message":"Add DIAG for CUDA (#17873)\n\n* Add DIAG for CUDA\n\n* Refactor parameters","tree":"8aad49c295e57879052947ed7057d5181b11bcd7"}},{"name":"diagmask.cu","mode":"0100644","size":1760,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"diagmask.cuh","mode":"0100644","size":150,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"dsv4-hc.cu","mode":"0100644","size":9683,"lastCommit":{"hash":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","oid":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","parents":["b2dd28a3b601417b8ea345c4db72df3048e1114f"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-07-17T00:33:33+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-17T00:33:33+08:00"},"message":"DeepseekV4: Add fused hyper-connection ops (#25585)\n\n* dsv4 hc-ops\n\n* add missing files;\n\n* add cparams\n\n* update rpc version\n\n* address review comments\n\n* address review comments","tree":"ca61063d563441412e69a532868a6918b1721dec"}},{"name":"dsv4-hc.cuh","mode":"0100644","size":292,"lastCommit":{"hash":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","oid":"0dc74e332edee2616e4d8d9ab3b68dfc340fc14a","parents":["b2dd28a3b601417b8ea345c4db72df3048e1114f"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-07-17T00:33:33+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-17T00:33:33+08:00"},"message":"DeepseekV4: Add fused hyper-connection ops (#25585)\n\n* dsv4 hc-ops\n\n* add missing files;\n\n* add cparams\n\n* update rpc version\n\n* address review comments\n\n* address review comments","tree":"ca61063d563441412e69a532868a6918b1721dec"}},{"name":"fattn-common.cuh","mode":"0100644","size":48331,"lastCommit":{"hash":"4a6a3344f7e58196980e085ddbf016f6227fae23","oid":"4a6a3344f7e58196980e085ddbf016f6227fae23","parents":["b60f98f9fc5f4b26c9d06b4800ab8d75d837a382"],"author":{"name":"sudoingX","email":"200180104+sudoingX@users.noreply.github.com","when":"2026-09-27T17:22:25+07:00"},"committer":{"name":"sudoingX","email":"200180104+sudoingX@users.noreply.github.com","when":"2026-09-27T17:22:25+07:00"},"message":"Fix: keep the q4_0 V dequant of the FA vector kernel in registers\n\ndequantize_V_q4_0 copied the quants into a local int and read them back\nthrough an int8_t pointer to it. nvcc 12.8 for sm_120 kept that local on\nthe stack, with 128 generic stores per KV step in the one-query q4_0\nkernel, and q4_0 decode fell to 12.4 tok/s at 64K depth on an RTX 5060 Ti\nagainst 30.5 tok/s with q8_0.\n\nBuild the value in a register from two 2-byte loads and take each byte\nwith a shift. Same arithmetic, byte-identical output; tg128 at 64K goes\nfrom 12.41 to 29.40 tok/s and the q4_0 kernels lose their stack at head\nsizes 64, 128 and 256.\n","tree":"8e85fd70db9c53903ab96f3fdeae85f35ee2bb6c"}},{"name":"fattn-mma-f16.cuh","mode":"0100644","size":124507,"lastCommit":{"hash":"0163094fa579c4547035f7700b1f6ca009fc8331","oid":"0163094fa579c4547035f7700b1f6ca009fc8331","parents":["6d1917060d72cfffc6b54391ef8006b7f51361b9"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: q8_0 decode/verify flash attention raw K/V lead staging (fattn-verify-lead)\n\nq8_0 decode/verify flash attention (the 4x8 tile) stages raw K/V with cp.async and starts the next tile's K and V a phase early. Bit-identical; decode +2.9% at 64K depth.\n","tree":"efbc443a668040590ad155d4e558404da5900100"}},{"name":"fattn-tile.cu","mode":"0100644","size":2146,"lastCommit":{"hash":"fa72aeccb23947074c12b5fec25f5b6ced28cbfe","oid":"fa72aeccb23947074c12b5fec25f5b6ced28cbfe","parents":["ed7adbfefd5e3af5f1c7c04ac67145783def85c5"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2026-07-24T17:53:54+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-24T17:53:54+02:00"},"message":"HIP: remove rocWMMA FlashAttention (#26046)","tree":"101f3999f40a1439a361469ff297fc582fefcd94"}},{"name":"fattn-tile.cuh","mode":"0100644","size":63102,"lastCommit":{"hash":"fa72aeccb23947074c12b5fec25f5b6ced28cbfe","oid":"fa72aeccb23947074c12b5fec25f5b6ced28cbfe","parents":["ed7adbfefd5e3af5f1c7c04ac67145783def85c5"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2026-07-24T17:53:54+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-24T17:53:54+02:00"},"message":"HIP: remove rocWMMA FlashAttention (#26046)","tree":"101f3999f40a1439a361469ff297fc582fefcd94"}},{"name":"fattn-vec.cuh","mode":"0100644","size":24989,"lastCommit":{"hash":"9e58d4d692ed3d350591cc86d06c73c61c122509","oid":"9e58d4d692ed3d350591cc86d06c73c61c122509","parents":["3571fa5435ac9ff243662b1caabc407e8d433c9d"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-06-03T13:56:42+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-03T13:56:42+02:00"},"message":"Avoid PDL race conditions by disabling __restrict__ when PDL is used (#24030)\n\n* Removes __restrict__ from PDL kernel headers due to incompatibility with\nPDL. Adds preprocessor directives based on arch in kernel body to add\n__restrict__ to retain performance on older architectures.\n\n* Simplifies new __restrict__ usage via macro\n\n* Add hopper to PDL __restrict__ fix.\n\nCo-authored-by: Oliver Simons \n\n---------\n\nCo-authored-by: Oliver Simons ","tree":"0cf5e574727793b7453ab0ebb783401697b161ec"}},{"name":"fattn.cu","mode":"0100644","size":28558,"lastCommit":{"hash":"d1b140ccb11a60c55e441e70ce6baa40a26ac902","oid":"d1b140ccb11a60c55e441e70ce6baa40a26ac902","parents":["6d595f4116cac1cb1daa915ea665678e4c6d7bca"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: pair-tiled 6-head GQA q8_0 prefill flash attention (fattn-q8-prefill)\n\nPrompt batches over a q8_0 cache tile the 6 query heads of a KV head in pairs instead of padding them to 8, stage raw q8_0 rows with cp.async and dequantize exactly in half2. Bit-identical; prefill +8% at 16K, +14% at 32K, +21% at 64K. Decode and verify keep the 8-wide tile.\n","tree":"09421db39f53aa256afc791cb6a7162a72eb1271"}},{"name":"fattn.cuh","mode":"0100644","size":271,"lastCommit":{"hash":"f8f0a47a55167bea25199d4d761372cd7cee76b7","oid":"f8f0a47a55167bea25199d4d761372cd7cee76b7","parents":["06938ac129e5feee1e731323e5c37dc973de5573"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-06-03T18:39:59+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-03T18:39:59+08:00"},"message":"cuda: reserve space for quantize kv-cache at startup (#23907)\n\n* cuda: reserve space for quantize kv-cache at startup\n\n* address review comments\n\n* remove forward decl\n\nCo-authored-by: Johannes Gäßler \n\n* remove assert in ggml-cuda.cu\n\nCo-authored-by: Johannes Gäßler \n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"27bea280c0dd18ddf3ba8445601b3a6eba92d9ce"}},{"name":"fill.cu","mode":"0100644","size":1110,"lastCommit":{"hash":"86a3f0fad8b153ac9396e1ac18e790e4179c53f2","oid":"86a3f0fad8b153ac9396e1ac18e790e4179c53f2","parents":["63908b631a2cef6491c2b6ce1c504f046660183f"],"author":{"name":"Sigbjørn Skjæret","email":"sigbjorn.skjaeret@scala.com","when":"2025-12-09T12:23:47+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-09T12:23:47+01:00"},"message":"ggml : allow fill node alloc inplace (#17870)\n\n","tree":"6ab82cfb29ea4b8d051287bbaab5e495b8e52729"}},{"name":"fill.cuh","mode":"0100644","size":99,"lastCommit":{"hash":"51e0c2d917c21826585e84be1c27f75147325de0","oid":"51e0c2d917c21826585e84be1c27f75147325de0","parents":["37a4f632442cb0e6d046b714d36cecb619833f4e"],"author":{"name":"Jay Zenith","email":"162098309+JayZenith@users.noreply.github.com","when":"2025-12-08T05:10:12-08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-08T21:10:12+08:00"},"message":"cuda : add FILL op support (#17851)\n\n* cuda : add FILL op support\n\n* cuda : add missing FILL op files","tree":"aaa936b9487e51e6385279f3c2535e26acdc214b"}},{"name":"fwht.cu","mode":"0100644","size":16574,"lastCommit":{"hash":"113d61c842e3889406777388f7bb929f288e12b0","oid":"113d61c842e3889406777388f7bb929f288e12b0","parents":["8ea08e9c7b8a8bfd72305b28efd7019c33dfa969"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: Hadamard transform writes per-128 q8 tiles directly (fwht-q8)\n\nThe Hadamard transform in front of a PTQ1_0 matmul that takes the MMQ path also writes the per-128 q8 tiles MMQ reads, so the matmul skips its own quantize pass (and the f32 copy when nothing else reads it). Bit-identical; +3% prefill.\n","tree":"c293c37ef5b6d25c91b4d4c27cea63a33e4b9d54"}},{"name":"fwht.cuh","mode":"0100644","size":1221,"lastCommit":{"hash":"113d61c842e3889406777388f7bb929f288e12b0","oid":"113d61c842e3889406777388f7bb929f288e12b0","parents":["8ea08e9c7b8a8bfd72305b28efd7019c33dfa969"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: Hadamard transform writes per-128 q8 tiles directly (fwht-q8)\n\nThe Hadamard transform in front of a PTQ1_0 matmul that takes the MMQ path also writes the per-128 q8 tiles MMQ reads, so the matmul skips its own quantize pass (and the f32 copy when nothing else reads it). Bit-identical; +3% prefill.\n","tree":"c293c37ef5b6d25c91b4d4c27cea63a33e4b9d54"}},{"name":"gated_delta_net.cu","mode":"0100644","size":21878,"lastCommit":{"hash":"c0209b63a39312975f3cbff7670afda41f653cd9","oid":"c0209b63a39312975f3cbff7670afda41f653cd9","parents":["ad2d86b8dc2153be655bbf209e8f85afa3b2908c"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: in-place GDN recurrent state reading from cache rows (gdn-rows)\n\nThe DeltaNet layers read each seq's recurrent state straight from its cache row instead of gathering it first (only while every seq's row is its own cell; otherwise the old gather). Bit-identical; MTP decode +1.6-2.3%. GGML_GDN_STATE_GATHER=1 restores the gather.\n","tree":"5e386426be0e2881da6097e48c87ccb8ffda868d"}},{"name":"gated_delta_net.cuh","mode":"0100644","size":689,"lastCommit":{"hash":"5a460dea9f961cdb508d58a6e7b0f9e259b4c19f","oid":"5a460dea9f961cdb508d58a6e7b0f9e259b4c19f","parents":["c8ae9a750c6c89dc928503a16175b7b3c2d224e4"],"author":{"name":"Gaurav Garg","email":"gaugarg@nvidia.com","when":"2026-07-03T14:36:29+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-03T14:36:29+05:30"},"message":"Remove redundant CUDA copies after gated_delta_net. (#23940)\n\n* Remove redundant CUDA copies after gated_delta_net.\n\nCurrently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.\n\nThe change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.\n\n* Address review comments","tree":"fcf2fae75d9af0512b7bb1d5c86f27ff2b612438"}},{"name":"getrows.cu","mode":"0100644","size":23595,"lastCommit":{"hash":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","oid":"01fd9521c92e7882a3fa1083bb933e3bd9305bef","parents":["633168fb66919408e7b649bcf7a651d065ebed3e"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-04T12:56:48-07:00"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@users.noreply.github.com","when":"2026-09-04T13:12:38-07:00"},"message":"ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) (#148)\n\n* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)\n\nUpstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a\nsingle fp16 scale per 256 weights. The ternary checkpoints this targets carry a\nscale per 128, so a 256-wide scale has to discard one of the two group scales it\nstraddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern\nis preserved exactly, but the smaller-alpha group of each block is reconstructed\nat the larger alpha, giving 6.43% weight error that passes into activations at\nfull strength because it is multiplicative on the weight rather than noise.\n\nPTQ1_0 keeps TQ1_0's trit packing and our group of 128:\n\n qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw\n\nagainst PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of\ninformation: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.\n\ntest-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute\nquantization error 0.008678 and dot product error 0.141111, versus tq1_0's\n0.008681 and 0.141345. A round trip of real folded ternary weights is\nbit-identical to the PQ2_0 reconstruction, so the only error either format\ncarries is the shared fp16 scale rounding.\n\nA distinct type id (143) rather than changing TQ1_0 in place, for the same reason\nPQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and\nreinterpreting its block width would silently misread every existing TQ1_0 gguf.\n\nTQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the\nstages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly\nits original behaviour.\n\nScope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so\nllama-quantize can target it. The vec_dot decodes a block to element order before\nthe dot, sharing the traversal with dequantize so the two cannot drift; it is\ngeneric C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,\nand the repack gemv/gemm paths, are not implemented.\n\n* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation\n\nThree plumbing gaps the traits entry alone did not cover, each of which failed\nend-to-end quantization in a different way:\n\n ggml_quantize_chunk - without a dispatch case the returned byte count did not\n match the row size and GGML_ASSERT(result == nrows *\n row_size) aborted a worker thread mid-tensor, leaving a\n truncated output file behind.\n ggml_ftype -> wtype - the ftype could not resolve to a ggml type.\n validate_row_data - \"invalid type 143\", so quantization refused the result\n it had just produced.\n\nVerified end to end on a 1.7B ternary model with the embedding and output tensors\npinned to f16 in both runs, so only the quantized tensors differ:\n\n quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller\n (1.75/2.125 bpw predicts 17.65%)\n perplexity, 4 chunks at ctx 512, identical to every printed digit:\n per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,\n final 12.6944 +/- 1.12596 for both\n\nThe perplexity run goes through the new CPU vec_dot, so that path is exercised\nrather than only the codec.\n\n* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0\n\nAdds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both\ndequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations\nfor f32 and f16, get_rows, the simdgroup constants and both pipeline-selection\nswitches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently\nskipped the new type and reported a clean pass.\n\nCorrect: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on\na 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed\ndigit, 12.6935 +/- 1.12567 for both.\n\nSlow, and the reason is structural. Measured on M5 Pro:\n\n PQ2_0 PTQ1_0\n pp128 4505 tok/s 2590 tok/s 0.57x\n tg32 182.5 tok/s 43.6 tok/s 0.24x\n\nBase-3 packing makes element order non-positional, so a thread owning 16\ncontiguous weights cannot index bytes directly, and threads owning different trit\nindices reload the same bytes. Hoisting the trit factor out of the inner loop for\nspans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap\nneeds the thread mapping changed so one thread consumes all five trits of a byte\nit loads once, instead of five threads each loading it for one trit.\n\nThis reproduces what the earlier TQ1_0 work already found on other backends:\nbase-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay\nfor itself at decode without that restructure. The 17.6% file-size reduction is\nreal and lossless; the speed is not there yet.\n\n* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0\n\nFour changes, each measured on Apple M5 Pro against a 1.7B ternary model, with\nMUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:\n\n 1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five\n trits of a byte it loads once, instead of eight threads each reloading the\n same qs bytes for one trit index; the block's 26 bytes are read once total\n 2. y staged in registers 136.4 -> 143.5 y was being re-read from device\n memory once per row; it is now gathered once per block and reused, with the\n ternary offset folded into a single sumy subtraction\n 3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real\n limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163\n 4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads\n 0 and 1 only, so those did 19 elements while the rest did 15 and the whole\n simdgroup waited; one element per thread makes it 16 each and drops yl to 16\n\nTrit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with\nv = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was\ntried and is not better for decode: it adds a dependent load per byte, 26 extra\nper block against PQ2_0's 32 total loads. It does help the mul_mm path, which\nmust produce 16 contiguous elements and cannot use the byte-owning mapping, so\ndequantize keeps the table.\n\nFinal, PQ2_0 vs PTQ1_0 on M5 Pro:\n\n pp128 4541 -> 3998 tok/s 0.88x\n tg32 186 -> 165 tok/s 0.89x\n size 357 -> 294 MiB 17.6% smaller, and lossless\n\nPrefill's remaining gap is structural: mul_mm's dequantize contract is 16\ncontiguous elements, so five threads still touch the same 16 bytes for different\ntrit indices. Closing it needs mul_mm itself to stage a decoded block in\nthreadgroup memory, which is a larger change than this.\n\n* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here\n\nAdds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit\naccessor, the float2 dequantize entry point, the six dequant dispatch sites in\nconvert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch\nsites, and both supports_op type lists.\n\n**This code has never been compiled or executed.** The machine it was written on\nis an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be\nbuilt and run on a CUDA host before anyone relies on it.\n\nWhat *is* verified is the part where a bug would actually hide. The device\naccessor's element mapping is pure integer logic, so it was transcribed verbatim\nto the host and checked against the CPU codec's own traversal over 2,560,000\nelement positions across 20,000 random blocks: zero mismatches. That check is\nkept as tests/test-ptq1_0-element-map.cpp so it can be rerun.\n\nTrit extraction uses the same base-3 remainder recurrence as the Metal path,\nt = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and\npacks them into an int as signed int8 lanes so accumulation still goes through\ndp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.\n\nMMQ is deliberately not implemented. Types without it fall back to dequantize\nplus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ\ntile loader blind against tile layouts I cannot test would be worse than leaving\nthe fallback in place.\n\n* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out\n\nN_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,\nso 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise\n(+/- 3.4 on a single measurement).\n\nFive further optimisations were tried and MEASURED, and four of them lost. Kept\nhere so nobody spends the same day twice:\n\n 256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain\n (helps mul_mm, which keeps it)\n independent pow3 extraction 159.8 vs 166.0 recurrence WORSE\n contiguous byte ownership 162.0 vs 165.3 scattered WORSE\n N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4\n N_SG = 8 161.1 worse than 1-2\n\nWhat that rules out. The lookup table shows decode is not bound by trit\narithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep\nserial dependency either -- four rows per thread already supply enough\nindependent work, so fewer ops beat more ILP. Contiguous byte runs made it worse\nbecause thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte\nbranch cost more than consolidating the loads saved.\n\nWhere the remaining 11% actually lives, per 128 weights:\n\n PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read\n PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read\n ALU 1.25x, bytes 0.82x\n\nA purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at\n1.21x. Measured 0.89x sits between them, which is about where this instruction\nand byte mix should land. Extracting a base-3 digit is inherently a multiply\nwhere a 2-bit field is a shift, so this gap is the price of packing at the\n1.585-bit information limit rather than on a 2-bit boundary, not an\nimplementation defect.\n\nThe one lever left with a real mechanism behind it is prefill, not decode. In\ndecode each row has its own weights so there is nothing to share, but mul_mm\nreuses a weight tile across many output columns, and its dequantize contract\nforces 16 contiguous elements, so five threads still touch the same 16 bytes.\nStaging a cooperatively decoded block in threadgroup memory would remove that\nredundancy outright. Prefill is at 0.87x (3946 vs 4492).\n\nPerplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.\n\n* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5\n\nSelf-review compared every switch that names PQ2_0 against those naming PTQ1_0\nand found the new type absent from ten support enumerations: seven in\nggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in\nggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible\nwith PQ2_0 there -- same block size, same to_float and from_float shape -- so it\nbelongs in each.\n\nClosing those gaps then exposed a real defect. Adding the type to the Metal\nsupport lists enabled the mul_mv_ext path for wider n, which immediately failed\nwith \"kernel not found in any metal library\" for\nkernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only\nr1_2 through r1_4 had been written, because the grep that enumerated them was\ntruncated by head. Instantiating r1_5 fixes it, and the instantiation set is now\nderived from PQ2_0's rather than transcribed.\n\nWorth noting the earlier 1251/1251 passes were not wrong, they were narrow: with\nthe type missing from the support lists the wider-n path was never selected, so\nthe absent kernel could not be reached. The gap and the defect concealed each\nother.\n\nAfter: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical\nto PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.\n\n* tests: verify the CUDA MMVQ dot numerically on the host\n\nThe CUDA path still cannot be compiled here, but its arithmetic can be checked.\nThis stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the\n4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor\nverbatim, and compares them against a float reference built by dequantising the\nblock and the q8_1 activations independently.\n\nResult over 5000 random weight blocks with random q8_1 activations:\n\n exact integer-accumulator checks 20000, mismatches 0\n error / sum of term magnitudes 2.399e-08\n\nThe integer test is the one that matters: the kernel accumulates in int via dp4a\nbefore scaling, so if the trit extraction, the int8 lane packing or the chunk-to-\nq8_1 pairing were wrong, that accumulator would differ. It does not, on any of\nthe 20000 chunks.\n\nA first pass reported 7.062e-05 \"relative error\" and looked like a failure. That\nwas cancellation: the reference total is a signed sum that can land near zero, so\ndividing by it inflates the ratio. Measured against the sum of term magnitudes\ninstead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32\nrounding of the four per-chunk products.\n\nTogether with tests/test-ptq1_0-element-map.cpp this leaves only compilation\nunproven for the CUDA path, not correctness of its math.\n\n* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW\n\nFirst compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and\nCMAKE_CUDA_ARCHITECTURES=120, failed with\n\n vecdotq.cuh(822): error: identifier \"ptq1_0_trit\" is undefined\n\nThe accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.\nPQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with\nbyte_perm -- so nothing in the file it was mirrored from exposed the dependency.\nBoth headers include common.cuh, so the helper belongs there.\n\nVerified on the 5090 after the fix:\n\n test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK\n 45 ptq1_0 cases OK, 0 failures\n host element-map test 2,560,000 positions, 0 mismatches\n host MMVQ dot test 20,000 integer checks, 0 mismatches\n\nThe two host tests were already green before this, which is the point worth\nrecording: they verified the arithmetic and could not have caught a missing\ndeclaration. Compilation was the only gate that would find this, and the earlier\ncommit saying the CUDA path had never been built was accurate about the risk.\n\n* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed\n\nAdds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit\naccessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone\ndequant shader, the shader-generator registration, and 16 pipeline sites in\nggml-vulkan.cpp mirrored from Q1_0's.\n\nMirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan\nkernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still\ndeclares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a\ngroup-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0\nwould have inherited that. Q1_0 also has no MMVQ entry, so this follows the same\ndequantize-based path rather than inventing one.\n\nOne divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block\n(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C\nfield order exactly. Getting that wrong misindexes every block silently instead of\nfailing, which is the same class of error as the Q2_0 breakage above.\n\nNot wired, and it is a real gap rather than an oversight: on-device quantization\n(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,\nwhich is a separate piece of work and is not needed to run a model -- weights are\nquantized offline by llama-quantize.\n\n**This has not been compiled or run.** There was no Vulkan toolchain on the\nmachine it was written on. It is pushed so it can be built and tested on a GPU\nbox, and it should be treated as unverified until test-backend-ops says otherwise.\n\n* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower\n\nBenchmarked the CUDA path for the first time. It is correct and slow, and the\nnumbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against\npq2_0 on the same run.\n\n n=1 n=2 n=3 n=8 n=512\n q1_0 8.88 9.82 13.05 30.14 225.4\n pq2_0 9.75 11.05 14.92 31.27 232.4\n ptq1_0 46.8 64.6 99.1 287.3 525.6\n\nThe per-chunk version asked \"which byte holds element e\" per element, a branch plus\na variable-length recurrence, and iqs differs per lane so the branch diverged. Taking\nthe whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26\nbytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.\n\nSo divergence was not the dominant cost, and the comment now says so. What remains is\nload shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,\nwhere pq2_0 issues eight dp4a calls over 4-byte words.\n\nThe fix is identified but deliberately not guessed at. Below element 80 the element\nstride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES\nat the same trit index. Four recurrences packed into one int would feed dp4a against a\n4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this\nsession has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that\npredictions about where GPU time goes are wrong about as often as they are right.\n\nCorrectness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090\nand an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.\n\nNote n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,\nwhich is consistent with the problem being in the MMVQ load pattern specifically.\n\n* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines\n\nThe Vulkan commit did not build. Three errors, one root cause each, all found by a\nteammate on an Intel Arc B390 and independently by windows CI:\n\n1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in\n vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but\n dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed\n the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which\n falls back to the scalar and coopmat1 paths that are actually implemented.\n\n2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in\n dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs\n and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour\n earlier, same helper, different backend**: defined in the dequantize chain, called\n from the matmul chain. Fixed the same way, with a shared header\n vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not\n duplicated: two copies would have to stay in step with the CPU codec in\n ggml-quants.c, and a divergence there appears as wrong matmul results, never as a\n build error.\n\n3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2\n pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a\n comment recording that the absence is deliberate. (1) and (3) had to be fixed\n together or one error just trades for the other.\n\nVerified locally this time, which is the other half of the lesson: glslc, molten-vk,\nvulkan-headers and vulkan-loader were already installed on the machine that wrote the\noriginal commit. It was pushed unverified on the false assumption that no toolchain\nwas present, and a local build finds all three in minutes.\n\n -DGGML_VULKAN=ON builds clean\n SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2\n test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression\n\nThat local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case \"not\nsupported\", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an\nApple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness\ncomes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases\nactually executed and 0 failures, on KHR_coopmat.\n\nStill absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),\ncoopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.\n\n* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path\n\nThe PTQ1_0 pipelines were created but the type was absent from every\nselection and support enumeration, so supports_op returned false, the\nscheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the\ngenerated shaders were unreachable. The build looked green while nothing\nexecuted on the backend. Add the type alongside GGML_TYPE_Q1_0 in the\nfive pipeline getters and the two supports_op gates.\n\nAdding it there alone is not safe, because PTQ1_0 is the only type in\nthose lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no\ndecoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots\nare still allocated by vk_matmul_pipeline2's constructor, so the mm getter\nwould return a non-null pipeline whose l/m/s are all null, and the\ncaller's \"mmp == nullptr\" fallback would not fire; the mul_mat_id getter\nwould instead trip GGML_ASSERT(support_fp32acc). Both getters now return\nnullptr when the cm2 slots are empty, which routes the caller to the\ndequant + f16 matmul fallback it already implements. That path needs\nggml_vk_get_to_fp16, which is one of the gates opened above.\n\nThe cpy, SET_ROWS and DUP gates are deliberately left alone:\ncopy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on\ndevice is unimplemented and a refusal there is correct.\n\nVerified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78\ndeclined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case\nfor case, byte-identical across two runs. The coopmat2 guard is reasoned\nfrom the pipeline-creation structure and is not covered by that run,\nwhich is a coopmat1 device; it has no coopmat2 hardware behind it yet.\n\n* cuda: accelerate PTQ1_0 MMVQ with packed DP4A\n\nAssisted-by: OpenAI Codex","tree":"8db3c0f9c4945a0216b85c2a7166448174b43af0"}},{"name":"getrows.cuh","mode":"0100644","size":622,"lastCommit":{"hash":"e1e8e0991ffd9e99a445c6812bb519d5bac9f4b5","oid":"e1e8e0991ffd9e99a445c6812bb519d5bac9f4b5","parents":["6f67cf1f480926391ad75ff746e0a021214bf70c"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2025-04-30T23:12:59+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-04-30T23:12:59+02:00"},"message":"CUDA: batched+noncont MMQ, refactor bs>1 MoE code (#13199)\n\n","tree":"a7887b8c3c27cb70688a08b195d13a69e4b3a6aa"}},{"name":"ggml-cuda.cu","mode":"0100644","size":262183,"lastCommit":{"hash":"6d1917060d72cfffc6b54391ef8006b7f51361b9","oid":"6d1917060d72cfffc6b54391ef8006b7f51361b9","parents":["c0209b63a39312975f3cbff7670afda41f653cd9"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: fused CUDA kernel for DeltaNet conv path (gdn-conv)\n\nOne CUDA kernel for the DeltaNet conv path (concat -> ssm_conv -> silu -> conv-state writes): the concat is never materialized; same arithmetic. Bit-identical; MTP decode +1.5%. GGML_CUDA_SSM_CONV_CONCAT_FUSION=0 turns it off.\n","tree":"b62a7b2173f152483ed21ab164952353e634072b"}},{"name":"gla.cu","mode":"0100644","size":3109,"lastCommit":{"hash":"ee7136c6d1e0ba7633294dad137b1573048031ec","oid":"ee7136c6d1e0ba7633294dad137b1573048031ec","parents":["c6860cc7346c90219475e4467bb8a288e0df975c"],"author":{"name":"Molly Sophia","email":"mollysophia379@gmail.com","when":"2025-01-10T09:58:08+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-01-10T09:58:08+08:00"},"message":"llama: add support for QRWKV6 model architecture (#11001)\n\nllama: add support for QRWKV6 model architecture (#11001)\r\n\r\n* WIP: Add support for RWKV6Qwen2\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* RWKV: Some graph simplification\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Add support for RWKV6Qwen2 with cpu and cuda GLA\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* RWKV6[QWEN2]: Concat lerp weights together to reduce cpu overhead\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix some typos\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* code format changes\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix wkv test & add gla test\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix cuda warning\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Update README.md\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Update ggml/src/ggml-cuda/gla.cu\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* Fix fused lerp weights loading with RWKV6\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* better sanity check skipping for QRWKV6 in llama-quant\r\n\r\nthanks @compilade\r\n\r\nSigned-off-by: Molly Sophia \r\nCo-authored-by: compilade \r\n\r\n---------\r\n\r\nSigned-off-by: Molly Sophia \r\nCo-authored-by: Georgi Gerganov \r\nCo-authored-by: compilade ","tree":"7aaf56a126b7ab6da25b789b041a8c6d5298ce5b"}},{"name":"gla.cuh","mode":"0100644","size":112,"lastCommit":{"hash":"ee7136c6d1e0ba7633294dad137b1573048031ec","oid":"ee7136c6d1e0ba7633294dad137b1573048031ec","parents":["c6860cc7346c90219475e4467bb8a288e0df975c"],"author":{"name":"Molly Sophia","email":"mollysophia379@gmail.com","when":"2025-01-10T09:58:08+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-01-10T09:58:08+08:00"},"message":"llama: add support for QRWKV6 model architecture (#11001)\n\nllama: add support for QRWKV6 model architecture (#11001)\r\n\r\n* WIP: Add support for RWKV6Qwen2\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* RWKV: Some graph simplification\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Add support for RWKV6Qwen2 with cpu and cuda GLA\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* RWKV6[QWEN2]: Concat lerp weights together to reduce cpu overhead\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix some typos\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* code format changes\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix wkv test & add gla test\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Fix cuda warning\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Update README.md\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* Update ggml/src/ggml-cuda/gla.cu\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* Fix fused lerp weights loading with RWKV6\r\n\r\nSigned-off-by: Molly Sophia \r\n\r\n* better sanity check skipping for QRWKV6 in llama-quant\r\n\r\nthanks @compilade\r\n\r\nSigned-off-by: Molly Sophia \r\nCo-authored-by: compilade \r\n\r\n---------\r\n\r\nSigned-off-by: Molly Sophia \r\nCo-authored-by: Georgi Gerganov \r\nCo-authored-by: compilade ","tree":"7aaf56a126b7ab6da25b789b041a8c6d5298ce5b"}},{"name":"im2col.cu","mode":"0100644","size":12729,"lastCommit":{"hash":"8e1f9d083453678808a26465a3563a19a0b49699","oid":"8e1f9d083453678808a26465a3563a19a0b49699","parents":["e93666076038c0bd26397feed6cfb8a6c6d04f74"],"author":{"name":"CrispStrobe","email":"154636388+CrispStrobe@users.noreply.github.com","when":"2026-05-11T19:48:29+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-11T19:48:29+02:00"},"message":"CUDA: handle OW > 65535 in im2col (2D and 3D) (#22944)\n\n`im2col_cuda` and `im2col_3d_cuda` both dispatch with\n`block_nums.y = OW`. CUDA caps grid Y at 65535. Conv1d encoders on\nraw 16 kHz audio with T > 65535 (~ 4 s) trip the limit -- e.g. SEANet\nat 11 s lands at OW = 176000 -- and the launch returns\n`invalid configuration argument`.\n\nClamp `block_nums.y` to `MIN(OW, MAX_GRIDDIM_Y)` and loop inside the\nkernel with stride `MAX_GRIDDIM_Y`. Same in-kernel stride pattern\nalready used for the z axis (`MAX_GRIDDIM_Z`). Both 2D `im2col_kernel`\nand 3D `im2col_3d_kernel` need the same fix. Bit-identical for\nOW <= 65535 (single iteration of the new outer loop).\n\nTested on T4 / Jetson Orin with a SEANet encoder running on 11 s /\n16 kHz audio (im2col reaching OW ~ 176000); pre-fix launch returns\n`invalid configuration argument`, post-fix runs to completion.\nExisting test-backend-ops im2col cases unchanged.","tree":"a1cd0fb939e5d6ab2b81c01fa7de127913236403"}},{"name":"im2col.cuh","mode":"0100644","size":218,"lastCommit":{"hash":"0a1b3982cd0bd18730d50a693053b88c13fd04a6","oid":"0a1b3982cd0bd18730d50a693053b88c13fd04a6","parents":["5421f63ab08ca1e1a093662a5ccd0117e461185f"],"author":{"name":"leejet","email":"leejet714@gmail.com","when":"2025-09-04T16:38:49+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-09-04T10:38:49+02:00"},"message":"ggml: add ops for WAN video model (cuda && cpu) (#15669)\n\n* add conv3d support\n\n* add ggml_pad_ext for cpu & cuda backend\n\n* cuda/cpu: add im2col_3d support\n\n* cuda: make im2col a little faster\n\n* fix cuda pad/scale/im2col3d\n\n* make im2col_3d faster\n\n* gguf: support loading tensors which n_dims > GGML_MAX_DIMS\n\n* fix cuda get_rows\n\n* avoid ggml_conv_3d conflict\n\n* correct GGML_OP_COUNT assertion\n\n* avoid build failure\n\n* avoid build failure on MacOS\n\n* cuda: remove unnecessary MIN define\n\n* fix cpu im2col_3d\n\n* adjust the code style\n\n* cuda: use simpler loop in get_rows\n\n* add test_im2col_3d to test-backend-ops\n\n* test-backend-ops.cpp: remove trailing whitespace\n\n* cpu: im2col_3d support non continuous src\n\nCo-authored-by: Jeff Bolz \n\n* fix test_im2col_3d\n\n* remove unused variables\n\n* cuda: get_rows: dfloat2 -> float2\n\n* add test_pad_ext to test-backend-ops.cpp\n\n* add gguf_init_from_file_ext impl\n\n* Revert \"gguf: support loading tensors which n_dims > GGML_MAX_DIMS\"\n\nThis reverts commit d8377a0a37f314bd3713fe043b4333ad661610c1.\n\n* Revert \"add gguf_init_from_file_ext impl\"\n\nThis reverts commit d9f1d13208c68ef83b3538201ac7f31614fb1994.\n\n* update ggml_backend_vk_device_supports_op\n\n* fix ggml_backend_vk_device_supports_op\n\n* update other backend supports op for ggml_pad_ext\n\n* metal/opencl/sycl/vulkan: fix GGML_OP_PAD check in supports_op\n\n---------\n\nCo-authored-by: Jeff Bolz ","tree":"96305d8651cc1fff58f242414c72e42489f42bf0"}},{"name":"lightning-indexer.cu","mode":"0100644","size":24211,"lastCommit":{"hash":"3b53219361a61b53e7741c479b81b755ec6096b1","oid":"3b53219361a61b53e7741c479b81b755ec6096b1","parents":["aff6eb6e7503538fec1532dec2f584bc7a4a4e4d"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-15T19:57:52+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-15T19:57:52+02:00"},"message":"cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (#25545)\n\n* cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)\n\n* chore : remove indentation of #pragma unroll\n\n* cuda : remove unnecessary kernel template declarations\n\n* cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.\n\n* cuda : relax MMA architecture requirements to Turing in lightning indexer implementation\n\n* chore : renamed variables\n\n* chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer()\n\n* chore : TODO for AMD rocWMMA\n\n* chore : whitespace formatting\n\n* chore : another variable rename to fix problems caused by shadowing\n\n* chore : yet another rename, this time uppercased all constants\n\n* cuda : added alignment checks for Q and K tensors in lightning indexer implementation\n\n---------\n\nCo-authored-by: Stanisław Szymczyk ","tree":"c8b11d1796e618218f5b27c151ed63da4d608b5c"}},{"name":"lightning-indexer.cuh","mode":"0100644","size":190,"lastCommit":{"hash":"3b53219361a61b53e7741c479b81b755ec6096b1","oid":"3b53219361a61b53e7741c479b81b755ec6096b1","parents":["aff6eb6e7503538fec1532dec2f584bc7a4a4e4d"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-15T19:57:52+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-15T19:57:52+02:00"},"message":"cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (#25545)\n\n* cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)\n\n* chore : remove indentation of #pragma unroll\n\n* cuda : remove unnecessary kernel template declarations\n\n* cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.\n\n* cuda : relax MMA architecture requirements to Turing in lightning indexer implementation\n\n* chore : renamed variables\n\n* chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer()\n\n* chore : TODO for AMD rocWMMA\n\n* chore : whitespace formatting\n\n* chore : another variable rename to fix problems caused by shadowing\n\n* chore : yet another rename, this time uppercased all constants\n\n* cuda : added alignment checks for Q and K tensors in lightning indexer implementation\n\n---------\n\nCo-authored-by: Stanisław Szymczyk ","tree":"c8b11d1796e618218f5b27c151ed63da4d608b5c"}},{"name":"mean.cu","mode":"0100644","size":3112,"lastCommit":{"hash":"e947228222147356bc7e64154d3439e142481632","oid":"e947228222147356bc7e64154d3439e142481632","parents":["29f1482221b68fdbf5bd9b762c9e3e350e21f1ec"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-05-20T13:59:02+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-20T13:59:02+02:00"},"message":"Programmatic Dependent Launch (PDL) for more performance on newer NVIDIA GPUs (Hopper+) (#22522)\n\n* Adds initial PDL setup.\n\n* Adds PDL barriers based on simple heuristic: place \"sync\" before first input pointer access, and \"launch\" after last write, e.g. to tensors like dst.\n\n* Further optimization pass of the first half of kernels\n\n* Optimized PDL barriers for the second batch of kernels\n\n* Further refinements after rebase.\n\n* Moves pdl logic to separate function, removes some whitespace\n\n* Strips post-hoc PDL logic\n\n* Adds stream capture PDL setup. Enrolls quantize_q8_1 to leverage pdl to\noverlap execution with previous kernels\n\n* Enrolls mul_mat_vec_q, rms_norm_f32 and k_bin_bcast (partly) into PDL\n\n* Enrolls mmvf, rope, set-rows and topk kernels for gpt-oss into PDL\n\n* Introduce ggml_cuda_kernel_launch, to abstract away cudaLaunchKernelEx,\nto enable hip/musa compatibility\n\n* Enrolls cpy_scalar_contiguous, k_get_rows_float and rms_norm_f32\n\n* Enrolls flash_attn_combine_results\n\n* Fix: Drops needless and broken check of CUDA arch for PDL. PDL either\nworks or is without effect.\n\n* Enrolls flash-attention kernels to pdl\n\n* Fix: inlines ggml_cuda_kernel_launch, and uses perfect forwarding for\nkernels args. This fixes PDL.\n\n* Perf: Enrolls k_bin_bcast variadic template invocation into PDL, via\nand template alias and template expansion\n\n* Enrolls all remaining kernels for qwen3-coder-next into PDL\n\n* Remove all PDL LC calls to create a baseline\n\n* Added LC according to internal guidance and tested kernel performance.\n\n* Enrols missing qwen3-5 kernels passively into PDL.\n\n* Kernel optimizations (LC signals) for qwen3.5\n\n* Enrolls ssm-scan kernels into PDL\n\n* Adds GGML_CUDA_PDL command line option to toggle PDL.\n\n* Fix: Ada and lower compilation by guarding PDL calls correctly\n\n* Cleanup: Removes commented out GGML_CUDA_PDL_LC\n\n* Cleanup: Removes experimental comments\n\n* Adds 90-virtual to build script so that Hopper GPUs can leverage PDL.\n\n* Adds stricter checks to enable PDL, adds env-check to disable it, and removes now superfluous compile option to enable PDL.\n\n* Fix: Correct PDL en/disablement based on device-side arch check. Host\nside check is UB. Required moving from macros to inlined functions\n\n* Fix: default-disable PDL. Enable by setting GGML_CUDA_ENABLE_PDL=1\n\n* Enable PDL by default for Hopper+ devices\n\n* Enrolls softcap_f32 and two flash_attn kernels into PDL.\n\n* Improves flash attn PDL barrier placement\n\n* Fix: Perf regression on ada; excludes ada and below from PDL launches\n\n* Improves some sync barrier placements\n\n* Drops superfluous constructor\n\n* Adds #endif guard comments\n\n* Reverts experimental change to top-k-moe.cu, which moved expensive allocations\nin front of the PDL barrier. It did not have a meaningful impact.\n\n* Exchanges GGML_CUDA_DISABLE_PDL with GGML_CUDA_PDL. IFF GGML_CUDA_PDL=0\nPDL is disabled\n\n* Revert \"Drops superfluous constructor\". Adds const to remaining\narguments\n\nThis reverts commit 12b1d250da0089ae02a9bb71bbb3fd6d70f6f2f1.\n\n* Cleanup: Removes and fixes some comments and whitespace\n\n* Clarifies comment of sync-barrier position\n\n* Relocates and refactors PDL launch functions and accessories\n\n* Adds error checking to the regular kernel launch path\n\n* Drops \"auto\" in favor of \"ggml_cuda_kernel_params\"\n\n* Adds \"const\" to ggml_cuda_kernel_launch_params\n\n* [Whitespace] Adds final newline to common.cuh to make editorconfig CI job happy","tree":"6bd686700efc2858d3b391fe3e51b1f8436b2e40"}},{"name":"mean.cuh","mode":"0100644","size":99,"lastCommit":{"hash":"aa064b2eb7f7779db5e094a9d8d66d5033557de2","oid":"aa064b2eb7f7779db5e094a9d8d66d5033557de2","parents":["aa0ef5c578eef4c2adc7be1282f21bab5f3e8d26"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-06-22T12:39:54+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-22T12:39:54+08:00"},"message":"CUDA: add mean operation (#14313)\n\n* CUDA: add mean operation\n\n* add back sum_rows_f32_cuda\n\n* Review: early exit if col!=0","tree":"9453e3e8eca9384c5973f3f6b138c2b171d0fa0a"}},{"name":"mma.cuh","mode":"0100644","size":63456,"lastCommit":{"hash":"3e037f313c2c4cfce897d9be8f43954283a61de1","oid":"3e037f313c2c4cfce897d9be8f43954283a61de1","parents":["d81e63dcfd104cbfd7409a2bc6c09706d4b313be"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2026-05-14T22:58:58+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-14T22:58:58+02:00"},"message":"HIP: RDNA3 mma FA, faster AMD transpose, tune AMD (#22880)\n\nAdds RDNA3 support to the CUDA mma FA kernel. To make the RDNA3 tensor cores work with the FP16 accumulation for VKQ the tiles they need to be 32 logical units long in direction of the attention head; for head sizes 80 and 112 that are not exactly divided by 32 the regular length of 16 with FP32 accumulation is used instead. The longer tiles also enable more efficient transposition for a warp size of 32 which is why it's also used for RDNA4. However, this scrambles the data layout of the accumulators along the attention head dimension. To prevent accidental misuse I added another entry to ggml_cuda_mma::data_layout.\n\nI also tuned the kernel parameters for RDNA3, RDNA4, and CDNA1 in general, during which I discovered that the kernel can be made to work for head sizes up to 256 for CDNA. For RDNA3/4 I was not able to get better performance that the tile kernel for head sizes > 128.","tree":"b2a6c13909f73b2cb1d596262cdde60403252add"}},{"name":"mmf.cu","mode":"0100644","size":7883,"lastCommit":{"hash":"5839ba352471b2a7b45e7ba401619a6896f10f8b","oid":"5839ba352471b2a7b45e7ba401619a6896f10f8b","parents":["a320cbfcb7056b7b81fb854d97fe01d0ea77c4b5"],"author":{"name":"Pranesh Gonegandla","email":"pranesh.iitp@gmail.com","when":"2026-07-16T07:02:25Z"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-16T09:02:25+02:00"},"message":"CUDA: dedup MoE gate/up activation quantization (#25441)\n\n* CUDA: dedup MoE gate/up activation quantization (fp4)\n\nFor MoE gate/up projections the src1 activation is broadcast across the\nrouted experts (ne11 == 1), so ids_src1 maps every one of a token's\nn_expert_used slots to the same physical row. The MMQ path therefore\nre-quantized each token's activation n_expert_used times.\n\nFor fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of\nonce per expert. For NVFP4 a single quantize+scatter kernel\n(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the\nresulting block_fp4_mmq straight to all n_expert_used slots, using an\ninverse token->compact-row map (build_tok2c). MXFP4, and\nGGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique\nrows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).\nBoth are bit-identical to the previous gather-then-quantize path (identical\nsource data, deterministic per-block quantization), verified by\ntest-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the\ndefault, gather, and per-expert paths) and by coherent end-to-end\ngeneration. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original\nper-expert path.\n\nSame-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192\n(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):\nactivation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused\nquantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The\nfused path avoids materializing and re-reading the 8x compact buffer,\nwriting the expert copies directly from registers.\n\n* CUDA: bounds-check token ids in build_tok2c_kernel\n\nGuard against malformed ids_src1: skip out-of-range token ids (t < 0 or\nt >= n_tokens) and drop entries beyond n_expert_used per token instead of\nwriting past the token's tok2c region. No behavior change for valid MoE\nrouting data; test-backend-ops MUL_MAT_ID 790/790.\n\n* Refactor the code based on review comments\n\n- Removed previously added kernels that were not necessary anymore\\\n- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.\n\n* Adding q8_1 support for dedup and addressing review comments\n\n* Add pragma unrolls\n\n* Remove redundant cudaMemsetAsync call\n\n* Removing follow up redundancies\n\n---------\n\nCo-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>","tree":"0c237a160ef38fcf3ccac256668ae324b0e2afc6"}},{"name":"mmf.cuh","mode":"0100644","size":45141,"lastCommit":{"hash":"1fd5f4803713ea3e1eda326483c9cc71a572cf02","oid":"1fd5f4803713ea3e1eda326483c9cc71a572cf02","parents":["210a6570ceda20c5d6439172c09ada08c3754cc9"],"author":{"name":"Anav Prasad","email":"anavp@nvidia.com","when":"2026-06-01T19:38:37-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-02T10:38:37+08:00"},"message":"clean up unused variables warnings (#23975)","tree":"bd95ddabe320bad6c6ede1aeef633ee8f0ffc4b8"}},{"name":"mmid.cu","mode":"0100644","size":8376,"lastCommit":{"hash":"5839ba352471b2a7b45e7ba401619a6896f10f8b","oid":"5839ba352471b2a7b45e7ba401619a6896f10f8b","parents":["a320cbfcb7056b7b81fb854d97fe01d0ea77c4b5"],"author":{"name":"Pranesh Gonegandla","email":"pranesh.iitp@gmail.com","when":"2026-07-16T07:02:25Z"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-16T09:02:25+02:00"},"message":"CUDA: dedup MoE gate/up activation quantization (#25441)\n\n* CUDA: dedup MoE gate/up activation quantization (fp4)\n\nFor MoE gate/up projections the src1 activation is broadcast across the\nrouted experts (ne11 == 1), so ids_src1 maps every one of a token's\nn_expert_used slots to the same physical row. The MMQ path therefore\nre-quantized each token's activation n_expert_used times.\n\nFor fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of\nonce per expert. For NVFP4 a single quantize+scatter kernel\n(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the\nresulting block_fp4_mmq straight to all n_expert_used slots, using an\ninverse token->compact-row map (build_tok2c). MXFP4, and\nGGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique\nrows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).\nBoth are bit-identical to the previous gather-then-quantize path (identical\nsource data, deterministic per-block quantization), verified by\ntest-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the\ndefault, gather, and per-expert paths) and by coherent end-to-end\ngeneration. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original\nper-expert path.\n\nSame-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192\n(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):\nactivation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused\nquantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The\nfused path avoids materializing and re-reading the 8x compact buffer,\nwriting the expert copies directly from registers.\n\n* CUDA: bounds-check token ids in build_tok2c_kernel\n\nGuard against malformed ids_src1: skip out-of-range token ids (t < 0 or\nt >= n_tokens) and drop entries beyond n_expert_used per token instead of\nwriting past the token's tok2c region. No behavior change for valid MoE\nrouting data; test-backend-ops MUL_MAT_ID 790/790.\n\n* Refactor the code based on review comments\n\n- Removed previously added kernels that were not necessary anymore\\\n- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.\n\n* Adding q8_1 support for dedup and addressing review comments\n\n* Add pragma unrolls\n\n* Remove redundant cudaMemsetAsync call\n\n* Removing follow up redundancies\n\n---------\n\nCo-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>","tree":"0c237a160ef38fcf3ccac256668ae324b0e2afc6"}},{"name":"mmid.cuh","mode":"0100644","size":278,"lastCommit":{"hash":"5839ba352471b2a7b45e7ba401619a6896f10f8b","oid":"5839ba352471b2a7b45e7ba401619a6896f10f8b","parents":["a320cbfcb7056b7b81fb854d97fe01d0ea77c4b5"],"author":{"name":"Pranesh Gonegandla","email":"pranesh.iitp@gmail.com","when":"2026-07-16T07:02:25Z"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-16T09:02:25+02:00"},"message":"CUDA: dedup MoE gate/up activation quantization (#25441)\n\n* CUDA: dedup MoE gate/up activation quantization (fp4)\n\nFor MoE gate/up projections the src1 activation is broadcast across the\nrouted experts (ne11 == 1), so ids_src1 maps every one of a token's\nn_expert_used slots to the same physical row. The MMQ path therefore\nre-quantized each token's activation n_expert_used times.\n\nFor fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of\nonce per expert. For NVFP4 a single quantize+scatter kernel\n(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the\nresulting block_fp4_mmq straight to all n_expert_used slots, using an\ninverse token->compact-row map (build_tok2c). MXFP4, and\nGGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique\nrows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).\nBoth are bit-identical to the previous gather-then-quantize path (identical\nsource data, deterministic per-block quantization), verified by\ntest-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the\ndefault, gather, and per-expert paths) and by coherent end-to-end\ngeneration. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original\nper-expert path.\n\nSame-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192\n(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):\nactivation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused\nquantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The\nfused path avoids materializing and re-reading the 8x compact buffer,\nwriting the expert copies directly from registers.\n\n* CUDA: bounds-check token ids in build_tok2c_kernel\n\nGuard against malformed ids_src1: skip out-of-range token ids (t < 0 or\nt >= n_tokens) and drop entries beyond n_expert_used per token instead of\nwriting past the token's tok2c region. No behavior change for valid MoE\nrouting data; test-backend-ops MUL_MAT_ID 790/790.\n\n* Refactor the code based on review comments\n\n- Removed previously added kernels that were not necessary anymore\\\n- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.\n\n* Adding q8_1 support for dedup and addressing review comments\n\n* Add pragma unrolls\n\n* Remove redundant cudaMemsetAsync call\n\n* Removing follow up redundancies\n\n---------\n\nCo-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>","tree":"0c237a160ef38fcf3ccac256668ae324b0e2afc6"}},{"name":"mmq-config-ampere.cuh","mode":"0100644","size":39552,"lastCommit":{"hash":"bdc23b56b4458b9f1655aec5287f3ab56ee8daaa","oid":"bdc23b56b4458b9f1655aec5287f3ab56ee8daaa","parents":["ea50aba8cd7ea261168346172e38f3d562b3d317"],"author":{"name":"Cary Palmer","email":"professorpalmer9@gmail.com","when":"2026-09-22T00:59:47-05:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-21T22:59:47-07:00"},"message":"cuda: branch-free PTQ1_0 MMQ tile loader and full Ampere tile table (2x prefill) (#214)\n\n* cuda: branch-free PTQ1_0 MMQ tile loader and full Ampere tile table (2x prefill)\n\nPrefill takes the MMQ path and PTQ1_0 ran it at half PQ2_0's speed from the same weights\n(RTX 4070: pp2048 630 vs ~1300 t/s). CUPTI put the whole gap in mul_mat_q, 2.7x the\nper-call time of the PQ2_0 instantiation for identical shapes.\n\n- mmq-config-ampere.cuh: PTQ1_0 was the only type capped at mmq_x = 64; add the 80/96/112/128\n entries every other type has. 630 -> 914 t/s.\n- mmq-load-tiles.cuh: load_tiles_ptq1_0 split a block's 8 lanes into three divergent branches\n (lanes 0-3: 5 trit-unpack iterations, lanes 4-5: 5 more, lane 6: 2, lane 7 idle), so each warp\n serialized 12 iterations for 5 of work. All lanes now run one identical 5-iteration loop on\n their own 32-bit word of the block; lane 6 walks the two qh bytes in both 16-bit halves and\n recombines adjacent digits with a single __byte_perm. Only smem store offsets differ per lane.\n 914 -> 1304 t/s.\n\nDecode untouched (66.9 t/s). test-backend-ops MUL_MAT 78/78, MUL_MAT_ID 75/75 PTQ1_0 shapes vs\nCPU; greedy 300-token generation identical; perplexity -c 2048 -b 2048 7.6740 vs 7.6742 stock.\n\n* cuda: drop the MMQ loader benchmark narrative from the comment.\n\nKeep the lane-mapping invariant only; the measurements stay in the PR body.\n\n---------\n\nCo-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com>","tree":"aa1594ebe1d920d827f8e8a87e8fb21bd8d84db7"}},{"name":"mmq-config-blackwell.cuh","mode":"0100644","size":3747,"lastCommit":{"hash":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","oid":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","parents":["00e8187618590ca52e6737f97d38173087458ebf"],"author":{"name":"Pooya Khosravi","email":"16091823+pooyakhosravi@users.noreply.github.com","when":"2026-09-08T16:14:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T16:14:36-07:00"},"message":"cuda: optimize Bonsai low-bit kernels for GB10 (#135)\n\n* cuda: optimize Bonsai low-bit kernels for GB10\n\nAssisted-by: OpenAI Codex\n\n* cuda: address GB10 review feedback\n\n* cuda: clarify async MMQ Y-tile addressing\n\n* cuda: prefetch Q1 MMQ tiles on GB10\n\n---------\n\nCo-authored-by: Pasha Khosravi ","tree":"251e06fc6526ab3112c75b4252bc409448c3919a"}},{"name":"mmq-config-cdna.cuh","mode":"0100644","size":16889,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-config-pascal.cuh","mode":"0100644","size":26236,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-config-rdna2.cuh","mode":"0100644","size":26488,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-config-rdna3-5.cuh","mode":"0100644","size":28447,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-config-rdna3.cuh","mode":"0100644","size":28445,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-config-rdna4.cuh","mode":"0100644","size":28445,"lastCommit":{"hash":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","oid":"8bbb28b7612ead8ed2e02e3c42d3a5c6f80bb345","parents":["ed4119b4bb1120067df32d53f14b37b520148267"],"author":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"committer":{"name":"Pasha Khosravi","email":"khosravipasha@gmail.com","when":"2026-08-25T08:13:31Z"},"message":"ggml: rename Q2_0_G128 to PQ2_0 (ggml type id 142 unchanged)\n","tree":"aba6da172c4dff451780fa6187af15f72c24d23b"}},{"name":"mmq-hopper-q1.cu","mode":"0100644","size":26213,"lastCommit":{"hash":"6ac5eb087009a762a3f61876f7dc152bc8600bf4","oid":"6ac5eb087009a762a3f61876f7dc152bc8600bf4","parents":["d8f26eec76da6d09bb708bcba51ef64b8cd868a3"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-13T21:02:26-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-13T21:02:26-07:00"},"message":"cuda: enable the Hopper wgmma Q1_0/PQ2_0 prefill path by default (#171)\n\n* cuda: enable the Hopper wgmma prefill path by default\n\nInverts the runtime gate from opt-in (GGML_HOPPER_Q1) to opt-out\n(GGML_HOPPER_Q1_DISABLE) for builds configured with GGML_CUDA_HOPPER_Q1.\nThe build-time flag is unchanged and still off by default, since it\nrequires a CUTLASS checkout.\n\nValidated internally: prefill improves for the shapes this path serves\nand decode is unaffected. A control arm with the path disabled matches\nbaseline, so the sm_90a plus CUTLASS build costs nothing by itself.\n\nThe per-128-K activation scale is coarser than q8_1's per-32, so results\nare not bit-identical. The deviation was measured against a same-path\ncontrol; it is small and the tail is bounded.\n\n* cuda: restrict the Hopper wgmma path to Hopper, and trim the file comment\n\nGGML_CUDA_CC_BLACKWELL is 1200, the sm120 family, not the start of\nBlackwell. Blackwell spans CC 1000, 1100 and 1200, so the old\ncc >= GGML_CUDA_CC_BLACKWELL bound let sm_100 and sm_110 reach an sm_90a\nwgmma kernel. That was unreachable while the path was opt-in and becomes\nreachable once it defaults on, so bound the dispatch to the 900 series.\n\nAlso use GGML_CUDA_CC_HOPPER instead of a literal 900. The macro does\nexist in this tree, contrary to the comment that was there.\n\nThe file comment is shortened to the concise form AGENTS.md asks for and\nno longer points at notes a reader of this repo cannot open.","tree":"905204e29243b7787590ad5912ccff113f37693b"}},{"name":"mmq-load-tiles.cuh","mode":"0100644","size":90591,"lastCommit":{"hash":"70bf64bab3f1db9e955e0143ea2ca94a0da85874","oid":"70bf64bab3f1db9e955e0143ea2ca94a0da85874","parents":["a3a22122eb800d4d7ffa104f3ad6547e692afa3e"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ cp.async double-buffering (mmq-pipeline)\n\nPTQ1_0 MMQ: activations double-buffered with cp.async and the next weight tile fetched into registers during compute. Bit-identical, +3.3% on the matmuls.\n","tree":"189acb482fee8c114a339d712c7e8f40c3522638"}},{"name":"mmq-ptq1_0.cuh","mode":"0100644","size":14077,"lastCommit":{"hash":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","oid":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","parents":["113d61c842e3889406777388f7bb929f288e12b0"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ main loop staged per 128-value block (mmq-staged)\n\nPTQ1_0 MMQ main loop staged one 128-value block at a time: one barrier per block, x and y copied with cp.async, trits decoded between the MMAs, B fragments via ldmatrix. Bit-identical; +10% prefill. GGML_CUDA_NO_PTQ1_0_STAGED=1 restores the old loop.\n","tree":"e7f41c3007366bbf2726c51866a49c069cdf41d3"}},{"name":"mmq-vec-dot.cuh","mode":"0100644","size":50263,"lastCommit":{"hash":"8ea08e9c7b8a8bfd72305b28efd7019c33dfa969","oid":"8ea08e9c7b8a8bfd72305b28efd7019c33dfa969","parents":["70bf64bab3f1db9e955e0143ea2ca94a0da85874"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ 128-value activation scaling and int32 accumulation (mmq-y128)\n\nPTQ1_0 MMQ quantizes activations with one scale per 128 values and accumulates a whole weight block in int32, so the float epilogue runs once per block instead of per 32: +15-18% on the matmuls, ~+11% prefill. Not bit-identical: PPL +0.12%, top token unchanged at 99.42% of positions. Decode and speculative verify (mat-vec) are untouched.\n","tree":"bf8c856dc0893803cca918758878cd344134e4eb"}},{"name":"mmq.cu","mode":"0100644","size":22642,"lastCommit":{"hash":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","oid":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","parents":["00e8187618590ca52e6737f97d38173087458ebf"],"author":{"name":"Pooya Khosravi","email":"16091823+pooyakhosravi@users.noreply.github.com","when":"2026-09-08T16:14:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T16:14:36-07:00"},"message":"cuda: optimize Bonsai low-bit kernels for GB10 (#135)\n\n* cuda: optimize Bonsai low-bit kernels for GB10\n\nAssisted-by: OpenAI Codex\n\n* cuda: address GB10 review feedback\n\n* cuda: clarify async MMQ Y-tile addressing\n\n* cuda: prefetch Q1 MMQ tiles on GB10\n\n---------\n\nCo-authored-by: Pasha Khosravi ","tree":"251e06fc6526ab3112c75b4252bc409448c3919a"}},{"name":"mmq.cuh","mode":"0100644","size":83455,"lastCommit":{"hash":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","oid":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","parents":["113d61c842e3889406777388f7bb929f288e12b0"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ main loop staged per 128-value block (mmq-staged)\n\nPTQ1_0 MMQ main loop staged one 128-value block at a time: one barrier per block, x and y copied with cp.async, trits decoded between the MMAs, B fragments via ldmatrix. Bit-identical; +10% prefill. GGML_CUDA_NO_PTQ1_0_STAGED=1 restores the old loop.\n","tree":"e7f41c3007366bbf2726c51866a49c069cdf41d3"}},{"name":"mmvf.cu","mode":"0100644","size":40951,"lastCommit":{"hash":"2a2e248d86b372a188b876b3aec16da1461e3a7e","oid":"2a2e248d86b372a188b876b3aec16da1461e3a7e","parents":["b7ccd1932fae922206c1a9ef3c49b87929fa64fb"],"author":{"name":"sudoingX","email":"200180104+sudoingX@users.noreply.github.com","when":"2026-09-19T08:22:44Z"},"committer":{"name":"sudoingX","email":"200180104+sudoingX@users.noreply.github.com","when":"2026-09-27T17:21:59+07:00"},"message":"Fix: use the mat-vec kernel for bf16 matrices under 64 rows at 2 to 8 columns\n\nThe qwen35 gated-delta-net gate projections are bf16 [5120 x 48]. At 2 to 8\ncolumns on Ampere they took a 10.5 us path where mul_mat_vec_f takes 4.4 to\n7.3 us (test-backend-ops perf, RTX 3060); at one column both already used\nmul_mat_vec_f (3.3 us). 96 calls per token on the 27B model.\n","tree":"0b9f624fa9ac09f47b35b4f582f4499e125d02d6"}},{"name":"mmvf.cuh","mode":"0100644","size":826,"lastCommit":{"hash":"8bece2eb20f0134632ae229849fbde6559882d36","oid":"8bece2eb20f0134632ae229849fbde6559882d36","parents":["a6fd8ca1fee621addff1695165414c4822fb08bf"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-02-03T23:31:23+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-02-03T23:31:23+08:00"},"message":"CUDA: use mmvq for mul-mat-id for small batch sizes (#18958)\n\n* CUDA: use mmvq for mul-mat-id for small batch sizes\n\n* add mmvq too\n\n* Fix perf issue on ampere. Use mmvf mm-id only for non-nvidia GPUs\n\n* templatize multi_token_path","tree":"88ee4a12467b368ad62526d378dc825e806b487c"}},{"name":"mmvq-ptq1_0.cuh","mode":"0100644","size":37797,"lastCommit":{"hash":"93aff088805f938bc4882062181f59c78facb40c","oid":"93aff088805f938bc4882062181f59c78facb40c","parents":["e69b3d844f11d05e3daaff3c0149d5483ea58a0e"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: PTQ1_0 tensor-core verify mat-vec register reduction (decode-verify)\n\nPTQ1_0 tensor-core verify mat-vec (MTP verify, 2-8 columns): the 32 residue slots fold in registers in warp_reduce_sum's pairing and only log2(warps) steps go through smem, with CTA geometry per shape. Bit-identical to the 1-column kernel; MTP decode +8.7% (two slots verifying +15%).\n","tree":"037813bc03d67e9b4fa562ce95d71cbcd49c21b6"}},{"name":"mmvq.cu","mode":"0100644","size":76540,"lastCommit":{"hash":"2ed33b9330e85ba919d3557306a6605e350887e9","oid":"2ed33b9330e85ba919d3557306a6605e350887e9","parents":["834c579f70d886d15edb1dcd4243e1cf3350775f"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 tensor-core mat-vec for 3-8 columns (ptq1-tc-verify)\n\nPTQ1_0 mat-vec for 3 to 8 columns on the tensor cores (mma.m16n8k32 over decoded trits), so a speculative verify costs about one column instead of ~1.8x at 4. Same bits as the dp4a kernel.\n","tree":"66f55188b574504cc347c7a45bd936b55fb449ef"}},{"name":"mmvq.cuh","mode":"0100644","size":988,"lastCommit":{"hash":"bc81d47aba663383dc8827bd3bc8c4a35f735d16","oid":"bc81d47aba663383dc8827bd3bc8c4a35f735d16","parents":["0b246862b98b452770a4ce3a87a9a89b6b28a2f1"],"author":{"name":"Jaden_Mach","email":"88880593+jadenmach2@users.noreply.github.com","when":"2026-05-28T08:50:25-04:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-28T14:50:25+02:00"},"message":"CUDA: route batch>=4 quantized matmul to MMQ on AMD MFMA hardware (#23227)\n\n* CUDA: per-quant MMVQ/MMQ batch threshold on AMD MFMA hardware\n\nThe dispatcher uses a single global threshold (MMVQ_MAX_BATCH_SIZE = 8)\nto choose between mul_mat_vec_q (per-row GEMV) and mul_mat_q (MFMA-tiled\nGEMM) for quantized matmul. On AMD CDNA, the optimal crossover differs\nsubstantially by quant family because the per-row GEMV cost is dominated\nby dequantisation, not the dot-product itself: K-quants pay a heavier\nsuper-block decode and so MMQ wins sooner; legacy and IQ quants have\nlean decode and stay ahead until the batch fully populates an MFMA tile.\n\nThis patch introduces ggml_cuda_should_use_mmvq(type, cc, ne11) -> bool,\nmirroring the existing ggml_cuda_should_use_mmq, and gates per-quant\nthresholds on amd_mfma_available(cc):\n\n Q3_K, Q4_K, Q5_K : MMVQ <= 3 (MMQ wins from batch=4: +5% .. +76%)\n Q2_K, Q6_K : MMVQ <= 5 (MMQ wins from batch=6: +8% .. +35%)\n others : MMVQ <= 8 (legacy & IQ regress under MMQ; unchanged)\n\nNon-AMD-MFMA paths (NVIDIA, RDNA, CDNA1 without MFMA) are byte-identical\nto master. GGML_CUDA_FORCE_MMVQ=1 restores the original global threshold\nfor A/B testing.\n\nMeasured on MI250X (gfx90a, ROCm 7.2.1) with Llama-3.2-3B-Instruct,\nllama-bench pp512 across all 20 supported quants, ubatch 1..8, 10 reps.\nFull table in PR description.\n\n Selected pp512 throughput (tok/s, ub=8):\n Q4_K_S: 559 -> 940 (+68%)\n Q5_K_S: 503 -> 884 (+76%)\n Q3_K_S: 629 -> 879 (+40%)\n Q2_K : 615 -> 809 (+32%)\n Q6_K : 582 -> 776 (+33%)\n\n Selected pp512 throughput (tok/s, ub=4):\n Q4_K_S: 444 -> 480 (+ 8%)\n Q4_0 : 682 -> 685 (+ 0%) (no regression - retains MMVQ)\n IQ4_XS: 706 -> 698 (- 1%) (no regression - retains MMVQ)\n\n* CUDA: address review — inline MMVQ batch table, drop env hatch & doc block\n\n* tune kernel selection logic for CDNA1\n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"a12ed106444f3aa042d1f7744db9f7dec7923d0f"}},{"name":"norm.cu","mode":"0100644","size":37529,"lastCommit":{"hash":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","oid":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","parents":["00e8187618590ca52e6737f97d38173087458ebf"],"author":{"name":"Pooya Khosravi","email":"16091823+pooyakhosravi@users.noreply.github.com","when":"2026-09-08T16:14:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T16:14:36-07:00"},"message":"cuda: optimize Bonsai low-bit kernels for GB10 (#135)\n\n* cuda: optimize Bonsai low-bit kernels for GB10\n\nAssisted-by: OpenAI Codex\n\n* cuda: address GB10 review feedback\n\n* cuda: clarify async MMQ Y-tile addressing\n\n* cuda: prefetch Q1 MMQ tiles on GB10\n\n---------\n\nCo-authored-by: Pasha Khosravi ","tree":"251e06fc6526ab3112c75b4252bc409448c3919a"}},{"name":"norm.cuh","mode":"0100644","size":1130,"lastCommit":{"hash":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","oid":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","parents":["00e8187618590ca52e6737f97d38173087458ebf"],"author":{"name":"Pooya Khosravi","email":"16091823+pooyakhosravi@users.noreply.github.com","when":"2026-09-08T16:14:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T16:14:36-07:00"},"message":"cuda: optimize Bonsai low-bit kernels for GB10 (#135)\n\n* cuda: optimize Bonsai low-bit kernels for GB10\n\nAssisted-by: OpenAI Codex\n\n* cuda: address GB10 review feedback\n\n* cuda: clarify async MMQ Y-tile addressing\n\n* cuda: prefetch Q1 MMQ tiles on GB10\n\n---------\n\nCo-authored-by: Pasha Khosravi ","tree":"251e06fc6526ab3112c75b4252bc409448c3919a"}},{"name":"opt-step-adamw.cu","mode":"0100644","size":2994,"lastCommit":{"hash":"8a43e940ab0daaff198809bf9277289994ec62f5","oid":"8a43e940ab0daaff198809bf9277289994ec62f5","parents":["5c9a8b22b10132db620529435e3cfa49304b65cc"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-11-16T22:17:59+02:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-11-17T08:30:29+02:00"},"message":"ggml: new optimization interface (ggml/988)\n","tree":"df6c2204baed8c0fab338c0a3d3b5d9807b526db"}},{"name":"opt-step-adamw.cuh","mode":"0100644","size":150,"lastCommit":{"hash":"424c5d00a9b97dd5559635872db9b57f87c23b02","oid":"424c5d00a9b97dd5559635872db9b57f87c23b02","parents":["a6809c6a2e8163cba296d4cc0c5cd4bbc8073638"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-09-20T19:04:44+03:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-09-20T21:15:05+03:00"},"message":"ggml/examples: add backend support for numerical optimization (ggml/949)\n\n* CUDA eval works\n\n* stochastic gradient descent op\n\n* Adam except decay\n\n* CUDA CROSS_ENTROPY_LOSS_BACK\n\n* CUDA mnist-fc training works\n\n* backend CLI arg\n\n* refactor gguf load\n\n* remove sched from opt_step_adam\n\n* implement l1 regularization (weight decay)\n\n* extra call to add optimizer\n\n* initialize gradients with ggml_graph_reset\n\n* gradient accumulation\n\n* increment iter per eval instead of epoch\n\n* adjust backend interfaces\n\n* fix ggml_graph_reset without backend\n\n* fix ggml graph export/import\n\n* fixup\n\n* rename\n\n* revert ggml_opt changes\n\n* more general CUDA repeat_back\n\n* update documentation, fix CNN\n\n* validation split\n\n* add clarifying comment\n\n* optimize PyTorch training\n\n* adjust buffer size, thread count\n\n* fix 0.0f validation split\n\n* Update examples/mnist/mnist-common.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* fix gradient accumulation\n\n* tensor flag for accumulators -> tensor hash set\n\n* Update include/ggml.h\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* fix test prints\n\n* Update src/ggml-backend.c\n\nCo-authored-by: Georgi Gerganov \n\n* better CUDA support for noncontiguous out_prod\n\n* add comment\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: slaren \n","tree":"6f3a1a31269e26c2ac6c650aeecd8d713357317d"}},{"name":"opt-step-sgd.cu","mode":"0100644","size":1759,"lastCommit":{"hash":"5cdb27e0917479d2d742cea7beee089574bb09fa","oid":"5cdb27e0917479d2d742cea7beee089574bb09fa","parents":["3ea913f1ce9567289aedd866a569dbab8fb8e419"],"author":{"name":"Jonathan Graehl","email":"99024+graehl@users.noreply.github.com","when":"2025-08-14T03:03:57-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-14T12:03:57+02:00"},"message":"finetune: SGD optimizer, more CLI args (#13873)\n\n* examples/finetune -opt SGD (stochastic gradient descent) memory opt\n\nadd unit tested GGML_OPT_OPTIMIZER_SGD to ggml - avoids allocating\nm, v tensors.\n\nsupport finetune.cpp arg -opt SGD (or sgd). (default adamw as before)\n\nllama 3.2-1b-F32 result: observed 11gb gpu ram (41 sec/epoch)\nwhen using SGD instead of 19gb (55 sec/epoch) using adamw.\n(wikipedia 100 lines finetune)\n\n(\nusing the same GPU memory, adamw can only do before OOM 512\nbatch/context, reaching:\ntrain: [███████▉] data=0000140/0000140 loss=0.02575±0.00099 acc=99.52±0.03% t=00:00:47 ETA=00:00:00\nval: [███████▉] data=0000008/0000008 loss=4.76565±0.28810 acc=41.46±0.77% t=00:00:00 ETA=00:00:00\n\nSGD is superior, though it converges slower, with max before OOM 1728\nbatch/context (esp see the better validation perf):\ntrain: [███████▉] data=0000039/0000039 loss=0.00371±0.00010 acc=99.96±0.01% t=00:00:41 ETA=00:00:00\nval: [███████▉] data=0000003/0000003 loss=5.11406±0.76034 acc=48.01±0.69% t=00:00:01 ETA=00:00:00\n)\n\nnote: when finetuning long enough (or w/ enough -lr),\nvalidation accuracy *eventually* drops ('catastrophic forgetting')\n\n-lr-half (halflife) option useful for SGD to avoid oscillation or\nsuper slow underdamped learning (makes setting -lr more forgiving).\nterminal -lr for now is set by lr-halvings i.e. if you want at most\n1/8 the inital -lr you set -lr-halvings 3.\n\nnote: objective loss not directly comparable between adamw, sgd? -\ncheck perplexity or accuracy or consider relative improvements\nfor convergence\n\nnew finetune args -wd 1e-9 to enable weight decay in sgd or adamw,\nand max -epochs N (default 2 as before)\n\ncache (1 - wd*alpha) in 'adamw' opt struct -\nno noticeable perf benefit, disabled (still done\nfor new SGD though)\n\nsince opt. memory is pre-allocated, the ggml_opt_get_optimizer_params\nwould probably be able to change between SGD and AdamW with each epoch\nbut would need to use adamw for the first (unconfirmed - no cmdline arg\nto set such a policy yet)\n\ntest-opt checks adamw as before and now sgd (except for a few disabled\ntests for sgd only; probably just needs logging values and adding\nalternate reference values); tolerance on the 'regression'\ntest is broader for sgd (so we don't need many more epochs)\n\n* Vulkan: Implement GGML_OP_OPT_STEP_SGD\n\n* tests: Fix OPT_STEP_SGD test-backend-ops\n\n* SGD op param store weight-decay and not 1-alpha*wd\n\n* minor + cosmetic changes\n\n* fix vulkan sgd\n\n* try CI fix\n\n---------\n\nCo-authored-by: 0cc4m \nCo-authored-by: Johannes Gäßler ","tree":"c84fd7efa0271134bb130ea261cfbdb548998459"}},{"name":"opt-step-sgd.cuh","mode":"0100644","size":146,"lastCommit":{"hash":"5cdb27e0917479d2d742cea7beee089574bb09fa","oid":"5cdb27e0917479d2d742cea7beee089574bb09fa","parents":["3ea913f1ce9567289aedd866a569dbab8fb8e419"],"author":{"name":"Jonathan Graehl","email":"99024+graehl@users.noreply.github.com","when":"2025-08-14T03:03:57-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-14T12:03:57+02:00"},"message":"finetune: SGD optimizer, more CLI args (#13873)\n\n* examples/finetune -opt SGD (stochastic gradient descent) memory opt\n\nadd unit tested GGML_OPT_OPTIMIZER_SGD to ggml - avoids allocating\nm, v tensors.\n\nsupport finetune.cpp arg -opt SGD (or sgd). (default adamw as before)\n\nllama 3.2-1b-F32 result: observed 11gb gpu ram (41 sec/epoch)\nwhen using SGD instead of 19gb (55 sec/epoch) using adamw.\n(wikipedia 100 lines finetune)\n\n(\nusing the same GPU memory, adamw can only do before OOM 512\nbatch/context, reaching:\ntrain: [███████▉] data=0000140/0000140 loss=0.02575±0.00099 acc=99.52±0.03% t=00:00:47 ETA=00:00:00\nval: [███████▉] data=0000008/0000008 loss=4.76565±0.28810 acc=41.46±0.77% t=00:00:00 ETA=00:00:00\n\nSGD is superior, though it converges slower, with max before OOM 1728\nbatch/context (esp see the better validation perf):\ntrain: [███████▉] data=0000039/0000039 loss=0.00371±0.00010 acc=99.96±0.01% t=00:00:41 ETA=00:00:00\nval: [███████▉] data=0000003/0000003 loss=5.11406±0.76034 acc=48.01±0.69% t=00:00:01 ETA=00:00:00\n)\n\nnote: when finetuning long enough (or w/ enough -lr),\nvalidation accuracy *eventually* drops ('catastrophic forgetting')\n\n-lr-half (halflife) option useful for SGD to avoid oscillation or\nsuper slow underdamped learning (makes setting -lr more forgiving).\nterminal -lr for now is set by lr-halvings i.e. if you want at most\n1/8 the inital -lr you set -lr-halvings 3.\n\nnote: objective loss not directly comparable between adamw, sgd? -\ncheck perplexity or accuracy or consider relative improvements\nfor convergence\n\nnew finetune args -wd 1e-9 to enable weight decay in sgd or adamw,\nand max -epochs N (default 2 as before)\n\ncache (1 - wd*alpha) in 'adamw' opt struct -\nno noticeable perf benefit, disabled (still done\nfor new SGD though)\n\nsince opt. memory is pre-allocated, the ggml_opt_get_optimizer_params\nwould probably be able to change between SGD and AdamW with each epoch\nbut would need to use adamw for the first (unconfirmed - no cmdline arg\nto set such a policy yet)\n\ntest-opt checks adamw as before and now sgd (except for a few disabled\ntests for sgd only; probably just needs logging values and adding\nalternate reference values); tolerance on the 'regression'\ntest is broader for sgd (so we don't need many more epochs)\n\n* Vulkan: Implement GGML_OP_OPT_STEP_SGD\n\n* tests: Fix OPT_STEP_SGD test-backend-ops\n\n* SGD op param store weight-decay and not 1-alpha*wd\n\n* minor + cosmetic changes\n\n* fix vulkan sgd\n\n* try CI fix\n\n---------\n\nCo-authored-by: 0cc4m \nCo-authored-by: Johannes Gäßler ","tree":"c84fd7efa0271134bb130ea261cfbdb548998459"}},{"name":"out-prod.cu","mode":"0100644","size":4930,"lastCommit":{"hash":"d9b6be07d0864ab09417b17ba36f9788087dd22c","oid":"d9b6be07d0864ab09417b17ba36f9788087dd22c","parents":["929d47a39163d67a2808413eaf8916097c4bb53c"],"author":{"name":"Alexander Heisler","email":"126129661+heislera763@users.noreply.github.com","when":"2026-08-20T03:27:51-04:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-20T10:27:51+03:00"},"message":"ggml-cuda: provide static workspace for cuBLAS handles (#26574)\n\n* provide static workspace for cuBLAS handles\n\n* account for concurrent streams when using GGML_CUDA_GRAPH_OPT\n\n* drop cublas_handle overloads and remove direct cublasSetStream calls\n\n* Update ggml/src/ggml-cuda/common.cuh\n\n---------\n\nCo-authored-by: Oliver Simons ","tree":"160a1277abe05ba464c2fbda31d805873ab56711"}},{"name":"out-prod.cuh","mode":"0100644","size":100,"lastCommit":{"hash":"424c5d00a9b97dd5559635872db9b57f87c23b02","oid":"424c5d00a9b97dd5559635872db9b57f87c23b02","parents":["a6809c6a2e8163cba296d4cc0c5cd4bbc8073638"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-09-20T19:04:44+03:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-09-20T21:15:05+03:00"},"message":"ggml/examples: add backend support for numerical optimization (ggml/949)\n\n* CUDA eval works\n\n* stochastic gradient descent op\n\n* Adam except decay\n\n* CUDA CROSS_ENTROPY_LOSS_BACK\n\n* CUDA mnist-fc training works\n\n* backend CLI arg\n\n* refactor gguf load\n\n* remove sched from opt_step_adam\n\n* implement l1 regularization (weight decay)\n\n* extra call to add optimizer\n\n* initialize gradients with ggml_graph_reset\n\n* gradient accumulation\n\n* increment iter per eval instead of epoch\n\n* adjust backend interfaces\n\n* fix ggml_graph_reset without backend\n\n* fix ggml graph export/import\n\n* fixup\n\n* rename\n\n* revert ggml_opt changes\n\n* more general CUDA repeat_back\n\n* update documentation, fix CNN\n\n* validation split\n\n* add clarifying comment\n\n* optimize PyTorch training\n\n* adjust buffer size, thread count\n\n* fix 0.0f validation split\n\n* Update examples/mnist/mnist-common.cpp\n\nCo-authored-by: Georgi Gerganov \n\n* fix gradient accumulation\n\n* tensor flag for accumulators -> tensor hash set\n\n* Update include/ggml.h\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: slaren \n\n* fix test prints\n\n* Update src/ggml-backend.c\n\nCo-authored-by: Georgi Gerganov \n\n* better CUDA support for noncontiguous out_prod\n\n* add comment\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: slaren \n","tree":"6f3a1a31269e26c2ac6c650aeecd8d713357317d"}},{"name":"pad.cu","mode":"0100644","size":4463,"lastCommit":{"hash":"a0d585537cb7f0352c4859acbad64d5084dbe964","oid":"a0d585537cb7f0352c4859acbad64d5084dbe964","parents":["98e57ca422c5adb33663a6406c2f2d5b7d255da7"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2026-02-10T08:07:16+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-02-10T08:07:16+02:00"},"message":"cuda : extend GGML_OP_PAD to work with non-cont src0 (#19429)\n\n* cuda : extend GGML_OP_PAD to work with non-cont src0\n\n* tests : add permuted pad","tree":"a091c5d4c1b8022a5a2eb89976680f273d5a5890"}},{"name":"pad.cuh","mode":"0100644","size":131,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"pad_reflect_1d.cu","mode":"0100644","size":3501,"lastCommit":{"hash":"0f7c69689f4e681948a6005adea9bee9c08ff903","oid":"0f7c69689f4e681948a6005adea9bee9c08ff903","parents":["835b2b915c52bcabcd688d025eacff9a07b65f52"],"author":{"name":"R0CKSTAR","email":"yeahdongcn@gmail.com","when":"2025-09-26T08:56:10+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-09-26T02:56:10+02:00"},"message":"musa: fix build warnings (#15611)\n\nSigned-off-by: Xiaodong Ye ","tree":"ec4b7154fe152dfa877955655b6f74999d957399"}},{"name":"pad_reflect_1d.cuh","mode":"0100644","size":153,"lastCommit":{"hash":"b1ab91821f980f8993423c3f2a82a0a0f60c09d2","oid":"b1ab91821f980f8993423c3f2a82a0a0f60c09d2","parents":["9ebebef62fd0adf8685874f154e227ea87b7c6f4"],"author":{"name":"Yavor Ivanov","email":"yavorgenadiev@gmail.com","when":"2025-08-22T14:06:29+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-22T13:06:29+02:00"},"message":"cuda : add Pad Reflect 1D support (#14659)\n\n* Add Pad Reflect 1D CUDA support\n\n* Update ggml/src/ggml-cuda/pad_reflect_1d.cu\n\nCo-authored-by: Johannes Gäßler \n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"0e1c8381a2ba6840478ab4a5ad68e543948a8ddb"}},{"name":"pool1d.cu","mode":"0100644","size":2664,"lastCommit":{"hash":"29ea9412a6b5e343dc6736a5a4349158cf78b30c","oid":"29ea9412a6b5e343dc6736a5a4349158cf78b30c","parents":["70adb1b4cea5ee39f867792c78dc59320921eda7"],"author":{"name":"Aman Karki","email":"itsamankarki@gmail.com","when":"2026-08-23T13:07:32+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-23T10:37:32+03:00"},"message":"cuda : add POOL_1D support (#27573)\n\n* cuda : add POOL_1D support\n\n* fix: add missing trailing newline for editorconfig compliance","tree":"46f1d989be1b3ead4ddc33d5648d08ddc2b0ecc9"}},{"name":"pool1d.cuh","mode":"0100644","size":137,"lastCommit":{"hash":"29ea9412a6b5e343dc6736a5a4349158cf78b30c","oid":"29ea9412a6b5e343dc6736a5a4349158cf78b30c","parents":["70adb1b4cea5ee39f867792c78dc59320921eda7"],"author":{"name":"Aman Karki","email":"itsamankarki@gmail.com","when":"2026-08-23T13:07:32+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-23T10:37:32+03:00"},"message":"cuda : add POOL_1D support (#27573)\n\n* cuda : add POOL_1D support\n\n* fix: add missing trailing newline for editorconfig compliance","tree":"46f1d989be1b3ead4ddc33d5648d08ddc2b0ecc9"}},{"name":"pool2d.cu","mode":"0100644","size":3306,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"pool2d.cuh","mode":"0100644","size":137,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"quantize.cu","mode":"0100644","size":37671,"lastCommit":{"hash":"8ea08e9c7b8a8bfd72305b28efd7019c33dfa969","oid":"8ea08e9c7b8a8bfd72305b28efd7019c33dfa969","parents":["70bf64bab3f1db9e955e0143ea2ca94a0da85874"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ 128-value activation scaling and int32 accumulation (mmq-y128)\n\nPTQ1_0 MMQ quantizes activations with one scale per 128 values and accumulates a whole weight block in int32, so the float epilogue runs once per block instead of per 32: +15-18% on the matmuls, ~+11% prefill. Not bit-identical: PPL +0.12%, top token unchanged at 99.42% of positions. Decode and speculative verify (mat-vec) are untouched.\n","tree":"bf8c856dc0893803cca918758878cd344134e4eb"}},{"name":"quantize.cuh","mode":"0100644","size":4337,"lastCommit":{"hash":"2ed33b9330e85ba919d3557306a6605e350887e9","oid":"2ed33b9330e85ba919d3557306a6605e350887e9","parents":["834c579f70d886d15edb1dcd4243e1cf3350775f"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 tensor-core mat-vec for 3-8 columns (ptq1-tc-verify)\n\nPTQ1_0 mat-vec for 3 to 8 columns on the tensor cores (mma.m16n8k32 over decoded trits), so a speculative verify costs about one column instead of ~1.8x at 4. Same bits as the dp4a kernel.\n","tree":"66f55188b574504cc347c7a45bd936b55fb449ef"}},{"name":"reduce_rows.cuh","mode":"0100644","size":1215,"lastCommit":{"hash":"9e58d4d692ed3d350591cc86d06c73c61c122509","oid":"9e58d4d692ed3d350591cc86d06c73c61c122509","parents":["3571fa5435ac9ff243662b1caabc407e8d433c9d"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-06-03T13:56:42+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-03T13:56:42+02:00"},"message":"Avoid PDL race conditions by disabling __restrict__ when PDL is used (#24030)\n\n* Removes __restrict__ from PDL kernel headers due to incompatibility with\nPDL. Adds preprocessor directives based on arch in kernel body to add\n__restrict__ to retain performance on older architectures.\n\n* Simplifies new __restrict__ usage via macro\n\n* Add hopper to PDL __restrict__ fix.\n\nCo-authored-by: Oliver Simons \n\n---------\n\nCo-authored-by: Oliver Simons ","tree":"0cf5e574727793b7453ab0ebb783401697b161ec"}},{"name":"roll.cu","mode":"0100644","size":2418,"lastCommit":{"hash":"0a5036bee9cfb946870689db4400e9e0d17844c9","oid":"0a5036bee9cfb946870689db4400e9e0d17844c9","parents":["8ad7b3e65b5834e5574c2f5640056c9047b5d93b"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-07-29T14:45:18+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-07-29T14:45:18+08:00"},"message":"CUDA: add roll (#14919)\n\n* CUDA: add roll\n\n* Make everything const, use __restrict__","tree":"8ea4437bb16c722191cfc19bade02b109a1ec44e"}},{"name":"roll.cuh","mode":"0100644","size":133,"lastCommit":{"hash":"0a5036bee9cfb946870689db4400e9e0d17844c9","oid":"0a5036bee9cfb946870689db4400e9e0d17844c9","parents":["8ad7b3e65b5834e5574c2f5640056c9047b5d93b"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-07-29T14:45:18+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-07-29T14:45:18+08:00"},"message":"CUDA: add roll (#14919)\n\n* CUDA: add roll\n\n* Make everything const, use __restrict__","tree":"8ea4437bb16c722191cfc19bade02b109a1ec44e"}},{"name":"rope.cu","mode":"0100644","size":44541,"lastCommit":{"hash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","oid":"fe8156f789011f6ea0baf6917ea09f88b89d9554","parents":["77acca437fb6dbe79c649ca69ff98b32bfac48c3"],"author":{"name":"Xuan-Son Nguyen","email":"son@huggingface.co","when":"2026-08-19T14:04:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-19T14:04:57+02:00"},"message":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization","tree":"45a80859202244449fe4586006e673c53336e861"}},{"name":"rope.cuh","mode":"0100644","size":486,"lastCommit":{"hash":"687e7789271ec1276e3470f158428e11a4f80b6f","oid":"687e7789271ec1276e3470f158428e11a4f80b6f","parents":["18f7ad7fc912444acc0f51995a4b8e45fd9a0cd4"],"author":{"name":"Rafail Giavrimis","email":"47496212+grafail@users.noreply.github.com","when":"2026-08-08T17:32:37+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-09T00:32:37+08:00"},"message":"CUDA: fuse rms_norm + mul + rope (+ view + set_rows) (#26767)\n\n* CUDA: fuse rms_norm + mul + rope (+ view + set_rows)\n\n* tests: add broadcast weight case to rms_norm_mul_rope\n\n* CUDA: check memory ranges before rms_norm rope fusion\n\n* CUDA: check memory ranges in rope set_rows fusion","tree":"aa97b9a3475b39a4efafdef3540af94452c08e4a"}},{"name":"scale.cu","mode":"0100644","size":2888,"lastCommit":{"hash":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","oid":"d8d96cf05115ae53b26d5730e234b2c262cb3de4","parents":["00e8187618590ca52e6737f97d38173087458ebf"],"author":{"name":"Pooya Khosravi","email":"16091823+pooyakhosravi@users.noreply.github.com","when":"2026-09-08T16:14:36-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-08T16:14:36-07:00"},"message":"cuda: optimize Bonsai low-bit kernels for GB10 (#135)\n\n* cuda: optimize Bonsai low-bit kernels for GB10\n\nAssisted-by: OpenAI Codex\n\n* cuda: address GB10 review feedback\n\n* cuda: clarify async MMQ Y-tile addressing\n\n* cuda: prefetch Q1 MMQ tiles on GB10\n\n---------\n\nCo-authored-by: Pasha Khosravi ","tree":"251e06fc6526ab3112c75b4252bc409448c3919a"}},{"name":"scale.cuh","mode":"0100644","size":135,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"set-rows.cu","mode":"0100644","size":15356,"lastCommit":{"hash":"ed8c26150e6b0ed6e2635cab75ace5ed121482ca","oid":"ed8c26150e6b0ed6e2635cab75ace5ed121482ca","parents":["90e0f5cfcb6cdb4b7b60a4f81b0a26e542149ad5"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-08T13:24:20+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-08T19:24:20+08:00"},"message":"cuda : add support for f16->f16 GGML_OP_SET_ROWS (#25367)","tree":"969877565f765ed34fb7f0887e76c4334a8c53f6"}},{"name":"set-rows.cuh","mode":"0100644","size":155,"lastCommit":{"hash":"7de5c7cab61d4da4387ed9b216f88b96297bcc2d","oid":"7de5c7cab61d4da4387ed9b216f88b96297bcc2d","parents":["8eff95544e817704d44bec20f9fc956ce76a33be"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-07-12T21:31:38+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-07-12T16:31:38+03:00"},"message":"CUDA: add set rows for f32 and f16 (#14551)\n\n* CUDA: add set rows for f32 and f16\n\n* Review: change kernel params, use strides from host\n\n* Use 1-d kernel\n\n* Review: use int64_t for blockDim.x, rename nb->s for clarity","tree":"ca2ffcf34bf44e2c398c93bb717aedd8a91e99ff"}},{"name":"set.cu","mode":"0100644","size":1268,"lastCommit":{"hash":"851553ea6b24cb39fd5fd188b437d777cb411de8","oid":"851553ea6b24cb39fd5fd188b437d777cb411de8","parents":["85a7d8677bf2200981e52f744a21d5267964ffcf"],"author":{"name":"YaelGitAccount","email":"38328157276@mby.co.il","when":"2025-10-28T21:10:28+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-10-28T20:10:28+01:00"},"message":"cuda: add SET operation support (#16804)\n\n* feat(cuda): add GGML_OP_SET support\n\nImplement CUDA kernel for SET operation with f32 support.\n\nAll tests passing (14598/14598).\n\n* cuda(set): add I32 support; keep F32\n\n* refactor(cuda): use ggml_cuda_cpy to unify SET operator logic and remove code duplication\n\n* Update ggml/src/ggml-cuda/ggml-cuda.cu\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Update ggml/src/ggml-cuda/set.cu\n\nCo-authored-by: Sigbjørn Skjæret \n\n---------\n\nCo-authored-by: Sigbjørn Skjæret ","tree":"a5edb95ed131f775173cb258c8589d0e5e3b59e3"}},{"name":"set.cuh","mode":"0100644","size":145,"lastCommit":{"hash":"851553ea6b24cb39fd5fd188b437d777cb411de8","oid":"851553ea6b24cb39fd5fd188b437d777cb411de8","parents":["85a7d8677bf2200981e52f744a21d5267964ffcf"],"author":{"name":"YaelGitAccount","email":"38328157276@mby.co.il","when":"2025-10-28T21:10:28+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-10-28T20:10:28+01:00"},"message":"cuda: add SET operation support (#16804)\n\n* feat(cuda): add GGML_OP_SET support\n\nImplement CUDA kernel for SET operation with f32 support.\n\nAll tests passing (14598/14598).\n\n* cuda(set): add I32 support; keep F32\n\n* refactor(cuda): use ggml_cuda_cpy to unify SET operator logic and remove code duplication\n\n* Update ggml/src/ggml-cuda/ggml-cuda.cu\n\nCo-authored-by: Sigbjørn Skjæret \n\n* Update ggml/src/ggml-cuda/set.cu\n\nCo-authored-by: Sigbjørn Skjæret \n\n---------\n\nCo-authored-by: Sigbjørn Skjæret ","tree":"a5edb95ed131f775173cb258c8589d0e5e3b59e3"}},{"name":"snake.cu","mode":"0100644","size":2821,"lastCommit":{"hash":"58e68df0f91dd16ff56423ee5ef44062ed73bdfc","oid":"58e68df0f91dd16ff56423ee5ef44062ed73bdfc","parents":["9b2925e1e01e24976b60538eb201617f86a8dd46"],"author":{"name":"Pascal","email":"admin@serveurperso.com","when":"2026-05-08T11:44:09+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-08T17:44:09+08:00"},"message":"cuda: fuse snake activation (mul, sin, sqr, mul, add) (#22667)\n\n* cuda: fuse snake activation (mul, sin, sqr, mul, add)\n\nAdd ggml_cuda_op_snake_fused with F32 / F16 / BF16 templates. The\nmatcher recognizes the naive 5 op decomposition emitted by audio\ndecoders (BigVGAN, Vocos) for snake activation\ny = x + sin(a*x)^2 * inv_b and rewrites it to a single elementwise\nkernel.\n\nAdd test_snake_fuse comparing CPU naive vs CUDA fused across\nF32 / F16 / BF16.\n\n* cuda: address review feedback from @am17an\n\nUse ggml_cuda_cast for F32/F16/BF16 conversions and rename\nkernel_snake to snake_kernel to match upstream conventions.\n\n* cuda: snake fusion fastdiv on T_len, Suggested-by: @am17an\n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Aman Gupta \n\n* cuda: snake fusion check add->type matches x->type\n\nAddress review feedback from @am17an\n\n* cuda: snake fusion check add->type matches x->type\n\nMoved for readability (equivalent)\nAddress review feedback from @am17an\n\n---------\n\nCo-authored-by: Aman Gupta ","tree":"e02ae1619ebd02292a519456d9e8b37f24afbc4b"}},{"name":"snake.cuh","mode":"0100644","size":366,"lastCommit":{"hash":"58e68df0f91dd16ff56423ee5ef44062ed73bdfc","oid":"58e68df0f91dd16ff56423ee5ef44062ed73bdfc","parents":["9b2925e1e01e24976b60538eb201617f86a8dd46"],"author":{"name":"Pascal","email":"admin@serveurperso.com","when":"2026-05-08T11:44:09+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-08T17:44:09+08:00"},"message":"cuda: fuse snake activation (mul, sin, sqr, mul, add) (#22667)\n\n* cuda: fuse snake activation (mul, sin, sqr, mul, add)\n\nAdd ggml_cuda_op_snake_fused with F32 / F16 / BF16 templates. The\nmatcher recognizes the naive 5 op decomposition emitted by audio\ndecoders (BigVGAN, Vocos) for snake activation\ny = x + sin(a*x)^2 * inv_b and rewrites it to a single elementwise\nkernel.\n\nAdd test_snake_fuse comparing CPU naive vs CUDA fused across\nF32 / F16 / BF16.\n\n* cuda: address review feedback from @am17an\n\nUse ggml_cuda_cast for F32/F16/BF16 conversions and rename\nkernel_snake to snake_kernel to match upstream conventions.\n\n* cuda: snake fusion fastdiv on T_len, Suggested-by: @am17an\n\n* Update tests/test-backend-ops.cpp\n\nCo-authored-by: Aman Gupta \n\n* cuda: snake fusion check add->type matches x->type\n\nAddress review feedback from @am17an\n\n* cuda: snake fusion check add->type matches x->type\n\nMoved for readability (equivalent)\nAddress review feedback from @am17an\n\n---------\n\nCo-authored-by: Aman Gupta ","tree":"e02ae1619ebd02292a519456d9e8b37f24afbc4b"}},{"name":"softcap.cu","mode":"0100644","size":1472,"lastCommit":{"hash":"e947228222147356bc7e64154d3439e142481632","oid":"e947228222147356bc7e64154d3439e142481632","parents":["29f1482221b68fdbf5bd9b762c9e3e350e21f1ec"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-05-20T13:59:02+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-20T13:59:02+02:00"},"message":"Programmatic Dependent Launch (PDL) for more performance on newer NVIDIA GPUs (Hopper+) (#22522)\n\n* Adds initial PDL setup.\n\n* Adds PDL barriers based on simple heuristic: place \"sync\" before first input pointer access, and \"launch\" after last write, e.g. to tensors like dst.\n\n* Further optimization pass of the first half of kernels\n\n* Optimized PDL barriers for the second batch of kernels\n\n* Further refinements after rebase.\n\n* Moves pdl logic to separate function, removes some whitespace\n\n* Strips post-hoc PDL logic\n\n* Adds stream capture PDL setup. Enrolls quantize_q8_1 to leverage pdl to\noverlap execution with previous kernels\n\n* Enrolls mul_mat_vec_q, rms_norm_f32 and k_bin_bcast (partly) into PDL\n\n* Enrolls mmvf, rope, set-rows and topk kernels for gpt-oss into PDL\n\n* Introduce ggml_cuda_kernel_launch, to abstract away cudaLaunchKernelEx,\nto enable hip/musa compatibility\n\n* Enrolls cpy_scalar_contiguous, k_get_rows_float and rms_norm_f32\n\n* Enrolls flash_attn_combine_results\n\n* Fix: Drops needless and broken check of CUDA arch for PDL. PDL either\nworks or is without effect.\n\n* Enrolls flash-attention kernels to pdl\n\n* Fix: inlines ggml_cuda_kernel_launch, and uses perfect forwarding for\nkernels args. This fixes PDL.\n\n* Perf: Enrolls k_bin_bcast variadic template invocation into PDL, via\nand template alias and template expansion\n\n* Enrolls all remaining kernels for qwen3-coder-next into PDL\n\n* Remove all PDL LC calls to create a baseline\n\n* Added LC according to internal guidance and tested kernel performance.\n\n* Enrols missing qwen3-5 kernels passively into PDL.\n\n* Kernel optimizations (LC signals) for qwen3.5\n\n* Enrolls ssm-scan kernels into PDL\n\n* Adds GGML_CUDA_PDL command line option to toggle PDL.\n\n* Fix: Ada and lower compilation by guarding PDL calls correctly\n\n* Cleanup: Removes commented out GGML_CUDA_PDL_LC\n\n* Cleanup: Removes experimental comments\n\n* Adds 90-virtual to build script so that Hopper GPUs can leverage PDL.\n\n* Adds stricter checks to enable PDL, adds env-check to disable it, and removes now superfluous compile option to enable PDL.\n\n* Fix: Correct PDL en/disablement based on device-side arch check. Host\nside check is UB. Required moving from macros to inlined functions\n\n* Fix: default-disable PDL. Enable by setting GGML_CUDA_ENABLE_PDL=1\n\n* Enable PDL by default for Hopper+ devices\n\n* Enrolls softcap_f32 and two flash_attn kernels into PDL.\n\n* Improves flash attn PDL barrier placement\n\n* Fix: Perf regression on ada; excludes ada and below from PDL launches\n\n* Improves some sync barrier placements\n\n* Drops superfluous constructor\n\n* Adds #endif guard comments\n\n* Reverts experimental change to top-k-moe.cu, which moved expensive allocations\nin front of the PDL barrier. It did not have a meaningful impact.\n\n* Exchanges GGML_CUDA_DISABLE_PDL with GGML_CUDA_PDL. IFF GGML_CUDA_PDL=0\nPDL is disabled\n\n* Revert \"Drops superfluous constructor\". Adds const to remaining\narguments\n\nThis reverts commit 12b1d250da0089ae02a9bb71bbb3fd6d70f6f2f1.\n\n* Cleanup: Removes and fixes some comments and whitespace\n\n* Clarifies comment of sync-barrier position\n\n* Relocates and refactors PDL launch functions and accessories\n\n* Adds error checking to the regular kernel launch path\n\n* Drops \"auto\" in favor of \"ggml_cuda_kernel_params\"\n\n* Adds \"const\" to ggml_cuda_kernel_launch_params\n\n* [Whitespace] Adds final newline to common.cuh to make editorconfig CI job happy","tree":"6bd686700efc2858d3b391fe3e51b1f8436b2e40"}},{"name":"softcap.cuh","mode":"0100644","size":158,"lastCommit":{"hash":"138b288b594eeb19d58e0ab0d74eae32009c9c20","oid":"138b288b594eeb19d58e0ab0d74eae32009c9c20","parents":["bbd0f917797e9d524680f1b30d34a46eb06d7651"],"author":{"name":"Sigbjørn Skjæret","email":"sigbjorn.skjaeret@scala.com","when":"2025-07-29T14:22:03+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-07-29T14:22:03+02:00"},"message":"cuda : add softcap fusion (#14907)\n\n","tree":"e7ec58cd5b3974615d60c7837902d482da85231f"}},{"name":"softmax.cu","mode":"0100644","size":17724,"lastCommit":{"hash":"9bd4c09ea571a9020f30eeef169b552625b5b5a4","oid":"9bd4c09ea571a9020f30eeef169b552625b5b5a4","parents":["0b14b87d7c20cb753b94b96854dd7b45306fc696"],"author":{"name":"Oliver Simons","email":"osimons@nvidia.com","when":"2026-08-03T14:22:44+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-03T14:22:44+02:00"},"message":"CUDA: Fix data-races when reusing SMEM in block_reduce (#26385)\n\n* CUDA: Fix data-races when reusing block_reduce\n\nblock_reduce currently doesn't resync after reading from SMEM, causing\npotential data-races when reusing SMEM for multiple reductions.\n\nOne may consider simply always adding this in block_reduce, but this\ncomes at a potential perf cost\n\n* double-buffering for single-row softmax\n\n* double-buffering for norm as well\n\n* Add comment\n\n* Add explanatory comment to block_reduce\n\n* Specify need for + do memory barrier only in multi-warp scenario\n\n* Implement review-suggestion from @gaugarg-nv","tree":"fa12c148da1dc06754aa780b471b5a7612711daa"}},{"name":"softmax.cuh","mode":"0100644","size":228,"lastCommit":{"hash":"9c8dcefe171ba70ed14f2a64d1433da12d0dd1c1","oid":"9c8dcefe171ba70ed14f2a64d1433da12d0dd1c1","parents":["681149ced2582f48c24792403df1307fed5eb951"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2025-01-16T16:43:38+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-01-16T16:43:38+01:00"},"message":"CUDA: backwards pass for misc. ops, add tests (#11257)\n\n* CUDA: backwards pass for misc. ops, add tests\r\n\r\n* remove restrict from pointers","tree":"8adc62b673c771f16215148a0b6d844e5e445d1c"}},{"name":"solve_tri.cu","mode":"0100644","size":11457,"lastCommit":{"hash":"d9b6be07d0864ab09417b17ba36f9788087dd22c","oid":"d9b6be07d0864ab09417b17ba36f9788087dd22c","parents":["929d47a39163d67a2808413eaf8916097c4bb53c"],"author":{"name":"Alexander Heisler","email":"126129661+heislera763@users.noreply.github.com","when":"2026-08-20T03:27:51-04:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-20T10:27:51+03:00"},"message":"ggml-cuda: provide static workspace for cuBLAS handles (#26574)\n\n* provide static workspace for cuBLAS handles\n\n* account for concurrent streams when using GGML_CUDA_GRAPH_OPT\n\n* drop cublas_handle overloads and remove direct cublasSetStream calls\n\n* Update ggml/src/ggml-cuda/common.cuh\n\n---------\n\nCo-authored-by: Oliver Simons ","tree":"160a1277abe05ba464c2fbda31d805873ab56711"}},{"name":"solve_tri.cuh","mode":"0100644","size":104,"lastCommit":{"hash":"cd0e3a7a3b93b5c2c2aeceeb778b931125b1125d","oid":"cd0e3a7a3b93b5c2c2aeceeb778b931125b1125d","parents":["efaaccdd69cd9db777584c2a062f70c0526a6fb5"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-11-28T05:15:32+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-11-28T12:15:32+08:00"},"message":"SOLVE_TRI CUDA kernel for small matrices (#17457)\n\n","tree":"5ed7f55c84993e187013fd4130fca6063793c5d0"}},{"name":"ssm-conv.cu","mode":"0100644","size":13277,"lastCommit":{"hash":"6d1917060d72cfffc6b54391ef8006b7f51361b9","oid":"6d1917060d72cfffc6b54391ef8006b7f51361b9","parents":["c0209b63a39312975f3cbff7670afda41f653cd9"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: fused CUDA kernel for DeltaNet conv path (gdn-conv)\n\nOne CUDA kernel for the DeltaNet conv path (concat -> ssm_conv -> silu -> conv-state writes): the concat is never materialized; same arithmetic. Bit-identical; MTP decode +1.5%. GGML_CUDA_SSM_CONV_CONCAT_FUSION=0 turns it off.\n","tree":"b62a7b2173f152483ed21ab164952353e634072b"}},{"name":"ssm-conv.cuh","mode":"0100644","size":1153,"lastCommit":{"hash":"6d1917060d72cfffc6b54391ef8006b7f51361b9","oid":"6d1917060d72cfffc6b54391ef8006b7f51361b9","parents":["c0209b63a39312975f3cbff7670afda41f653cd9"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: fused CUDA kernel for DeltaNet conv path (gdn-conv)\n\nOne CUDA kernel for the DeltaNet conv path (concat -> ssm_conv -> silu -> conv-state writes): the concat is never materialized; same arithmetic. Bit-identical; MTP decode +1.5%. GGML_CUDA_SSM_CONV_CONCAT_FUSION=0 turns it off.\n","tree":"b62a7b2173f152483ed21ab164952353e634072b"}},{"name":"ssm-scan.cu","mode":"0100644","size":41617,"lastCommit":{"hash":"d9b6be07d0864ab09417b17ba36f9788087dd22c","oid":"d9b6be07d0864ab09417b17ba36f9788087dd22c","parents":["929d47a39163d67a2808413eaf8916097c4bb53c"],"author":{"name":"Alexander Heisler","email":"126129661+heislera763@users.noreply.github.com","when":"2026-08-20T03:27:51-04:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-20T10:27:51+03:00"},"message":"ggml-cuda: provide static workspace for cuBLAS handles (#26574)\n\n* provide static workspace for cuBLAS handles\n\n* account for concurrent streams when using GGML_CUDA_GRAPH_OPT\n\n* drop cublas_handle overloads and remove direct cublasSetStream calls\n\n* Update ggml/src/ggml-cuda/common.cuh\n\n---------\n\nCo-authored-by: Oliver Simons ","tree":"160a1277abe05ba464c2fbda31d805873ab56711"}},{"name":"ssm-scan.cuh","mode":"0100644","size":103,"lastCommit":{"hash":"250d7953e829ff0e16074510fef0e7cec696461b","oid":"250d7953e829ff0e16074510fef0e7cec696461b","parents":["403fbacbbcd3650abcc53e84f5e6bf88e51dd6f5"],"author":{"name":"a3sh","email":"38979186+A3shTnT@users.noreply.github.com","when":"2025-04-01T00:05:13+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-03-31T18:05:13+02:00"},"message":"ggml : faster ssm scan (#10558)\n\n* faster ssm_scan\n\n* delete unused commnet\n\n* clang format\n\n* add space\n\n* modify unnecessary calculations\n\n* faster ssm conv implementatioin\n\n* modify file name with dash","tree":"dab49b46276a9216a9dd669a3177fba6f7924316"}},{"name":"sum.cu","mode":"0100644","size":1364,"lastCommit":{"hash":"6028bf74351d35a06bd98498624f8c2f029f7d1a","oid":"6028bf74351d35a06bd98498624f8c2f029f7d1a","parents":["bc5182272c373267352bc689e5fca276934bea2d"],"author":{"name":"Oliver Simons","email":"osimons@nvidia.com","when":"2025-08-13T10:04:46+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-08-13T10:04:46+02:00"},"message":"CUDA: Optimize `reduce_rows_f32` kernel, leading up to 25x perf improvement on kernel-level and 10% perf increase for Gemma3n (#15132)\n\n* Factor out `reduce_rows_f32` from common.cuh\n\nThis increases iteration cycle speed by not having to recompile\nevery kernel all the time\n\n* Hide memory-latency by loop unrolling in reduce_rows_f32\n\n* Further optimizations to `reduce_rows_f32`\n\n1. Increase threadblock size to better hide latency of memory requests.\n As a consequence of bigger threadblocks, do 2-step summation, using\n shared memory to communicate results between invocations\n2. Use sum_temp array to reduce waits on sum\n3. Adjust num_unroll to reflext bigger threadblock\n4. Improve default block_dims, increase support for more block_dims\n\n* Add perf tests for `reduce_rows_f32` kernel\n\n* Add heuristic to toggle 128/512 threads based on sm count\n\nBreak even point was the minimum of the following multiples.\n\n| GPU Model | Nrow SM Count Multiple |\n| ----------- | ----------- |\n| RTX 4000 SFF ADA | 2.0x |\n| RTX 6000 ADA | 2.5x |\n| RTX PRO 6000 Blackwell Max-Q | 3.04x |\n| RTX PRO 4500 Blackwell\t| 3.15x |\n\n* Ensure perf gains also for small ncols and large nrows\n\nAlternative to this, one could have also made the number of unrollings\ntemplate-able, but that would require compiling the kernel multiple\ntimes, increasing binary size unnecessarily\n\n* Modify perf and unit-tests\n\n* Apply auto-formatting by clang\n\n* Fix CI build failure\n\nSee https://github.com/ggml-org/llama.cpp/actions/runs/16798370266/job/47573716079?pr=15132#step:7:486\nBuilding with VS generator worked though.\n\n* Remove sm_count property from `ggml_backend_cuda_context`\n\nRequested by @JohannesGaessler, and should fix remaining CI issues as a\nside-effect\n\n* Add CUB-based implementation for GGML_OP_MEAN\n\nCurrently this branch is only executed for nrows==1\n\n* Add heuristics to execute CUB branch only when it brings perf\n\nHeuristics were determined on the following HW:\n\n* RTX 4000 SFF ADA\n* RTX 6000 ADA\n* RTX PRO 6000 Blackwell Max-Q\n* RTX PRO 4500 Blackwell\n\n* Add unit-test for CUB-based mean\n\nTests should run with CUDA Graphs enabled per default on NVGPUs\n\n* Rename `USE_CUB` to `GGML_CUDA_USE_CUB`\n\nSuggested by @JohannesGaessler\n\n* Unindent Preprocessor directives\n\nSee\nhttps://github.com/ggml-org/llama.cpp/pull/15132#discussion_r2269213506","tree":"88c2d461b389ce54d08bedc7b658a8e3df2a417c"}},{"name":"sum.cuh","mode":"0100644","size":210,"lastCommit":{"hash":"202084d31d4247764fc6d6d40d2e2bda0c89a73a","oid":"202084d31d4247764fc6d6d40d2e2bda0c89a73a","parents":["dbbebcab339c7d63d3806e8f32574bb9aad9a694"],"author":{"name":"Johannes Gäßler","email":"johannesg@5d6.de","when":"2024-09-03T17:21:46+02:00"},"committer":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-09-08T11:05:55+03:00"},"message":"tests: add gradient tests for all backends (ggml/932)\n\n* tests: add gradient checking to test-backend-ops\n\n* remove old comment\n\n* reorder includes\n\n* adjust SIN/COS parameters\n\n* add documentation, use supports_op if possible\n","tree":"ccf7cfc841fcfb38809b51fc40b279c609ad5e26"}},{"name":"sumrows.cu","mode":"0100644","size":2313,"lastCommit":{"hash":"e947228222147356bc7e64154d3439e142481632","oid":"e947228222147356bc7e64154d3439e142481632","parents":["29f1482221b68fdbf5bd9b762c9e3e350e21f1ec"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-05-20T13:59:02+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-20T13:59:02+02:00"},"message":"Programmatic Dependent Launch (PDL) for more performance on newer NVIDIA GPUs (Hopper+) (#22522)\n\n* Adds initial PDL setup.\n\n* Adds PDL barriers based on simple heuristic: place \"sync\" before first input pointer access, and \"launch\" after last write, e.g. to tensors like dst.\n\n* Further optimization pass of the first half of kernels\n\n* Optimized PDL barriers for the second batch of kernels\n\n* Further refinements after rebase.\n\n* Moves pdl logic to separate function, removes some whitespace\n\n* Strips post-hoc PDL logic\n\n* Adds stream capture PDL setup. Enrolls quantize_q8_1 to leverage pdl to\noverlap execution with previous kernels\n\n* Enrolls mul_mat_vec_q, rms_norm_f32 and k_bin_bcast (partly) into PDL\n\n* Enrolls mmvf, rope, set-rows and topk kernels for gpt-oss into PDL\n\n* Introduce ggml_cuda_kernel_launch, to abstract away cudaLaunchKernelEx,\nto enable hip/musa compatibility\n\n* Enrolls cpy_scalar_contiguous, k_get_rows_float and rms_norm_f32\n\n* Enrolls flash_attn_combine_results\n\n* Fix: Drops needless and broken check of CUDA arch for PDL. PDL either\nworks or is without effect.\n\n* Enrolls flash-attention kernels to pdl\n\n* Fix: inlines ggml_cuda_kernel_launch, and uses perfect forwarding for\nkernels args. This fixes PDL.\n\n* Perf: Enrolls k_bin_bcast variadic template invocation into PDL, via\nand template alias and template expansion\n\n* Enrolls all remaining kernels for qwen3-coder-next into PDL\n\n* Remove all PDL LC calls to create a baseline\n\n* Added LC according to internal guidance and tested kernel performance.\n\n* Enrols missing qwen3-5 kernels passively into PDL.\n\n* Kernel optimizations (LC signals) for qwen3.5\n\n* Enrolls ssm-scan kernels into PDL\n\n* Adds GGML_CUDA_PDL command line option to toggle PDL.\n\n* Fix: Ada and lower compilation by guarding PDL calls correctly\n\n* Cleanup: Removes commented out GGML_CUDA_PDL_LC\n\n* Cleanup: Removes experimental comments\n\n* Adds 90-virtual to build script so that Hopper GPUs can leverage PDL.\n\n* Adds stricter checks to enable PDL, adds env-check to disable it, and removes now superfluous compile option to enable PDL.\n\n* Fix: Correct PDL en/disablement based on device-side arch check. Host\nside check is UB. Required moving from macros to inlined functions\n\n* Fix: default-disable PDL. Enable by setting GGML_CUDA_ENABLE_PDL=1\n\n* Enable PDL by default for Hopper+ devices\n\n* Enrolls softcap_f32 and two flash_attn kernels into PDL.\n\n* Improves flash attn PDL barrier placement\n\n* Fix: Perf regression on ada; excludes ada and below from PDL launches\n\n* Improves some sync barrier placements\n\n* Drops superfluous constructor\n\n* Adds #endif guard comments\n\n* Reverts experimental change to top-k-moe.cu, which moved expensive allocations\nin front of the PDL barrier. It did not have a meaningful impact.\n\n* Exchanges GGML_CUDA_DISABLE_PDL with GGML_CUDA_PDL. IFF GGML_CUDA_PDL=0\nPDL is disabled\n\n* Revert \"Drops superfluous constructor\". Adds const to remaining\narguments\n\nThis reverts commit 12b1d250da0089ae02a9bb71bbb3fd6d70f6f2f1.\n\n* Cleanup: Removes and fixes some comments and whitespace\n\n* Clarifies comment of sync-barrier position\n\n* Relocates and refactors PDL launch functions and accessories\n\n* Adds error checking to the regular kernel launch path\n\n* Drops \"auto\" in favor of \"ggml_cuda_kernel_params\"\n\n* Adds \"const\" to ggml_cuda_kernel_launch_params\n\n* [Whitespace] Adds final newline to common.cuh to make editorconfig CI job happy","tree":"6bd686700efc2858d3b391fe3e51b1f8436b2e40"}},{"name":"sumrows.cuh","mode":"0100644","size":212,"lastCommit":{"hash":"aa064b2eb7f7779db5e094a9d8d66d5033557de2","oid":"aa064b2eb7f7779db5e094a9d8d66d5033557de2","parents":["aa0ef5c578eef4c2adc7be1282f21bab5f3e8d26"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2025-06-22T12:39:54+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-06-22T12:39:54+08:00"},"message":"CUDA: add mean operation (#14313)\n\n* CUDA: add mean operation\n\n* add back sum_rows_f32_cuda\n\n* Review: early exit if col!=0","tree":"9453e3e8eca9384c5973f3f6b138c2b171d0fa0a"}},{"name":"template-instances","mode":"0040000","size":0,"lastCommit":{"hash":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","oid":"6d595f4116cac1cb1daa915ea665678e4c6d7bca","parents":["113d61c842e3889406777388f7bb929f288e12b0"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:56-04:00"},"message":"bonsai: PTQ1_0 MMQ main loop staged per 128-value block (mmq-staged)\n\nPTQ1_0 MMQ main loop staged one 128-value block at a time: one barrier per block, x and y copied with cp.async, trits decoded between the MMAs, B fragments via ldmatrix. Bit-identical; +10% prefill. GGML_CUDA_NO_PTQ1_0_STAGED=1 restores the old loop.\n","tree":"e7f41c3007366bbf2726c51866a49c069cdf41d3"}},{"name":"top-k.cu","mode":"0100644","size":4533,"lastCommit":{"hash":"074944998d3f25e7001ede30d152b59dff741c8c","oid":"074944998d3f25e7001ede30d152b59dff741c8c","parents":["3de7dd4c8f5d9806279249310b6c3db24a1a67ab"],"author":{"name":"fairydreaming","email":"166155368+fairydreaming@users.noreply.github.com","when":"2026-07-09T20:07:12+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-09T20:07:12+02:00"},"message":"ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (#24776)\n\n* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage\n\n* ggml : allocate tmp_dst only only once before the loop\n\n* chore : whitespaces\n\nCo-authored-by: Georgi Gerganov \n\n* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations\n\n* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement\n\nCo-authored-by: Johannes Gäßler \n\n* chore : replace ternary operators with min/max\n\n---------\n\nCo-authored-by: Stanisław Szymczyk \nCo-authored-by: Georgi Gerganov \nCo-authored-by: Johannes Gäßler ","tree":"3099742364a1901d11ed17e56650db687d59b2a7"}},{"name":"top-k.cuh","mode":"0100644","size":100,"lastCommit":{"hash":"d3dce4e0a5f2fd497a312a3152f2eab939f0d519","oid":"d3dce4e0a5f2fd497a312a3152f2eab939f0d519","parents":["4974bf53cf14073c7b66e1151348156aabd42cb8"],"author":{"name":"Daniel Bevenius","email":"daniel.bevenius@gmail.com","when":"2026-01-04T21:22:16+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-01-04T22:22:16+02:00"},"message":"sampling : add support for backend sampling (#17004)\n\n* sampling : add support for backend sampling\n\nThis commit adds support for performing sampling operations on the\nbackend (e.g. GPU) as part of the model computation graph.\n\nThe motivation for this feature is to enable sampling to be performed\ndirectly on the backend as part of the computation graph being executed,\nallowing for some or all of the sampling to be done on the backend.\n\nFor example, the backend sampler chain might select/sample a token\ndirectly in which case only the sampled token needs to be transferred\nfrom device memory to host memory.\n\nIt is also possible for the backend samplers to perform filtering of\nthe logits, or compute and filter the probability distribution, in\nwhich case only the filtered logits or probabilites need to be\ntransferred back to system memory for further processing by CPU\nsamplers.\n\nCurrently the backend sampling works in a similar manner to how\npooling works, it is a function that is called by build_graph and the\nsampler operations become part of the models computation graph.\n\n* llama-cli : add backend sampler configuration\n\n* server : add backend sampling options/configuration\n\n* webui : add backend sampling options\n\n* ggml : add initial cumsum implementation for CUDA\n\n* sampling : enable all backend sampler tests\n\nThis commit enables all exisiting backend sampler tests in the\ntest-backend-sampler. Previously, some tests were disabled because\nthere were missing ggml operation implementations.\n\n* graph : do not include llama-model.h\n\n* sampling : always expose sampled_ids\n\nThis commit precomputes and caches the full-vocab token id list in\nllama_context's constructor, so llama_get_backend_sampled_token_ids_ith\nalways returns a valid pointer.\n\nThe motivation for this is that this enables both common/sampling.cpp\nand src/llama-sampling.cpp can simplify their logic.\n\nNot all backends samplers that process logits need to set the\nsampled_tokens_id as they may not change the order of the logits, for\nexample the temperature sampler only scales the logits but does not\nchange their order. Simliar the logit bias sampler only adds bias to\nspecific token ids but does not change the order of the logits. In\nthese cases there will not be a device to host copy of the sampled\ntoken ids, and this is the use case where having this precomputed\nlist is useful.\n\n* sampling : ensure at most one output token per seq\n\nThis commit adds a check in the batch allocator to ensure that when\nbackend sampling is enabled, at most one output token is specified per\nsequence.\n\n* CUDA: Optimize argsort for gpu-based token sampling\n\nArgsort is used for top-k currently. WE optimize argsort by 2 things:\n\n1. Use `DeviceRadixSort` for single-row/sequence to parallelize it\n across our SMs\n2. Use `DeviceSegmentedSort` for multi-row/sequence as this is the\n correct entrypoint (the function chooses different execution paths,\n it contains `DeviceSegmentedRadixSort` as one of the paths and will\n choose the best one according to heuristics.\n https://nvidia.github.io/cccl/cub/api/structcub_1_1DeviceSegmentedSort.html#overview\n\nSome perf numbers for a RTX PRO 6000:\n\nOn the kernel level, tested with\n`GGML_CUDA_DISABLE_GRAPHS=1 ./test-backend-ops -o ARGSORT perf`\nBefore:\n```\n ARGSORT(type=f32,ne=[65000,16,1,1],order=0): 4130 runs - 359.24 us/run\n ARGSORT(type=f32,ne=[200000,1,1,1],order=0): 8192 runs - 861.34 us/run\n ARGSORT(type=f32,ne=[200000,16,1,1],order=0): 1343 runs - 1020.01 us/run\n```\n\nAfter:\n```\n ARGSORT(type=f32,ne=[65000,16,1,1],order=0): 4130 runs - 312.41 us/run\n ARGSORT(type=f32,ne=[200000,1,1,1],order=0): 16384 runs - 63.48 us/run\n ARGSORT(type=f32,ne=[200000,16,1,1],order=0): 1343 runs - 874.36 us/run\n```\n\n---\nOn the model level, tested with\n`llama-cli -m gpt-oss-20b-mxfp4.gguf -n 200 -p \"What is\nthe Capital of Sweden?\" -no-cnv -fa 1 --backend-sampling`\n\nBefore:\n```\nllama_perf_sampler_print: sampling time = 0.25 ms / 207 runs ( 0.00 ms per token, 824701.20 tokens per second)\nllama_perf_context_print: load time = 18215.58 ms\nllama_perf_context_print: prompt eval time = 28.20 ms / 7 tokens ( 4.03 ms per token, 248.19 tokens per second)\nllama_perf_context_print: eval time = 714.79 ms / 199 runs ( 3.59 ms per token, 278.40 tokens per second)\nllama_perf_context_print: total time = 857.62 ms / 206 tokens\n```\n\nAfter\n```\nllama_perf_sampler_print: sampling time = 0.25 ms / 207 runs ( 0.00 ms per token, 828000.00 tokens per second)\nllama_perf_context_print: load time = 18366.92 ms\nllama_perf_context_print: prompt eval time = 35.92 ms / 7 tokens ( 5.13 ms per token, 194.87 tokens per second)\nllama_perf_context_print: eval time = 532.79 ms / 199 runs ( 2.68 ms per token, 373.50 tokens per second)\nllama_perf_context_print: total time = 683.65 ms / 206 tokens\n```\n\n* sampling : remove version from sampler chain\n\nThis commit removes the version field from the sampler chain and instead\nused the sampler pointer itself for change detection.\n\n* sampling : always populate logits for sampled probs\n\nThis commit updates common/sampler.cpp set_logits and\nsrc/llama-sampling.cpp llama_sampler_sample to always populate the\nlogits field when backend sampled probabilities are available.\n\nThe motivation for this is that this ensure that CPU sampler always have\naccess to the logits values even when probabilites have been produced by\nbackend samplers.\n\n* sampling : simplify backend sampling logic decode\n\nThis commit tries to simplify the backend sampling logic in\nllama_context::decode.\n\n* squash! sampling : simplify backend sampling logic decode\n\nFix condition to check if backend actually sampled tokens, not just that\nbackend samplers are available.\n\n* common : fix regression caused by extra memory allocations during sampling\n\n* squash! sampling : simplify backend sampling logic decode\n\nThe commit fixes a variable shadowing issue in the\n`llama_context::decode` function which was introduced in a previous\nrefactoring.\n\n* squash! common : fix regression caused by extra memory allocations during sampling\n\nApply the same changes to llama-sampling.cpp, llama_sampler_sample as\nwere applied in commit 38f408c25.\n\n* sampling : introduce sampling_info struct\n\nThis commit introduces a sampling_info struct to encapsulate all\nbackend sampling related data within the llama_context class.\n\nIt also updates to use more descriptive names for sampled tokens and\ncandidates in the backend sampler ggml data structure.\n\n* sampling : return early if backend sampling is disabled\n\n* sampling : use pinned memory for backend sampling buffers\n\n* common, tools : refactor model loading to support backend samplers\n\nThis commit refactors the model loading process in common/common.cpp\nto enable backend sampler to be configure prior to the llama_context\ncreation.\n\nThe motivation for this change is that just being able to set/reset the\nbackend samplers after the llama_context has been created will cause a\nresize to occur in llama_context::output_reserve which we want to avoid.\n\n* sampling : add stride variable for clarity\n\n* sampling: clarify candidate ids usage in comments\n\n* sampling : fix copying both sampled tokens and logits/probs from backend\n\nThis commit fixes the issue where both sampled tokens and logits/probs\nwere not being copied correctly from the backend to the host when\nmultiple backend samplers were used.\n\nA test for this scenario has also been added to ensure that both types\nof data are copied correctly when different backend samplers are\nemployed.\n\n* tests : cleanup test-backend-sampler.cpp\n\n* common : remove build-info.cpp from commit [no ci]\n\nThis file was generated during the build process and should not be\nincluded in previous commits.\n\n* sampling : cleanup and clarify output_reserve\n\n* sampling : remove redundant checks for stride and size [no ci]\n\n* sampling : add debug log when backend sampler selects token\n\nThis commit adds a debug log statement in the llama_sampler_sample\nto indicate when a backend sampler has selected a token for a given\nindex.\n\nThe modification helps in tracing the sampling process and understanding\nthe flow of control when backend samplers are used.\n\n* examples : update batched to use backend sampling\n\nThis commit updates the batched example to demonstrate how to use\nbackend samplers.\n\n* llama-cli : fix dangling reference to sampler config\n\n* common : initialize backend samplers\n\n* samplers : add missing cont\n\n* sampling : add assertions for contiguous tensors in async copy functions\n\n* examples : add info about hybrid sampling in batched [no ci]\n\n* sampling : remove backend-dist option (wip)\n\nThis commit removes the `--backend-dist` option and instead uses the\nconfigured --samplers chain to determine which samplers run on the\nbackend.\n\nBackend sampling is still enabled using With `--backend_sampling`, and\nthe sampler chain, either explictly specified using `--samplers` or the\ndefault, is automatically analyzed to determine which samplers can run\non the backend. The system finds the longest contiguous chain of\nbackend supported samplers from the start of the sampler sequence.\nFor example:\n\n* If the chain is `top-k -> temperature -> top-p`, and both `top-k` and\n `temperature` are backend-supported but `top-p` is not, then `top-k`\n and `temperature` will run on the backend, while `top-p` and\n subsequent samplers run on the CPU.\n\n* If all configured samplers are supported, the final distribution\n sampling will also happen on the backend, transferring only the\n sampled token IDs back to the host.\n\n* If the sampler chain starts with an unsupported sampler (e.g.,\n `penalties`), all sampling runs on the CPU. Note that this is\n currently the case with the default sampler so to use backend sampling\n it is required to specify a sampler chain. See below for an example.\n\nThe following shows how llama-cli can be run with backend sampling:\n```console\n$ llama-cli -m models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf \\\n --prompt 'What is the capital of Sweden?' \\\n -n 20 \\\n -no-cnv \\\n --verbose-prompt \\\n -ngl 40 \\\n --backend-sampling \\\n --samplers 'top_k;temperature'\n```\nIn this case the all sampling will happen on the backend since both\n`top_k` and `temperature` are supported backend samplers.\n\nTo enable a partial backend sampling (hybrid sampling), for example\nrunning `top_k` and `temperature` on the backend and `typ_p` on the CPU\nthe following sampler chain could be specified:\n```console\n$ llama-cli -m models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf \\\n --prompt 'What is the capital of Sweden?' \\\n -n 20 \\\n -no-cnv \\\n --verbose-prompt \\\n -ngl 40 \\\n --backend-sampling \\\n --samplers 'top_k;temperature;top_p'\n```\n\nIf this looks good then I'll follow up with updates the llama-cli and\nllama-server documentation to reflect these changes.\n\n* CUDA: Add top-k implementation\n\n* sampling : add min-p backend sampler\n\n* Use `FetchContent` over CPM as it's bundled with CMake\n\nThanks @ggerganov for the suggestion\n\n* common : add get_active_samplers function to check enabled samplers\n\nThis commit adds a function to check if a sampler is actually enabled,\nmeaning that it does not have values that disables its effect. This is\nthen used by the backend samplers initialization to avoid considering\nsamplers that are not enabled when determining the split point between\nthem.\n\nThe motivation for this is that this allows the default sampler chain\nfor `--samplers` to be used and any sampler that is not enabled will not\ncause the backend samplers to be skipped.\nFor example, before this change if the penalties sampler was included in\nthe samplers list but had default values that disable it, it would cause\nthe backend samplers to be skipped entirely.\n\nThis commit also contains some refactoring to remove some code\nduplication.\n\n* cuda : fix editorconfig-checker warning\n\n* sampling : use argmax for min-p sampling\n\n* sampling : fix temperature check to allow zero temperature\n\nThis commit modifies the temperature sampling check to allow a\ntemperature value of zero. Previously, the check only allowed\npositive temperature values, which excluded the valid case of\nzero temperature.\n\nThe motivation for this is to enable a zero temperature setting which is\nalso currently causing the following test to fail:\n```console\n(venv) $ cd tools/server/tests\n(venv) $ ./tests.sh unit/test_basic.py::test_load_split_model\n```\n\n* cuda : fix top-k compilation when CUB is unavailable\n\nThis commit adds a macro guard around argsort_f32_i32_cuda_cub usage\nin the top-k fallback path, falling back to bitonic sort when\nGGML_CUDA_USE_CUB is not defined.\n\nThe motivation for this is that some environments like AMD HIP\ndo not have CUB available, causing compilation failure.\n\nRefs: https://github.com/ggml-org/llama.cpp/actions/runs/19728226426/job/56523606840#step:6:208\n\n* sampling : add comments about backend sampler [no ci]\n\nThis commit adds a comment to llama_context's constructor explaining why\nbackend samplers are initialized early in the process.\n\n* sampling : remove backend sampling chain from common_sampler\n\nThis commit removes the backend sampling chain from the common_sampler\nstructure and related functions.\n\nThe motivation for this change is that the backend samplers are not\ncurrently set on the context, and if they are they would cause the\na graph reallocation to occur. Instead, the intialization is handled\nlike it currently is by llama_context's constructor.\n\n* Fix top-k comp & behavior for non-CUB path\n\nSome changes were made in 5ea3be265ba6f8916daf52e19e3fb8efe9a03637\nwhich were incomplete. In the case of non-CUB, bitonic sort and its\nlimitations of ncols < 1024 have to apply, similar to argsort.cu\n\n* sampling : support intermixed backend/cpu samplers\n\nThis commit updates the backend sampling implementation to support\nintermixed usage of backend and CPU samplers within the same batch.\n\nThe initial implementation was developed as an all-or-nothing solution:\neither perform backend sampling for the entire batch, or perform CPU\nsampling for the entire batch.\n\nThe motivation for this change is to support batches with mixed\nsequences. For example, we may have a backend sampler configured for\nsequence 0, while sequence 1 in the same batch uses CPU sampling. This\nwas not supported in the initial implementation.\n\nThis issue manifested in llama-server with the webui: decoding with\nbackend samplers would work initially, but after changing to CPU\nsampling, a slot (sequence) could still be using a backend sampler.\nThis meant that logits in output_reserve would not be allocated,\nresulting in an error.\n\nThe solution in this commit inspects the batch to determine which\nsampling modes are needed and allocates buffers accordingly. However,\nthere is a known inefficiency: when we have intermixed backend/CPU\nsamplers in the same batch, we currently copy all logits to the host,\neven for sequences using backend samplers.\n\nAdded test_backend_cpu_mixed_batch to verify correct behavior with\nmixed backend/CPU samplers in a single batch, including dynamic\nsampler switching between decode calls.\n\n* squash! sampling : support intermixed backend/cpu samplers\n\nAdd check that logits is not null which is can happen for embeddings.\n\n* squash! sampling : support intermixed backend/cpu samplers\n\nFix llama-save-load-state which currently fails by handling the case\nwhen batch.logits is nullptr (like when loading state) by allocating\nspace for all outputs as CPU logits.\n\n* refactor : simplify and improve memory management\n\n* Add initial version for top-p sampling\n\nAs we only support static graphs for the time and we don't know the size\nof the output of top-p, we have to do value-scaling same as for min-p\noperator.\n\nFurther improvements can be applied to the unit-test (i.e. check for\nequivalence of top_p happening on backend with top_p happening on cpu)\nand also by constructing candidates and sorting those as opposed to\nreversing the sort of the logits (this would be arange +\nget_rows instead of argsort + get_rows)\n\n* sampling : use logits directly for min-p filtering\n\n* sampling : simplify\n\n* llama : simplify\n\n* llama : cleanup + naming\n\n* llama : call backend_init once\n\n* llama : reserve graphs with samplers\n\n* llama : naming\n\n* cont : naming\n\n* sampling : lower log level for output buffer reallocations [no ci]\n\nThis commit changes the logging level for output buffer reallocations\nin the llama_context::output_reserve function from INFO to DEBUG.\n\nThe motivation for this is that it currently logs to info and when\nenabling verbose logging for llama-cli this will get mixed with the\noutput, for example:\n\n```console\nWhat is the capital of Sweden?output_reserve: reallocating output buffer from size 0.58 MiB to 1.74 MiB\n 1. Stockholm\n2\\. Helsinki\nBased are the options\n1. Stockholm\nExplanation: Stockholm is the capital of\n...\n```\n\n* Fix backend_top_p_sampler\n\nsoftmax(softmax) will return uniform distribution, so we should not\nreturn the softmax but the logits instead.\n\n* Factor out `ggml_sort` into its own function\n\n* Make backend's top_p sampler inclusive\n\nIn addition to match the algorithm proposed in the original\n[paper](https://arxiv.org/abs/1904.09751), this resolves the edge-case\nwhere `max_p is > top_p` for a single logit, where the mask would\notherwise be empty (and we thus sample from the whole vocabulary with\nequal likelihood)\n\n* common : simplify sampler chain initialization\n\n* sampling : do not create empty samplers\n\n* sampling : fix top_p empty condition\n\n* examples : remove outdated backend sampling section\n\nThis commit removes the outdated section about using backend samplers\nfrom the README.md file in the examples/batched.\n\n* sampling : fix backend temp sampler for zero temperature\n\nThis commit fixes the implementation of the temperature-based sampler\nfor the case when the temperature is set to zero. This now correctly\nselects the most probable token by masking out all other tokens in the\nlogits.\n\n* CUDA: Move cccl fetch to after cuda has been enabled in CMakeLists.txt\n\nThis will allow cccl to set build flags for the CUDA compiler, required\ne.g. for MSVC compat, see also\nhttps://github.com/NVIDIA/cccl/pull/6791\n\n* CUDA: Use standard-compliant preprocessor for MSVC builds\n\nWorkarounds of https://github.com/NVIDIA/cccl/pull/6791 will not be\nbackported to CCCL 3.2, only the diagnostics/error messages will:\nhttps://github.com/NVIDIA/cccl/pull/6827\n\n* CUDA: Update CCCL's rc candidate\n\n* squash! sampling : fix backend temp sampler for zero temperature\n\nThis modifies the parent commit to simply return the most probably token\ninstead of masking the logits.\n\n* sampling : implement temp_ext_backend sampling\n\nThis commit implements the apply function for the extended temperature\nsampling.\n\n* sampling : minor cleanup\n\n* sampling : stop short if backend sampler sampled a token\n\nThis commit modifies the graph building logic to immediately continue\nwhen a token has already been sampled by the backend sampler.\n\nIt also updates the test for backend temporary sampling to include\ntop-k and distribution samplers in the chain to verify that they are not\nproducing any logits (they are not run).\n\n* Revert \"sampling : stop short if backend sampler sampled a token\"\n\nThis reverts commit 87b2719eca55b30afff600fc7f61c6cce9452cbf.\n\n* sampling : fix backend temp sampling to use logits masking\n\n* sampling : simplify temp sampling\n\n* sampling : remove redundant calls to ggml_build_forward_expand\n\n* sampling : check backend support during init\n\n* cont : keep backend sampling disabled for now\n\n* sampling : fix outputs and device checks\n\n* sampling : fix candidates logic\n\n* Add perf-tests for CUMSUM\n\n* Readd `cub::DeviceScan::InclusiveSum`-based CumSum\n\nFor single rows and large columns doing a for-loop over the function\n`cub::DeviceScan::InclusiveSum` offered by CUB outperforms the\n`cumsum_cub_kernel` where `cub::BlockScan` is used.\n\nNumbers before this change\n\n Backend 1/3: CUDA0\n Device description: NVIDIA RTX 6000 Ada Generation\n Device memory: 48510 MB (48039 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 311258 runs - 3.26 us/run - 2048 kB/run - 599.76 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 229390 runs - 4.40 us/run - 5120 kB/run - 1110.23 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 37583 runs - 29.63 us/run - 6250 kB/run - 201.18 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 892819 runs - 1.12 us/run - 1 kB/run - 0.85 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 450505 runs - 2.25 us/run - 8 kB/run - 3.39 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 155629 runs - 6.61 us/run - 32 kB/run - 4.62 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 81910 runs - 12.60 us/run - 64 kB/run - 4.85 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 49146 runs - 23.99 us/run - 128 kB/run - 5.09 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 24573 runs - 47.10 us/run - 256 kB/run - 5.18 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 16382 runs - 93.57 us/run - 512 kB/run - 5.22 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 8191 runs - 184.79 us/run - 1024 kB/run - 5.29 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 8191 runs - 280.43 us/run - 1562 kB/run - 5.31 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 2148 runs - 2771.23 us/run - 15625 kB/run - 5.38 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 458696 runs - 2.21 us/run - 4 kB/run - 1.73 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 360404 runs - 2.82 us/run - 32 kB/run - 10.83 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 147438 runs - 7.12 us/run - 128 kB/run - 17.15 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 81910 runs - 12.90 us/run - 256 kB/run - 18.92 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 49146 runs - 24.32 us/run - 512 kB/run - 20.08 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 24573 runs - 47.28 us/run - 1024 kB/run - 20.66 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 16382 runs - 93.21 us/run - 2048 kB/run - 20.96 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 8191 runs - 185.04 us/run - 4096 kB/run - 21.11 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 5369 runs - 282.08 us/run - 6250 kB/run - 21.13 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 537 runs - 2806.46 us/run - 62500 kB/run - 21.26 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 458696 runs - 2.20 us/run - 8 kB/run - 3.47 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 360404 runs - 2.82 us/run - 64 kB/run - 21.66 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 147438 runs - 7.12 us/run - 256 kB/run - 34.28 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 81910 runs - 12.90 us/run - 512 kB/run - 37.84 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 49146 runs - 24.32 us/run - 1024 kB/run - 40.15 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 47.28 us/run - 2048 kB/run - 41.31 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 93.20 us/run - 4096 kB/run - 41.92 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 8194 runs - 185.05 us/run - 8192 kB/run - 42.22 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 5370 runs - 282.15 us/run - 12500 kB/run - 42.26 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 269 runs - 4067.61 us/run - 125000 kB/run - 29.36 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 303067 runs - 3.32 us/run - 16 kB/run - 4.60 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 303067 runs - 3.32 us/run - 128 kB/run - 36.76 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 147438 runs - 7.17 us/run - 512 kB/run - 68.13 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 81910 runs - 12.90 us/run - 1024 kB/run - 75.68 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 49146 runs - 24.33 us/run - 2048 kB/run - 80.28 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 47.30 us/run - 4096 kB/run - 82.59 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 93.24 us/run - 8192 kB/run - 83.80 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 6147 runs - 185.07 us/run - 16384 kB/run - 84.45 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 4029 runs - 282.40 us/run - 25000 kB/run - 84.46 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 270 runs - 4118.40 us/run - 250000 kB/run - 58.11 GB/s\n Backend CUDA0: OK\nBackend 2/3: CUDA1\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96677 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 368595 runs - 2.73 us/run - 2048 kB/run - 715.83 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 216282 runs - 4.72 us/run - 5120 kB/run - 1035.32 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 32214 runs - 34.33 us/run - 6250 kB/run - 173.64 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 810909 runs - 1.24 us/run - 1 kB/run - 0.77 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 401359 runs - 2.52 us/run - 8 kB/run - 3.03 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 139247 runs - 7.44 us/run - 32 kB/run - 4.10 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 73719 runs - 14.27 us/run - 64 kB/run - 4.28 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 40955 runs - 27.24 us/run - 128 kB/run - 4.48 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 24573 runs - 53.46 us/run - 256 kB/run - 4.57 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 16382 runs - 105.29 us/run - 512 kB/run - 4.64 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 8191 runs - 210.15 us/run - 1024 kB/run - 4.65 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 8191 runs - 318.22 us/run - 1562 kB/run - 4.68 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 2148 runs - 3142.23 us/run - 15625 kB/run - 4.74 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 303067 runs - 3.34 us/run - 4 kB/run - 1.14 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 253921 runs - 4.03 us/run - 32 kB/run - 7.58 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 73719 runs - 14.96 us/run - 256 kB/run - 16.32 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 40955 runs - 28.66 us/run - 512 kB/run - 17.04 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 24573 runs - 54.21 us/run - 1024 kB/run - 18.01 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 16382 runs - 106.49 us/run - 2048 kB/run - 18.34 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 8191 runs - 210.88 us/run - 4096 kB/run - 18.52 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 5369 runs - 321.77 us/run - 6250 kB/run - 18.53 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 537 runs - 3191.79 us/run - 62500 kB/run - 18.69 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 376786 runs - 2.67 us/run - 8 kB/run - 2.86 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 245730 runs - 4.10 us/run - 64 kB/run - 14.90 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 122865 runs - 8.20 us/run - 256 kB/run - 29.79 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 65528 runs - 16.38 us/run - 512 kB/run - 29.82 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 40955 runs - 28.69 us/run - 1024 kB/run - 34.04 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 55.28 us/run - 2048 kB/run - 35.33 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 108.50 us/run - 4096 kB/run - 36.00 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 8194 runs - 213.75 us/run - 8192 kB/run - 36.55 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 5370 runs - 326.31 us/run - 12500 kB/run - 36.54 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 538 runs - 3252.68 us/run - 125000 kB/run - 36.72 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 303067 runs - 3.32 us/run - 16 kB/run - 4.60 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 253921 runs - 4.06 us/run - 128 kB/run - 30.09 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 122865 runs - 8.20 us/run - 512 kB/run - 59.57 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 65528 runs - 16.38 us/run - 1024 kB/run - 59.63 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 28.69 us/run - 2048 kB/run - 68.09 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 55.28 us/run - 4096 kB/run - 70.67 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 108.50 us/run - 8192 kB/run - 72.02 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 6147 runs - 213.60 us/run - 16384 kB/run - 73.17 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 4029 runs - 326.04 us/run - 25000 kB/run - 73.15 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 270 runs - 5458.69 us/run - 250000 kB/run - 43.84 GB/s\n\n----\nNumbers after:\n\nBackend 1/3: CUDA0\n Device description: NVIDIA RTX 6000 Ada Generation\n Device memory: 48510 MB (48039 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 311258 runs - 3.25 us/run - 2048 kB/run - 601.62 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 229390 runs - 4.40 us/run - 5120 kB/run - 1110.14 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 37583 runs - 29.67 us/run - 6250 kB/run - 200.89 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 892819 runs - 1.12 us/run - 1 kB/run - 0.85 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 458696 runs - 2.21 us/run - 8 kB/run - 3.45 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 376786 runs - 2.66 us/run - 32 kB/run - 11.46 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 393168 runs - 2.59 us/run - 64 kB/run - 23.57 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 393168 runs - 2.59 us/run - 128 kB/run - 47.15 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 376786 runs - 2.69 us/run - 256 kB/run - 90.69 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 327640 runs - 3.06 us/run - 512 kB/run - 159.65 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 311258 runs - 3.28 us/run - 1024 kB/run - 297.77 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 270303 runs - 3.74 us/run - 1562 kB/run - 398.14 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 137472 runs - 7.35 us/run - 15625 kB/run - 2026.94 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 876437 runs - 1.14 us/run - 4 kB/run - 3.33 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 442314 runs - 2.28 us/run - 32 kB/run - 13.39 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 155629 runs - 6.69 us/run - 128 kB/run - 18.24 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 81910 runs - 12.53 us/run - 256 kB/run - 19.49 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 49146 runs - 24.18 us/run - 512 kB/run - 20.20 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 65528 runs - 15.34 us/run - 1024 kB/run - 63.66 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 73719 runs - 14.76 us/run - 2048 kB/run - 132.35 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 65528 runs - 16.01 us/run - 4096 kB/run - 244.07 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 64428 runs - 16.51 us/run - 6250 kB/run - 360.97 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 33831 runs - 29.59 us/run - 62500 kB/run - 2016.08 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 868246 runs - 1.16 us/run - 8 kB/run - 6.59 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 442314 runs - 2.28 us/run - 64 kB/run - 26.76 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 155629 runs - 6.69 us/run - 256 kB/run - 36.48 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 81910 runs - 12.53 us/run - 512 kB/run - 38.97 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 49146 runs - 24.17 us/run - 1024 kB/run - 40.41 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 47.53 us/run - 2048 kB/run - 41.10 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 61.25 us/run - 4096 kB/run - 63.77 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 32776 runs - 31.79 us/run - 8192 kB/run - 245.82 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 32220 runs - 32.90 us/run - 12500 kB/run - 362.35 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 6725 runs - 151.99 us/run - 125000 kB/run - 785.77 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 851864 runs - 1.18 us/run - 16 kB/run - 12.97 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 442314 runs - 2.30 us/run - 128 kB/run - 53.13 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 155629 runs - 6.68 us/run - 512 kB/run - 73.13 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 81910 runs - 12.68 us/run - 1024 kB/run - 77.00 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 24.56 us/run - 2048 kB/run - 79.53 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 47.52 us/run - 4096 kB/run - 82.21 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 93.44 us/run - 8192 kB/run - 83.62 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 16392 runs - 63.36 us/run - 16384 kB/run - 246.68 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 16116 runs - 65.25 us/run - 25000 kB/run - 365.53 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 3375 runs - 304.46 us/run - 250000 kB/run - 785.98 GB/s\n Backend CUDA0: OK\nBackend 2/3: CUDA1\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96677 MB free)\n\n CUMSUM(type=f32,ne=[128,128,4,4]): 376786 runs - 2.69 us/run - 2048 kB/run - 727.04 GB/s\n CUMSUM(type=f32,ne=[2048,16,5,4]): 216282 runs - 4.64 us/run - 5120 kB/run - 1053.30 GB/s\n CUMSUM(type=f32,ne=[20000,10,4,1]): 32214 runs - 34.21 us/run - 6250 kB/run - 174.27 GB/s\n CUMSUM(type=f32,ne=[128,1,1,1]): 819100 runs - 1.22 us/run - 1 kB/run - 0.78 GB/s\n CUMSUM(type=f32,ne=[1024,1,1,1]): 409550 runs - 2.47 us/run - 8 kB/run - 3.09 GB/s\n CUMSUM(type=f32,ne=[4096,1,1,1]): 303067 runs - 3.31 us/run - 32 kB/run - 9.21 GB/s\n CUMSUM(type=f32,ne=[8192,1,1,1]): 237539 runs - 4.33 us/run - 64 kB/run - 14.08 GB/s\n CUMSUM(type=f32,ne=[16384,1,1,1]): 237539 runs - 4.33 us/run - 128 kB/run - 28.17 GB/s\n CUMSUM(type=f32,ne=[32768,1,1,1]): 188393 runs - 5.37 us/run - 256 kB/run - 45.47 GB/s\n CUMSUM(type=f32,ne=[65536,1,1,1]): 188393 runs - 5.41 us/run - 512 kB/run - 90.20 GB/s\n CUMSUM(type=f32,ne=[131072,1,1,1]): 188393 runs - 5.41 us/run - 1024 kB/run - 180.41 GB/s\n CUMSUM(type=f32,ne=[200000,1,1,1]): 188393 runs - 5.41 us/run - 1562 kB/run - 275.27 GB/s\n CUMSUM(type=f32,ne=[2000000,1,1,1]): 128880 runs - 7.76 us/run - 15625 kB/run - 1920.33 GB/s\n CUMSUM(type=f32,ne=[128,4,1,1]): 802718 runs - 1.26 us/run - 4 kB/run - 3.03 GB/s\n CUMSUM(type=f32,ne=[1024,4,1,1]): 401359 runs - 2.51 us/run - 32 kB/run - 12.18 GB/s\n CUMSUM(type=f32,ne=[4096,4,1,1]): 139247 runs - 7.51 us/run - 128 kB/run - 16.26 GB/s\n CUMSUM(type=f32,ne=[8192,4,1,1]): 73719 runs - 14.17 us/run - 256 kB/run - 17.23 GB/s\n CUMSUM(type=f32,ne=[16384,4,1,1]): 40955 runs - 27.37 us/run - 512 kB/run - 17.84 GB/s\n CUMSUM(type=f32,ne=[32768,4,1,1]): 40955 runs - 26.33 us/run - 1024 kB/run - 37.10 GB/s\n CUMSUM(type=f32,ne=[65536,4,1,1]): 40955 runs - 26.19 us/run - 2048 kB/run - 74.59 GB/s\n CUMSUM(type=f32,ne=[131072,4,1,1]): 40955 runs - 26.35 us/run - 4096 kB/run - 148.26 GB/s\n CUMSUM(type=f32,ne=[200000,4,1,1]): 42952 runs - 24.18 us/run - 6250 kB/run - 246.51 GB/s\n CUMSUM(type=f32,ne=[2000000,4,1,1]): 32757 runs - 31.01 us/run - 62500 kB/run - 1923.68 GB/s\n CUMSUM(type=f32,ne=[128,8,1,1]): 786336 runs - 1.28 us/run - 8 kB/run - 5.95 GB/s\n CUMSUM(type=f32,ne=[1024,8,1,1]): 393168 runs - 2.57 us/run - 64 kB/run - 23.73 GB/s\n CUMSUM(type=f32,ne=[4096,8,1,1]): 131056 runs - 7.67 us/run - 256 kB/run - 31.82 GB/s\n CUMSUM(type=f32,ne=[8192,8,1,1]): 73719 runs - 14.43 us/run - 512 kB/run - 33.84 GB/s\n CUMSUM(type=f32,ne=[16384,8,1,1]): 40955 runs - 27.90 us/run - 1024 kB/run - 35.01 GB/s\n CUMSUM(type=f32,ne=[32768,8,1,1]): 24573 runs - 54.63 us/run - 2048 kB/run - 35.75 GB/s\n CUMSUM(type=f32,ne=[65536,8,1,1]): 16382 runs - 72.24 us/run - 4096 kB/run - 54.08 GB/s\n CUMSUM(type=f32,ne=[131072,8,1,1]): 20485 runs - 52.66 us/run - 8192 kB/run - 148.37 GB/s\n CUMSUM(type=f32,ne=[200000,8,1,1]): 21480 runs - 48.00 us/run - 12500 kB/run - 248.42 GB/s\n CUMSUM(type=f32,ne=[2000000,8,1,1]): 16140 runs - 61.99 us/run - 125000 kB/run - 1926.51 GB/s\n CUMSUM(type=f32,ne=[128,16,1,1]): 786336 runs - 1.28 us/run - 16 kB/run - 11.90 GB/s\n CUMSUM(type=f32,ne=[1024,16,1,1]): 393168 runs - 2.57 us/run - 128 kB/run - 47.57 GB/s\n CUMSUM(type=f32,ne=[4096,16,1,1]): 131056 runs - 7.65 us/run - 512 kB/run - 63.83 GB/s\n CUMSUM(type=f32,ne=[8192,16,1,1]): 73719 runs - 14.42 us/run - 1024 kB/run - 67.74 GB/s\n CUMSUM(type=f32,ne=[16384,16,1,1]): 40955 runs - 27.87 us/run - 2048 kB/run - 70.09 GB/s\n CUMSUM(type=f32,ne=[32768,16,1,1]): 24573 runs - 54.54 us/run - 4096 kB/run - 71.63 GB/s\n CUMSUM(type=f32,ne=[65536,16,1,1]): 12291 runs - 107.53 us/run - 8192 kB/run - 72.66 GB/s\n CUMSUM(type=f32,ne=[131072,16,1,1]): 10245 runs - 105.10 us/run - 16384 kB/run - 148.70 GB/s\n CUMSUM(type=f32,ne=[200000,16,1,1]): 10744 runs - 95.36 us/run - 25000 kB/run - 250.11 GB/s\n CUMSUM(type=f32,ne=[2000000,16,1,1]): 5400 runs - 186.97 us/run - 250000 kB/run - 1279.90 GB/s\n\n* sampling : expand support (wip)\n\n* tests : fix memory leaks\n\n* cont : fixes\n\n* tests : check temp back to 0.0\n\n* sampling : fix top-p\n\n* sampling : handle n_probs case\n\n* server : handle unsupported cases\n\n* metal : print node names for debugging\n\n* ggml : remove redundant src in ggml_cast\n\n* ggml-alloc : fix reuse-parent logic for misaligned sizes\n\n* Revert \"ggml : remove redundant src in ggml_cast\"\n\nThis reverts commit 62d1b0082dbad699fbeea85a096bc334e3c1c0e6.\n\n* CUDA: Add Cooperative-Groups-based parallelization of ncols in softmax\n\nOld implementation parallelizes rows across SMs, which does not fit the\nneeds of backend-sampling (where we have ncols >> nrows and thus want to\nparallelize ncols across SMs)\n\n* Add TODOs to and adjust heuristics of row-wise soft_max in CUDA\n\nHeuristics were selected based on the following numbers:\n\n```\n-- Before\nBackend 1/2: CUDA0\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96691 MB free)\n\n SOFT_MAX(type=f32,ne=[4096,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 2236 runs - 450.34 us/run - 655360 kB/run - 1401.20 GB/s\n SOFT_MAX(type=f32,ne=[12888,256,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 17748 runs - 56.80 us/run - 128880 kB/run - 2168.19 GB/s\n SOFT_MAX(type=f32,ne=[77,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 57204 runs - 18.35 us/run - 12320 kB/run - 640.57 GB/s\n SOFT_MAX(type=f32,ne=[1024,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9840 runs - 102.46 us/run - 81920 kB/run - 763.45 GB/s\n SOFT_MAX(type=f32,ne=[77,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98064 runs - 10.25 us/run - 6160 kB/run - 573.43 GB/s\n SOFT_MAX(type=f32,ne=[256,256,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98310 runs - 10.25 us/run - 10240 kB/run - 953.20 GB/s\n SOFT_MAX(type=f32,ne=[64,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 5.99 us/run - 640 kB/run - 101.84 GB/s\n SOFT_MAX(type=f32,ne=[77,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 5.97 us/run - 770 kB/run - 123.02 GB/s\n SOFT_MAX(type=f32,ne=[8192,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 6.00 us/run - 64 kB/run - 10.16 GB/s\n SOFT_MAX(type=f32,ne=[8192,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 163820 runs - 6.12 us/run - 256 kB/run - 39.91 GB/s\n SOFT_MAX(type=f32,ne=[8192,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.88 us/run - 1024 kB/run - 141.92 GB/s\n SOFT_MAX(type=f32,ne=[16384,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n SOFT_MAX(type=f32,ne=[16384,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 114674 runs - 8.87 us/run - 512 kB/run - 55.06 GB/s\n SOFT_MAX(type=f32,ne=[16384,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 2048 kB/run - 190.82 GB/s\n SOFT_MAX(type=f32,ne=[32768,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 21.37 us/run - 256 kB/run - 11.43 GB/s\n SOFT_MAX(type=f32,ne=[32768,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 22.54 us/run - 1024 kB/run - 43.33 GB/s\n SOFT_MAX(type=f32,ne=[32768,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 23.92 us/run - 4096 kB/run - 163.32 GB/s\n SOFT_MAX(type=f32,ne=[65536,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 32764 runs - 38.94 us/run - 512 kB/run - 12.54 GB/s\n SOFT_MAX(type=f32,ne=[65536,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24573 runs - 41.94 us/run - 2048 kB/run - 46.57 GB/s\n SOFT_MAX(type=f32,ne=[65536,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24582 runs - 43.09 us/run - 8192 kB/run - 181.32 GB/s\n SOFT_MAX(type=f32,ne=[131072,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 16382 runs - 74.56 us/run - 1024 kB/run - 13.10 GB/s\n SOFT_MAX(type=f32,ne=[131072,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 16382 runs - 79.85 us/run - 4096 kB/run - 48.92 GB/s\n SOFT_MAX(type=f32,ne=[131072,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 12294 runs - 82.41 us/run - 16384 kB/run - 189.64 GB/s\n SOFT_MAX(type=f32,ne=[262144,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8191 runs - 145.16 us/run - 2048 kB/run - 13.46 GB/s\n SOFT_MAX(type=f32,ne=[262144,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8194 runs - 155.46 us/run - 8192 kB/run - 50.26 GB/s\n SOFT_MAX(type=f32,ne=[262144,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 7175 runs - 160.70 us/run - 32768 kB/run - 194.56 GB/s\n SOFT_MAX(type=f32,ne=[524288,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8191 runs - 285.81 us/run - 4096 kB/run - 13.67 GB/s\n SOFT_MAX(type=f32,ne=[524288,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 4098 runs - 306.91 us/run - 16384 kB/run - 50.92 GB/s\n SOFT_MAX(type=f32,ne=[524288,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 3591 runs - 317.06 us/run - 65536 kB/run - 197.32 GB/s\n\n-- After\nBackend 1/2: CUDA0\n Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition\n Device memory: 97250 MB (96691 MB free)\n\n SOFT_MAX(type=f32,ne=[4096,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 2236 runs - 450.67 us/run - 655360 kB/run - 1400.15 GB/s\n SOFT_MAX(type=f32,ne=[12888,256,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 17748 runs - 56.97 us/run - 128880 kB/run - 2161.50 GB/s\n SOFT_MAX(type=f32,ne=[77,4096,5,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 57204 runs - 18.35 us/run - 12320 kB/run - 640.36 GB/s\n SOFT_MAX(type=f32,ne=[1024,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9840 runs - 102.46 us/run - 81920 kB/run - 763.42 GB/s\n SOFT_MAX(type=f32,ne=[77,1024,10,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98064 runs - 10.25 us/run - 6160 kB/run - 573.43 GB/s\n SOFT_MAX(type=f32,ne=[256,256,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98310 runs - 10.25 us/run - 10240 kB/run - 953.21 GB/s\n SOFT_MAX(type=f32,ne=[64,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 7.00 us/run - 640 kB/run - 87.26 GB/s\n SOFT_MAX(type=f32,ne=[77,64,20,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.99 us/run - 770 kB/run - 105.05 GB/s\n SOFT_MAX(type=f32,ne=[8192,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 172011 runs - 6.02 us/run - 64 kB/run - 10.13 GB/s\n SOFT_MAX(type=f32,ne=[8192,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 163820 runs - 6.12 us/run - 256 kB/run - 39.87 GB/s\n SOFT_MAX(type=f32,ne=[8192,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 147438 runs - 6.91 us/run - 1024 kB/run - 141.40 GB/s\n SOFT_MAX(type=f32,ne=[16384,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 128 kB/run - 14.89 GB/s\n SOFT_MAX(type=f32,ne=[16384,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 114674 runs - 8.79 us/run - 512 kB/run - 55.54 GB/s\n SOFT_MAX(type=f32,ne=[16384,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 2048 kB/run - 190.82 GB/s\n SOFT_MAX(type=f32,ne=[32768,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 131056 runs - 8.11 us/run - 256 kB/run - 30.12 GB/s\n SOFT_MAX(type=f32,ne=[32768,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 22.54 us/run - 1024 kB/run - 43.33 GB/s\n SOFT_MAX(type=f32,ne=[32768,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 49146 runs - 23.32 us/run - 4096 kB/run - 167.50 GB/s\n SOFT_MAX(type=f32,ne=[65536,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.19 us/run - 512 kB/run - 59.63 GB/s\n SOFT_MAX(type=f32,ne=[65536,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 40955 runs - 24.59 us/run - 2048 kB/run - 79.43 GB/s\n SOFT_MAX(type=f32,ne=[65536,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 24582 runs - 43.21 us/run - 8192 kB/run - 180.84 GB/s\n SOFT_MAX(type=f32,ne=[131072,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.19 us/run - 1024 kB/run - 119.25 GB/s\n SOFT_MAX(type=f32,ne=[131072,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 40955 runs - 24.59 us/run - 4096 kB/run - 158.87 GB/s\n SOFT_MAX(type=f32,ne=[131072,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 12294 runs - 82.37 us/run - 16384 kB/run - 189.74 GB/s\n SOFT_MAX(type=f32,ne=[262144,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 122865 runs - 8.20 us/run - 2048 kB/run - 238.28 GB/s\n SOFT_MAX(type=f32,ne=[262144,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 36873 runs - 28.66 us/run - 8192 kB/run - 272.61 GB/s\n SOFT_MAX(type=f32,ne=[262144,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 9225 runs - 108.51 us/run - 32768 kB/run - 288.13 GB/s\n SOFT_MAX(type=f32,ne=[524288,1,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 98292 runs - 10.24 us/run - 4096 kB/run - 381.65 GB/s\n SOFT_MAX(type=f32,ne=[524288,4,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 32784 runs - 31.74 us/run - 16384 kB/run - 492.43 GB/s\n SOFT_MAX(type=f32,ne=[524288,16,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): 8721 runs - 121.20 us/run - 65536 kB/run - 516.19 GB/s\n```\n\n* Fix compiler warnings by casting `const` away\n\n* llama : require backend samplers to be of type llama_sampler_chain\n\n* sampling : use host buffer type for inputs\n\n* Try fixing HIP build errors by adding corresponding #defines\n\nWill likely have to disable for MUSA as I didn't find any docs online\n\n* Fix launch logic when supports_cooperative_launch=false\n\n* Disable cooperative groups for musa\n\nDidn't find any doc online, so I don't even know if they support this\n\n* server : reconnect the backend_sampling setting in the WebUI\n\n* graph : make the compute graph constant with respect to active samplers\n\n* batch : fix sequence id ownage\n\n* graph : respect sampler order for graph reuse\n\n* HIP/MUSA: fix build for backend sampling\n\n* sampling : optimize logit_bias sampler\n\n* cont : fix build\n\n* sampling : generic ggml op support detection\n\n* sampling : fix greedy\n\n* tests : run backend sampler tests always on the CPU\n\n* Apply suggestions from code review\n\nCo-authored-by: Johannes Gäßler \n\n* webui : fix lint\n\n* Fix data-race in `soft_max_f32_parallelize_cols_single_row`\n\nBy using `tmp_vals` to store both max values and exponential\naccumulator there was a potential data-race, where the exponential accumulator\nfor a given CTA may have written to `tmp_vals` before all others CTAs have\nread the max value from it.\n\nTo avoid a third g.sync(), an additional temporary data-storage was\nadded. Given that there are syncs in place after writing to gmem, it is\nguaranteed that the previous values for sums/max were read by all CTAs now.\n\n* Apply automated code-formating to softmax.cu\n\n* llama : clarify backend_accept/backend_set_input comments [no ci]\n\n* llama : fix typo in comment [no ci]\n\n* tests : use smart pointers for backend samplers\n\n* tests : use smart pointers for model and context\n\n* tests : remove vocab member from test_model_context\n\nAlso includes some minor cleanups related to nullptr checks.\n\n* tests : extract batch info update to separate method\n\n* tests : fix batch token position tracking in test_backend_sampler.cpp\n\n* tests : add --device option support to backend sampler tests\n\nThis commit adds support for specifying a device to run the test on.\n\n* common : disable backend sampling when grammar is involved\n\n* Fix different RNG-states between backend-sampling and llama-sampling\n\nBy default, we perform a warm-up step where the ggml_cgraph is computed\nonce. For backend-sampling, this graph contains the sampler, and thus\nthe RNG state of the backend's dist sampler is advanced once.\n\nSolution to this is to reset the samplers after the warmup has finished\n\n* Make backend dist sampler use same rnd's as dist sampler\n\nWe sample in double precision and cast to float to match rnd numbers of\nllama_dampler_dist which uses double precision (sampling from\nstd::uniform_real_distribution and\nstd::uniform_real_distribution with same rng will produce\ndifferent sequences).\n\n* Update CCCL version to v3.2.0-rc2\n\n* Build with CCCL 3.2 for CUDA backends\n\nGives best perf for backend-sampling on CUDA. Flag can be removed once\nCCCL 3.2 is bundled within CTK and that CTK version is used in llama.cpp\n\n* tests : revert server test changes (no longer needed)\n\n* ggml : include cub/cub.cuh instead of block_scan.cuh\n\nThis commit updates the include directive in cumsum.cu to use\ncub/cub.cuh instead of cub/block/block_scan.cuh.\n\nThe motivation of this change is that without it compilation fails\nwith the following error:\n```console\n/llama.cpp/ggml/src/ggml-cuda/cumsum.cu(196): error: name followed by \"::\" must be a class or namespace name\n cub::DeviceScan::InclusiveSum(nullptr,\n ^\n\n/llama.cpp/ggml/src/ggml-cuda/cumsum.cu(207): error: name followed by \"::\" must be a class or namespace name\n cub::DeviceScan::InclusiveSum((void *) tmp_alloc.get(), tmp_size, src, dst, ne, stream);\n ^\n\n2 errors detected in the compilation of \"/llama.cpp/ggml/src/ggml-cuda/cumsum.cu\".\ngmake[2]: *** [ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/build.make:317: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/cumsum.cu.o] Error 2\n```\nCommit 83b3b1c271c78bd77664120431aa8c354d68daac (\"cuda: optimize\ncumsum cub path (#18362)\") updated the include directive replacing\ndevice_scan.cuh which is causing this issue.\n\nThis commit uses cub/cub.cuh umbrella header which is consistent with\nother files in the ggml-cuda directory like mean.cu, sum.cu, etc.\n\n* arg : add shorthand for --backend-sampling\n\n* ci : add server workflow with backend sampling\n\n* sampling : fix reshapes\n\n* server : remove printfs\n\n* sampling : zero-initialize input buffers\n\n* minor : add comments + some cleanup\n\n* llama : assert at most one output token per sequence\n\n* tests : add more top_k tests\n\n* CUDA: Fix non-determinism of CUB-based Top-K\n\nDeviceTopK::MaxPairs is an iterative algorithm, where `d_keys_out` is\nwritten after every iteration. As a consequence, it must not overlap\nwith `d_keys_in`, or otherwise undefined behavior occurs (keys are no\nlonger unique in d_keys_in and may map to different values between\niterations)\n\n* CUDA: Optimize index of top_k_cub\n\nBy using the fancy\n[`counting_iterator`](https://nvidia.github.io/cccl/thrust/api/classthrust_1_1counting__iterator.html#classthrust_1_1counting__iterator)\nexposed by CCCL, we can avoid materializing the index to GPU memory,\nsaving VRAM + 1 kernel invocation\n\n* Apply code-formatting to top-k.cu\n\n* CUDA: Remove obsolete temp_keys from CUB\n\nSince we use cuda::discard_iterator to avoid writing out the keys, we\ncan directly pass in src instead of copying it to `temp_keys`\n\n* minor : cleanup, TODOs, etc.\n\n---------\n\nCo-authored-by: Georgi Gerganov \nCo-authored-by: Oliver Simons \nCo-authored-by: Johannes Gäßler ","tree":"e9f980f624ccdc6f2afe7c6ea75cd3f403d54b35"}},{"name":"topk-moe.cu","mode":"0100644","size":17417,"lastCommit":{"hash":"846e991ec3c7ccec49112ff2c5b00b710e5f551d","oid":"846e991ec3c7ccec49112ff2c5b00b710e5f551d","parents":["fb0e6b621917488d623437349fb5361e0ac21c70"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-07-22T00:30:01+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-22T00:30:01+08:00"},"message":"cuda: add sqrt_softplus in topk-moe for dsv4 (#25896)","tree":"39ea83f4104bfa59e193440324c1ea9b1cf810d2"}},{"name":"topk-moe.cuh","mode":"0100644","size":1039,"lastCommit":{"hash":"846e991ec3c7ccec49112ff2c5b00b710e5f551d","oid":"846e991ec3c7ccec49112ff2c5b00b710e5f551d","parents":["fb0e6b621917488d623437349fb5361e0ac21c70"],"author":{"name":"Aman Gupta","email":"amangupta052@gmail.com","when":"2026-07-22T00:30:01+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-07-22T00:30:01+08:00"},"message":"cuda: add sqrt_softplus in topk-moe for dsv4 (#25896)","tree":"39ea83f4104bfa59e193440324c1ea9b1cf810d2"}},{"name":"tri.cu","mode":"0100644","size":5359,"lastCommit":{"hash":"96fe9badfc5235ff0a049aca647bff8c448055aa","oid":"96fe9badfc5235ff0a049aca647bff8c448055aa","parents":["bde188d60f58012ada0725c6dd5ba7c69fe4dd87"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-12-04T22:19:51+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-04T22:19:51+01:00"},"message":"Add support for CUMSUM and TRI for CUDA. (#17584)\n\n* Add support for CUMSUM and TRI for CUDA.\n\n* Minor optimizations.\n\n* Correct warp_prefix_inclusive_sum in float2 variant to return float2\n\n* Optimize TRI\n\n* Whitespace\n\n* Fix strides.\n\n* Implement double loop\n\n* Whitespace\n\n* Fix HIP compilation bugs\n\n* Optimizations + big case performance tests\n\n* Implement using CUB with fallback to custom kernel\n\n* Remove error message.\n\n* Fixes from code review\n\n* Comment out CPU-unsupported F16/BF16 cases to fix CI\n\n* Fine, you win :P\n\n* Fix last cast, use NO_DEVICE_CODE and GGML_UNUSED_VARS\n\n* Vary warp-size based on physical warp size\n\n* Add GGML_UNUSED_VARS in tri as well\n\n* Use constexpr and call prefix_inclusive with warp_size template param\n\n* Update ggml/src/ggml-cuda/cumsum.cu\n\nCo-authored-by: Johannes Gäßler \n\n* Apply suggestions from code review\n\nCo-authored-by: Johannes Gäßler \n\n* Change to tid % warp_size\n\n* Fix strides; hardcode mask; add ggml_lane_mask_t\n\n* Missing renames, remove unused get_warp_mask(), explicit calls to ggml_cuda_info()\n\n* Too hasty...\n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"21a2f9aca78c491c0ee1df33efbcc51d4d2e03f3"}},{"name":"tri.cuh","mode":"0100644","size":131,"lastCommit":{"hash":"96fe9badfc5235ff0a049aca647bff8c448055aa","oid":"96fe9badfc5235ff0a049aca647bff8c448055aa","parents":["bde188d60f58012ada0725c6dd5ba7c69fe4dd87"],"author":{"name":"Piotr Wilkin (ilintar)","email":"piotr.wilkin@syndatis.com","when":"2025-12-04T22:19:51+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-12-04T22:19:51+01:00"},"message":"Add support for CUMSUM and TRI for CUDA. (#17584)\n\n* Add support for CUMSUM and TRI for CUDA.\n\n* Minor optimizations.\n\n* Correct warp_prefix_inclusive_sum in float2 variant to return float2\n\n* Optimize TRI\n\n* Whitespace\n\n* Fix strides.\n\n* Implement double loop\n\n* Whitespace\n\n* Fix HIP compilation bugs\n\n* Optimizations + big case performance tests\n\n* Implement using CUB with fallback to custom kernel\n\n* Remove error message.\n\n* Fixes from code review\n\n* Comment out CPU-unsupported F16/BF16 cases to fix CI\n\n* Fine, you win :P\n\n* Fix last cast, use NO_DEVICE_CODE and GGML_UNUSED_VARS\n\n* Vary warp-size based on physical warp size\n\n* Add GGML_UNUSED_VARS in tri as well\n\n* Use constexpr and call prefix_inclusive with warp_size template param\n\n* Update ggml/src/ggml-cuda/cumsum.cu\n\nCo-authored-by: Johannes Gäßler \n\n* Apply suggestions from code review\n\nCo-authored-by: Johannes Gäßler \n\n* Change to tid % warp_size\n\n* Fix strides; hardcode mask; add ggml_lane_mask_t\n\n* Missing renames, remove unused get_warp_mask(), explicit calls to ggml_cuda_info()\n\n* Too hasty...\n\n---------\n\nCo-authored-by: Johannes Gäßler ","tree":"21a2f9aca78c491c0ee1df33efbcc51d4d2e03f3"}},{"name":"tsembd.cu","mode":"0100644","size":1797,"lastCommit":{"hash":"3913f8730ec6d6245480affc30ae3049107956f4","oid":"3913f8730ec6d6245480affc30ae3049107956f4","parents":["76888d202ed2b835ae19ea9f9db6baf39e419297"],"author":{"name":"Daniel Bevenius","email":"daniel.bevenius@gmail.com","when":"2025-09-16T15:25:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-09-16T15:25:57+02:00"},"message":"ggml : fix padding in timestep embedding kernels (#15932)\n\n* ggml : remove adding extra dim timestep embedding\n\nThis commit updates the ggml_timestep_embedding function to no longer\nadd an extra dimension when the specified dimension is odd.\n\nThe motivation for this change is that this introduces an unnecessary\ndimension when the dimension is odd, which caused an issue in the\nkernels which were not expecting this extra dimension and it resulted in\nuninitialized memory for the second to last dimension.\n\n* ggml-cuda : fix padding in timestep embedding kernel\n\nThis commit removes the zeroing out of the last dimension now that we\nare not adding the extra padding dimension.\n\n* ggml-metal : fix padding in timestep embedding kernel\n\nThis commit fixes the zero padding for odd dimensions in\nthe timestep embedding kernel\n\n* ggml-opencl : fix padding in timestep embedding kernel\n\nThis commit fixes the zero padding for odd dimensions in\nthe timestep embedding kernel.\n\n* ggml-sycl : fix padding in timestep embedding kernel\n\nThis commit fixes the zero padding for odd dimensions in\nthe timestep embedding kernel.\n\n* ggml-vulkan : fix padding in timestep embedding kernel\n\nThis commit fixes the zero padding for odd dimensions in\nthe timestep embedding kernel.\n\n* ggml-cpu : fix padding in timestep embedding function\n\nThis commit removes the zeroing out of the last dimension now that we\nare not adding the extra padding dimension.","tree":"6af7c3058fe149279844f55239d90330b02f032b"}},{"name":"tsembd.cuh","mode":"0100644","size":161,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"unary.cu","mode":"0100644","size":22444,"lastCommit":{"hash":"e947228222147356bc7e64154d3439e142481632","oid":"e947228222147356bc7e64154d3439e142481632","parents":["29f1482221b68fdbf5bd9b762c9e3e350e21f1ec"],"author":{"name":"Andreas Kieslinger","email":"47689530+aendk@users.noreply.github.com","when":"2026-05-20T13:59:02+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-05-20T13:59:02+02:00"},"message":"Programmatic Dependent Launch (PDL) for more performance on newer NVIDIA GPUs (Hopper+) (#22522)\n\n* Adds initial PDL setup.\n\n* Adds PDL barriers based on simple heuristic: place \"sync\" before first input pointer access, and \"launch\" after last write, e.g. to tensors like dst.\n\n* Further optimization pass of the first half of kernels\n\n* Optimized PDL barriers for the second batch of kernels\n\n* Further refinements after rebase.\n\n* Moves pdl logic to separate function, removes some whitespace\n\n* Strips post-hoc PDL logic\n\n* Adds stream capture PDL setup. Enrolls quantize_q8_1 to leverage pdl to\noverlap execution with previous kernels\n\n* Enrolls mul_mat_vec_q, rms_norm_f32 and k_bin_bcast (partly) into PDL\n\n* Enrolls mmvf, rope, set-rows and topk kernels for gpt-oss into PDL\n\n* Introduce ggml_cuda_kernel_launch, to abstract away cudaLaunchKernelEx,\nto enable hip/musa compatibility\n\n* Enrolls cpy_scalar_contiguous, k_get_rows_float and rms_norm_f32\n\n* Enrolls flash_attn_combine_results\n\n* Fix: Drops needless and broken check of CUDA arch for PDL. PDL either\nworks or is without effect.\n\n* Enrolls flash-attention kernels to pdl\n\n* Fix: inlines ggml_cuda_kernel_launch, and uses perfect forwarding for\nkernels args. This fixes PDL.\n\n* Perf: Enrolls k_bin_bcast variadic template invocation into PDL, via\nand template alias and template expansion\n\n* Enrolls all remaining kernels for qwen3-coder-next into PDL\n\n* Remove all PDL LC calls to create a baseline\n\n* Added LC according to internal guidance and tested kernel performance.\n\n* Enrols missing qwen3-5 kernels passively into PDL.\n\n* Kernel optimizations (LC signals) for qwen3.5\n\n* Enrolls ssm-scan kernels into PDL\n\n* Adds GGML_CUDA_PDL command line option to toggle PDL.\n\n* Fix: Ada and lower compilation by guarding PDL calls correctly\n\n* Cleanup: Removes commented out GGML_CUDA_PDL_LC\n\n* Cleanup: Removes experimental comments\n\n* Adds 90-virtual to build script so that Hopper GPUs can leverage PDL.\n\n* Adds stricter checks to enable PDL, adds env-check to disable it, and removes now superfluous compile option to enable PDL.\n\n* Fix: Correct PDL en/disablement based on device-side arch check. Host\nside check is UB. Required moving from macros to inlined functions\n\n* Fix: default-disable PDL. Enable by setting GGML_CUDA_ENABLE_PDL=1\n\n* Enable PDL by default for Hopper+ devices\n\n* Enrolls softcap_f32 and two flash_attn kernels into PDL.\n\n* Improves flash attn PDL barrier placement\n\n* Fix: Perf regression on ada; excludes ada and below from PDL launches\n\n* Improves some sync barrier placements\n\n* Drops superfluous constructor\n\n* Adds #endif guard comments\n\n* Reverts experimental change to top-k-moe.cu, which moved expensive allocations\nin front of the PDL barrier. It did not have a meaningful impact.\n\n* Exchanges GGML_CUDA_DISABLE_PDL with GGML_CUDA_PDL. IFF GGML_CUDA_PDL=0\nPDL is disabled\n\n* Revert \"Drops superfluous constructor\". Adds const to remaining\narguments\n\nThis reverts commit 12b1d250da0089ae02a9bb71bbb3fd6d70f6f2f1.\n\n* Cleanup: Removes and fixes some comments and whitespace\n\n* Clarifies comment of sync-barrier position\n\n* Relocates and refactors PDL launch functions and accessories\n\n* Adds error checking to the regular kernel launch path\n\n* Drops \"auto\" in favor of \"ggml_cuda_kernel_params\"\n\n* Adds \"const\" to ggml_cuda_kernel_launch_params\n\n* [Whitespace] Adds final newline to common.cuh to make editorconfig CI job happy","tree":"6bd686700efc2858d3b391fe3e51b1f8436b2e40"}},{"name":"unary.cuh","mode":"0100644","size":4273,"lastCommit":{"hash":"86db42e97f6f20330b1a54653eeff6814162c39b","oid":"86db42e97f6f20330b1a54653eeff6814162c39b","parents":["6217b49583432f55014c2a0551f453d42b300530"],"author":{"name":"Anav Prasad","email":"anavp@nvidia.com","when":"2026-04-23T02:28:56Z"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-04-23T10:28:56+08:00"},"message":"CUDA: fuse relu + sqr (#22249)","tree":"8b2975cc0865f2a678df66cf894dc13a6b0d1783"}},{"name":"upscale.cu","mode":"0100644","size":13433,"lastCommit":{"hash":"2ba719519d950c5a62c00cdb8b119cc0914c1fa3","oid":"2ba719519d950c5a62c00cdb8b119cc0914c1fa3","parents":["7f8ef50cce40e3e7e4526a3696cb45658190e69a"],"author":{"name":"Tarek Dakhran","email":"tarek@liquid.ai","when":"2025-11-30T21:57:31+01:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-11-30T21:57:31+01:00"},"message":"model: LFM2-VL fixes (#17577)\n\n* Adjust to pytorch\n\n* Add antialiasing upscale\n\n* Increase number of patches to 1024\n\n* Handle default marker insertion for LFM2\n\n* Switch to flag\n\n* Reformat\n\n* Cuda implementation of antialias kernel\n\n* Change placement in ops.cpp\n\n* consistent float literals\n\n* Pad only for LFM2\n\n* Address PR feedback\n\n* Rollback default marker placement changes\n\n* Fallback to CPU implementation for antialias implementation of upscale","tree":"80ce04d09afefdede9acc0bf865435e62d4f7b78"}},{"name":"upscale.cuh","mode":"0100644","size":139,"lastCommit":{"hash":"f3f65429c44bb195a9195bfdc19a30a79709db7b","oid":"f3f65429c44bb195a9195bfdc19a30a79709db7b","parents":["88540445615e77a0177fcca43aaa8e9d8eea6864"],"author":{"name":"Georgi Gerganov","email":"ggerganov@gmail.com","when":"2024-06-26T18:33:02+03:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2024-06-26T18:33:02+03:00"},"message":"llama : reorganize source code + improve CMake (#8006)\n\n* scripts : update sync [no ci]\r\n\r\n* files : relocate [no ci]\r\n\r\n* ci : disable kompute build [no ci]\r\n\r\n* cmake : fixes [no ci]\r\n\r\n* server : fix mingw build\r\n\r\nggml-ci\r\n\r\n* cmake : minor [no ci]\r\n\r\n* cmake : link math library [no ci]\r\n\r\n* cmake : build normal ggml library (not object library) [no ci]\r\n\r\n* cmake : fix kompute build\r\n\r\nggml-ci\r\n\r\n* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE\r\n\r\nggml-ci\r\n\r\n* move public backend headers to the public include directory (#8122)\r\n\r\n* move public backend headers to the public include directory\r\n\r\n* nix test\r\n\r\n* spm : fix metal header\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov \r\n\r\n* scripts : fix sync paths [no ci]\r\n\r\n* scripts : sync ggml-blas.h [no ci]\r\n\r\n---------\r\n\r\nCo-authored-by: slaren ","tree":"3d44ce270dd666d8343013fb2f363eb90e9abcbf"}},{"name":"vecdotq.cuh","mode":"0100644","size":59914,"lastCommit":{"hash":"3b19c377d18157bfec39ec71bad193e9ef000cf2","oid":"3b19c377d18157bfec39ec71bad193e9ef000cf2","parents":["25092e7d5d39d22556c97310461be075b5261449"],"author":{"name":"Ankit Khandelwal","email":"ankk98@gmail.com","when":"2026-09-23T21:34:45+05:30"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-23T09:04:45-07:00"},"message":"rocm: vectorized HIP PTQ1_0 vec_dot (#211)\n\n* vectorized HIP PTQ1_0 vec_dot\n\nThe model file stores weights as densely packed base-3 digits (trits). To use\nthem, the GPU must unpack each trit and multiply-accumulate it with the\nactivation. The old AMD code did this one trit at a time in a long serial\nchain (128 slow steps per block). The new code unpacks 4 trits at once using\nthe GPU's byte-shuffle and 4-wide dot-product instructions. Same math, ~3x\nless instruction work, which is why decode got ~88% faster. A wrong guess\nabout a shuffle instruction's argument order was caught by on-device probes\nand unit tests before it could do harm; the corrected convention is now\ndocumented in a code comment.\n\n* tests : cover PTQ1_0 HIP vector unpack in host dot test\n\nTranscribe the vector idiom (unsigned digits, bias fold, swapped-operand\nperm, qh tail) and assert it against the scalar reference over all 256\nbyte values at every word position. Unwrap the vecdotq comment to one\nsentence per line.\n\nAssisted-by: Opencode with Muse Spark\n\n* ggml-cuda : point PTQ1_0 HIP branch at its host test mirror","tree":"d02c966f3ce7ebea98bdace166e3f148e79ca3b9"}},{"name":"vendors","mode":"0040000","size":0,"lastCommit":{"hash":"3dd767559a1b35fbad6cc8921ff639797dd07755","oid":"3dd767559a1b35fbad6cc8921ff639797dd07755","parents":["7d2da03143f05baf2686a59b5a69371206f4348e"],"author":{"name":"bri-prism","email":"288398250+bri-prism@users.noreply.github.com","when":"2026-09-07T19:45:49-07:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-09-07T19:45:49-07:00"},"message":"cuda: query the device type with cudaDeviceGetAttribute, not cudaGetDeviceProperties (#163)\n\nggml_backend_cuda_device_get_type read the whole cudaDeviceProp to test one\nflag. The full property read costs ~660 us per call; the hybrid-attention\ngraph builder asks for the device type of every device on every recurrent\nlayer of every graph build (dozens of calls per graph), which an H100 nsys\nprofile put at 409 cudaGetDeviceProperties calls / 337 ms per llama-bench run\nagainst 25 / 22 ms for upstream, about 0.3 ms per decoded token.\n\ncudaDeviceGetAttribute(cudaDevAttrIntegrated) answers the same question in\n~20 ns. HIP gets the attribute alias; MUSA resolves the CUDA name already.","tree":"12610f9ee560f7ee0ef15ab9b6f51301e449ed97"}},{"name":"wkv.cu","mode":"0100644","size":9615,"lastCommit":{"hash":"5988633170f66d71570275b89f5c78e0c236e67a","oid":"5988633170f66d71570275b89f5c78e0c236e67a","parents":["f785fc9ea485e6cfdda129978310aa52939c3619"],"author":{"name":"0","email":"1939455790@qq.com","when":"2026-08-12T01:46:23+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-11T20:46:23+03:00"},"message":"cuda : add warp-per-row wkv7 kernel for single-token decode (#26111)","tree":"46187ad0896ec74c0bb1cd044d23b4e7063e7feb"}},{"name":"wkv.cuh","mode":"0100644","size":218,"lastCommit":{"hash":"7dfad387e3f6ac98d383ded2d175eb59736a3993","oid":"7dfad387e3f6ac98d383ded2d175eb59736a3993","parents":["60c902926c928f9c2cd6390ce411876f92feeaf3"],"author":{"name":"Molly Sophia","email":"mollysophia379@gmail.com","when":"2025-03-18T07:27:50+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2025-03-18T07:27:50+08:00"},"message":"llama: Add support for RWKV v7 architecture (#12412)\n\n* ggml: Add op l2_norm\n\nSigned-off-by: Molly Sophia \n\n* ggml: Add op rwkv_wkv7\n\nSigned-off-by: Molly Sophia \n\n* llama: Add support for RWKV7 and ARWKV7 models\n\nSigned-off-by: Molly Sophia \n\n* llama: fix inference with RWKV6Qwen2\n\nSigned-off-by: Molly Sophia \n\n* llama: add more (a)rwkv7 variants in size\n\nSigned-off-by: Molly Sophia \n\n* Apply code-format changes\n\nSigned-off-by: Molly Sophia \n\n* fix MUSA build\n\nSigned-off-by: Molly Sophia \n\n* llama: fix shape error with rwkv using llama-parallel\n\nSigned-off-by: Molly Sophia \n\n---------\n\nSigned-off-by: Molly Sophia ","tree":"5a29b2c9766b2001f89a2a4b99fff1a33dfd1cf4"}}],"lastCommit":{"hash":"c0209b63a39312975f3cbff7670afda41f653cd9","oid":"c0209b63a39312975f3cbff7670afda41f653cd9","parents":["ad2d86b8dc2153be655bbf209e8f85afa3b2908c"],"author":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"committer":{"name":"nekomimi.pet","email":"meowskulls@nekomimi.pet","when":"2026-10-13T19:13:57-04:00"},"message":"bonsai: in-place GDN recurrent state reading from cache rows (gdn-rows)\n\nThe DeltaNet layers read each seq's recurrent state straight from its cache row instead of gathering it first (only while every seq's row is its own cell; otherwise the old gather). Bit-identical; MTP decode +1.6-2.3%. GGML_GDN_STATE_GATHER=1 restores the gather.\n","tree":"5e386426be0e2881da6097e48c87ccb8ffda868d"}}