{"entries":[{"name":".clang-format","mode":"0100644","size":4376,"lastCommit":{"hash":"890f1a27ed4810d9655fc3f748d9b6bc483932fb","oid":"890f1a27ed4810d9655fc3f748d9b6bc483932fb","parents":["58728bdbf0421f3f6d4407436ccbadf0ee90ab53"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-06-17T14:11:21+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-17T09:11:21+03:00"},"message":"openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off (#24503)\n\n* Add interface is_model_splitted() to check the c-graph is splited or not\n\n* Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs()\n\n* Only do this for fallback sub graph\n\n* Move dynamic dims compute in graph missmatch\n\n* ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models\n\n* ggml-openvino:add comments\n\n* ggml-openvino: override VIEW op_case to 0 for split model inputs\n\n* openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend\n\n* Enable additional mul_mat tests and add tensor data saving function (#81)\n\n* ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling\n\n* OpenVINO: add NORM/TANH support and rework SOFT_MAX translation\n\n* ggml-openvino: extend VIEW handling\n\n* Enable -fa off (#118)\n\n* Enable --context-shift\n\n* Fix llm param compute error for normal softmax not the softmax in attention\n\n* OpenVINO backend: fix error for attention size compute in llm param\n\n* use tensor->extra in infer_request i/o\n\n* OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand\n\n* OpenVINO backend: clean unused code\n\n* 1to1 match op update (#146)\n\n* added translate_1to1_match_1_input function and updated gelu and tanh translations\n\n* Remove unused translation function calls\n\n---------\n\nCo-authored-by: Mustafa Cavus \n\n* initial gemma4 support\n\n* removed hardcoded names for kv cache slicing\n\n* OpenVINO backend: Add new attention pattern for llm parameters compute\n\n* flash attn Q shape static conversion\n\n* Remove slice in permute translation when n_seq is 1\n\n* return optional in extract_layer_from_name\n\n* OpenVINO backend: refactor VIEW related operation (#148)\n\n* OpenVINO backend: refactor VIEW related operation\n\n* Enable VIEW handling in following ops\n\n* OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO\n\n* OpenVINO backend: Add ops l2_norm & pad\n\n* OpenVINO backend does not support CPY with non-contiguous data or mismatched types\n\n* add op SSM_CONV GATED_DELTA_NET\n\n* OpenVINO backend: fix error for bf16 in OV gpu plugin\n\n* reverted static Q input shape for attention layer\n\n* OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case\n\n* Disable remote tensor due to bug in ov gpu\n\n* Disable n_token > 1 GATED_DELTA_NET on gpu\n\n* OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row\n\n* OpenVINO backend: clean code\n\n* OpenVINO backend: enable view + norm/rms_norm\n\n* OpenVINO backend: concat op\n\n* OpenVINO backend: argsort op\n\n* OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS\n\n* Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct\n\n* OpenVINO backend: enable sum_rows\n\n* OpenVINO backend: enable clamp\n\n* OpenVINO backend: enable DIV\n\n* OpenVINO backend: enable GGML_OP_MUL_MAT_ID\n\n* OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed\n\n* OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed\n\n* OpenVINO backend: fix issue in mul_mat_id\n\n* OpenVINO backend: Disable DIV with broadcast on GPU\n\n* OpenVINO backend: update DIV\n\n* use ov internal op GatedDeltaNet\n\n* OpenVINO backend: enable llama erch test qwen3next\n\n* OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope\n\n* OpenVINO backend: fix error\n\n* suggested changes, need review\n\n* suggested changes, need review\n\n* OpenVINO backend: clean unused code & fix build warning\n\n* OpenVINO backend: enable minicpm3 for arch test\n\n* Disable GDN op (#177)\n\n* disable gated_delta_net\n\n* update stateful_kv_size correctly in mismatch case\n\n* OpenVINO backend: enable arch test for qwen3vl\n\n* OpenVINO backend: enable cohere2 for arch test\n\n* OpenVINO backend: enable t5 for arch test\n\n* OpenVINO backend: enable jamba for arch test\n\n* OpenVINO backend: remove warning for tmp\n\n* OpenVINO backend: enable kimi-linear for arch test\n\n* Remove unused\n\n* Fix gpt-oss accuracy issue\n\n* OpenVINO backend: enable arctic for arch test\n\n* OpenVINO backend: enable grok for arch test\n\n* Gemma4 initial npu support (#179)\n\n* Initiall gemma4 npu support\n\n* temp. fix for gemma4 accuracy bug on npu\n\n* Remove hardcoded names for npu-fold handling\n\n* revert static n tokens for cont translation as it is not needed\n\n* removed unused variable\n\n* ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference.\n\n* Revert \"Gemma4 initial npu support (#179)\"\n\nThis reverts commit 0d29a9c4a52dc2c8aa52990f1a3854cfb01768ad.\n\n* OpenVINO backend: disable debug log print\n\n* Update TBB discovery. Delegated to OpenVINOs own config.\n\n* OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1\n\n* OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend\n\n* Add raw ov infer profiling metric\n\n* Add OV raw infer time metric to static compute path\n\nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\n\n* Modify precision of static profiling\n\n* update to OV 2026.2, add OV windows CI\n\n* fix editorconfig-checks\n\n* Initiall gemma4 npu support\n\n* temp. fix for gemma4 accuracy bug on npu\n\n* Remove hardcoded names for npu-fold handling\n\n* revert static n tokens for cont translation as it is not needed\n\n* removed unused variable\n\n* test-llama-archs fix\n\n* Fix gemma4 flash_attn fallback\n\n* support im2col\n\n* fix code style\n\n* disable add_rope_sin_cos optimization\n\n* stateless boradcast and rope optimizations\n\n* Enable manual gqa attn by default for stateless gpu\n\n* manual gqa: fixed static batch\n\n* gemma4 llama-bench ctx update fix\n\n* Update OV win CI\n\n* stateful rope fusion temp. fix\n\n* OpenVINO backend: Conslolidate supported ops\n\n* Exclude unsupported GGML_OP_SUB cases\n\n* Exclude unsupported TOPK_MOE cases\n\n* OpenVINO Backend: MUL_MAT enhancements\n\n* Update OV CI\n\n* support f16 mask input for npu\n\n* Make GGML_OPENVINO_* env vars usage uniform\n\nStandardize all GGML_OPENVINO_* env flags:\npositive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable.\nThis fixes cases where text values or empty strings enabled features.\n\n* OpenVINO backend: Enhance envvar handling\n\n* more cleanup\n\n* move ggml_openvino_env_flag to appropriate place\n\n* OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix\n\n* ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data\n\n* Update openvino.Dockerfile\n\nUse BuildKit cache mounts for faster Docker rebuilds.\r\nUse apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF.\n\n* ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers\n\nReplace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates.\n\n* OpenVINO backend: Enable GGML_OP_ADD_ID\n\n* Uptade openvino backend clamg-format\n\n* clang-format\n\n* Update OPENVINO.md (#211)\n\n* OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision\n\n* Remove strict concurrency for gpu-openvino-low-perf\n\n* Update openvino CI keynames; add ccache-clear\n\n* Apply suggestions from code review\n\nCo-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>\n\n* Fix formatting\n\n---------\n\nCo-authored-by: Xuejun Zhai \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Xuejun \nCo-authored-by: Wang Yang \nCo-authored-by: Ravi Panchumarthy \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>\nCo-authored-by: Mostafa Faheem \nCo-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>","tree":"4a0947d86935051575826f9bed7a2d804a7b9e55"}},{"name":"CMakeLists.txt","mode":"0100644","size":755,"lastCommit":{"hash":"890f1a27ed4810d9655fc3f748d9b6bc483932fb","oid":"890f1a27ed4810d9655fc3f748d9b6bc483932fb","parents":["58728bdbf0421f3f6d4407436ccbadf0ee90ab53"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-06-17T14:11:21+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-06-17T09:11:21+03:00"},"message":"openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off (#24503)\n\n* Add interface is_model_splitted() to check the c-graph is splited or not\n\n* Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs()\n\n* Only do this for fallback sub graph\n\n* Move dynamic dims compute in graph missmatch\n\n* ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models\n\n* ggml-openvino:add comments\n\n* ggml-openvino: override VIEW op_case to 0 for split model inputs\n\n* openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend\n\n* Enable additional mul_mat tests and add tensor data saving function (#81)\n\n* ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling\n\n* OpenVINO: add NORM/TANH support and rework SOFT_MAX translation\n\n* ggml-openvino: extend VIEW handling\n\n* Enable -fa off (#118)\n\n* Enable --context-shift\n\n* Fix llm param compute error for normal softmax not the softmax in attention\n\n* OpenVINO backend: fix error for attention size compute in llm param\n\n* use tensor->extra in infer_request i/o\n\n* OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand\n\n* OpenVINO backend: clean unused code\n\n* 1to1 match op update (#146)\n\n* added translate_1to1_match_1_input function and updated gelu and tanh translations\n\n* Remove unused translation function calls\n\n---------\n\nCo-authored-by: Mustafa Cavus \n\n* initial gemma4 support\n\n* removed hardcoded names for kv cache slicing\n\n* OpenVINO backend: Add new attention pattern for llm parameters compute\n\n* flash attn Q shape static conversion\n\n* Remove slice in permute translation when n_seq is 1\n\n* return optional in extract_layer_from_name\n\n* OpenVINO backend: refactor VIEW related operation (#148)\n\n* OpenVINO backend: refactor VIEW related operation\n\n* Enable VIEW handling in following ops\n\n* OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO\n\n* OpenVINO backend: Add ops l2_norm & pad\n\n* OpenVINO backend does not support CPY with non-contiguous data or mismatched types\n\n* add op SSM_CONV GATED_DELTA_NET\n\n* OpenVINO backend: fix error for bf16 in OV gpu plugin\n\n* reverted static Q input shape for attention layer\n\n* OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case\n\n* Disable remote tensor due to bug in ov gpu\n\n* Disable n_token > 1 GATED_DELTA_NET on gpu\n\n* OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row\n\n* OpenVINO backend: clean code\n\n* OpenVINO backend: enable view + norm/rms_norm\n\n* OpenVINO backend: concat op\n\n* OpenVINO backend: argsort op\n\n* OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS\n\n* Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct\n\n* OpenVINO backend: enable sum_rows\n\n* OpenVINO backend: enable clamp\n\n* OpenVINO backend: enable DIV\n\n* OpenVINO backend: enable GGML_OP_MUL_MAT_ID\n\n* OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed\n\n* OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed\n\n* OpenVINO backend: fix issue in mul_mat_id\n\n* OpenVINO backend: Disable DIV with broadcast on GPU\n\n* OpenVINO backend: update DIV\n\n* use ov internal op GatedDeltaNet\n\n* OpenVINO backend: enable llama erch test qwen3next\n\n* OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope\n\n* OpenVINO backend: fix error\n\n* suggested changes, need review\n\n* suggested changes, need review\n\n* OpenVINO backend: clean unused code & fix build warning\n\n* OpenVINO backend: enable minicpm3 for arch test\n\n* Disable GDN op (#177)\n\n* disable gated_delta_net\n\n* update stateful_kv_size correctly in mismatch case\n\n* OpenVINO backend: enable arch test for qwen3vl\n\n* OpenVINO backend: enable cohere2 for arch test\n\n* OpenVINO backend: enable t5 for arch test\n\n* OpenVINO backend: enable jamba for arch test\n\n* OpenVINO backend: remove warning for tmp\n\n* OpenVINO backend: enable kimi-linear for arch test\n\n* Remove unused\n\n* Fix gpt-oss accuracy issue\n\n* OpenVINO backend: enable arctic for arch test\n\n* OpenVINO backend: enable grok for arch test\n\n* Gemma4 initial npu support (#179)\n\n* Initiall gemma4 npu support\n\n* temp. fix for gemma4 accuracy bug on npu\n\n* Remove hardcoded names for npu-fold handling\n\n* revert static n tokens for cont translation as it is not needed\n\n* removed unused variable\n\n* ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference.\n\n* Revert \"Gemma4 initial npu support (#179)\"\n\nThis reverts commit 0d29a9c4a52dc2c8aa52990f1a3854cfb01768ad.\n\n* OpenVINO backend: disable debug log print\n\n* Update TBB discovery. Delegated to OpenVINOs own config.\n\n* OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1\n\n* OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend\n\n* Add raw ov infer profiling metric\n\n* Add OV raw infer time metric to static compute path\n\nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\n\n* Modify precision of static profiling\n\n* update to OV 2026.2, add OV windows CI\n\n* fix editorconfig-checks\n\n* Initiall gemma4 npu support\n\n* temp. fix for gemma4 accuracy bug on npu\n\n* Remove hardcoded names for npu-fold handling\n\n* revert static n tokens for cont translation as it is not needed\n\n* removed unused variable\n\n* test-llama-archs fix\n\n* Fix gemma4 flash_attn fallback\n\n* support im2col\n\n* fix code style\n\n* disable add_rope_sin_cos optimization\n\n* stateless boradcast and rope optimizations\n\n* Enable manual gqa attn by default for stateless gpu\n\n* manual gqa: fixed static batch\n\n* gemma4 llama-bench ctx update fix\n\n* Update OV win CI\n\n* stateful rope fusion temp. fix\n\n* OpenVINO backend: Conslolidate supported ops\n\n* Exclude unsupported GGML_OP_SUB cases\n\n* Exclude unsupported TOPK_MOE cases\n\n* OpenVINO Backend: MUL_MAT enhancements\n\n* Update OV CI\n\n* support f16 mask input for npu\n\n* Make GGML_OPENVINO_* env vars usage uniform\n\nStandardize all GGML_OPENVINO_* env flags:\npositive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable.\nThis fixes cases where text values or empty strings enabled features.\n\n* OpenVINO backend: Enhance envvar handling\n\n* more cleanup\n\n* move ggml_openvino_env_flag to appropriate place\n\n* OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix\n\n* ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data\n\n* Update openvino.Dockerfile\n\nUse BuildKit cache mounts for faster Docker rebuilds.\r\nUse apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF.\n\n* ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers\n\nReplace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates.\n\n* OpenVINO backend: Enable GGML_OP_ADD_ID\n\n* Uptade openvino backend clamg-format\n\n* clang-format\n\n* Update OPENVINO.md (#211)\n\n* OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision\n\n* Remove strict concurrency for gpu-openvino-low-perf\n\n* Update openvino CI keynames; add ccache-clear\n\n* Apply suggestions from code review\n\nCo-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>\n\n* Fix formatting\n\n---------\n\nCo-authored-by: Xuejun Zhai \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Xuejun \nCo-authored-by: Wang Yang \nCo-authored-by: Ravi Panchumarthy \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>\nCo-authored-by: Mostafa Faheem \nCo-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>","tree":"4a0947d86935051575826f9bed7a2d804a7b9e55"}},{"name":"ggml-decoder.cpp","mode":"0100644","size":83922,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"ggml-decoder.h","mode":"0100644","size":18462,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"ggml-openvino-extra.cpp","mode":"0100644","size":18332,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"ggml-openvino-extra.h","mode":"0100644","size":9976,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"ggml-openvino.cpp","mode":"0100644","size":63434,"lastCommit":{"hash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","oid":"fe8156f789011f6ea0baf6917ea09f88b89d9554","parents":["77acca437fb6dbe79c649ca69ff98b32bfac48c3"],"author":{"name":"Xuan-Son Nguyen","email":"son@huggingface.co","when":"2026-08-19T14:04:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-19T14:04:57+02:00"},"message":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization","tree":"45a80859202244449fe4586006e673c53336e861"}},{"name":"ggml-quants.cpp","mode":"0100644","size":60462,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"ggml-quants.h","mode":"0100644","size":8507,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"model-cache.cpp","mode":"0100644","size":9631,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"model-cache.h","mode":"0100644","size":3128,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"openvino","mode":"0040000","size":0,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"utils.cpp","mode":"0100644","size":67582,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}},{"name":"utils.h","mode":"0100644","size":7940,"lastCommit":{"hash":"aee56b3abfeb134218fe5c496c329a1faac1ef28","oid":"aee56b3abfeb134218fe5c496c329a1faac1ef28","parents":["a97123e497968f3440264c0464a7adc7c999c027"],"author":{"name":"Zijun Yu","email":"zijun.yu@intel.com","when":"2026-08-14T01:30:48+08:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-13T20:30:48+03:00"},"message":"OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)\n\n* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support\n\n* OpenVINO backend: disable TOPK_MOE op test\n\n* OpenVINO Backend: Add op FILL support\n\n* OpenVINO backend: enable set rows with multi dims\n\n* fix the name missmatch in setrow + view\n\n* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID\n\n* OpenVINO Backend: enable SQR & SQRT\n\n* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer\n\n* OpenVINO backend: enable fallback for openVINO to CPU backend\n\n* OpenVINO backend: fix accurace issue in gemma3n arch test\n\n* fix mpt failed case\n\n* OpenVINO backend: clean nodeinfo\n\n* OpenVINO Backend: enable zero-size copy for view\n\n* add concat ssm_conv in compute_dynamic_dim\n\nenable qwen35\n\nFix after rebase\n\nremove logging\n\n* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region\n\n* OpenVINO backend: fix CPY op test failed issue\n\n* OpenVINO backend: fix GATED_DELTA_NET op test failed issue\n\n* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph\n\n* handle qwen35 dynamic clearing of cache correctly\n\n* Enable qwen35 dense multi seq\n\n* Fix qwen35 9b gqa\n\n* Fix after rebase\n\n* Disable SOLVE_TRI\n\n* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)\n\nIn stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,\nhead_size]) into the Multiply against the rank-4 cos/sin tables\n([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the\nOpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage\noutput (e.g. Phi-3-mini). Lift the data to rank-4 before the split/\nMultiply so the operands are equal-rank, matching what the TYPE_NORMAL\nbranch already does. CPU and stateless paths are unaffected.\n\nPhi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:\n before: PPL = 27120.43\n after: PPL = 6.2263 (CPU reference: 6.2251)\n\n* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update\n\n* OpenVINO Backenb: remove changes in llama.cpp\n\n* Doc change (use x64 Native Tools Command Prompt for VS)\n\n* Cleaner sentence\n\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\n\n* OpenVINO Backend: cache key upgrade includes all src name\n\n* OpenVINO Backend: enable llama arch test on ci\n\n* OpenVINO Backend: move parameter node creating from decoder into translate\n\n* OpenVINO Backend: create extra input ov node move from decoder to translate\n\n* fix for op regression due to is_model_splitted\n\n* openvino: fix CPY writeback for recurrent state rollback\n\nDetect the rollback conv/gdn state writeback CPY nodes structurally\ninstead of by tensor name, since the rollback path in\nbuild_conv_state does not call cb() and left the nodes unnamed. Add\nper-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the\ncached IR handles any kv head, sequence count and snapshot slot for\nboth the conv state and the GDN state writeback.\n\nAssisted-by: GitHub Copilot\n\n* qwen35 moe\n\n* optimize MoE expert aggregation with ReduceSum\n\n* Skip GET_ROWS inaccurate test\n\n* openvino: fallback dynamic MUL_MAT_ID shapes\n\n* OpenVINO Backend: fix error in arch test model mpt\n\n* fix error caused by cpy in arch test model kimi-linear\n\n* OpenVINO Backend: fix error in arch test model minimax-m3\n\n* openvino: fix GPU mul_mat_id op tests\n\n* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU\n\nThe OpenVINO weight Constants are zero-copy views into host buffers\nallocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU\nthe plugin holds its own device copy after compile_model, so these host\npages are dead weight for inference. For a 1B Q4_K_M model this leaves\n~850 MB of host RSS resident that the GPU path never reads again.\n\nAdd an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s\nthe registered host weight buffers once the model is compiled, dropping\ntheir resident pages while keeping the mappings valid (ggml still owns the\nlifetime; tensors still point in). Measured steady-state RSS drops from\n~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged\nthroughput and correct output.\n\nThe GPU backend uses a single dynamic-shape model for both prefill and\ndecode, so a graph is compiled once and reused; the only event that forces\na recompile is clear_caches() on backend teardown. The change therefore:\n - releases on the first cache-hit (model compiled, plugin has its copy);\n - pins the compiled-model cache across backend teardown so a later\n context reuses it instead of recompiling against the dropped pages;\n - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model\n load, both of which would otherwise read zeroed weights or silently\n reuse the wrong compiled graph.\n\nScope/limitations (all fail loud, never silently wrong): GPU only (the CPU\nplugin reads the host Constants at inference time), one model per process,\nand stable graph shapes. This reduces steady-state RSS, not the transient\ncompile-time peak. All changes are confined to the OpenVINO backend.\n\n* ggml-openvino: stream weight requantization to cut the compile-time RSS peak\n\nrequantize_to_buffers() dequantized the entire tensor to a temporary\nstd::vector of n_elements before requantizing. For token_embd.weight\n(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is\nthe single largest contributor to the OpenVINO compile-time memory peak --\nit also fires twice for token_embd (once at load, once at graph build,\nbecause token_embd is loaded via a CPU/mmap buffer and not cached as an OV\nweight extra).\n\nStream the dequant instead: process a fixed window of complete rows\n(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk\nstraight into the output buffers. The transient F32 footprint is now\nCHUNK_ROWS*ne0 floats regardless of tensor size.\n\nquantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk\nwrites its weights/scales/zp at the correct block. Streaming is applied to\nthe Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)\npath keeps the whole-array call because it packs two weights per byte with\nrunning zp ORs, and a fallback handles any future target whose block size\ndoes not divide a row.\n\nMeasured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);\n8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged\n(\"capital of France is Paris\"); throughput unchanged. Unlike\nGGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just\nsteady-state, and needs no env flag. All changes confined to the OpenVINO\nbackend.\n\n* ggml-openvino: avoid redundant token_embd requantization at compile\n\ntoken_embd.weight is referenced twice in the graph path: as the GET_ROWS\nembedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on\nevery weight-node build, and is_model_splitted() built a full (naive) set of\nweight nodes just to test name membership — each requant is a ~1-2 GB F32\ndequant of the 262M-element embedding.\n\nTwo changes:\n- Add collect_weight_names(): a name-only collector for topology checks.\n is_model_splitted() now uses it instead of create_weight_nodes(cgraph,\n true), so the splitted-check no longer triggers any weight extraction.\n- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime\n cache keyed by tensor->data. These tensors have no OV buffer context to own\n a cached extra, so without this they were rebuilt on every (re)compile;\n prefill and decode graphs now share one build (verified: 2nd graph hits the\n cache instead of re-requantizing).\n\nPeak RSS is unchanged (the streaming-requant commit already removed the F32\ntransient); this removes redundant compile-time work. Output verified\nunchanged (\"capital of France is Paris\"). Confined to the OpenVINO backend.\n\n* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM\n\nThe streaming requantization and the non-OpenVINO-buffer weight-node cache\n(plus the name-only is_model_splitted path that pairs with it) are now opt-in\nvia GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()\nfully materializes the F32 buffer and weights are rebuilt per compile exactly\nas before; when set, the streaming path and the cross-compile weight cache\nare used.\n\nDefault off keeps behavior identical to upstream unless explicitly enabled.\nVerified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output\n\"capital of France is Paris\" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,\nadded earlier, remains a separate opt-in for the steady-state release.)\n\n* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)\n\nThe plugin-level ov::cache_dir caches the compiled blob keyed by the OV\nmodel, but producing that model still runs the full frontend every time:\nweight requantization (incl. the large token_embd F32 transient) and the\nggml->OV graph conversion. This adds an opt-in frontend cache keyed off a\nfingerprint computed directly from the ggml cgraph, so a hit imports a\npreviously exported CompiledModel and skips requant + convert + compile\nentirely.\n\nKey (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per\nnode op/name), a sampled per-weight fingerprint (name/shape/type + bounded\nhead+tail byte sample), and blob-affecting config (device, flash-attn, rope\nparams, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar\nmanifest stores every weight's fingerprint and is re-verified on load, so a\nsampled-hash collision cannot cause a wrong-model hit (verified: two\ndifferent quantizations of the same model produce distinct cache entries).\n\nFlow (dynamic single-model path only; split models defer to ov::cache_dir):\non a verified hit, core.import_model() restores the CompiledModel and a\nlightweight decoder is built with a names-only weight map (membership is all\nthe decoder needs for I/O mapping; weights live in the imported model). On a\nmiss, compile as usual then export the blob (atomic temp+rename, manifest\nwritten first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/\nCACHE_MODE are stripped from the config used for the cached compile and the\nimport — a blob compiled with cache_dir set cannot be re-imported.\n\nMeasured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s\n(~2.4x faster compile phase). Output verified unchanged on cold and warm,\nstandalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default\noff; confined to the OpenVINO backend.\n\n* ggml-openvino: harden frontend model cache correctness\n\nThe frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.\n\nAdd a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.\n\nDISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.\n\nAlso make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.\n\nWhile touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.\n\n* ggml-openvino: add memory optimization umbrella switch\n\nAdd GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.\n\nCentralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.\n\nKeep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.\n\n* ggml-openvino: rename compiled model cache env\n\nRename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.\n\nUpdate the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.\n\n* docs: document OpenVINO memory optimization env vars\n\nAdd runtime configuration entries for the newly recognized OpenVINO environment variables.\n\nDocument GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.\n\nDocument GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.\n\n* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug\n\n1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash\non decode for models that pad the token embedding (n_embd -> n_embd_inp).\nPAD never reorders/merges dims, so it keeps the same dynamic dim index as\nits source.\n\n2. process_view_input_new() chained VIEW inputs through src[0] (the\nimmediate op-graph parent) using offsets treated as relative to that\nparent. But ggml_tensor::view_offs is always absolute from the true root\nallocation (ggml collapses VIEW-of-VIEW chains internally). For the\nper-layer deepstack view (\"embd (view)\", whose src[0] is \"embd\" - itself\nan already-narrowed, zero-offset VIEW of the padded root, with the SAME\nggml shape as the deepstack view but a different absolute offset), this\ncaused an out-of-bounds re-slice that silently fell back to returning the\nwrong (already-resolved sibling) tensor. In practice every deepstack ADD\nended up adding the real base token embedding into the residual stream\ninstead of zero, corrupting generation (\"Hello my name is 1000000...\"\ninstead of coherent text). Fixed by detecting this pattern (same shape as\nthe immediate src, different absolute offset) and re-slicing directly\nfrom the untouched root tensor using the innermost view's absolute\noffset.\n\nAlso adds a GGML_OPENVINO_DEBUG_NODE=,,... env var that attaches\nextra debug Result nodes for arbitrary intermediate tensors, without binding\nthem to any ggml buffer (avoiding the risk of reading a ggml buffer that has\nsince been overwritten by a later in-place op). This was instrumental in\ndiagnosing bug #2 above and is left in as a general-purpose debugging aid.\n\n* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes\n\nIMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into\nne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape\npath, inp_pos was padded/shaped as if it held a single plane, which\ninterleaved padding across the 4 planes and desynced later reshapes\nfrom the rest of the (chunk_size-wide) graph.\n\n- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout\n- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)\n or n_planes (decode) instead of assuming 1 value per token\n- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size\n independently instead of one flat block\n- get_ov_input_tensor_static_decode(): copy n_planes contiguous values\n instead of asserting/copying a single scalar\n\n* disable test-llama-archs tests.\n\n* openvino: gate fallback with env var\n\n* Revert changes in test-llama-archs\n\n* Apply editor config\n\n* reject CPY with quantized destination as unsupported\n\n---------\n\nCo-authored-by: Xuejun \nCo-authored-by: Mustafa Cavus \nCo-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>\nCo-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>\nCo-authored-by: suryasidd \nCo-authored-by: Mustafa Cavus \nCo-authored-by: Ravi Panchumarthy ","tree":"b9187c88960210741647e52919622dd27ae67c39"}}],"lastCommit":{"hash":"fe8156f789011f6ea0baf6917ea09f88b89d9554","oid":"fe8156f789011f6ea0baf6917ea09f88b89d9554","parents":["77acca437fb6dbe79c649ca69ff98b32bfac48c3"],"author":{"name":"Xuan-Son Nguyen","email":"son@huggingface.co","when":"2026-08-19T14:04:57+02:00"},"committer":{"name":"GitHub","email":"noreply@github.com","when":"2026-08-19T14:04:57+02:00"},"message":"ggml: add ggml_rope_set_offset (+ metal support) (#27120)\n\n* add params\n\n* cpu kernel\n\n* metal kernel\n\n* add test backend ops\n\n* gate other backends\n\n* ggml: (cuda) support ggml_rope_set_offset (#27121)\n\n* rm cuda supports_op guard, fix webgpu clang-format\n\n* ggml: support ggml_rope_set_offset on vulkan (#27344)\n\n* ggml: support ggml_rope_set_offset on vulkan\n\n* remove inplace optimization","tree":"45a80859202244449fe4586006e673c53336e861"}}