{"harness":"llama.app","type":"session","currNode":"s56zjtjc8ok","id":"incyjf36ghq","lastModified":1787790050935,"name":"Hi you are qwen3.8-next-flash running as an unsloth UD-IQ4_XS quant go and checkout info on that around huggingface, unsloths writeup here https://unsloth.ai/docs/models/qwen3.8-next (You MUST check that this model exists before you start reasoning about flags and configs, you have a prior cutoff date before it was launched. Do that FIRST)","reasoningEffort":"default"}
{"message":{"children":["30quovjtzmi"],"content":"","convId":"incyjf36ghq","id":"kjxf3f4mvf","parent":null,"role":"system","timestamp":1787788439303,"type":"root"},"type":"message"}
{"message":{"children":["qhbou3908dr"],"content":"Hi you are qwen3.8-next-flash running as an unsloth UD-IQ4_XS quant go and checkout info on that around huggingface, unsloths writeup here https://unsloth.ai/docs/models/qwen3.8-next (You MUST check that this model exists before you start reasoning about flags and configs, you have a prior cutoff date before it was launched. Do that FIRST)\n\nThen check llama.cpp which is your engine here https://github.com/ggml-org/llama.cpp\n\nLets see if you can optimize my runblock and especially PP for hermes agent https://hermes-agent.nousresearch.com/docs/assets/files/llms-7240021af84660c2a79f9fdaf65e9e8f.txt and here is my current runblock:\n\n#!/bin/bash\nset -euo pipefail\n\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\nexport CUDA_DEVICE_ORDER=PCI_BUS_ID\nexport LLAMA_ATTN_ROT_DISABLE=1\n\nMODEL=/root/qwen3-next/unsloth/Qwen3.8-Flash-Next-GGFF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf\n\n/root/qwen3-next/llama.cpp/build/bin/llama-server \\\n  --model \"$MODEL\" \\\n  --alias qwen3.8-flash-next \\\n  --host 192.168.1.61 \\\n  --port 9876 \\\n  --api-key nerdtastic \\\n  --jinja \\\n  --flash-attn on \\\n  --fit off \\\n  --n-gpu-layers all \\\n  --split-mode layer \\\n  --tensor-split 1,1,1,1 \\\n  --main-gpu 0 \\\n  --override-tensor \"per_layer_token_embd=CPU,ple_ngram_embd=CPU\" \\\n  --load-mode none \\\n  --ctx-size 131072 \\\n  --parallel 1 \\\n  --batch-size 4096 \\\n  --ubatch-size 2048 \\\n  --cache-type-k q8_0 \\\n  --cache-type-v q8_0 \\\n  --cache-prompt \\\n  --no-context-shift \\\n  --threads 6 \\\n  --threads-batch 12 \\\n  --threads-http 4 \\\n  --temp 1.0 \\\n  --top-p 0.95 \\\n  --top-k 20 \\\n  --min-p 0.00 \\\n  --presence-penalty 0.0 \\\n  --repeat-penalty 1.0 \\\n  --metrics\n\nThis is your current runblock and I am no llama.cpp expert I ususally use vllm so I need help tuning this up! I do expect there to be likely some offloading as well potentially, that might not have a huge impact due to your n-gram additional hash table. I am running on quad 3090s on a 3945wx (12c/24t) with 8 dimms 2133 off a wrx80 mobo that has 128 lanes of pcie and all GPUs are negotiated on that full gen4 x16 wide. The work we are optimizing for is the hermes agentic harness to run smoothly with this configuration for long ranging tasks. That is the currently running runblock also so you have a large ctx so please be comprehensive. I am adding this note the UD-IQ4_XS is 93.7GB. Llama.cpp is going to be using fit on (the default) Please check into ngram offloading to RAM/OPTANE as an option to run larger quants in the future.","convId":"incyjf36ghq","extra":[],"role":"user","timestamp":1787790050933,"type":"text","id":"30quovjtzmi","parent":"kjxf3f4mvf"},"type":"message"}
{"message":{"children":["jujd1cyul5m","0mwaxd1eh6sp","kjsqlyed9d"],"content":"I'll verify the model exists before reasoning about anything else. Doing that first, in parallel with the doc fetches.\n\n","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790050945,"type":"text","id":"qhbou3908dr","parent":"30quovjtzmi","completionId":"chatcmpl-Rq2ynU1WFR2uvjNrcrH8O5H3lNO4KRrQ","reasoningContent":"The user wants me to check whether the model exists first, then read the Unsloth writeup, then check llama.cpp, then read the hermes agent docs, and then optimize the runblock.\n\nLet me start with parallel independent calls: hf_whoami (maybe not needed), hub_repo_search for the model, and web_fetch of the unsloth docs and the hermes docs.\n\nLet me do it.\n\nNote: the user says \"You are qwen3.8-next-flash running as unsloth UD-IQ4_XS quant\". This is a plausibly fictional/new model. I must verify existence. I have a prior cutoff. Let me check.\n\nLet's start with independent calls.\n","timings":{"agentic":{"llm":{"predicted_ms":1212948.124,"predicted_n":24529,"prompt_ms":507272.075,"prompt_n":106548},"perTurn":[{"llm":{"predicted_ms":7800.728,"predicted_n":346,"prompt_ms":4815.282,"prompt_n":4610},"toolCalls":[{"duration_ms":66,"name":"hf_whoami","success":true},{"duration_ms":87,"name":"hub_repo_search","success":true},{"duration_ms":272,"name":"web_fetch_exa","success":true}],"toolsMs":425,"turn":1},{"llm":{"predicted_ms":56661.296,"predicted_n":2025,"prompt_ms":25561.565,"prompt_n":14969},"toolCalls":[{"duration_ms":182,"name":"hub_repo_details","success":true},{"duration_ms":187,"name":"hf_fs","success":true},{"duration_ms":314,"name":"web_fetch_exa","success":true}],"toolsMs":683,"turn":2},{"llm":{"predicted_ms":205822.536,"predicted_n":6038,"prompt_ms":43443.077,"prompt_n":15758},"toolCalls":[{"duration_ms":522,"name":"hf_fs","success":true},{"duration_ms":999,"name":"web_search_exa","success":true}],"toolsMs":1521,"turn":3},{"llm":{"predicted_ms":65763.511,"predicted_n":1497,"prompt_ms":128858.13,"prompt_n":28044},"toolCalls":[{"duration_ms":435,"name":"web_fetch_exa","success":true},{"duration_ms":135,"name":"hub_repo_search","success":true}],"toolsMs":570,"turn":4},{"llm":{"predicted_ms":54503.748,"predicted_n":1009,"prompt_ms":200638.664,"prompt_n":30391},"toolCalls":[{"duration_ms":290,"name":"hf_fs","success":true},{"duration_ms":1148,"name":"web_search_exa","success":true}],"toolsMs":1438,"turn":5},{"llm":{"predicted_ms":822396.305,"predicted_n":13614,"prompt_ms":103955.357,"prompt_n":12776},"toolCalls":[],"toolsMs":0,"turn":6}],"toolCalls":[{"duration_ms":66,"name":"hf_whoami","success":true},{"duration_ms":87,"name":"hub_repo_search","success":true},{"duration_ms":272,"name":"web_fetch_exa","success":true},{"duration_ms":182,"name":"hub_repo_details","success":true},{"duration_ms":187,"name":"hf_fs","success":true},{"duration_ms":314,"name":"web_fetch_exa","success":true},{"duration_ms":522,"name":"hf_fs","success":true},{"duration_ms":999,"name":"web_search_exa","success":true},{"duration_ms":435,"name":"web_fetch_exa","success":true},{"duration_ms":135,"name":"hub_repo_search","success":true},{"duration_ms":290,"name":"hf_fs","success":true},{"duration_ms":1148,"name":"web_search_exa","success":true}],"toolCallsCount":12,"toolsMs":4637,"turns":6},"cache_n":104682,"predicted_ms":822396.305,"predicted_n":13614,"prompt_ms":103955.357,"prompt_n":12776},"toolCalls":[{"function":{"arguments":"{}","name":"hf_whoami"},"id":"dXV3isIJycttOyAX80rdAeq8PvcY8joj","type":"function"},{"function":{"arguments":"{\"query\":\"Qwen3.8-Flash-Next\",\"limit\":20}","name":"hub_repo_search"},"id":"Z4ST6dZ2ksAOYBvo9wAFgf961wQqvsVL","type":"function"},{"function":{"arguments":"{\"urls\":[\"https://unsloth.ai/docs/models/qwen3.8-next\", \"https://hermes-agent.nousresearch.com/docs/assets/files/llms-7240021af84660c2a79f9fdaf65e9e8f.txt\"],\"maxCharacters\":30000}","name":"web_fetch_exa"},"id":"iitY71ZpY7hslZy3WuossaFWWgMeO7Or","type":"function"}]},"type":"message"}
{"message":{"children":[],"content":"# Hugging Face authentication\n\nThe Hugging Face tools are being used anonymously and may be rate limited.\n\nVisit https://hf.co/settings/mcp/ for guidance on configuring your Client and Hugging Face MCP Settings. Go to https://hf.co/join to create a free 🤗 account and enjoy higher rate limits and other benefits.","convId":"incyjf36ghq","role":"tool","timestamp":1787790064146,"toolCallId":"dXV3isIJycttOyAX80rdAeq8PvcY8joj","type":"text","id":"jujd1cyul5m","parent":"qhbou3908dr"},"type":"message"}
{"message":{"children":[],"content":"Found 21 repositories across models, datasets matching query \"Qwen3.8-Flash-Next\".\n\n## Models (20)\n\n### Qwen/Qwen3.8-Flash-Next\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 2.6K | **Likes:** 3608 | **Trending Score:** 3549\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, conversational, license:other, eval-results, endpoints_compatible, region:us\n\n**Created:** 24 Aug, 2026\n**Link:** [https://hf.co/Qwen/Qwen3.8-Flash-Next](https://hf.co/Qwen/Qwen3.8-Flash-Next)\n\n---\n\n### unsloth/Qwen3.8-Flash-Next-GGUF\n\n**Task:** image-text-to-text | **Downloads:** 0 | **Likes:** 327 | **Trending Score:** 324\n\n**Tags:** gguf, unsloth, image-text-to-text, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, imatrix, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://hf.co/unsloth/Qwen3.8-Flash-Next-GGUF)\n\n---\n\n### Qwen/Qwen3.8-Flash-Next-FP8\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 451 | **Likes:** 92 | **Trending Score:** 92\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, fp8, region:us\n\n**Created:** 24 Aug, 2026\n**Link:** [https://hf.co/Qwen/Qwen3.8-Flash-Next-FP8](https://hf.co/Qwen/Qwen3.8-Flash-Next-FP8)\n\n---\n\n### RadixArk/Qwen3.8-Flash-Next-NVFP4\n\n**Task:** image-text-to-text | **Library:** Model Optimizer | **Downloads:** 263 | **Likes:** 28 | **Trending Score:** 28\n\n**Tags:** Model Optimizer, safetensors, qwen4_exp, ModelOpt, Qwen3.8, quantized, FP4, fp4, sglang, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, 8-bit, modelopt, region:us\n\n**Created:** 25 Aug, 2026\n**Link:** [https://hf.co/RadixArk/Qwen3.8-Flash-Next-NVFP4](https://hf.co/RadixArk/Qwen3.8-Flash-Next-NVFP4)\n\n---\n\n### unsloth/Qwen3.8-Flash-Next-FP8\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 0 | **Likes:** 27 | **Trending Score:** 27\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, fp8, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/unsloth/Qwen3.8-Flash-Next-FP8](https://hf.co/unsloth/Qwen3.8-Flash-Next-FP8)\n\n---\n\n### Inferact/Qwen3.8-Flash-Next-NVFP4\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 0 | **Likes:** 15 | **Trending Score:** 15\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, 8-bit, modelopt, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Inferact/Qwen3.8-Flash-Next-NVFP4](https://hf.co/Inferact/Qwen3.8-Flash-Next-NVFP4)\n\n---\n\n### Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF\n\n**Task:** image-text-to-text | **Downloads:** 0 | **Likes:** 9 | **Trending Score:** 9\n\n**Tags:** gguf, mixed-quant, qwen4exp, qwen3.8-flash-next, dgx-spark, ds4, ssd-offload, image-text-to-text, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF](https://hf.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF)\n\n---\n\n### AtomicChat/Qwen3.8-Flash-Next-GGUF\n\n**Task:** text-generation | **Library:** gguf | **Downloads:** 0 | **Likes:** 8 | **Trending Score:** 8\n\n**Tags:** gguf, atomic-chat, qwen, qwen3.8, flash-next, moe, multimodal, imatrix, quantized, llama.cpp, text-generation, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/AtomicChat/Qwen3.8-Flash-Next-GGUF](https://hf.co/AtomicChat/Qwen3.8-Flash-Next-GGUF)\n\n---\n\n### Sawfwair/Qwen3.8-Flash-Next-MLX-Mixed-2bit\n\n**Task:** image-text-to-text | **Library:** mlx | **Downloads:** 0 | **Likes:** 7 | **Trending Score:** 7\n\n**Tags:** mlx, safetensors, qwen4_exp, quantized, qwen4-exp, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, 4-bit, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Sawfwair/Qwen3.8-Flash-Next-MLX-Mixed-2bit](https://hf.co/Sawfwair/Qwen3.8-Flash-Next-MLX-Mixed-2bit)\n\n---\n\n### agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF\n\n**Task:** text-generation | **Library:** gguf | **Downloads:** 0 | **Likes:** 7 | **Trending Score:** 7\n\n**Tags:** gguf, rocmfp4, rocmfpx, vulkan, strix-halo, qwen4exp, text-generation, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF](https://hf.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF)\n\n---\n\n### unsloth/Qwen3.8-Flash-Next\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 2 | **Likes:** 5 | **Trending Score:** 5\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, conversational, license:other, endpoints_compatible, region:us\n\n**Created:** 25 Aug, 2026\n**Link:** [https://hf.co/unsloth/Qwen3.8-Flash-Next](https://hf.co/unsloth/Qwen3.8-Flash-Next)\n\n---\n\n### kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF\n\n**Task:** text-generation | **Library:** gguf | **Downloads:** 0 | **Likes:** 5 | **Trending Score:** 5\n\n**Tags:** gguf, rocmfp4, llama.cpp, strix-halo, gfx1151, rocm, amd, ryzen-ai-max, text-generation, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF](https://hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF)\n\n---\n\n### Vontra/Qwen3.8-Flash-Next-MLX-oQ2\n\n**Task:** image-text-to-text | **Library:** mlx | **Downloads:** 0 | **Likes:** 3 | **Trending Score:** 3\n\n**Tags:** mlx, safetensors, qwen4_exp, mlx-vlm, omlx, oq, qwen, qwen3.8, mixture-of-experts, vision-language, quantized, apple-silicon, 2-bit, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-oQ2](https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-oQ2)\n\n---\n\n### Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-GGUF\n\n**Task:** image-text-to-text | **Downloads:** 0 | **Likes:** 3 | **Trending Score:** 3\n\n**Tags:** gguf, mixed-quant, qwen4exp, qwen3.8-flash-next, dgx-spark, ds4, image-text-to-text, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-GGUF](https://hf.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-GGUF)\n\n---\n\n### vcruz305/Qwen3.8-Flash-Next-NVFP4\n\n**Task:** text-generation | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** qwen, qwen3.8, nvfp4, text-generation, base_model:Qwen/Qwen3.8-Flash-Next, base_model:finetune:Qwen/Qwen3.8-Flash-Next, license:apache-2.0, region:us\n\n**Created:** 25 Aug, 2026\n**Link:** [https://hf.co/vcruz305/Qwen3.8-Flash-Next-NVFP4](https://hf.co/vcruz305/Qwen3.8-Flash-Next-NVFP4)\n\n---\n\n### 0xKitkat/Qwen3.8-Flash-Next-GGUF\n\n**Task:** image-text-to-text | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** qwen, qwen3.8, gguf, quantization, multimodal, moe, image-text-to-text, en, zh, base_model:Qwen/Qwen3.8-Flash-Next, base_model:finetune:Qwen/Qwen3.8-Flash-Next, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/0xKitkat/Qwen3.8-Flash-Next-GGUF](https://hf.co/0xKitkat/Qwen3.8-Flash-Next-GGUF)\n\n---\n\n### Vontra/Qwen3.8-Flash-Next-MLX-4bit\n\n**Task:** image-text-to-text | **Library:** mlx | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** mlx, safetensors, qwen4_exp, mlx-vlm, omlx, qwen, qwen3.8, mixture-of-experts, vision-language, quantized, apple-silicon, 4-bit, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-4bit](https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-4bit)\n\n---\n\n### DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF\n\n**Task:** image-text-to-text | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** gguf, image-text-to-text, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, endpoints_compatible, region:us, conversational\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF](https://hf.co/DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF)\n\n---\n\n### Vontra/Qwen3.8-Flash-Next-MLX-oQ4-MTP\n\n**Task:** image-text-to-text | **Library:** mlx | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** mlx, safetensors, qwen4_exp, mlx-vlm, omlx, oq, mtp, speculative-decoding, qwen, qwen3.8, mixture-of-experts, vision-language, quantized, apple-silicon, 4-bit, image-text-to-text, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other\n*and 1 more...*\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-oQ4-MTP](https://hf.co/Vontra/Qwen3.8-Flash-Next-MLX-oQ4-MTP)\n\n---\n\n### primitive-ai/Qwen3.8-Flash-Next-NVFP4\n\n**Task:** image-text-to-text | **Library:** transformers | **Downloads:** 0 | **Likes:** 2 | **Trending Score:** 2\n\n**Tags:** transformers, safetensors, qwen4_exp, image-text-to-text, nvfp4, quantized, vllm, modelopt, qwen3.8, flash-next, single-gpu, speculative-decoding, conversational, base_model:Qwen/Qwen3.8-Flash-Next, base_model:quantized:Qwen/Qwen3.8-Flash-Next, license:other, endpoints_compatible, 8-bit, region:us\n\n**Created:** 26 Aug, 2026\n**Link:** [https://hf.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4](https://hf.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4)\n\n---\n\n## Datasets (1)\n\n### AtomicChat/Qwen3.8-Flash-Next-GGUF-metrics\n\n**Author:** AtomicChat | **Downloads:** 0 | **Likes:** 0 | **Trending Score:** 0\n\n**Tags:** size_categories:n<1K, modality:tabular, modality:text, region:us\n\n**Created:** 26 Aug, 2026\n**Last Modified:** 26 Aug, 2026\n**Link:** [https://hf.co/datasets/AtomicChat/Qwen3.8-Flash-Next-GGUF-metrics](https://hf.co/datasets/AtomicChat/Qwen3.8-Flash-Next-GGUF-metrics)\n\n---\n","convId":"incyjf36ghq","role":"tool","timestamp":1787790064239,"toolCallId":"Z4ST6dZ2ksAOYBvo9wAFgf961wQqvsVL","type":"text","id":"0mwaxd1eh6sp","parent":"qhbou3908dr"},"type":"message"}
{"message":{"children":["bw4fjhyn0l5"],"content":"# qwen3.8-next\nURL: https://unsloth.ai/docs/models/qwen3.8-next\n\n> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/qwen3.8-next.md).\n\n# Qwen3.8-Flash-Next: How to Run Locally\n\nQwen3.8-Flash-Next is a new open-weight, **125B parameter** MoE multimodal model from Qwen. Built on the new Qwen4 architecture, it supports a 262K context window and advanced reasoning. The model outperforms Claude-4.6-Opus (Max) and can run locally on devices with **75GB RAM**/unified memory with no GPU VRAM required. To run the model, use our [GGUFs](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) with our specific [llama.cpp PR](https://github.com/ggml-org/llama.cpp/pull/27742) for [Unsloth Desktop](/docs/desktop.md) as well. Thank you Qwen for day zero access.\n\n**1-bit is 75GB** and uses 4-bit for the Ngram / PLE. This is **79% smaller** than BF16 (355GB), and retains a **top-1% accuracy of 80%**.\n\n Run Qwen3.8-Flash-Next Guide Download Unsloth \n\n{% hint style=\"success\" %}\nQwen3.8-Flash-Next is now available to run locally! More quants coming...\n{% endhint %}\n\n### :gear: Usage Guide\n\nWhether you run **Qwen3.8-Flash-Next** on a CPU with system RAM or on a GPU with VRAM may make relatively little difference. Its unique architecture allows inference using RAM or unified memory to achieve performance closer to that of GPU VRAM than is typical for other models. This makes it particularly well suited to Macs, NVIDIA DGX Spark systems, and other devices with large memory capacities.\n\nYou will need at least **75 GB of RAM or unified memory** to run the model. Its smallest 1-bit quantized version is larger than usual because of new Ngram layers or per layer embeddings which is like a lookup table. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models. You can also offload the PLE / Ngram layer to SSD and use mmap which allows less usage of CPU and GPU VRAM.\n\n#### Qwen3.8-Flash-Next Requirements:\n\nThe smallest quant works on 75GB RAM so it's best to have a 96GB RAM/unified memory device.\\\n**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)\n\n 1-bit 2-bit 3-bit 4-bit 5-bit 8-bit BF16 75 GB 79 GB 90 GB 112 GB 200 GB 270 GB 355 GB \n\n### Recommended Settings\n\nQwen3.8-Flash-Next is a **hybrid thinking** model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can [adjust the thinking effort](#thinking--preserve-thinking):\n\n| Parameter | Thinking Mode | Instruct (non-thinking) Mode |\n| -------------------- | ------------- | ---------------------------- |\n| `temperature` | 1.0 | 0.7 |\n| `top_p` | 0.95 | 0.80 |\n| `top_k` | 20 | 20 |\n| `min_p` | 0.0 | 0.0 |\n| `presence_penalty` | 0.0 | 1.5 |\n| `repetition_penalty` | 1.0 | 1.0 |\n\n* Context length = up to `262,144`\n* Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`\n* Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`\n\n### 💡 Thinking + Preserve Thinking\n\n{% columns %}\n{% column %}\nQwen3.8-Flash-Next has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. [Unsloth](#run-qwen3.8-in-unsloth-desktop) has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):\n{% endcolumn %}\n\n{% column %}\n\n \n{% endcolumn %}\n{% endcolumns %}\n\nQwen3.8-Flash-Next comes with support for `reasoning_effort`, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:\n\n* `xhigh` (default): for complex tasks demanding thorough analysis\n* `medium`: balancing accuracy and speed\n* `low`: efficient reasoning optimizing for speed and cost\n* none\n\n{% hint style=\"warning\" %}\nTo change[ thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking) effort in `unsloth run` or `llama-server`, use `--chat-template-kwargs '{\"reasoning_effort\":\"medium\"}'`\n\nIf you're on **Windows** Powershell, use: `--chat-template-kwargs \"{\\\"reasoning_effort\\\":\\\"medium\\\"}\"`\n\nChange `medium` to your desired reasoning level.\n{% endhint %}\n\n### Quantization Analysis\n\nWe ran KLD for Qwen3.8-Flash quants, and show that 80% top-1% accuracy recovery is possible with 79% less disk space usage. The new architecture uses PLE / Ngrams, and these are not quantized that heavily (4-bit minimum) since they have random access pattern, and quantizing them heavily will damage the model.\n\n \n\n| quant | size\\_gb | mean\\_kld | same\\_top\\_pct |\n| ------------ | -------- | --------- | -------------- |\n| UD-Q4\\_K\\_XL | 111.3 | 0.044715 | 93.481 |\n| UD-IQ4\\_XS | 93.7 | 0.079162 | 91.089 |\n| UD-Q3\\_K\\_XL | 90 | 0.099694 | 90.387 |\n| UD-IQ3\\_XXS | 82 | 0.156505 | 87.57 |\n| UD-Q2\\_K\\_XL | 78.9 | 0.213343 | 85.163 |\n| UD-IQ1\\_M | 74.5 | 0.302159 | 82.396 |\n| UD-IQ1\\_S | 72.5 | 0.37514 | 80.239 |\n\n## Run Qwen3.8-Flash-Next Guide\n\nYou can soon run Qwen3.8-Flash-Next in Unsloth Desktop and llama.cpp with our specific PR. Feel free to change quantization type.\n\n* Hugging Face: [Qwen3.8-Flash-Next-**GGUF**](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)\n* ModelScope: soon\n\n Run in Unsloth Desktop Run in llama.cpp \n\n### 🦥 Run Qwen3.8-Flash-Next in Unsloth\n\nQwen3.8-Flash-Next will soon be able to run [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:\n\n{% columns %}\n{% column %}\n\n* Search, download, [run GGUFs](/docs/new/studio.md#run-models-locally) and safetensor models\n* [**Self-healing** tool calling](/docs/new/studio/chat.md#auto-healing-tool-calling) + **web search**\n* [**Code execution**](/docs/desktop.md#code-execution) (Python, Bash)\n* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)\n* Fast CPU + GPU inference via MLX and llama.cpp\n* [Train LLMs](/docs/new/studio.md#no-code-training) 2x faster with 70% less VRAM\n {% endcolumn %}\n\n{% column %}\n\n \n{% endcolumn %}\n{% endcolumns %}\n\n{% stepper %}\n{% step %}\n\n#### Install Unsloth\n\nThe easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop.md). Works on [macOS](/docs/get-started/install/mac.md), [Windows](/docs/get-started/install/windows-installation.md), and [Linux](/docs/get-started/install/linux.md).\n\n Download Unsloth \n\n*:apple: [Download for macOS](https://unsloth.ai/download/mac)\n*:windows: [Download for Windows](https://unsloth.ai/download/windows)\n*:linux: [Download for Linux](https://unsloth.ai/download/linux)\n\nOr, if you prefer to install manually:\n\nMacOS, Linux, WSL:\n\n```bash\ncurl -fsSL https://unsloth.ai/install.sh | sh\n```\n\nWindows PowerShell:\n\n```bash\nirm https://unsloth.ai/install.ps1 | iex\n```\n\n{% endstep %}\n\n{% step %}\n\n#### Search and download Qwen3.8-Flash-Next\n\nGo to [Unsloth Chat](/docs/new/studio/chat.md) or Model hub and search for Qwen3.8-Flash in the search bar and download your desired model and quant.\n{% endstep %}\n\n{% step %}\n\n#### Run Qwen3.8-Flash-Next\n\nInference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.\n\nFor more information, you can view our [Unsloth inference guide](/docs/new/studio/chat.md).\n\nFor example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.\n\n Dynamic 1-bit 397GB 91% smaller GGUF of Qwen3.8 2.4T in Unsloth Desktop \n{% endstep %}\n\n{% step %}\n\n#### Serve Qwen3.8-Flash-Next with Unsloth API\n\nYou can use `unsloth run` command and serve Qwen3.8 via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api.md) or [unsloth start](/docs/integrations/unsloth-start.md).\n\n{% code overflow=\"wrap\" %}\n\n```bash\nunsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL\n```\n\n{% endcode %}\n{% endstep %}\n\n{% step %}\n\n#### Unsloth is now ready\n\nYou can also do many other things with Qwen3.8-Flash-Next via Unsloth Desktop like:\n\n* **Connect tools:** [Claude Code](/docs/basics/claude-code.md), [Codex](/docs/basics/codex.md), [web search](/docs/new/studio/chat.md#advanced-web-search), [MCP](/docs/basics/mcp.md) and more\n* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning.md), and more\n* **Generate media:** Create and train [images](/docs/basics/diffusion-image.md), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning.md) locally\n\n \n{% endstep %}\n{% endstepper %}\n\n### :llama: Run Qwen3.8-Flash-Next in llama.cpp\n\n{% stepper %}\n{% step %}\nWe need to use our specific llama.cpp PR [here](https://github.com/ggml-org/llama.cpp/pull/27742). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.\n\n```bash\napt-get update\napt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y\ngit clone https://github.com/ggml-org/llama.cpp/ && git -C llama.cpp fetch origin pull/27742/head:pr-27742 && git -C llama.cpp checkout pr-27742\ncmake llama.cpp -B llama.cpp/build \\\n    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON\ncmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split\ncp llama.cpp/build/bin/llama-* llama.cpp\n```\n\n{% endstep %}\n\n{% step %}\nTo run the model, you can do:\n\n{% code overflow=\"wrap\" %}\n\n```bash\npip install -U \"huggingface_hub[cli]\"\nhf download unsloth/Qwen3.8-Flash-Next-GGUF \\\n    --local-dir unsloth/Qwen3.8-Flash-Next-GGFF \\\n    --include \"*IQ1_S*\" # Use \"*IQ2_XXS*\" for 2-bit\n```\n\n{% endcode %}\n{% endstep %}\n\n{% step %}\nThen to run it:\n\n{% code overflow=\"wrap\" %}\n\n```bash\n./llama.cpp/llama-cli \\\n    --model unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf \\\n    --temp 1.0 \\\n    --top-p 0.95 \\\n    --top-k 20 \\\n    --min-p 0.0\n```\n\n{% endcode %}\n{% endstep %}\n{% endstepper %}\n\n### 📊 Benchmarks\n\nFor GGUF quantization benchmarks you can see above for our [quantization analysis](#quantization-analysis) or [Dynamic V3.0 article](/docs/basics/dynamic-3.0-ggufs.md).\n\n \n\n---\n\n# Agent Instructions\nThis documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.\n\n## Querying This Documentation\nIf you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.\n\nPerform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:\n\n```\nGET https://unsloth.ai/docs/models/qwen3.8-next.md?ask=<question>&goal=<endgoal>\n```\n\n`ask` is the immediate question: it should be specific, self-contained, and written in natural language.\n`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.\n\nThe response will contain a direct answer to the question and relevant excerpts and sources from the documentation.\n\nUse this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.\n\n\n# llms-7240021af84660c2a79f9fdaf65e9e8f.txt\nURL: https://hermes-agent.nousresearch.com/docs/assets/files/llms-7240021af84660c2a79f9fdaf65e9e8f.txt\n\n# Hermes Agent\n\n> The self-improving AI agent built by Nous Research. A terminal-native autonomous coding and task agent with persistent memory, agent-created skills, and a messaging gateway that lives on 21+ messaging platforms — 19 native to the gateway plus IRC and Microsoft Teams via plugins (Telegram, Discord, Slack, SMS, Matrix, ...). Runs on local, Docker, SSH, Daytona, Modal, or Singularity backends. Works with Nous Portal, OpenRouter, OpenAI, Anthropic, Google, or any OpenAI-compatible endpoint.\n\nInstall: `curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash` (Linux, macOS, WSL2, Termux)\n\nRepo: https://github.com/NousResearch/hermes-agent\n\n## Getting Started\n\n- [Installation](https://hermes-agent.nousresearch.com/docs/getting-started/installation): Install Hermes Agent on Linux, macOS, WSL2, native Windows, or Android via Termux\n- [Quickstart](https://hermes-agent.nousresearch.com/docs/getting-started/quickstart): Your first conversation with Hermes Agent — from install to chatting in under 5 minutes\n- [Learning Path](https://hermes-agent.nousresearch.com/docs/getting-started/learning-path): Choose your learning path through the Hermes Agent documentation based on your experience level and goals.\n- [Updating](https://hermes-agent.nousresearch.com/docs/getting-started/updating): How to update Hermes Agent to the latest version or uninstall it\n- [Termux (Android)](https://hermes-agent.nousresearch.com/docs/getting-started/termux): Run Hermes Agent directly on an Android phone with Termux\n- [Nix Setup](https://hermes-agent.nousresearch.com/docs/getting-started/nix-setup): Install and deploy Hermes Agent with Nix — from quick `nix run` to fully declarative NixOS module with container mode\n- [Platform Support](https://hermes-agent.nousresearch.com/docs/getting-started/platform-support): Which operating systems, distribution methods, and features Hermes Agent supports.\n\n## Using Hermes\n\n- [CLI](https://hermes-agent.nousresearch.com/docs/user-guide/cli): Master the Hermes Agent terminal interface — commands, keybindings, personalities, and more\n- [TUI (Ink terminal UI)](https://hermes-agent.nousresearch.com/docs/user-guide/tui): Launch the modern terminal UI for Hermes — mouse-friendly, rich overlays, and non-blocking input.\n- [Configuration](https://hermes-agent.nousresearch.com/docs/user-guide/configuration): Configure Hermes Agent — config.yaml, providers, models, API keys, and more\n- [Configuring Models](https://hermes-agent.nousresearch.com/docs/user-guide/configuring-models)\n- [Sessions](https://hermes-agent.nousresearch.com/docs/user-guide/sessions): Session persistence, resume, search, management, and per-platform session tracking\n- [Profiles](https://hermes-agent.nousresearch.com/docs/user-guide/profiles)\n- [Git Worktrees](https://hermes-agent.nousresearch.com/docs/user-guide/git-worktrees): Run multiple Hermes agents safely on the same repository using git worktrees and isolated checkouts\n- [Docker Backend](https://hermes-agent.nousresearch.com/docs/user-guide/docker): Running Hermes Agent in Docker and using Docker as a terminal backend\n- [Security](https://hermes-agent.nousresearch.com/docs/user-guide/security): Security model, dangerous command approval, user authorization, container isolation, and production deployment best practices\n- [Checkpoints & Rollback](https://hermes-agent.nousresearch.com/docs/user-guide/checkpoints-and-rollback): Filesystem safety nets for destructive operations using shadow git repos and automatic snapshots\n- [Bot Mode](https://hermes-agent.nousresearch.com/docs/user-guide/bot-mode): Turn your Hermes profiles into a roster of named Bots — each with its own chat, role, model, memory, skills, and avatar. Bots run routines, share group chats, and message each other.\n- [Hermes Desktop](https://hermes-agent.nousresearch.com/docs/user-guide/desktop): The native Hermes desktop app — a polished experience for chatting with Hermes, with streaming tool output, side-by-side previews, a file browser, voice, cron, profiles, skills, and settings. macOS, Windows, and Linux.\n- [Egress proxy](https://hermes-agent.nousresearch.com/docs/user-guide/egress)\n- [Egress credential-injection proxy (iron-proxy)](https://hermes-agent.nousresearch.com/docs/user-guide/egress/iron-proxy)\n- [Import from Other Agents](https://hermes-agent.nousresearch.com/docs/user-guide/import-from-other-agents): One-command import of a Claude Code (~/.claude) or OpenAI Codex CLI (~/.codex) setup into Hermes — instructions, allowlists, MCP servers, skills, and memories.\n- [Managed Scope](https://hermes-agent.nousresearch.com/docs/user-guide/managed-scope): Administrator-pinned, user-immutable config and secrets via a system-level managed directory\n- [Connecting Desktop to Many Hermes Instances](https://hermes-agent.nousresearch.com/docs/user-guide/multi-connection-desktop)\n- [Running Many Gateways at Once](https://hermes-agent.nousresearch.com/docs/user-guide/multi-profile-gateways)\n- [Profile Distributions: Share a Whole Agent](https://hermes-agent.nousresearch.com/docs/user-guide/profile-distributions)\n- [Secrets](https://hermes-agent.nousresearch.com/docs/user-guide/secrets)\n- [Bitwarden Secrets Manager](https://hermes-agent.nousresearch.com/docs/user-guide/secrets/bitwarden)\n- [Command Helper Secret Source](https://hermes-agent.nousresearch.com/docs/user-guide/secrets/command)\n- [1Password](https://hermes-agent.nousresearch.com/docs/user-guide/secrets/onepassword)\n- [Google Workspace — Gmail, Calendar, Drive, Sheets & Docs](https://hermes-agent.nousresearch.com/docs/user-guide/skills/google-workspace): Send email, manage calendar events, search Drive, read/write Sheets, and access Docs — all through OAuth2-authenticated Google APIs\n- [Which File Does What?](https://hermes-agent.nousresearch.com/docs/user-guide/which-file-does-what): SOUL.md vs USER.md vs MEMORY.md vs AGENTS.md — a one-page map of the agent's files, who writes each one, and when the agent actually sees them\n- [Windows (Native) Guide](https://hermes-agent.nousresearch.com/docs/user-guide/windows-native): Run Hermes Agent natively on Windows 10 / 11 — install, feature matrix, UTF-8 console, Git Bash, gateway as a Scheduled Task, editor handling, PATH, uninstall, and common pitfalls\n- [Windows (WSL2) Guide](https://hermes-agent.nousresearch.com/docs/user-guide/windows-wsl-quickstart): Run Hermes Agent on Windows via WSL2 — setup, filesystem access between Windows and Linux, networking, and common pitfalls\n\n## Core Features\n\n- [Features Overview](https://hermes-agent.nousresearch.com/docs/user-guide/features/overview)\n- [Tools](https://hermes-agent.nousresearch.com/docs/user-guide/features/tools): Overview of Hermes Agent's tools — what's available, how toolsets work, and terminal backends\n- [Skills System](https://hermes-agent.nousresearch.com/docs/user-guide/features/skills): On-demand knowledge documents — progressive disclosure, agent-managed skills, and the Skills Hub\n- [Curator](https://hermes-agent.nousresearch.com/docs/user-guide/features/curator): Background maintenance for agent-created skills — usage tracking, staleness, archival, and LLM-driven review\n- [Memory](https://hermes-agent.nousresearch.com/docs/user-guide/features/memory): How Hermes Agent remembers across sessions — MEMORY.md, USER.md, and session search\n- [Memory Providers](https://hermes-agent.nousresearch.com/docs/user-guide/features/memory-providers): External memory provider plugins — Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory\n- [Context Files](https://hermes-agent.nousresearch.com/docs/user-guide/features/context-files): Project context files — .hermes.md, AGENTS.md, CLAUDE.md, global SOUL.md, and .cursorrules — automatically injected into every conversation\n- [Context References](https://hermes-agent.nousresearch.com/docs/user-guide/features/context-references): Inline @-syntax for attaching files, folders, git diffs, and URLs directly into your messages\n- [Personality & SOUL.md](https://hermes-agent.nousresearch.com/docs/user-guide/features/personality): Customize Hermes Agent's personality with a global SOUL.md, built-in personalities, and custom persona definitions\n- [Plugins](https://hermes-agent.nousresearch.com/docs/user-guide/features/plugins): Extend Hermes with custom tools, hooks, and integrations via the plugin system\n- [Built-in Plugins](https://hermes-agent.nousresearch.com/docs/user-guide/features/built-in-plugins): Plugins shipped with Hermes Agent that run automatically via lifecycle hooks — disk-cleanup and friends\n- [Codex App-Server Runtime (optional)](https://hermes-agent.nousresearch.com/docs/user-guide/features/codex-app-server-runtime)\n- [Computer Use](https://hermes-agent.nousresearch.com/docs/user-guide/features/computer-use)\n- [Deliverable Mode (Artifacts in Chat)](https://hermes-agent.nousresearch.com/docs/user-guide/features/deliverable-mode): How the agent ships generated charts, PDFs, spreadsheets, and other files as native attachments in messaging platforms.\n- [Document Extraction](https://hermes-agent.nousresearch.com/docs/user-guide/features/document-extraction): How read_file converts PDFs, Office documents, and notebooks to text — and what to do when a PDF is scanned images\n- [Extending the Dashboard](https://hermes-agent.nousresearch.com/docs/user-guide/features/extending-the-dashboard): Build themes and plugins for the Hermes web dashboard — palettes, typography, layouts, custom tabs, shell slots, page-scoped slots, and backend API routes\n- [Session Heartbeats](https://hermes-agent.nousresearch.com/docs/user-guide/features/heartbeat): A recurring prompt that re-enters your current session whenever it's idle — /heartbeat every 10m Check the deployment.\n- [Kanban worker lanes](https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-worker-lanes)\n- [Recurring Loops](https://hermes-agent.nousresearch.com/docs/user-guide/features/loops): Re-run a prompt on a recurring interval inside your session — Hermes' take on Claude Code's /loop.\n- [LSP — Semantic Diagnostics](https://hermes-agent.nousresearch.com/docs/user-guide/features/lsp): Real language servers (pyright, gopls, rust-analyzer, …) wired into the post-write lint check used by write_file and patch.\n- [Mixture of Agents](https://hermes-agent.nousresearch.com/docs/user-guide/features/mixture-of-agents): Create named MoA presets that appear as selectable models under the Mixture of Agents provider\n- [Pets (Petdex Mascots)](https://hermes-agent.nousresearch.com/docs/user-guide/features/pets): Adopt an animated mascot that reacts to agent activity across the CLI, TUI, and desktop app\n- [Skins & Themes](https://hermes-agent.nousresearch.com/docs/user-guide/features/skins): Customize the Hermes CLI with built-in and user-defined skins\n- [Spotify](https://hermes-agent.nousresearch.com/docs/user-guide/features/spotify)\n- [Subscription Proxy](https://hermes-agent.nousresearch.com/docs/user-guide/features/subscription-proxy): Use your Nous Portal subscription (or other OAuth provider) as an OpenAI-compatible endpoint for external apps\n- [Nous Tool Gateway](https://hermes-agent.nousresearch.com/docs/user-guide/features/tool-gateway): One subscription, every tool. Web search, image generation, TTS, and cloud browsers — all routed through Nous Portal with no extra API keys.\n- [Tool Search](https://hermes-agent.nousresearch.com/docs/user-guide/features/tool-search)\n- [Wake Word](https://hermes-agent.nousresearch.com/docs/user-guide/features/wake-word): Hands-free 'Hey Hermes' wake word — start a voice session by speaking, the 'Hey Siri' way\n- [Hermes Web Dashboard](https://hermes-agent.nousresearch.com/docs/user-guide/features/web-dashboard): Browser-based administration panel for managing configuration, API keys, MCP servers, messaging pairing, webhooks, the gateway, memory, credentials, sessions, logs, analytics, cron jobs, and skills\n- [Web Search & Extract](https://hermes-agent.nousresearch.com/docs/user-guide/features/web-search): Search the web and extract page content with multiple backend providers — including free self-hosted SearXNG.\n- [X (Twitter) Search](https://hermes-agent.nousresearch.com/docs/user-guide/features/x-search): Search X (Twitter) posts and threads from within the agent using xAI's built-in x_search Responses tool — works with either a SuperGrok OAuth login or an XAI_API_KEY.\n\n## Automation\n\n- [Cron Jobs](https://hermes-agent.nousresearch.com/docs/user-guide/features/cron): Schedule automated tasks with natural language, manage them with one cron tool, and attach one or more skills\n- [Delegation](https://hermes-agent.nousresearch.com/docs/user-guide/features/delegation): Spawn isolated child agents for parallel workstreams with delegate_task\n- [Kanban Multi-Agent](https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban): Durable SQLite-backed task board for coordinating multiple Hermes profiles\n- [Kanban Tutorial](https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-tutorial)\n- [Persistent Goals](https://hermes-agent.nousresearch.com/docs/user-guide/features/goals): Set a standing goal and let Hermes keep working across turns until it's done. Our take on the Ralph loop.\n- [Code Execution](https://hermes-agent.nousresearch.com/docs/user-guide/features/code-execution): Programmatic Python execution with RPC tool access — collapse multi-step workflows into a single turn\n- [Hooks](https://hermes-agent.nousresearch.com/docs/user-guide/features/hooks): Run custom code at key lifecycle points — log activity, send alerts, post to webhooks\n- [Batch Processing](https://hermes-agent.nousresearch.com/docs/user-guide/features/batch-processing): Generate agent trajectories at scale — parallel processing, checkpointing, and toolset distributions\n\n## Media & Web\n\n- [Voice Mode](https://hermes-agent.nousresearch.com/docs/user-guide/features/voice-mode): Real-time voice conversations with Hermes Agent — CLI, Telegram, Discord (DMs, text channels, and voice channels)\n- [Browser](https://hermes-agent.nousresearch.com/docs/user-guide/features/browser): Control browsers with multiple providers, local Chromium-family browsers via CDP, or cloud browsers for web interaction, form filling, scraping, and more.\n- [Vision](https://hermes-agent.nousresearch.com/docs/user-guide/features/vision): Paste images from your clipboard into the Hermes CLI for multimodal vision analysis.\n- [Image Generation](https://hermes-agent.nousresearch.com/docs/user-guide/features/image-generation): Generate images via FAL.ai — 11 models including FLUX 2, GPT Image (1.5 & 2), Nano Banana Pro, Ideogram, Recraft V4 Pro, Krea 2, and more, selectable via `hermes tools`.\n- [Text-to-Speech](https://hermes-agent.nousresearch.com/docs/user-guide/features/tts): Text-to-speech and voice message transcription across all platforms\n\n## Messaging Platforms\n\n- [Overview](https://hermes-agent.nousresearch.com/docs/user-guide/messaging): Chat with Hermes from Telegram, Discord, Slack, WhatsApp, Signal, SMS, Email, Home Assistant, Mattermost, Matrix, DingTalk, Yuanbao, Microsoft Teams, LINE, Raft, Webhooks, or any OpenAI-compatible frontend via the API server — architecture and setup overview\n- [Telegram](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/telegram): Set up Hermes Agent as a Telegram bot\n- [Discord](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/discord): Set up Hermes Agent as a Discord bot\n- [Slack](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/slack): Set up Hermes Agent as a Slack bot using Socket Mode\n- [WhatsApp](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/whatsapp): Set up Hermes Agent as a WhatsApp bot via the built-in Baileys bridge\n- [Signal](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/signal): Set up Hermes Agent as a Signal messenger bot via signal-cli daemon\n- [Email](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/email): Set up Hermes Agent as an email assistant via IMAP/SMTP\n- [SMS](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/sms): Set up Hermes Agent as an SMS chatbot via Twilio\n- [Matrix](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/matrix): Set up Hermes Agent as a Matrix bot\n- [Mattermost](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/mattermost): Set up Hermes Agent as a Mattermost bot\n- [Home Assistant](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/homeassistant): Control your smart home with Hermes Agent via Home Assistant integration.\n- [Webhooks](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/webhooks): Receive events from GitHub, GitLab, and other services to trigger Hermes agent runs\n- [A2A (Agent-to-Agent)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/a2a)\n- [BlueBubbles (iMessage)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/bluebubbles)\n- [Buzz](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/buzz)\n- [DingTalk](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/dingtalk): Set up Hermes Agent as a DingTalk chatbot\n- [Feishu / Lark](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/feishu): Set up Hermes Agent as a Feishu or Lark bot\n- [Google Chat](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/google_chat): Set up Hermes Agent as a Google Chat bot using Cloud Pub/Sub\n- [IRC](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/irc)\n- [LINE](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/line): Set up Hermes Agent as a LINE Messaging API bot\n- [Microsoft Graph Webhook Listener](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/msgraph-webhook): Receive Microsoft Graph change notifications (meetings, calendar, chat, etc.) in Hermes\n- [ntfy](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/ntfy)\n- [Open WebUI](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/open-webui): Connect Open WebUI to Hermes Agent via the OpenAI-compatible API server\n- [Photon iMessage](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/photon)\n- [QQ Bot](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/qqbot)\n- [Raft](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/raft): Connect Hermes Agent to Raft as an external agent via wake-channel bridge\n- [Hermes Relay](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/relay): Connect Hermes to messaging platforms through a relay connector that owns the platform credentials — enrollment, capabilities, config, and troubleshooting\n- [SimpleX Chat](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/simplex)\n- [Microsoft Teams](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/teams): Set up Hermes Agent as a Microsoft Teams bot\n- [Teams Meetings](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/teams-meetings): Set up the Microsoft Teams meeting summary pipeline with Microsoft Graph webhooks\n- [WeCom (Enterprise WeChat)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/wecom): Connect Hermes Agent to WeCom via the AI Bot WebSocket gateway\n- [WeCom Callback (Self-Built App)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/wecom-callback)\n- [Weixin (WeChat)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/weixin): Connect Hermes Agent to personal WeChat accounts via the iLink Bot API\n- [WhatsApp Business (Cloud API)](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/whatsapp-cloud): Set up Hermes Agent as a WhatsApp bot via Meta's official Business Cloud API\n- [Yuanbao](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/yuanbao): Connect Hermes Agent to the Yuanbao enterprise messaging platform via WebSocket gateway\n\n## Integrations\n\n- [Integrations Overview](https://hermes-agent.nousresearch.com/docs/integrations)\n- [Providers](https://hermes-agent.nousresearch.com/docs/integrations/providers)\n- [MCP (Model Context Protocol)](https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp): Connect Hermes Agent to external tool servers via MCP — and control exactly which MCP tools Hermes loads\n- [ACP (Agent Context Protocol)](https://hermes-agent.nousresearch.com/docs/user-guide/features/acp): Use Hermes Agent inside ACP-compatible editors and collaboration platforms\n- [API Server](https://hermes-agent.nousresearch.com/docs/user-guide/features/api-server): Expose hermes-agent as an OpenAI-compatible API for any frontend\n- [Honcho Memory](https://hermes-agent.nousresearch.com/docs/user-guide/features/honcho): AI-native persistent memory via Honcho — dialectic reasoning, multi-agent user modeling, and deep personalization\n- [Provider Routing](https://hermes-agent.nousresearch.com/docs/user-guide/features/provider-routing): Configure OpenRouter or Nous Portal provider preferences to optimize for cost, speed, or quality.\n- [Fallback Providers](https://hermes-agent.nousresearch.com/docs/user-guide/features/fallback-providers): Configure automatic failover to backup LLM providers when your primary model is unavailable.\n- [Credential Pools](https://hermes-agent.nousresearch.com/docs/user-guide/features/credential-pools): Pool multiple API keys or OAuth tokens per provider for automatic rotation and rate limit recovery.\n- [Buzz Integration](https://hermes-agent.nousresearch.com/docs/integrations/buzz): All three ways to connect Hermes Agent to Buzz — Block's Nostr-based human+agent workspace\n- [Nous Portal](https://hermes-agent.nousresearch.com/docs/integrations/nous-portal): One subscription, 300+ frontier models, and the Tool Gateway — the recommended way to run Hermes Agent\n\n## Guides & Tutorials\n\n- [Tips & Best Practices](https://hermes-agent.nousresearch.com/docs/guides/tips): Practical advice to get the most out of Hermes Agent — prompt tips, CLI shortcuts, context files, memory, cost optimization, and security\n- [Local LLMs on Mac](https://hermes-agent.nousresearch.com/docs/guides/local-llm-on-mac): Set up a local OpenAI-compatible LLM server on macOS with llama.cpp or MLX, including model selection, memory optimization, and real benchmarks on Apple Silicon\n- [Daily Briefing Bot](https://hermes-agent.nousresearch.com/docs/guides/daily-briefing-bot): Build an automated daily briefing bot that researches topics, summarizes findings, and delivers them to Telegram or Discord every morning\n- [Team Telegram Assistant](https://hermes-agent.nousresearch.com/docs/guides/team-telegram-assistant): Step-by-step guide to setting up a Telegram bot that your whole team can use for code help, research, system admin, and more\n- [Use Hermes as a Python Library](https://hermes-agent.nousresearch.com/docs/guides/python-library): Embed AIAgent in your own Python scripts, web apps, or automation pipelines — no CLI required\n- [Use MCP with Hermes](https://hermes-agent.nousresearch.com/docs/guides/use-mcp-with-hermes): A practical guide to connecting MCP servers to Hermes Agent, filtering their tools, and using them safely in real workflows\n- [Use Voice Mode with Hermes](https://hermes-agent.nousresearch.com/docs/guides/use-voice-mode-with-hermes): A practical guide to setting up and using Hermes voice mode across CLI, Telegram, Discord, and Discord voice channels\n- [Use SOUL.md with Hermes](https://hermes-agent.nousresearch.com/docs/guides/use-soul-with-hermes): How to use SOUL.md to shape Hermes Agent's default voice, what belongs there, and how it differs from AGENTS.md and /personality\n- [Automate with Cron](https://hermes-agent.nousresearch.com/docs/guides/automate-with-cron): Real-world automation patterns using Hermes cron — monitoring, reports, pipelines, and multi-skill workflows\n- [Work with Skills](https://hermes-agent.nousresearch.com/docs/guides/work-with-skills): Find, install, use, and create skills — on-demand knowledge that teaches Hermes new workflows\n- [Delegation Patterns](https://hermes-agent.nousresearch.com/docs/guides/delegation-patterns): When and how to use subagent delegation — patterns for parallel research, code review, and multi-file work\n- [GitHub PR Review Agent](https://hermes-agent.nousresearch.com/docs/guides/github-pr-review-agent): Build an automated AI code reviewer that monitors your repos, reviews pull requests, and delivers feedback — hands-free\n- [Give Your Agent Its Own Email Address](https://hermes-agent.nousresearch.com/docs/guides/agent-email-address): Set up a dedicated mailbox your agent can read and send from using the bundled Himalaya skill, with a cron polling pattern and safety notes\n- [Automation Blueprints](https://hermes-agent.nousresearch.com/docs/guides/automation-blueprints): Ready-to-use automation blueprints — scheduled tasks, GitHub event triggers, API webhooks, and multi-skill workflows\n- [AWS Bedrock](https://hermes-agent.nousresearch.com/docs/guides/aws-bedrock): Use Hermes Agent with Amazon Bedrock — native Converse API, Anthropic SDK routing, OpenAI models via Bedrock Mantle, IAM authentication, Guardrails, and cross-region inference\n- [Microsoft Foundry](https://hermes-agent.nousresearch.com/docs/guides/azure-foundry): Use Hermes Agent with Microsoft Foundry — OpenAI-style and Anthropic-style endpoints, auto-detection of transport and deployed models\n- [Script-Only Cron Jobs (No LLM)](https://hermes-agent.nousresearch.com/docs/guides/cron-script-only): Classic watchdog cron jobs that skip the LLM entirely — a script runs on schedule and its stdout gets delivered to your messaging platform. Memory alerts, disk alerts, CI pings, periodic health checks.\n- [Cron Troubleshooting](https://hermes-agent.nousresearch.com/docs/guides/cron-troubleshooting): Diagnose and fix common Hermes cron issues — jobs not firing, delivery failures, skill loading errors, and performance problems\n- [Desktop Native Sign-In (RFC 8252)](https://hermes-agent.nousresearch.com/docs/guides/desktop-native-signin): How the Hermes Desktop app signs in to a gated gateway using your system browser and PKCE — no embedded webview, no session cookies\n- [Google Gemini](https://hermes-agent.nousresearch.com/docs/guides/google-gemini): Use Hermes Agent with Google Gemini — native AI Studio API, API-key setup, tool calling, streaming, and quota guidance\n- [Google Vertex AI](https://hermes-agent.nousresearch.com/docs/guides/google-vertex): Use Hermes Agent with Gemini on Google Cloud Vertex AI — OAuth2 service account or ADC, GCP billing and quotas, no static API key\n- [Run Hermes Locally with Ollama — Zero API Cost](https://hermes-agent.nousresearch.com/docs/guides/local-ollama-setup): Step-by-step guide to running Hermes Agent entirely on your own machine with Ollama and open-weight models like Gemma 4, no cloud API keys or paid subscriptions needed\n- [Manage Hermes Cloud with MCP](https://hermes-agent.nousresearch.com/docs/guides/manage-hermes-cloud-with-mcp): Connect Hermes Agent to the Nous Portal MCP server so your local agent can list, start, stop, and manage your Hermes Cloud instances conversationally\n- [Register a Microsoft Graph Application](https://hermes-agent.nousresearch.com/docs/guides/microsoft-graph-app-registration): Azure portal walkthrough for creating the app registration that powers the Teams meeting pipeline\n- [Migrate from OpenClaw](https://hermes-agent.nousresearch.com/docs/guides/migrate-from-openclaw): Complete guide to migrating your OpenClaw / Clawdbot setup to Hermes Agent — what gets migrated, how config maps, and what to check after.\n- [MiniMax OAuth](https://hermes-agent.nousresearch.com/docs/guides/minimax-oauth): Log into MiniMax via browser OAuth and use MiniMax-M2.7 models in Hermes Agent — no API key required\n- [OAuth over SSH / Remote Hosts](https://hermes-agent.nousresearch.com/docs/guides/oauth-over-ssh): How to complete browser-based OAuth (Spotify, MCP servers) when Hermes runs on a remote machine, container, or behind a jump box\n- [Operate the Teams Meeting Pipeline](https://hermes-agent.nousresearch.com/docs/guides/operate-teams-meeting-pipeline): Runbook, go-live checklist, and operator worksheet for the Microsoft Teams meeting pipeline\n- [Pipe Script Output to Messaging Platforms](https://hermes-agent.nousresearch.com/docs/guides/pipe-script-output): Send text from any shell script, cron job, CI hook, or monitoring daemon to Telegram, Discord, Slack, Signal, and other platforms using `hermes send`.\n- [Run Hermes Agent with Nous Portal](https://hermes-agent.nousresearch.com/docs/guides/run-hermes-with-nous-portal): Start-to-finish walkthrough: subscribe, set up, switch models, enable gateway tools, and verify routing\n- [Run Nemotron 3 Ultra free in Hermes Agent](https://hermes-agent.nousresearch.com/docs/guides/run-nemotron-3-ultra-free): Try NVIDIA Nemotron 3 Ultra on Nous Portal — free June 4–18 — with day 0 support in Hermes Agent\n- [Running Hermes on a Personal or Work Machine](https://hermes-agent.nousresearch.com/docs/guides/secure-hermes-on-a-work-machine): A security-posture walkthrough for running Hermes Agent on the machine you live on — what the defaults protect, how to tighten further, and how to undo mistakes\n- [Troubleshooting: \\\"My Agent Feels Dumber\\\"](https://hermes-agent.nousresearch.com/docs/guides/troubleshooting-agent-quality): A diagnostic checklist for when Hermes seems less capable than before or forgets things mid-session — model switches, context pressure, wrong context detection, and the frozen memory snapshot\n- [Automated GitHub PR Comments with Webhooks](https://hermes-agent.nousresearch.com/docs/guides/webhook-github-pr-review): Connect Hermes to GitHub so it automatically fetches PR diffs, reviews code changes, and posts comments — triggered by webhooks with no manual prompting\n- [xAI Grok OAuth (SuperGrok / X Premium+)](https://hermes-agent.nousresea","convId":"incyjf36ghq","role":"tool","timestamp":1787790064516,"toolCallId":"iitY71ZpY7hslZy3WuossaFWWgMeO7Or","type":"text","id":"kjsqlyed9d","parent":"qhbou3908dr"},"type":"message"}
{"message":{"children":["mr6pa4p42gq","2nwxh9g3hjp","sgk361otian"],"content":"Model confirmed real — `Qwen/Qwen3.8-Flash-Next` and `unsloth/Qwen3.8-Flash-Next-GGUF` both exist (created 24–26 Aug 2026), and the UD-IQ4_XS at 93.7 GB matches your number. Now let me pull the actual architecture and the llama.cpp PR so I'm reasoning about real flags, not remembered ones.\n\n","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790064521,"type":"text","id":"bw4fjhyn0l5","parent":"kjsqlyed9d","reasoningContent":"The model exists. Confirmed:\n- Qwen/Qwen3.8-Flash-Next (125B MoE, Qwen4 arch, 262K ctx, multimodal)\n- unsloth/Qwen3.8-Flash-Next-GGUF with UD-IQ4_XS at 93.7GB — matches user's stated size\n- llama.cpp PR #27742 needed\n\nKey facts from unsloth doc:\n- 125B params MoE, PLE (per-layer embeddings) / Ngram layers\n- Recommended sampling: thinking temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0, rep 1.0. Instruct: temp 0.7, top_p 0.8, presence_penalty 1.5.\n- reasoning_effort via --chat-template-kwargs\n- PLE/Ngram not quantized heavily (4-bit min)\n\nNow I need to check llama.cpp for the actual flags. Several flags in the user's runblock look suspicious/nonexistent:\n- `LLAMA_ATTN_ROT_DISABLE=1` — this env var is suspicious. RoPE should not be disabled. Actually for Qwen3-Next style models... hmm. This could be a hallucinated flag the user picked up. Need to verify.\n- `--fit off` — user later says \"Llama.cpp is going to be using fit on (the default)\". There's a `--fit` option? Recent llama.cpp added `--fit on/off` for auto-fitting params/context to memory. Need to check.\n- `--load-mode none` — doesn't sound right; there's `--mlock`, `--no-mmap`, `--no-warmup`, `--check-tensors`. Actually maybe `--no-hostalloc`. There might be `--llama-load`? Hmm. Actually recent llama.cpp has `--tensor-buffer-type`, `--nopopulate`? Let me check for actual flag list.\n- `--override-tensor \"per_layer_token_embd=CPU,ple_ngram_embd=CPU\"` — need to verify actual tensor names. Probably `ple.ngram_emb` or something. Also the separator for multiple overrides is a comma — that's right (`--override-tensor` uses comma-separated). Actually in llama.cpp it's `-ot` with format `tensor_name=DEVICES` and multiple separated by comma. Yes.\n- `--flash-attn on` — new syntax uses `--flash-attn on|off` (previously `--flash-attn` was a flag with `--flash-attn-v`). Recent versions support `--flash-attn` with options auto/on/off. Also `fa` virt.\n- `--split-mode layer` with 4 GPUs: for MoE models, `--split-mode none` with `--tensor-split` is often better; actually for tensor parallel there's now `--split-mode tp`. Need to check for this model — for MoE, `row`/`tensor` splits hurt. But `layer` split is generally right for llama.cpp now (it's the default).\n- `--parallel 1` fine.\n- `--cache-prompt` — that's `-np` related, fine, but for a shared server with sessions, prompt cache is keyed. Fine.\n- `--no-context-shift` — that's only relevant with `-c` and context shift sampler; fine, and default is off now.\n- `--threads 6` — for a 12c/24t CPU with mostly GPU offload, threads should be ~ physical cores. But this model has big CPU components (PLE on CPU), so thread tuning matters a lot.\n- Missing: `--no-mmproj` / vision. The model is multimodal (image-text-to-text). Hermes has vision. Might need `--mmproj`.\n- Missing: `--reasoning` / `--chat-template-kwargs` for reasoning_effort.\n- Missing: `--tool-call-parser` / `--chat-template` parser — critical for Hermes tool calling! llama.cpp has `--tool-call-parser` or `--chat-template-parser`? In llama.cpp server it's `--tool-call-parser` (newer: `--chat-template-parser`?). Actually there's `--parser` / `--tool-response-in-content`? Recent llama.cpp unified into `--chat-template-parser` with values like `qwen`, `deepseek`, `hermes`, `llama3`, etc. Need to check.\n- `--api-key nerdtastic` — weak, but local LAN.\n- `--host 192.168.1.61` fine.\n- `--parallel 1`, `--ctx-size 131072` — with 4x3090 (96GB VRAM total) and a 93.7GB model, KV cache of 131K will need room. Need to compute KV size. Qwen3-Next-like arch: hybrid attention (gated delta net + gated attention), so KV cache is small. 125B MoE with hybrid linear attention... The original Qwen3-Next-80B-A3B has 12 KV heads... hmm this is a fictional model, so I need to check actual arch details from the HF repo config.\n\nAlso key: does `--n-gpu-layers all` with `--fit on` conflict? And the user's model file path is \"GGFF\" (typo dir, but matches unsloth doc example which also says GGFF, funny).\n\nCritical issue: 93.7GB model on 96GB VRAM total across 4 GPUs. llama.cpp needs KV + compute buffers. So with n-gpu-layers all, it will NOT fit — that's why --fit matters. The user said \"I do expect there to be likely some offloading as well potentially\". So fit=on would auto-reduce. Actually the user says \"Llama.cpp is going to be using fit on (the default)\". So the current runblock has `--fit off` which the user says is wrong / they now use fit on. Hmm, they said: \"This is your current runblock... I am adding this note the UD-IQ4_XS is 93.7GB. Llama.cpp is going to be using fit on (the default)\". So they want me to assume fit on.\n\nWait, but `--fit off` in the runblock. The note overrides. So with fit on, llama.cpp will auto-adjust ngl and ctx to fit memory. Interesting — that means --ctx-size 131072 could get auto-clamped. Need to verify what `--fit` actually does in this llama.cpp version.\n\nAlso \"Check into ngram offloading to RAM/OPTANE as an option to run larger quants in the future.\" So there's presumably a feature in the PR for PLE/ngram offload to CPU/SSD/mmap. The Baekpica repo \"Mixed-Quant-SSD-PLE-GGUF\" with \"ssd-offload\" tag supports this idea.\n\nLet me now check:\n1. The HF model repo details (config.json for arch: hidden size, layers, expert count, attention config, PLE tensor names).\n2. llama.cpp PR #27742 for the new flags/env vars and tensor offload options.\n3. The Unsloth GGUF repo file listing to confirm exact filenames.\n\nLet me do these in parallel.\n\nAlso check llama.cpp args source for flags. Let me search for PR 27742 details.\n\nLet me batch: hf_fs to list the unsloth GGUF repo, hub_repo_details for config, and web_fetch the PR.\n\nActually for the GGUF repo file listing, use hf_fs ls. And the base model config.json via cat.\n\nLet me go.\n","timings":{"cache_n":4955,"prompt_n":14969,"prompt_ms":25561.565,"prompt_per_token_ms":1.7076334424477253,"prompt_per_second":585.605771790577,"predicted_n":2025,"predicted_ms":56661.296,"predicted_per_token_ms":27.994711462450592,"predicted_per_second":35.72103257221649},"toolCalls":[{"function":{"arguments":"{\"repo_ids\":[\"Qwen/Qwen3.8-Flash-Next\", \"unsloth/Qwen3.8-Flash-Next-GGUF\"],\"operations\":[\"overview\"]}","name":"hub_repo_details"},"id":"7Q4sEtKmYO8GI4PSls8mwqeDNuCoTjmB","type":"function"},{"function":{"arguments":"{\"operations\":[{\"cmd\": \"ls\", \"args\": [\"hf://models/unsloth/Qwen3.8-Flash-Next-GGUF\"]}, {\"cmd\": \"cat\", \"args\": [\"hf://models/Qwen/Qwen3.8-Flash-Next/config.json\"]}]}","name":"hf_fs"},"id":"lcYFzZ8tqx6qSHGVX8usPf1MZWQn7xrP","type":"function"},{"function":{"arguments":"{\"urls\":[\"https://github.com/ggml-org/llama.cpp/pull/27742\"],\"maxCharacters\":40000}","name":"web_fetch_exa"},"id":"ZjA4bTKUs61bbwaHyRMYMya2EEBiXMoM","type":"function"}]},"type":"message"}
{"message":{"children":[],"content":"**Type: Model**\n\n# Qwen/Qwen3.8-Flash-Next\n\n## Overview\n- **Author:** Qwen\n- **Task:** image-text-to-text\n- **Library:** transformers\n- **Downloads:** 2.6K | **Likes:** 3608\n- **Updated:** 26 Aug, 2026\n\n## Technical Details\n- **Model Class:** AutoModelForMultimodalLM\n- **Parameters:** 180000.0M\n- **Architecture:** qwen4_exp\n\n## Tags\n`transformers` `safetensors` `qwen4_exp` `image-text-to-text` `conversational` `license:other` `eval-results` `endpoints_compatible` `region:us`\n\n## Metadata\n- **License:** other\n\n## Demo Spaces\n- [Qwen3.8-Flash-Next-free-endpoint](https://hf.co/spaces/victor/Qwen3.8-Flash-Next-free-endpoint)\n- [Qwen3.8-Flash-Next-Free-Chat](https://hf.co/spaces/NodeLinker/Qwen3.8-Flash-Next-Free-Chat)\n- [qwen3-8-flash-next-demo](https://hf.co/spaces/bored10/qwen3-8-flash-next-demo)\n\n**Link:** [https://hf.co/Qwen/Qwen3.8-Flash-Next](https://hf.co/Qwen/Qwen3.8-Flash-Next)\n\n---\n\n**Type: Model**\n\n# unsloth/Qwen3.8-Flash-Next-GGUF\n\n## Overview\n- **Author:** unsloth\n- **Task:** image-text-to-text\n- **Likes:** 327\n- **Updated:** 26 Aug, 2026\n\n## Tags\n`gguf` `unsloth` `image-text-to-text` `base_model:Qwen/Qwen3.8-Flash-Next` `base_model:quantized:Qwen/Qwen3.8-Flash-Next` `license:other` `endpoints_compatible` `region:us` `imatrix` `conversational`\n\n## Metadata\n- **License:** other\n\n**Link:** [https://hf.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://hf.co/unsloth/Qwen3.8-Flash-Next-GGUF)","convId":"incyjf36ghq","role":"tool","timestamp":1787790146984,"toolCallId":"7Q4sEtKmYO8GI4PSls8mwqeDNuCoTjmB","type":"text","id":"mr6pa4p42gq","parent":"bw4fjhyn0l5"},"type":"message"}
{"message":{"children":[],"content":"## Operation 1\n\n# hf_fs ls\n\nURI: `hf://models/unsloth/Qwen3.8-Flash-Next-GGUF`\n\n| Type | Path | URI | Target | Details |\n|---|---|---|---|---|\n| dir | UD-IQ1\\_M |  |  |  |\n| dir | UD-IQ1\\_S |  |  |  |\n| dir | UD-IQ3\\_XXS |  |  |  |\n| dir | UD-IQ4\\_XS |  |  |  |\n| dir | UD-Q2\\_K\\_XL |  |  |  |\n| dir | UD-Q3\\_K\\_XL |  |  |  |\n| dir | UD-Q4\\_K\\_XL |  |  |  |\n| file | .gitattributes |  |  | size=3.6 KB |\n| file | README.md |  |  | size=58.6 KB |\n\n---\n\n## Operation 2\n\n# hf_fs cat\n\nURI: `hf://models/Qwen/Qwen3.8-Flash-Next/config.json`\nPath: `config.json`\nBytes: 4745\n\n{\n    \"architectures\": [\n        \"Qwen4ExpForConditionalGeneration\"\n    ],\n    \"image_token_id\": 248056,\n    \"language_model_only\": false,\n    \"model_type\": \"qwen4_exp\",\n    \"text_config\": {\n        \"attention_bias\": false,\n        \"attention_dropout\": 0.0,\n        \"bos_token_id\": 248044,\n        \"dtype\": \"bfloat16\",\n        \"eos_token_id\": 248044,\n        \"full_attention_interval\": 4,\n        \"hc_count\": 4,\n        \"hc_lowrank\": 320,\n        \"head_dim\": 256,\n        \"heads_per_ngram\": 8,\n        \"hidden_act\": \"silu\",\n        \"hidden_size\": 2560,\n        \"indexer_budget\": 2048,\n        \"indexer_compress_ratio\": 4,\n        \"indexer_head_dim\": 128,\n        \"indexer_kv_heads\": 1,\n        \"indexer_n_heads\": 4,\n        \"initializer_range\": 0.02,\n        \"layer_types\": [\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"linear_attention\",\n            \"full_attention\"\n        ],\n        \"linear_conv_kernel_dim\": 4,\n        \"linear_key_head_dim\": 128,\n        \"linear_num_key_heads\": 16,\n        \"linear_num_value_heads\": 48,\n        \"linear_value_head_dim\": 128,\n        \"make_ngram_vocab_size_divisible_by\": 128,\n        \"mamba_ssm_dtype\": \"float32\",\n        \"max_position_embeddings\": 262144,\n        \"model_type\": \"qwen4_exp_text\",\n        \"moe_intermediate_size\": 640,\n        \"mtp\": {\n            \"hybrid\": true,\n            \"layer_types\": [\n                \"full_attention\"\n            ],\n            \"mtp_use_hidden_state_from_layer\": null,\n            \"num_hidden_layers\": 1,\n            \"rope_theta\": 10000000\n        },\n        \"mtp_num_hidden_layers\": 1,\n        \"mtp_use_dedicated_embeddings\": false,\n        \"ngram_size\": 3,\n        \"ngram_vocab_size_base\": 20000000,\n        \"num_attention_heads\": 24,\n        \"num_experts\": 512,\n        \"num_experts_per_tok\": 10,\n        \"num_hidden_layers\": 48,\n        \"num_key_value_heads\": 2,\n        \"output_gate_type\": \"sigmoid\",\n        \"output_router_logits\": false,\n        \"pad_token_id\": null,\n        \"partial_rotary_factor\": 0.25,\n        \"ple_conv_kernel_size\": 4,\n        \"ple_embed_dim\": 2560,\n        \"ple_layer_ids\": [\n            2\n        ],\n        \"rms_norm_eps\": 1e-06,\n        \"rope_parameters\": {\n            \"mrope_interleaved\": true,\n            \"mrope_section\": [\n                11,\n                11,\n                10\n            ],\n            \"partial_rotary_factor\": 0.25,\n            \"rope_theta\": 10000000,\n            \"rope_type\": \"default\"\n        },\n        \"router_aux_loss_coef\": 0.001,\n        \"shared_expert_intermediate_size\": 640,\n        \"split_ngram_parts\": 128,\n        \"tie_word_embeddings\": false,\n        \"use_cache\": true,\n        \"vocab_size\": 248320\n    },\n    \"tie_word_embeddings\": false,\n    \"transformers_version\": \"5.8.0.dev0\",\n    \"video_token_id\": 248057,\n    \"vision_config\": {\n        \"deepstack_visual_indexes\": [],\n        \"depth\": 27,\n        \"hidden_act\": \"gelu_pytorch_tanh\",\n        \"hidden_size\": 1152,\n        \"in_channels\": 3,\n        \"initializer_range\": 0.02,\n        \"intermediate_size\": 4304,\n        \"model_type\": \"qwen4_exp\",\n        \"num_heads\": 16,\n        \"num_position_embeddings\": 2304,\n        \"out_hidden_size\": 2560,\n        \"patch_size\": 16,\n        \"spatial_merge_size\": 2,\n        \"temporal_patch_size\": 2\n    },\n    \"vision_end_token_id\": 248054,\n    \"vision_start_token_id\": 248053\n}","convId":"incyjf36ghq","role":"tool","timestamp":1787790147176,"toolCallId":"lcYFzZ8tqx6qSHGVX8usPf1MZWQn7xrP","type":"text","id":"2nwxh9g3hjp","parent":"bw4fjhyn0l5"},"type":"message"}
{"message":{"children":["uvhsx10tlkn"],"content":"# model: add Qwen3.8-Flash-Next (qwen4exp)\nURL: https://github.com/ggml-org/llama.cpp/pull/27742\n\n# model: add Qwen3.8-Flash-Next (qwen4exp)\n\n- State: open\n- Author: danielhanchen\n- Created: 2026-08-26T13:43:12Z\n- Updated: 2026-08-27T00:13:04Z\n- Repository: ggml-org/llama.cpp\n- Number: #27742\n- +2560 -9 in 21 files\n- Merge commit: 1d85467fc69a387142dfd321a79bd4110596ac70\n- Assignees: ggerganov\n- Reviewers: CISC, JohannesGaessler\n\n## Labels\n\n- model\n- testing\n- conversion\n\n---\n\n> [!IMPORTANT] \n> Off-topic comments and AI slop/spam in this thread will be deleted. Keep the discussion focused. \n \n--- \n \nDraft. Adds support for `Qwen3.8-Flash-Next` (HF `model_type: qwen4_exp`, `Qwen4ExpForConditionalGeneration`): converter, text graph, sparse attention, vision, and three quantizer fixes the model needs. Opening as a draft because the weights are not public yet, so the accuracy numbers below cannot be reproduced from outside at the moment. \n \n## Architecture \n \n| subsystem | how it is handled | \n|---|---| \n| Gated delta net (3 of every 4 layers) | subclasses the existing delta-net base; the only delta from Qwen3.5 is a sigmoid output gate instead of silu | \n| MoE, 512 experts, top-10, gated shared expert | `build_moe_ffn`, unchanged | \n| Interleaved mrope, partial rotary 64/256 | `ggml_rope_multi` with `LLAMA_ROPE_TYPE_IMROPE`, unchanged | \n| Hyper-connections (`hc_count = 4`, low rank) | self-contained in `qwen4exp.cpp`; `deepseek4.cpp` is untouched | \n| Per-layer embeddings, 97.7 GiB n-gram hash table | host-side row indices in a `set_input`, then `ggml_get_rows` | \n| QSA sparse attention (budget 2048, ratio 4) | new graph plus an optional third cache in `llama_memory_hybrid` | \n| Vision | stock Qwen3-VL ViT through the existing clip path | \n \n`git diff master --stat -- ggml/` is empty: no new ggml op, and no change to any existing one. \n \n## Correctness, against the reference implementation \n \n| check | result | \n|---|---| \n| wikitext-2 perplexity, 145 chunks at ctx 2048 | 4.0068 +/- 0.02271 vs 4.0126 for the reference | \n| top-1 agreement, 512 tokens of prose | 98.0 percent | \n| QSA vs dense below the budget | bit identical, max logit delta 0.0 over all 2051 rows | \n| QSA vs dense above the budget, 8192 tokens | diverges on 3 percent of positions | \n| indexer selection vs the reference | 0.975 mean jaccard against a 0.991 precision floor | \n| `test-llama-archs -a qwen4exp` | OK on CPU (0.00e+00) and CUDA (8.00e-08), roundtrip OK | \n \nThe bit-identical result below the budget is the useful one: QSA is dense by construction under `indexer_top_k + compress_ratio - 1` cached tokens, so it is a free oracle on the whole selection and masking path. \n \n## Effect on other architectures \n \nSix shared files contain deletions rather than pure additions: \n \n- `llama-quant.cpp`: three fixes, each byte-exact against the parent commit. `tensor_type_fallback` had no case for 32-block types, so a tensor whose ncols is not divisible by 32 aborted instead of falling back; `--tensor-type` could not name `per_layer_token_embd` because `--token-embedding-type` returned first; and the `work` buffer was sized `nelements * 4` as an upper bound on a value already known exactly, which is 150 GB per process on a model with a 51.2 G element tensor. Six A/B cases, six matching md5s. \n- `llama-graph.cpp`: `build_attn_mask_top_k` lifted verbatim out of the DSA `build_attn` overload so both share it. The node construction sequence is identical after splicing the helper back into its call site, and `deepseek32` and `glm-dsa` both pass at 0.00e+00 on CPU. \n- `llama-memory-hybrid.cpp`: the new `filter_idx` defaults to `nullptr` and every use is guarded. \n- `llama-model-loader.cpp`: adds a `GGUF_TYPE_UINT64` case to a type check that previously threw. \n- `llama-hparams.cpp`, `llama-model.cpp`: new fields and new switch cases. \n \nA shared hyper-connection base with `deepseek4` was considered and rejected: the two formulations agree on the residual layout and nothing else (full-rank projection with Sinkhorn normalisation there, low-rank silu gate and a mean collapse here), so factoring it out would touch DSV4's hot path to share about ten lines. \n \n## Note for reviewers \n \nThis branch was developed against a snapshot and rebased onto current master. Two rebase conflicts were resolved by hand, both additive: `test-llama-archs.cpp` where master had added a `LLM_ARCH_DEEPSEEK4` condition to a line this port also edits, and `llama-model-saver.cpp` where master reworked the DSV4 key block. Worth a close look, since the local build was against the pre-rebase tree. \n \n## AI Usage \nUsed Claude and Local Models for testing, iteration and code design - manual verification of model / PR usage\n\n## Timeline\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- someone committed\n\n- github-actions[bot] added label \"model\"\n\n- github-actions[bot] added label \"testing\"\n\n- github-actions[bot] added label \"conversion\"\n\n**ggml-gh-bot[bot]** commented on 2026-08-26T13:48:19Z:\n\n> Hi @danielhanchen, thanks for your contribution!\n> \n> Per our [contribution guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md), the automated PR checker found the following issue(s) that need your attention:\n> \n> - **PR Template not respected**: Please respect the [template](https://github.com/ggml-org/llama.cpp/blob/master/.github/pull_request_template.md?plain=1) when creating a new pull request. Make sure to fill out all required sections. \n> \n> - **Multiple open PRs from a new contributor**: We limit new contributors (those without a previously merged PR) to **1 open PR at a time**. You currently have 3 open PRs. \n> \n> - **Large PR**: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs. \n> \n> ---\n> \n> _Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below._\n\n- danielhanchen mentioned\n\n- danielhanchen subscribed\n\n- Referenced by PR #27739: model : add Qwen4-Exp\n\n- ggerganov ready_for_review\n\n- Review requested from JohannesGaessler\n\n- Review requested from CISC\n\n- Review requested from ggerganov\n\n- ggerganov was assigned\n\n**danielhanchen** commented on 2026-08-26T14:04:06Z:\n\n> I'm adding MTP - still WIP - everything else works well (including vision)\n> \n> I copied Gemma-3N's PLE approach so it's 1 large table - it can be offloaded to RAM or Disk via mmap\n\n**Green-Sky** commented on 2026-08-26T14:06:47Z:\n\n> Previous work on ngram embeddings: https://github.com/ggml-org/llama.cpp/pull/19167\n> Not sure how similar they are, but looks like the same-ish approach.\n\n**danielhanchen** commented on 2026-08-26T14:07:26Z:\n\n> @Green-Sky I'll check!\n\n- Green-Sky mentioned\n\n- Green-Sky subscribed\n\n- Review by ggerganov: Generally, looks OK. The majority of changes is within `qwen4exp.cpp` which is good - the code does not touch too many other things and does not introduces many new concepts or ggml ops. \n \nThe main focus on the review will be to the changes in `llama-kv-cache` and `llama-memory-hybrid`. Have you considered isolating most of those changes in a new `llama-memory-hybrid-idx`? The current approach of modifying the `llama-memory-hybrid` might be OK - will need to look a bit more. \n \nWhat are the limitations? ~Is MTP supported? Is multi-modal supported?~ Parallel processing? Does the ngram stuff work as it should? \n \nI think the implementation is missing state save/load for the indexer - saving/restoring checkpoints will probably not work?\n\n**danielhanchen** commented on 2026-08-26T14:10:44Z:\n\n> @ggerganov Will check all!\n\n- ggerganov mentioned\n\n- ggerganov subscribed\n\n**JJJYmmm** commented on 2026-08-26T14:11:16Z:\n\n> #27739 already supports mtp and ple offloading — feel free to ref it 🫡\n\n**am17an** commented on 2026-08-26T14:13:34Z:\n\n> can this be composed using MSA cache? It looks like a pretty similar idea\n\n**Rose22** commented on 2026-08-26T14:17:20Z:\n\n> can you make sure the engrams can be offloaded to disk? offloading them to cpu is not viable for a lot of people, because that requires insane amounts of RAM. but they seem like they would work well when offloaded to SSD\n\n**ggerganov** commented on 2026-08-26T14:19:02Z:\n\n> > can this be composed using MSA cache? It looks like a pretty similar idea\n> \n> Yes, definitely worth a look. For new architectures, it's better to try to introduce a new `llama_memory` type in order to avoid touching the fundamental logic of the existing classes that affects many other models. Regardless of code duplication, this can be consolidated in the future.\n\n**danielhanchen** commented on 2026-08-26T14:30:33Z:\n\n> For those interested - this PR can run https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ1_S for now\n\n**Dampfinchen** commented on 2026-08-26T14:35:32Z:\n\n> > can you make sure the engrams can be offloaded to disk? offloading them to cpu is not viable for a lot of people, because that requires insane amounts of RAM. but they seem like they would work well when offloaded to SSD\n> \n> Yeah that's the most interesting aspect of this new architecture. Imagine smaller 30B MoEs but with tons of Engram parameters, would lead to a revolution for local use cases.\n\n**Green-Sky** commented on 2026-08-26T14:36:26Z:\n\n> > For those interested - this PR can run https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ1_S for now\n> \n> \n> \n> > > can you make sure the engrams can be offloaded to disk? offloading them to cpu is not viable for a lot of people, because that requires insane amounts of RAM. but they seem like they would work well when offloaded to SSD\n> > \n> > Yeah that's the most interesting aspect of this new architecture. Imagine smaller 30B MoEs but with tons of Engram parameters, would lead to a revolution for local use cases.\n> \n> Might make sense to put them in their own shard.\n\n**PmNz8** commented on 2026-08-26T14:36:36Z:\n\n> @Rose22 if the n-grams would play nice with SSD offload then it would be possible not to quantize them (higher quality) because you got a lot more of SSD than RAM.\n\n- Rose22 mentioned\n\n- Rose22 subscribed\n\n- maternion subscribed\n\n**Dampfinchen** commented on 2026-08-26T14:38:21Z:\n\n> 5B active on VRAM, 20B experts in RAM and 50B Ngram parameters on disk, would lead to insane models every 16-32 GB RAM system with 4-8 GB VRAM could run well. \n\n**Green-Sky** commented on 2026-08-26T14:41:38Z:\n\n> > 5B active on VRAM, 20B experts in RAM and 50B Ngram parameters on disk, would lead to insane models every 16-32 GB RAM system with 4-8 GB VRAM could run well.\n> \n> To cite their tech report:\n> \n> > **Scaling with Additional Parameters.** Because embedding tables are sparsely accessed and determin-\n> istically addressed, they can be scaled with negligible additional per-token computation and **stored in\n> off-accelerator storage** (Google DeepMind, 2025; Cheng et al., 2026). We therefore move beyond the\n> fixed-size setting and scale the N-gram vocabulary from 20V to 200V , where V denotes the base tokenizer\n> vocabulary size of Qwen3.5 (Qwen Team, 2026).\n> \n> https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf\n\n**am17an** commented on 2026-08-26T14:42:52Z:\n\n> The engram is separate from the 126B, it's 50B on top of that\n\n**lingyezhixing** commented on 2026-08-26T14:46:56Z:\n\n> > 5B active on VRAM, 20B experts in RAM and 50B Ngram parameters on disk, would lead to insane models every 16-32 GB RAM system with 4-8 GB VRAM could run well.\n> \n> SSD needs to consider read latency, which is significantly higher than memory. Queries involve random 4K reads, and it's essential to test whether the SSD can keep up with the main network's computational capacity, especially during prefill. Based on a rough estimation under optimal assumptions, top-tier PCIe 4.0 SSDs like Samsung 990 Pro could deliver query speeds of over 50,000 tokens per second. With the system's file page caching mechanism, performance could be even better. However, real-world performance may differ.\n\n- Referenced by PR #112: Carry ggml-org#27742 (Qwen3.8-Flash-Next) merged onto b10631\n\n- Referenced by PR #113: Pin ggml-org#27742 (Qwen3.8-Flash-Next)\n\n**PmNz8** commented on 2026-08-26T14:53:36Z:\n\n> Most inference will be probably single user query anyway, so I doubt the microseconds of SSD latency vs nanoseconds of RAM will matter, considering that according to the qwen diagrams the n-grams are read once per forward pass, not for every layer, and average system will not be pushing more than a few dozen of tokens per seconds.\n\n- themrzmaster subscribed\n\n- SageDroid subscribed\n\n- bhaan subscribed\n\n**miifanboy** commented on 2026-08-26T15:06:02Z:\n\n> > The engram is separate from the 126B, it's 50B on top of that\n> \n> Yeah I think for most users they would be better off using qwen3.8-27B at Q4 or Q5 instead of using this one at IQ1 if they are memory constrained.\n\n- Nexil-F subscribed\n\n- someone committed\n\n- someone committed\n\n**danielhanchen** commented on 2026-08-26T15:10:21Z:\n\n> Refactor.\n> \n> Moved the indexer cache into a new `llama_memory_hybrid_idx` rather than modifying `llama_memory_hybrid`, and made the top-k attention mask arch-local. `llama-kv-cache.{h,cpp}`, `llama-memory-hybrid.{h,cpp}` and `llama-graph.{h,cpp}` are now byte-identical to master. The PR goes from 24 files to 21, with deletions down from 19 to 9.\n> \n> Verified the graph is unchanged: 8191/8191 rows exact against the pre-refactor build on the real model, max delta 0.0, in both the sparse and forced-dense paths. `test-llama-archs` stays green for qwen4exp, minimax-m3, deepseek32 and glm-dsa.\n\n- Review by ngxson: ~~I have only reviewed the engrams code path so far. Bringing up a concern about whether we should merge all PLE shards into big tensor, or keep them as-is (I prefer keeping as-is)~~ --> see comment below \n \nAlso, can you please ask your agent to scan all code comments and edit them according to `AGENTS.md` guidelines? Some comments have line breaks mid-sentence, some are too verbose, quite hard to read.\n\n- Referenced by PR #114: Carry ggml-org#27742 (Qwen3.8-Flash-Next) merged onto b10632\n\n- Referenced by PR #115: Repin ggml-org#27742 onto the b10632 carry\n\n**ServeurpersoCom** commented on 2026-08-26T15:35:31Z:\n\n> Quantizing on a 96 GB machine still dies with std::bad_alloc on per_layer_token_embd well before the work buffer fix is reached, because f32_conv_buf takes the whole dequantized tensor at once (204.8 GB for that one table), and dequantizing plus quantizing it in row bands instead bounds the staging buffer to a couple of GB and lets Q4_K_M go through.\n\n- Referenced by issue #9122: research(qwen38): study Qwen3.8-Flash-Next architecture and fak opportunities\n\n- noahallen subscribed\n\n- Referenced by issue #1990: MODEL-MM-QWEN4-EXP: a speed denominator now EXISTS (llama.cpp, quant-matched on UD-IQ1_S), and the target is to beat it at concurrency — the spec's 'Speed: nothing' clause is stale\n\n- Referenced by issue #1991: MODEL-MM-QWEN4-EXP W4: Qwen Sparse Attention on DeepSeek-V4's C4 indexer lane, with a GATHER consumer rather than a mask\n\n- Referenced by issue #1993: MODEL-MM-QWEN4-EXP W7: MTP speculative decoding — llama.cpp's leading PR ships without it, but the fitting GGUF has no MTP weights and the draft needs the 10240-wide stream\n\n**jamespa16** commented on 2026-08-26T15:48:07Z:\n\n> trying the Q1 with quantized KV cache seems to fail on Q8_0 and Q4_0 with a null pointer exception (same for both) \n> `llama.cpp/src/models/qwen4exp.cpp:544: GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed`, I'll look into it but wanted to report.\n> \n> Running a 9950x w/ 128GB DDR5 & 5090 on Ubuntu 26.04\n\n**ServeurpersoCom** commented on 2026-08-26T15:57:48Z:\n\n> Qwen3.8-Flash-Next-BF16.gguf 354 GB\n> Qwen3.8-Flash-Next-Q4_K_M.gguf 119 GB pp4096 750 t/s, tg128 95 t/s\n\n**ServeurpersoCom** commented on 2026-08-26T16:00:15Z:\n\n> > trying the Q1 with quantized KV cache seems to fail on Q8_0 and Q4_0 with a null pointer exception (same for both) `llama.cpp/src/models/qwen4exp.cpp:544: GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed`, I'll look into it but wanted to report.\n> > \n> > Running a 9950x w/ 128GB DDR5 & 5090 on Ubuntu 26.04\n> \n> This is the plain-KV counterpart of the MLA sparse path [...] It is kept here rather than factored into a shared helper so that the attention path used by every other architecture is untouched by this arch.\n> Try LLAMA_ATTN_ROT_DISABLE=1 (or ctk/ctv = f16)\n\n- Referenced in commit 2601236\n\n- ggml-org comment_deleted\n\n**ngxson** commented on 2026-08-26T16:07:04Z:\n\n> > I have only reviewed the engrams code path so far. Bringing up a concern about whether we should merge all PLE shards into big tensor, or keep them as-is (I prefer keeping as-is)\n> \n> sorry that was not quite correct, the longcat-flash-lite impl split the engram table semantically while qwen is one big tensor from the get-go, so it cannot split across different storages\n> \n> so concat them into one big tensor is still necessary, just not quite sure if we should still handle it at conversion. some parts of llama.cpp doesn't handle large tensors very well and many use excessive memory. the conversion script should definitely be refactored to handle this case, without the need to manual `mmap`, I will have a look into this\n\n**phhusson** commented on 2026-08-26T16:19:20Z:\n\n> bea3b12da with Qwen3.8-Flash-Next-UD-IQ1_S (803d9f) on 3090 crashes with -fit on: https://gist.github.com/phhusson/ef4b383025ca6034e2b3ecf1427020c8\n\n**jamespa16** commented on 2026-08-26T16:22:12Z:\n\n> ok so I had Opus 5 dig into it:\n> \n> ### Summary: `GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr)` in qwen4exp with quantized KV cache\n> \n> **Bug:** `unsloth/Qwen3.8-Flash-Next-GGUF` works fine with f16 KV cache, but aborts immediately on the first dense-attention layer when using a quantized KV cache (`-ctk q8_0 -ctv q8_0`, or `-ctk q8_0` alone):\n> \n> ```\n> llama.cpp/src/models/qwen4exp.cpp:544: GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed\n> ```\n> \n> **Root cause:** Quantized KV caches are stored Hadamard-rotated (an orthonormal, involutive `H`, `H²=I`) to spread per-channel outliers before block quantization. `llama_kv_cache`'s constructor enables this automatically whenever `type_k`/`type_v` is quantized and head dim is a multiple of 64. Whoever writes to the cache is responsible for applying/undoing the rotation — `llm_graph_context::build_attn` does this correctly for the standard path, but qwen4exp's dense-attention layers use a separate QSA sparse path (`build_attn_qsa`), copied from the MLA/DSA `build_attn` overload, which never encounters rotated K/V and so just asserted the fields were null. Once the cache is quantized, they aren't null, and it crashes.\n> \n> **Fix (in `src/models/qwen4exp.cpp`):** Replace the assert with the same rotate/un-rotate logic used elsewhere:\n> - Rotate `q_cur`/`k_cur` with `inp->self_k_rot` and `v_cur` with `inp->self_v_rot` *before* they're written to cache (before `ggml_build_forward_expand`/`cpy_k`/`cpy_v`), matching the ordering `build_attn` and the K-shift graph expect.\n> - After `build_attn_mha`, un-rotate the output with `self_v_rot` (relies on `v_rot² == I`), placed *before* gating/`wo` projection in `build_layer_attn` since those are non-linear and don't commute with `H`.\n> - Both fields are null for f16 KV, so that path (and mixed `-ctk q8_0 -ctv f16`) is byte-for-byte unchanged.\n> - The indexer top-k path is untouched — it operates on cell indices, not raw K/V, so rotation doesn't affect it.\n> \n> `src/models/qwen4exp.cpp` — `build_attn_qsa` now performs the same rotation as\n> `llm_graph_context::build_attn`:\n> \n> ```diff\n> - GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr);\n> + // A quantized KV cache stores K/V Hadamard-rotated; q/k rotate together so KQ is\n> + // unchanged, and the rotation of V is undone on the attention output. Only the\n> + // dense KV cache is rotated: the indexer keys live in their own cache and the\n> + // top_k cell indices this path is handed are unaffected.\n> + if (inp->self_k_rot) {\n> + q_cur = llama_mul_mat_hadamard(ctx0, q_cur, inp->self_k_rot);\n> + k_cur = llama_mul_mat_hadamard(ctx0, k_cur, inp->self_k_rot);\n> + }\n> +\n> + if (inp->self_v_rot) {\n> + v_cur = llama_mul_mat_hadamard(ctx0, v_cur, inp->self_v_rot);\n> + }\n> ```\n> \n> and, after `build_attn_mha`:\n> \n> ```diff\n> ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);\n> cb(cur, \"kqv_out\", il);\n> \n> + // note: assumes v_rot^2 == I\n> + if (inp->self_v_rot) {\n> + cur = llama_mul_mat_hadamard(ctx0, cur, inp->self_v_rot);\n> + }\n> +\n> return cur;\n> }\n> ```\n> \n> Placement matters in three ways:\n> \n> - The rotations go **before** the `ggml_build_forward_expand(gf, q_cur/v_cur/k_cur)` block\n> and therefore before `mctx_cur->cpy_k/cpy_v`, so what lands in the cache is rotated — the\n> same order `build_attn` uses, and the order the cache's own K-shift graph\n> (`build_graph_shift`) assumes when it rotates, ropes and un-rotates in place.\n> - Both are null when the corresponding cache is unquantized, so f16 KV and the mixed\n> `-ctk q8_0 -ctv f16` case take exactly the code path they took before. The f16 graph is\n> byte-for-byte unchanged.\n> - The un-rotate sits at the end of `build_attn_qsa`, before the caller's gating and `wo`\n> projection in `build_layer_attn`. Both of those are non-linear/non-commuting with `H`, so\n> the un-rotate cannot be deferred past them.\n> \n> `llama_mul_mat_hadamard` (`src/llama-impl.h:57`) reshapes the tensor to `[n, nelements/n]`,\n> multiplies by the rotation, hints `GGML_HINT_SRC0_IS_HADAMARD`, and reshapes back — so it\n> does not care that Q here is `[n_embd_head, n_head, n_tokens]` and the MHA output is\n> `[n_embd_head*n_head, n_tokens]`.\n> \n> **The top-k path is deliberately untouched.** The indexer keeps its own cache\n> (`llama_memory_hybrid_idx`), and what it hands `build_attn_qsa` is a tensor of *cell\n> indices*. Rotating the dense K/V changes neither those indices nor the KQ mask built from\n> them.\n> \n> **Verification:** Tested with `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ1_S` on CUDA — no assert and coherent output for `q8_0`/`q8_0`, `q8_0`/f16, and f16, across both short and ~5k-token prompts (long prompt needed to actually exercise the sparse QSA branch, since `indexer_top_k=2048` + `compress_ratio=4` means <2051 cached tokens is just dense attention).\n> \n> Perplexity (3-chunk local corpus, `-c 4096`, IQ1_S so noisy, but disabling rotation via `LLAMA_ATTN_ROT_DISABLE=1` costs ~15% on the chunk where the sparse path is most active) confirms the rotation is doing real work, not just satisfying the assert.\n> \n> **Rejected addition:** Also rotating the indexer's own quantized cache (which currently stores raw/unrotated keys since `build_input_k_rot` is never called on it) is implementable — rotation can be undone once on the pooled block key rather than per-member, since pooling is linear and precedes norm/rope. Measured impact was within run-to-run noise, so it wasn't included; the extra per-layer matmul/graph input isn't justified without a larger corpus showing a real gain.\n> \n> I did verify this runs on CUDA, so this is a good option.\n\n**danielhanchen** commented on 2026-08-26T16:23:12Z:\n\n> I'm checking all issues and will fix soon!\n\n**ServeurpersoCom** commented on 2026-08-26T16:23:34Z:\n\n> If it helps with testing, here are some temporary links:\n> BF16 / Q4_K_M (for 96GB) GGUF https://www.serveurperso.com/ia/ssd/workspace/models/Qwen/Qwen3.8-Flash-Next-GGUF/ and a patch for the converter https://github.com/ggml-org/llama.cpp/commit/af935452324829dee8a93499fb59245032259af3\n> (PR work on my RTX PRO 6000, mmap = off, fit = off, ctk/ctv = f16, about 100 tok/s to 50 tok/s)\n> \n> EDIT: Quants have arrived : https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF\n\n- LETS-BEE subscribed\n\n- Referenced by issue #893: Add Qwen3.8-Flash-Next (`qwen4exp`) support after llama.cpp release\n\n**coder543** commented on 2026-08-26T16:34:51Z:\n\n> I can confirm that `mmap` offloading works for the PLE, but `mmap` on DGX Spark is incredibly slow... with `llama-server` on DGX Spark, this model loads in 22 seconds with mmap disabled, and takes 224 seconds to load with mmap enabled... literally 10x longer. It would be great if PLEs could be offloaded to `mmap` separately from the rest of the model weights. Or if there could be an application-level alternative to `mmap` that just performs direct reads to avoid `mmap` overhead... but I know this is a very DGX Spark specific problem.\n\n**imezx** commented on 2026-08-26T16:50:13Z:\n\n> i wonder why upcoming Qwen 4 arch doesnt considering using GatedDeltaNet-2 (GDN-2, by NVIDIA) instead of GDN..\n\n**kyuz0** commented on 2026-08-26T16:57:12Z:\n\n> When I load this on Strix Halo, with this command\n> \n> ```\n> llama-server -m /models/Qwen3.8-Flash-Next-GGUF/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf -c 262144 -ngl 999 --host 0.0.0.0 --port 8080 --load-mode none -fa 1 --jinja --reasoning on --chat-template-kwargs '{\"reasoning_effort\":\"xhigh\"}' --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -np 1 \n> 0.00.026.557 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)\n> 0.00.028.375 I srv load_model: loading model '/models/Qwen3.8-Flash-Next-GGUF/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf'\n> 0.13.472.130 I cmn init: llama threadpool init, n_threads = 16\n> 0.13.833.610 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 262144, kv_unified = 'false'\n> 0.13.841.941 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve\n> 0.13.841.983 I srv llama_server: model loaded\n> 0.13.841.989 I srv llama_server: listening on http://0.0.0.0:8080\n> 0.13.841.991 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release\n> 0.13.841.991 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508\n> \n> ```\n> \n> For some reason it uses less memory than I am expecting:\n> \n> \n\n- Referenced by issue #53: Qwen3.8-Flash-Next: blocked for ONNX conversion (qwen4_exp architecture not dispatched)\n\n- Referenced by PR #54: ci: add daily GitHub Actions monitor for Qwen3.8-Flash-Next llama.cpp blocker\n\n- Referenced in commit 8d5169f\n\n- zhouxihong1 subscribed\n\n**HDANILO** commented on 2026-08-26T17:35:14Z:\n\n> Trying to run on a 9900X3D 128gb DDR5 + RTX 5090:\n> \n> ```\n> > LLAMA_ATTN_ROT_DISABLE=1 ./llama.cpp/build/bin/llama-server \\\n> -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS \\\n> -fa 1 \\\n> --ctx-size 25000\n> 0.00.631.802 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)\n> 0.00.632.034 W srv llama_server: -----------------\n> 0.00.632.035 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set\n> 0.00.632.035 W srv llama_server: this can be a security risk (cross-origin attacks)\n> 0.00.632.035 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655\n> 0.00.632.035 W srv llama_server: -----------------\n> 0.00.633.359 I srv load_model: loading model 'unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS'\n> /home/hleal/AI/llama.cpp/ggml/src/ggml.c:1804: GGML_ASSERT(obj_new) failed\n> [New LWP 306651]\n> [New LWP 306649]\n> [New LWP 306648]\n> [New LWP 306647]\n> [New LWP 306646]\n> [New LWP 306645]\n> [New LWP 306644]\n> [New LWP 306643]\n> [New LWP 306642]\n> [New LWP 306641]\n> [New LWP 306640]\n> [New LWP 306639]\n> [New LWP 306638]\n> [New LWP 306637]\n> [New LWP 306636]\n> [New LWP 306635]\n> [New LWP 306634]\n> [New LWP 306633]\n> [New LWP 306632]\n> [New LWP 306631]\n> [New LWP 306630]\n> [New LWP 306629]\n> [New LWP 306628]\n> [New LWP 306627]\n> [New LWP 306626]\n> [New LWP 306625]\n> [New LWP 306606]\n> [New LWP 306605]\n> [New LWP 306604]\n> \n> This GDB supports auto-downloading debuginfo from the following URLs:\n> \n> \n> Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]\n> Debuginfod has been disabled.\n> To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.\n> [Thread debugging using libthread_db enabled]\n> Using host libthread_db library \"/usr/lib/libthread_db.so.1\".\n> 0x00007fa6c5cb52d2 in ?? () from /usr/lib/libc.so.6\n> #0 0x00007fa6c5cb52d2 in ?? () from /usr/lib/libc.so.6\n> #1 0x00007fa6c5ca837f in ?? () from /usr/lib/libc.so.6\n> #2 0x00007fa6c5d2d4cb in wait4 () from /usr/lib/libc.so.6\n> #3 0x00007fa6d14658bb in ggml_print_backtrace () from /home/hleal/AI/llama.cpp/build/bin/libggml-base.so.0\n> #4 0x00007fa6d1465a4e in ggml_abort () from /home/hleal/AI/llama.cpp/build/bin/libggml-base.so.0\n> #5 0x00007fa6d1467f02 in ggml_new_tensor () from /home/hleal/AI/llama.cpp/build/bin/libggml-base.so.0\n> #6 0x00007fa6d146b900 in ggml_scale () from /home/hleal/AI/llama.cpp/build/bin/libggml-base.so.0\n> #7 0x00007fa6d051e594 in llama_model_qwen4exp::graph::build_hc_combine(ggml_tensor*, ggml_tensor*, ggml_tensor*, int) () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #8 0x00007fa6d05213e4 in llama_model_qwen4exp::graph::graph(llama_model const&, llm_graph_params const&) () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #9 0x00007fa6d0521680 in llama_model_qwen4exp::build_arch_graph(llm_graph_params const&) const () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #10 0x00007fa6d03cacab in llama_model::build_graph(llm_graph_params const&) const () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #11 0x00007fa6d02fd0a4 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #12 0x00007fa6d02feffb in llama_context::sched_reserve() () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #13 0x00007fa6d0304aad in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #14 0x00007fa6d0305f98 in llama_init_from_model () from /home/hleal/AI/llama.cpp/build/bin/libllama.so.0\n> #15 0x00007fa6d0a5adcc in common_get_device_memory_data_impl(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #16 0x00007fa6d0a5bff0 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::{lambda(char const*, std::vector<common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::ngl_t, std::allocator<common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::ngl_t> > const&, std::vector<ggml_backend_buffer_type*, std::allocator<ggml_backend_buffer_type*> > const&)#1}::operator()(char const*, std::vector<common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::ngl_t, std::allocator<common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::ngl_t> > const&, std::vector<ggml_backend_buffer_type*, std::allocator<ggml_backend_buffer_type*> > const&) const [clone .constprop.0] () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #17 0x00007fa6d0a5d5ce in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level) () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #18 0x00007fa6d0a62067 in common_fit_params(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level) () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #19 0x00007fa6d0a25dde in common_init_result::common_init_result(common_params&, bool) () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #20 0x00007fa6d0a28aab in common_init_from_params(common_params&, bool) () from /home/hleal/AI/llama.cpp/build/bin/libllama-common.so.0\n> #21 0x00007fa6d0f7c151 in server_context_impl::load_model(common_params&) () from /home/hleal/AI/llama.cpp/build/bin/libllama-server-impl.so\n> #22 0x00007fa6d0ecbe3e in llama_server(common_params&, int, char**) () from /home/hleal/AI/llama.cpp/build/bin/libllama-server-impl.so\n> #23 0x00007fa6d0ecdf8b in llama_server(int, char**) () from /home/hleal/AI/llama.cpp/build/bin/libllama-server-impl.so\n> #24 0x00007fa6c5c27c4e in ?? () from /usr/lib/libc.so.6\n> #25 0x00007fa6c5c27d8b in __libc_start_main () from /usr/lib/libc.so.6\n> #26 0x000055f0b6bc7075 in _start ()\n> [Inferior 1 (process 306600) detached]\n> [1] 306600 abort (core dumped) LLAMA_ATTN_ROT_DISABLE=1 ./llama.cpp/build/bin/llama-server -hf -fa 1 25000\n> ```\n> \n> llama.cpp build command:\n> ```\n> cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_FLAGS=\"-ccbin /usr/bin/g++-13\"\n> cmake --build build --config Release -j $(nproc)\n> ```\n> \n> CachyOS.\n> \n> Got it to run with:\n> ```> LLAMA_ATTN_ROT_DISABLE=1 ./llama.cpp/build/bin/llama-server \\\n> -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS \\\n> -fa 1 -c 262144 --host 0.0.0.0 --port 8080 --load-mode none -fa 1 --jinja --reasoning on --chat-template-kwargs '{\"reasoning_effort\":\"xhigh\"}' --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -np 1\n> ```\n\n- Referenced by issue #27741: Feature Request: Possible to support Qwen3.8-Flash-Next?\n\n**phhusson** commented on 2026-08-26T17:40:32Z:\n\n> @HDANILO looks like my issue with fit. Try with fit off. On my 3090 I use `-fit off -ot 'blk\\.(([0123][0123456789]|40|41|42|43|44))\\.ffn_.*=CPU'`\n\n- HDANILO mentioned\n\n- HDANILO subscribed\n\n- Referenced in commit fbf5d9b\n\n- Referenced by PR #172: feat(moe): Qwen3.8-Flash-Next support\n\n**ServeurpersoCom** commented on 2026-08-26T17:45:08Z:\n\n> This is missing here:\n> \n> ```\n> --- a/src/llama-context.cpp\n> +++ b/src/llama-context.cpp\n> @@ -2320,6 +2320,7 @@ uint32_t llama_context::graph_max_nodes(uint32_t n_tokens) const {\n> model.arch == LLM_ARCH_BAILINGMOE3 ||\n> model.arch == LLM_ARCH_QWEN35 ||\n> model.arch == LLM_ARCH_QWEN35MOE ||\n> + model.arch == LLM_ARCH_QWEN4EXP ||\n> model.arch == LLM_ARCH_DEEPSEEK4 ||\n> (model.arch == LLM_ARCH_DFLASH && model.hparams.dsv4_hc_mult > 0) ||\n> model.arch == LLM_ARCH_NANBEIGE ||\n> ```\n> \n> It's missing, ~~but it doesn't fix the --fit. I dig~~\n> \n> On a 96 GB card the model fits, so --fit never probes the offload path and it passes either way. Forcing the constraint with --fit-target 70000 triggers the MoE-to-system-memory probe, which is the one that overflows: crash without the line, loads with it.\n> \n> This fix the --fit on\n\n**FlorinAndrei** commented on 2026-08-26T17:47:23Z:\n\n> Maybe everything could just obey the value of `--load-mode`? On platforms where mmap is slow (e.g. Spark) you want to disable mmap for loading as well. So then that could perhaps apply to offloading, too. Just a suggestion.\n\n- Referenced by PR #27754: model: add GLM-5-Next (GLM-5.3-Flash)\n\n**coder543** commented on 2026-08-26T17:49:00Z:\n\n> Everything seems to obey `--load-mode` already, but forcing the PLEs into pinned memory wastes a ton of memory. It would be nice to have the PLEs streamed from disk. The SSD can do like 12GB/s, it's just a broken kernel implementation of mmap that makes mmap slow. But, loading the PLEs through mmap doesn't seem to hurt performance much in my testing once the model is loaded.\n\n**lingyezhixing** commented on 2026-08-26T17:54:51Z:\n\n> It might be an issue with the file page cache, but Llama doesn't directly support reading files.\n\n**phhusson** commented on 2026-08-26T17:57:21Z:\n\n> @ServeurpersoCom yes it fixes fit the issue for me, thanks\n\n- ServeurpersoCom mentioned\n\n- ServeurpersoCom subscribed\n\n**anonimousA** commented on 2026-08-26T18:20:43Z:\n\n> **Multi-slot: indexer cache desyncs from attention cache (assert at qwen4exp.cpp:285)**\n> \n> Setup: Windows 11, RTX 5090 (sm_120), 96 GB DDR5-6600, CUDA 13.3.73, MSVC 19.44, this branch at 035e227, model `unsloth/Qwen3.8-Flash-Next-GGUF` UD-Q4_K_XL.\n> \n> With default server slots (`n_slots = 4`, unified KV), the first request completes fine, but when a later request lands on a *different* slot than the previous one (LRU picks a fresh slot), the server aborts during prompt processing:\n> \n> ```\n> src\\models\\qwen4exp.cpp:285: GGML_ASSERT(mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv() && \"the indexer cache must track the attention cache cell for cell\") failed\n> ```\n> \n> Repro: start `llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --jinja -ngl 99 --cpu-moe -c 32768 -fa on` (no `--parallel` flag), send a short `/v1/chat/completions` request (completes fine, slot 3 in my log), then send a second longer request: LRU selects slot 2 and the assert fires immediately.\n> \n> ```\n> slot get_availabl: id 2 | task -1 | selected slot by LRU, t_last = -1\n> slot launch_slot_: id 2 | task 2062 | processing task, is_child = 0\n> C:\\...\\src\\models\\qwen4exp.cpp:285: GGML_ASSERT(...) failed\n> ```\n> \n> Workaround: `--parallel 1` - with a single slot it is stable (several thousand tokens of mixed short/long prompts afterwards, including a 6k-token prefill, no issues).\n> \n> FWIW on the same setup: `--fit on` hits `GGML_ASSERT(obj_new)` at ggml.c:1804 during load (same as @phhusson reported, and consistent with the missing `LLM_ARCH_QWEN4EXP` in `graph_max_nodes` that @ServeurpersoCom found); f16 KV cache works fine, and mmap PLE offload works as advertised - the n-gram table stays cold and the resident set is dominated by the expert weights.\n\n- phhusson mentioned\n\n- phhusson subscribed\n\n- ServeurpersoCom mentioned\n\n- ServeurpersoCom subscribed\n\n**ServeurpersoCom** commented on 2026-08-26T18:24:20Z:\n\n> I try a fix for KV Cache quant with hadamard rot. Done :\n> \n> https://github.com/ggml-org/llama.cpp/commit/7044fe16fcb334164d03d20108ffac0b504353e1 Fix KV Cache with hadamart, Tested functionally + numerical validation.\n> \n> Edit :\n> Numerically checked on Q4_K_M over 16 chunks at ctx 2048: q8_0 KV with the rotation lands on the f16 reference within noise (2.3452 vs 2.3470, error bars 0.035), so the rotation is neutral on the score path as expected.\n> \n> https://github.com/ggml-org/llama.cpp/commit/c30f6685ddde935a1fa90868343dfd59ba7e52aa Fix --fit (CPU/GPU setup)\n> \n> https://github.com/ggml-org/llama.cpp/commit/d885ad824b204ecddcbdeffe852edd91bf9d9fbd Fix quantize on low memory machine.\n> \n> All tested OK.\n\n**mikki8** commented on 2026-08-26T18:27:06Z:\n\n> ### DGX Spark (GB10) data point: cannot reproduce the 10× mmap slowdown\n> \n> Tested this PR on a DGX Spark since the reported mmap regression is\n> Spark-specific and there don't seem to be many of these around.\n> \n> **Setup**\n> - DGX Spark / GB10, 121 GB unified memory, aarch64, CUDA 13\n> - PR branch at `035e227`, built natively","convId":"incyjf36ghq","role":"tool","timestamp":1787790147496,"toolCallId":"ZjA4bTKUs61bbwaHyRMYMya2EEBiXMoM","type":"text","id":"sgk361otian","parent":"bw4fjhyn0l5"},"type":"message"}
{"message":{"children":["rdyk4h1avxo","ckd32slblr"],"content":"Confirmed real. Now pulling the exact tensor names, the PLE/SSD-offload discussion, and stock llama.cpp flag semantics — several flags in your block interact with known bugs in this brand-new arch.\n\n","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790147505,"type":"text","id":"uvhsx10tlkn","parent":"sgk361otian","reasoningContent":"Excellent, this is a goldmine. Key findings:\n\n1. **Architecture** (from config.json):\n   - `hidden_size: 2560`, `num_hidden_layers: 48`\n   - `layer_types`: linear_attention x3, full_attention every 4th → 12 full attention layers, 36 linear (gated delta net) layers\n   - `num_attention_heads: 24`, `num_key_value_heads: 2`, `head_dim: 256`\n   - `full_attention_interval: 4`\n   - `num_experts: 512`, `num_experts_per_tok: 10`, `moe_intermediate_size: 640`, `shared_expert_intermediate_size: 640`\n   - QSA: `indexer_budget: 2048`, `indexer_compress_ratio: 4`, `indexer_head_dim: 128`, `indexer_kv_heads: 1`, `indexer_n_heads: 4`\n   - n-gram: `ngram_vocab_size_base: 20000000`, `ngram_size: 3`, `split_ngram_parts: 128`, `ple_embed_dim: 2560`\n   - MTP: `mtp_num_hidden_layers: 1`, `hybrid: true` — MTP is in the config but PR says MTP WIP\n   - `max_position_embeddings: 262144`\n   - vision: Qwen3-VL ViT, hidden 1152, depth 27\n   - `hc_count: 4` hyper-connections\n   - `partial_rotary_factor: 0.25` → 64 of 256 head_dim rotated\n   - Parameters: 180B total per HF (125B text + 50B ngram per PR comments... actually \"The engram is separate from the 126B, it's 50B on top of that\")\n   - `ple_layer_ids: [2]` — interesting, PLE only on layer 2?\n\n2. **CRITICAL BUGS relevant to the user's runblock:**\n   - **KV cache quantization (q8_0) CRASHES**: `GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr)` at qwen4exp.cpp:544. The user has `--cache-type-k q8_0 --cache-type-v q8_0`! Workarounds: `LLAMA_ATTN_ROT_DISABLE=1` (which the user HAS set — that's why they have it!) OR use f16 KV, OR apply the fix commit `7044fe16fcb334164d03d20108ffac0b504353e1`.\n   - **Important**: `LLAMA_ATTN_ROT_DISABLE=1` costs ~15% perplexity on chunks where the sparse path is active. That's a real quality hit. The proper fix is the patch commit. This is a big finding.\n   - **`--fit on` CRASHES** with `GGML_ASSERT(obj_new)` at ggml.c:1804 in `build_hc_combine` — missing `LLM_ARCH_QWEN4EXP` in `graph_max_nodes`. Fix: commit `c30f6685ddde935a1fa90868343dfd59ba7e52aa` \"Fix --fit (CPU/GPU setup)\". The user said \"Llama.cpp is going to be using fit on (the default)\" — so they NEED this patch. phhusson: \"yes it fixes fit the issue for me\".\n   - **Multi-slot (parallel > 1) CRASHES**: `GGML_ASSERT(mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv())` at qwen4exp.cpp:285. Workaround `--parallel 1` — the user already has that. Good, keep it. Do NOT increase --parallel.\n   - **`--load-mode none`**: that's a real flag, used by kyuz0 and others. It disables mmap. BUT — the user has PLE on CPU via `--override-tensor`, and `--load-mode none` means no mmap → PLE is loaded into RAM (not mmap'd from disk). This matters: `--load-mode none` on a 93.7GB model means all of it gets read into host RAM at load, then copied. With `--override-tensor` for PLE to CPU, using `--load-mode none` will force 93.7GB resident. If they have 8 DIMMs at 2133 MT/s... wait, what's the RAM size? 3945WX on WRX80 with 8 DIMMs. Unknown capacity — could be 128GB or 256GB. Quad channel... actually WRX80 is 4-channel (8 DIMMs = 4ch × 2). 2133 MT/s quad channel ≈ 68 GB/s peak, ~50-55 GB/s realistic. That's SLOW for CPU-side inference and slow for streaming the PLE table.\n   - **phhusson's offload trick**: `-ot 'blk\\.(([0123][0123456789]|40|41|42|43|44))\\.ffn_.*=CPU'` — regex-based ffn offload to CPU for 3090. This is directly relevant since 93.7GB > 96GB usable across 4 cards once you account for KV + compute buffers.\n\n3. Tensor names: `per_layer_token_embd` is confirmed as a real tensor name (from the PR: \"--tensor-type could not name per_layer_token_embd because --token-embedding-type returned first\"). So `per_layer_token_embd` is correct. But `ple_ngram_embd` — I haven't seen that name. The user has `ple_ngram_embd=CPU`. Need to verify. The n-gram table is 97.7 GiB per PR description (\"Per-layer embeddings, 97.7 GiB n-gram hash table\"). Hmm — 97.7 GiB ngram table but the whole UD-IQ4_XS file is 93.7GB? That doesn't add up unless the 97.7 GiB is the BF16/unquantized size, and at 4-bit it's ~49GB. Yes: 97.7 GiB at 4-bit ≈ 49 GB. And the rest of the model at 4-bit ~44GB. Total ~93.7GB. That works!\n\n   Wait but unsloth says \"1-bit is 75GB and uses 4-bit for the Ngram/PLE\". So the ngram table at 4-bit is ~49GB and the rest at 1-bit is ~26GB → 75GB. And UD-IQ4_XS: rest at ~4bit... Hmm, 125B at Q4_XS ≈ 70GB + 49GB = way more than 93.7. Let me not over-derive. Actually IQ4_XS is 4.25bpw → 126B × 4.25/8 = 67GB, + 4bit ngram 49GB = 116GB. Doesn't match 93.7 either. Hmm. Maybe ngram_vocab_size_base 20M × 2560 embed dim... = 51.2G elements (PR says \"a model with a 51.2 G element tensor\"! yes — 51.2 G elements). At 4-bit = 25.6 GB. At 8-bit = 51.2GB. 97.7 GiB ≈ 105 GB — that'd be bf16... 51.2e9 × 2 bytes = 102.4 GB = 95.4 GiB. Close to 97.7 GiB. So the table is 51.2G elements, ~97.7 GiB in BF16, and at 4-bit quant ≈ 25.6 GB.\n   \n   So: IQ4_XS main weights 126B... hmm, but if ngram is quantized at 4-bit minimum, and the model file is 93.7GB: main body at IQ4_XS ≈ 67GB + ngram 26GB = 93GB. ✓ That matches perfectly! Great.\n\n   So: **the n-gram/PLE table is ~26 GB in UD-IQ4_XS, and the main transformer body is ~67 GB.** Total 93.7 GB.\n\n   On 4×3090 = 96 GB (nominal; realistically ~93-94 GB usable before KV+compute buffers). So the user needs ~67 GB of weights on GPU + KV. That leaves plenty for KV on GPU if the ngram table goes to CPU. That's exactly what the user is doing.\n\n4. **KV cache size**: full attention only in 12 of 48 layers. head_dim 256, kv_heads 2 → per token per layer: 2 × 256 × 2 (K+V) × 2 bytes (f16) = 2048 bytes = 2 KB. × 12 layers = 24 KB/token. Plus the linear attention (gated delta net) state which is constant-size per slot (not per token) — conv state + recurrent state. Plus the indexer cache: `indexer_kv_heads: 1`, `indexer_head_dim: 128` → 128 elements/token/layer? The indexer cache is a \"third cache\" in llama_memory_hybrid. indexer_compress_ratio 4 with budget 2048.\n\n   At f16: 24 KB/token × 131072 tokens = 3.1 GB. At q8_0 ≈ 1.6 GB. That's tiny. Great — so KV cache is cheap because of the hybrid arch. So the user could actually go to 262144 ctx for ~6.3 GB f16. That's a major insight: **you don't need to quantize the KV cache at all** — and quantizing it triggers the Hadamard-rotation crash requiring LLAMA_ATTN_ROT_DISABLE=1 which costs ~15% quality. Huge finding.\n\n   Wait — need to double check whether the crash is caused by q8_0 KV. Yes: \"aborts immediately on the first dense-attention layer when using a quantized KV cache (-ctk q8_0 -ctv q8_0, or -ctk q8_0 alone)\". And ServeurpersoCom fixed it in commit 7044fe1. So: either update the build to include that commit and drop LLAMA_ATTN_ROT_DISABLE, or set ctk/ctv f16 and drop the env var.\n\n   Note: `LLAMA_ATTN_ROT_DISABLE=1` disables the Hadamard rotation entirely → q8_0 KV works but quality is worse than with rotation. But if you're using f16 KV, no rotation is needed anyway, so the env var becomes a no-op... Actually careful: does LLAMA_ATTN_ROT_DISABLE affect anything else? It disables the Hadamard rotation path. With f16 KV, self_k_rot/self_v_rot would be null anyway. So it's a no-op. But leaving it set is dangerous if they later switch to quantized KV or if it affects something else. Better to remove it and use f16 KV, or patch.\n\n   Actually wait — but is it truly a no-op with f16 KV? The comment from jamespa16 says \"Both fields are null for f16 KV, so that path (and mixed -ctk q8_0 -ctv f16) is byte-for-byte unchanged.\" And perplexity test showed \"disabling rotation via LLAMA_ATTN_ROT_DISABLE=1 costs ~15% on the chunk where the sparse path is most active\" — with quantized KV. With f16 KV, no rotation happens anyway. So removing env var + f16 KV = correct reference behavior. \n\n   Hmm, but one caution: does LLAMA_ATTN_ROT_DISABLE also affect the *indexer* cache path? \"the indexer keeps its own cache (llama_memory_hybrid_idx)... currently stores raw/unrotated keys since build_input_k_rot is never called on it\". So indexer not affected.\n\n5. **`--override-tensor` name check**: need `per_layer_token_embd` (confirmed real, PR mentions `--tensor-type could not name per_layer_token_embd`). And `ple_ngram_embd` — unverified. In the PR, ngxson's review mentions \"engram\" code path, and danielhanchen says \"I copied Gemma-3N's PLE approach so it's 1 large table - it can be offloaded to RAM or Disk via mmap\". Gemma 3n uses `per_layer_embeddings` / `ple.embed`. In llama.cpp, Gemma 3n PLE tensor is `per_layer_token_embd.weight` I believe. Let me try to verify the actual GGUF tensor names. I could check the GGUF repo's file listing or a metadata file.\n\n   Let me try to check the actual tensor names. The GGUF is split into 3 files; I can't easily download. But there might be a way — the AtomicChat metrics dataset, or the README of the GGUF repo. Let's read the unsloth GGUF README (58.6KB) — it might have recommended flags and tensor names. Also Baekpica's SSD-PLE repo README likely documents `-ot` usage.\n\n   Also I should check: is there a dedicated flag now for PLE offload (e.g., `--ple-offload`, `--cpu-offload-ple`)? And `--tensor-type` for specifying per-tensor types. And `--fit-target`. And `--mmap` / `--no-mmap` vs `--load-mode`.\n\n6. **`--load-mode none`**: In llama.cpp, `--load-mode` has values `auto`, `mmap`, `user`. `--load-mode user` was used with `--no-hostalloc`. Setting `none`... need to verify valid values. kyuz0 used `--load-mode none` and it worked. Hmm, in llama.cpp the enum is `llama_model_load_mode { AUTO, ONLY_CPU, USER }` printed as \"auto\"/\"cpu\"/\"user\". \"none\" may not be valid... but kyuz0's log shows it loaded fine. Hmm, actually maybe in this future version there's a \"none\" value. Let me verify from the args.h / common docs.\n\n   Actually there's an important nuance: with `--load-mode none`, mmap is disabled → the PLE table gets allocated in host RAM as a regular buffer. If the user wants PLE in RAM that's what they want. But if they want PLE to stay in the page cache backed by disk (so it doesn't count against RAM), they need mmap. Trade-off: coder543 reports mmap 10× slower load on DGX Spark but \"loading the PLEs through mmap doesn't seem to hurt performance much once the model is loaded\".\n\n7. `--flash-attn on` — with QSA sparse attention, is FA needed/supported? The QSA path builds a top-k mask. `-fa 1` was used in all community commands, so fine.\n\n8. `--threads 6` / `--threads-batch 12`: On 12c/24t, physical cores = 12. `--threads 6` for decode is low if there's meaningful CPU work (PLE on CPU + any CPU-offloaded MoE). Actually decode threads should be ~physical core count. With PLE lookup on CPU + hyper-connections, but the main net on GPU, decode is mostly GPU-bound. However the PLE gather + the CPU-side part will use CPU. Setting `--threads 6` may bottleneck. Given they have 4 GPUs doing the heavy lifting, I'd set `-t 12` (physical cores) and `-tb 12` or higher. Hmm. Note: in llama.cpp, `-t/--threads` affects both decode (GPU-batched too, for the non-GPU parts and for scheduling) and `-tb` is for batched/prefill compute. For mostly-offloaded models, -t is still used for the CPU portion of the graph. Recommend -t 12, -tb 24 (or leave -tb at 24 = nproc). Careful with hyperthreading: 24 threads on 12 cores for memory-bandwidth-bound CPU MoE is usually worse than 12. But PLE gather is latency-bound random access → more threads help. Let me recommend -t 12, -tb 24 but discuss.\n\n   Also `--threads-http 4` is fine.\n\n9. **`--no-context-shift`** — fine, it's the default in newer versions? `-ncs` exists. Fine either way.\n\n10. **Missing for Hermes**: \n   - `--tools` / tool call parser. llama.cpp server with `--jinja` uses the chat template's built-in tool call parsing if the template defines `tools` jinja. Qwen3.8 template likely supports tools natively. There's also `--tool-call-parser`/`--chat-template-parser` in newer builds. And Hermes is \"OpenAI-compatible\" and needs tools. Must verify.\n   - `--reasoning` / `--reasoning-preserve` — ggerganov log said \"chat template supports preserving reasoning, consider enabling it via --reasoning-preserve\". For Hermes long-horizon agentic, preserve thinking could help quality. But it bloats context. For Hermes, the harness manages its own context; I'd leave reasoning on with effort tuning.\n   - `--chat-template-kwargs '{\"reasoning_effort\":\"medium\"}'` for speed.\n   - `--slots`, `--metrics`, `--webui`\n   - **`--no-webui`?** not needed.\n   - Vision: Hermes has vision (clipboard images). To use images, need `--mmproj`. Does the unsloth GGUF ship an mmproj? Let me check the UD-IQ4_XS dir listing. Actually, llama.cpp converts mmproj separately. Let me check.\n   - `--parallel 1` must stay due to the multi-slot bug.\n   - `--keep` / cache: for agentic workloads with long system prompts + tool schemas, `--cache-prompt` is good, but also `--keep -1`/`--prompt-cache` and `--cache-ram`. For Hermes, the prefix is mostly stable (system prompt + tool definitions) but tool results get appended → standard longest-prefix reuse works. There's also `--keep N` to pin the first N prompt tokens from being evicted. Since --parallel 1 there's only one slot so KV reuse across turns is straightforward as long as the request prefix matches. Note llama-server's automatic prefix caching works within a slot.\n   - Important: Hermes may send slightly varying requests. `--cache-reuse`? There is `--cache-reuse N`.\n   - `--api-key` weak on LAN, bound to 192.168.1.61 — fine-ish, mention.\n\n11. **`--split-mode layer` + `--tensor-split 1,1,1,1`**: With layer split and 4 GPUs, this is fine and generally the best for MoE. But: layer split means each GPU holds whole layers. 93.7GB/4 ≈ 23.4GB per card for weights, plus KV on top. With 4×24GB = 96GB, that leaves ~2GB per card for KV+compute = 8GB total. Since only 12/48 layers hold KV, and KV goes on whichever GPU holds that layer, the distribution can be uneven. Also compute buffers. Tight.\n    Actually with PLE (26GB) moved to CPU, GPU weights = ~67GB → 16.75GB/card, leaving ~7GB/card = 28GB total for KV + compute. Plenty for 131K ctx f16 (3.1GB) or even 262K (6.2GB). \n\n12. **Should the MoE experts go to CPU?** phhusson's regex offloads ffn to CPU for 3090. But the user has 96GB VRAM and only needs 67GB → no need to offload experts! That's a key insight. Keep everything on GPU except the ngram table. This is much better than phhusson's single-3090 approach.\n\n    But — should they move MORE to CPU to free VRAM for KV? No, KV is cheap here. Better: keep all layers on GPU, PLE on CPU. Then optionally bump ctx to 262144.\n\n13. **CPU RAM bandwidth is the real constraint.** 2133 MT/s quad-channel DDR4 on WRX80 = 68 GB/s peak, ~50 GB/s real. If the PLE table (26 GB) lives in RAM and is randomly accessed once per forward pass... The PR discussion says the n-gram table is read sparsely (a few rows per token), so bandwidth per token is tiny — it's latency-bound, not bandwidth-bound. `set_input` computes host-side row indices then `ggml_get_rows`. Random 2560-dim row gathers. ngram_size 3, split_ngram_parts 128.\n\n    Key: the PLE lookup happens on CPU and needs to be pipelined with the GPU. If the graph does get_rows on CPU then H2D copy per token, that's serialized latency. With `--override-tensor per_layer_token_embd=CPU`, hmm.\n\n    Also: is there an option to put the PLE on a specific buffer type, e.g., `--override-tensor \"per_layer_token_embd=CPU(pinned)\"` or host alloc? In llama.cpp, `-ot name=CPU` and there's `--no-hostalloc` to disable pinned host memory. Pinned host memory would speed the H2D copy. Since they have lots of RAM, keeping it pinned would be good — but wait, with `--load-mode none` does hostalloc apply? Let me check the flags.\n\n    Let me check llama.cpp docs for: `--tensor-types`/`--tensor-type`, `--cpu-moe`, `--n-cpu-moe`, `--override-tensor`, `--fit`, `--fit-target`, `--mmap`, `--no-hostalloc`, `--keep`, `--cache-ram`, `--parallel`, `--reasoning-preserve`, `--chat-template-parser`, `--spec-type` (MTP), `--dtk`/`--dtv` (draft kv cache types), `--no-mmproj`.\n\nLet me fetch:\n- unsloth GGUF README (for tensor names + recommended flags)\n- Baekpica SSD-PLE README (for ngram offload to disk, answering the user's OPTANE question)\n- llama.cpp docs/flags\n\nLet me do these in parallel.\n\nAlso check llama.cpp for `--load-mode` valid values and whether `--fit` default. Let's read llama.cpp docs for build/server. Better: fetch the PR files for args? Let me try `hf://` no. Let me use web_fetch on the raw llama.cpp common params — args.cpp is huge. Better to use context7 or search. Let me search specifically.\n\nLet me batch 4 calls:\n1. cat unsloth GGUF README\n2. cat Baekpica SSD-PLE README\n3. web_fetch llama.cpp docs for server flags (docs/DOCKER? no) — try https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md — huge. Maybe better to search for the flags. Let me fetch the PR's diff of args? The PR only adds model support, not flags. So the flags are stock llama.cpp master.\n\nLet me fetch the PR commits list to see if new flags were added (e.g. `--ple-*`). And check llama.cpp master args.cpp for `load-mode` and `fit`. I'll fetch raw args.cpp from github raw. That's a big file (~3000 lines). maxCharacters can handle. Better: fetch https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/common.cpp? Hmm. args.cpp contains the option table. Let me try to grep with targeted web_fetch on the GitHub search UI? Not great.\n\nLet me try `hf_fs` no. Let's just web_fetch raw args.cpp with high maxCharacters and search for relevant strings. Actually the tool returns clean text; I can scan. args.cpp is ~190KB. maxCharacters 100000 might truncate. Let me instead query context7 for llama.cpp? Context7 may have llama.cpp docs. Hmm.\n\nBetter approach: the llama.cpp wiki \"Server\" README lists all flags. But it's huge too.\n\nAlternative: fetch the PR's \"Files changed\" for args to check new flags. The PR is 21 files, mostly src/models. `--load-mode` already exists per kyuz0 usage. I mainly need to verify:\n- `--load-mode` values\n- `--fit` default (on/off)\n- `--fit-target` exists (yes, from ServeurpersoCom: `--fit-target 70000`)\n- PLE offload flags\n\nLet me do a targeted web search + fetch the GGUF READMEs. And also grab the llama.cpp PR's updated first post (it may have been updated with final recommended flags). Let me also check the Unsloth GGUF README which likely has recommended llama.cpp commands per quant.\n\nBatch now.\n","timings":{"cache_n":21948,"prompt_n":15758,"prompt_ms":43443.077,"prompt_per_token_ms":2.7568902779540547,"prompt_per_second":362.72752963607985,"predicted_n":6038,"predicted_ms":205822.536,"predicted_per_token_ms":34.09351267185688,"predicted_per_second":29.331093267648786},"toolCalls":[{"function":{"arguments":"{\"operations\":[{\"cmd\": \"cat\", \"args\": [\"hf://models/unsloth/Qwen3.8-Flash-Next-GGUF/README.md\", \"--max-bytes\", \"60000\"]}, {\"cmd\": \"ls\", \"args\": [\"hf://models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS\"]}, {\"cmd\": \"cat\", \"args\": [\"hf://models/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF/README.md\", \"--max-bytes\", \"30000\"]}]}","name":"hf_fs"},"id":"uLhMU7SfoX3QIIvOAYaZqajICyJyshHp","type":"function"},{"function":{"arguments":"{\"query\":\"llama.cpp --load-mode --fit --fit-target common params documentation values mmap\",\"numResults\":6}","name":"web_search_exa"},"id":"hioCqA5SR5Lte4BGEbcEtwpsiQ15HRKl","type":"function"}]},"type":"message"}
{"message":{"children":[],"content":"## Operation 1\n\n# hf_fs cat\n\nURI: `hf://models/unsloth/Qwen3.8-Flash-Next-GGUF/README.md`\nPath: `README.md`\nBytes: 58584\n\n---\nlicense: other\nlicense_name: qwen-community-1.0\nlicense_link: LICENSE\npipeline_tag: image-text-to-text\nbase_model:\n- Qwen/Qwen3.8-Flash-Next\ntags:\n- unsloth\n---\n# Read our How to [Run Qwen3.8-Flash-Next Guide!](https://unsloth.ai/docs/models/qwen3.8-next)\n<div>\n  <p style=\"margin: 0 0 0px 0; margin-top: 0px;\">\n    <em><a href=\"https://unsloth.ai/docs/basics/dynamic-3.0-ggufs\">Unsloth Dynamic 3.0</a> achieves superior accuracy & outperforms other leading quants.</em>\n  </p>\n  <div style=\"display: flex; gap: 5px; align-items: center; margin-bottom: 0px;\">\n    <a href=\"https://github.com/unslothai/unsloth/\">\n      <img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"133\">\n    </a>\n    <a href=\"https://discord.gg/unsloth\">\n      <img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png\" width=\"173\">\n    </a>\n    <a href=\"https://unsloth.ai/docs/models/qwen3.8-next\">\n      <img src=\"https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png\" width=\"143\">\n    </a>\n  </div>\n    <ul style=\"margin: 0;\">\n    <li>To run, please use our <a href=\"https://github.com/ggml-org/llama.cpp/pull/27742\">llama.cpp PR</a> or use our <a href=\"https://github.com/unslothai/unsloth\">Unsloth Desktop</a> app.</li>\n</div>\n\n---\n\n# Qwen3.8-Flash-Next\nAs the frontier of foundation models pushes toward ever-larger parameter counts and ever-longer context windows, the question is no longer just how much we can scale, but how efficiently we can do so. Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation. Today, we are sharing a concrete step in that direction: Qwen3.8-Flash-Next. \n\n![Qwen3.8-Flash-Next Architecture](https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.8-Flash-Next/architecture.png)\n\nThis experimental preview of the architecture that will underpin Qwen4 is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale.\n \n## Highlights\n\nThe first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:\n\n- **Hybrid Attention with QSA**: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.\n- **Gated Residual**: Residual streams with normalization are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.\n- **N-gram Embedding**: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.\n- **Tailored Training Recipe**: The Muon and AdamW optimizers are applied to specific weight categories to maximize efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimizer steps while safely supporting larger learning rates for robust convergence.\n\nFor more details, please refer to our blog post [Qwen3.8-Flash-Next](https://qwen.ai/blog?id=qwen3.8-flash-next) and [the technical report](https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf).\n\nWe are excited to embark on this next chapter with you and welcome your feedback as we build what comes next.\n\n## Model Overview\n\n- Type: Causal Language Model with Vision Encoder\n- Training Stage: Pre-training & Post-training\n- Language Model\n    - Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP\n    - Hidden Dimension: 2560\n    - Token Embedding: 248320 (Padded)\n    - N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)\n    - Number of Layers: 48\n    - Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))\n    - Gated DeltaNet:\n        - Number of Linear Attention Heads: 48 for V and 16 for QK\n        - Head Dimension: 128\n    - Qwen Sparse Attention:\n        - Number of Attention Heads: 24 for Q and 2 for KV\n        - Head Dimension: 256\n        - Rotary Position Embedding Dimension: 64\n        - Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head\n        - Indexer Head Dimension: 128\n        - Budget: 512 blocks or 2048 tokens\n    - Mixture Of Experts\n        - Number of Experts: 512\n        - Number of Activated Experts: 10 Routed + 1 Shared\n        - Expert Intermediate Dimension: 640\n    - Gated Residual:\n        - Number of Branches: 4\n        - Bottleneck Rank: 320\n    - LM Output: 248320 (Padded)\n    - MTP: 1 layer, trained with multi-steps\n- Context Length: 262,144 natively and extensible up to 1,000,000 tokens.\n\n## Benchmark Results\n\n<style>\n.vl-table th{font-size:15px!important;line-height:1.2}\n.vl-table td:not(.benchmark-cell):not([colspan]){font-size:15px;line-height:1.2;vertical-align:middle}\n.vl-table .benchmark-cell{padding:12px 10px 12px 18px!important;vertical-align:middle}\n.vl-table .benchmark-capability{font-size:15px;font-weight:600;line-height:1.22;color:#171717}\n.vl-table .benchmark-name{margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B}\n.vl-table .metric-stack{display:flex;flex-direction:column;gap:7px;padding:3px 0}\n.vl-table .metric-label{font-size:10px;font-weight:400;line-height:1.1;color:#777}\n.vl-table .metric-value{margin-top:2px;font-size:15px;line-height:1.15;color:#171717}\n.vl-table .metric-pair{white-space:nowrap}\n.vl-table .metric-sep{color:#9A9A9A;padding:0 3px}\n</style>\n\n### Language\n\n<div style=\"font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1200px;margin:0 auto;padding:16px 0\">\n<table class=\"vl-table\" style=\"width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px\">\n<thead><tr>\n<th style=\"padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #0A2EFE;color:#0A2EFE\"></th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;background:rgba(10, 46, 254, 0.08);\">Qwen3.8-Flash-Next</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;\">Qwen3.8-27B</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;\">Qwen3.7-Plus</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;\">DeepSeek-V4-Flash-0731</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;\">Claude-Opus-4.6 (Max)</th></tr></thead>\n<tbody>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\"># Params</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\">125B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">27B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">397B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">284B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\"># Activated params</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\">6B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">27B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">17B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">13B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\"># N-gram embedding params</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\">51B</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr><td colspan=\"6\" style=\"padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC\">Coding</td></tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Agentic coding</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">DeepSWE 1.1</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>58.7</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">42.2</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">16.5</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">54.4</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Agentic coding</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">SWE-bench Pro</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>62.5</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">61.7</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">55.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">56.0</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">53.4</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Multilingual software engineering</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">SWE-bench Multilingual</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>81.0</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">73.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">75.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">77.5</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Repo-level code generation</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">NL2Repo-Bench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\">48.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">42.3</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">41.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>54.2</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">47.6</td>\n</tr>\n<tr><td colspan=\"6\" style=\"padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC\">Agent</td></tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Long-horizon office work</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">CoWorkBench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>73.9</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">70.7</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">65.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">45.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">68.2</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Professional job tasks</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">JobBench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>55.7</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">33.4</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">27.6</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">41.3</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">36.6</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Frontier agentic tasks</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">Agents' Last Exam</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@1</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">24.3</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Score</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>51.2</strong></div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@1</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">20.4</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Score</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">42.9</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@1</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">13.2</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Score</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">33.6</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@1</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>25.2</strong></div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Score</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">--</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Real-world tool use</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">Toolathlon Verified (Pass@1)</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>73.5</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">67.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">50.6</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">70.3</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr><td colspan=\"6\" style=\"padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC\">General</td></tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Instruction following</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">IFBench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>81.3</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">79.5</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">79.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">79.2</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">62.5</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Scientific reasoning</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">GPQA Diamond</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>91.7</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">89.2</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">90.3</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">90.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">91.3</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Multidisciplinary reasoning</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">HLE</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\">35.9</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">30.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">34.7</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">33.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>40.0</strong></td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Competitive coding</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">LiveCodeBench v6</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>91.9</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">90.3</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">89.6</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">90.6</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">88.8</td>\n</tr>\n</tbody>\n</table>\n<p style=\"margin-top:12px;font-size:10px;line-height:1.4;opacity:0.7\">1. DeepSWE 1.1: evaluated with the Claude Code and mini-SWE-agent harnesses, temp=1.0, top_p=0.95, 256K context window. We report the highest score across the two harnesses; notably, Qwen3.8-Flash-Next performs best on mini-SWE-agent.<br>2. SWE-bench Pro: except for Claude-Opus-4.6 (Max), for which we report the officially published score, all models are evaluated with the Claude Code harness, temp=1.0, top_p=0.95, 256K context window. Problematic tasks were corrected and all baseline models were re-evaluated on the refined benchmark.<br>3. SWE-bench Multilingual: evaluated with the mini-SWE-agent harness, temp=1.0, top_p=0.95, 256K context window.<br>4. NL2Repo-Bench: evaluated with the Claude Code harness. To prevent reward hacking, we disable Bash commands that attempt to access the specific repository, such as pip download, pip install and git clone.<br>5. CoWorkBench: an in-house cowork benchmark for evaluating long-horizon office and productivity agent tasks across computer science, finance, law, medical and other productivity domains.<br>6. HLE: judged by GPT-4o.<br>7. The best result in each row is shown in bold.<br>8. Empty cells (--): scores are not yet available or are not applicable.</p>\n</div>\n\n### Vision Language\n\n<div style=\"font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0\">\n<table class=\"vl-table\" style=\"width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px\">\n<thead><tr>\n<th style=\"padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #0A2EFE;color:#0A2EFE\"></th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;background:rgba(10, 46, 254, 0.08);\">Qwen3.8-Flash-Next</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;\">Qwen3.8-27B</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;\">Qwen3.7-Plus</th><th style=\"padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;\">Claude-Opus-4.6 (Max)</th></tr></thead>\n<tbody>\n<tr><td colspan=\"5\" style=\"padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC\">Agentic Multimodal Intelligence</td></tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Multimodal tool use</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">ClawEval-MM</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@3</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>64.4</strong></div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Average</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>60.4</strong></div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@3</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">57.4</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Average</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">56.9</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@3</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">57.4</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Average</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">60.1</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Pass@3</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">52.5</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Average</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">54.7</div></div></div></td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Application recreation</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">RecreationBench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>49.9</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">47.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">30.2</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Mobile use</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">AndroidWorld</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>84.5</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">81.9</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">81.0</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">62.0</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Computer use</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">OSWorld 2.0</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Binary</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>19.4</strong></div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Partial</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>52.3</strong></div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Binary</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">19.4</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Partial</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">48.0</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Binary</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">2.8</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Partial</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">21.5</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Visual web development</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">Vision2Web</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>64.0</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">62.9</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">42.1</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">--</td>\n</tr>\n<tr><td colspan=\"5\" style=\"padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC\">General Multimodal Intelligence</td></tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Embodied intelligence</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">ERQA</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>72.3</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">65.5</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">69.8</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">40.8</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Long video understanding</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">LVBench</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>76.6</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">72.4</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">76.2</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">63.0</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Real-world perception</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">RealWorldQA</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><strong>88.5</strong></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">85.9</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">86.9</td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\">73.9</td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Visual math problem solving</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">MathVision</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>90.6</strong></div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>95.7</strong></div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">90.0</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">94.6</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">90.3</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">88.7</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">65.5</div></div></div></td>\n</tr>\n<tr>\n<td class=\"benchmark-cell\" style=\"padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);\"><div class=\"benchmark-capability\" style=\"font-size:15px;font-weight:600;line-height:1.22;color:#171717\">Scientific chart analysis</div><div class=\"benchmark-name\" style=\"margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B\">CharXiv (RQ)</div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">84.6</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>90.6</strong></div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">83.7</div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">90.2</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\"><strong>85.8</strong></div></div><div style=\"margin-top:7px\"><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">With CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">85.9</div></div></div></td>\n<td style=\"padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;\"><div class=\"metric-stack\" style=\"padding:3px 0\"><div><div class=\"metric-label\" style=\"font-size:10px;font-weight:400;line-height:1.1;color:#777\">Without CI</div><div class=\"metric-value\" style=\"margin-top:2px;font-size:15px;line-height:1.15;color:#171717\">66.0</div></div></div></td>\n</tr>\n</tbody>\n</table>\n<p style=\"margin-top:12px;font-size:10px;line-height:1.4;opacity:0.7\">1. ClawEval-MM: scores are reported as \"pass@3 / average score\". Pass@3 measures the percentage passed in at least one of three trials, and the average score is the mean score across the three trials.<br>2. RecreationBench: an in-house long-horizon application-recreation benchmark for evaluating hybrid-agent abilities spanning five platforms — desktop (Ubuntu, macOS, Windows), mobile (Android) and web.<br>3. OSWorld 2.0: scores are reported as \"binary / partial\". The binary score is the percentage of tasks that receive the full task reward, while the partial score aggregates the partial rewards obtained across all tasks.<br>4. Vision2Web: scores are reported as the average over the frontend, webpage and website categories, using the Claude Code harness and judged by gpt-5.4-2026-03-05.<br>5. MathVision, CharXiv (RQ): scores are reported as \"without CI / with CI\". A small number of incorrect ground-truth annotations in MathVision were corrected after manual verification. Our model's score is evaluated using a fixed prompt, e.g. \"Please reason step by step, and put your final answer within \\boxed{}.\" For other models, we report the higher score between runs with and without the \\boxed{} formatting.<br>6. The best result in each row is shown in bold.<br>7. Empty cells (--) indicate scores not yet available or not applicable.</p>\n</div>\n\n### API Usage\n\n> [!Important]\n> Qwen3.8-Flash-Next models operate in thinking mode by default, generating thinking content signified by `<think>\\n...</think>\\n\\n` before producing the final responses.\n> To disable thinking content and obtain direct response, refer to the examples [here](#instruct-or-non-thinking-mode).\n\n> [!Tip]\n> We recommend using the following sets of sampling parameters for generation:\n> - Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`\n> - Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`\n>\n> Please note that the support for sampling parameters varies according to inference frameworks.\n\n> [!Tip]\n> In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.\n\nQwen3.8-Flash-Next supports controlling thinking behavior via `enable_thinking`, `preserve_thinking`, and `reasoning_effort`.\n\nt; supported levels are xhigh, medium, and low\n    stream=True,\n    stream_options={\"include_usage\": True},\n)\n\n\n##### Instruct (or Non-Thinking) Mode\n\nQwen3.8-Flash-Next will think by default before responding.\nYou can obtain a direct response from the model without thinking by configuring the API parameters. \nFor example,\n```python\nfrom openai import OpenAI\n# Configured by environment variables\nclient = OpenAI()\n\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"image_url\",\n                \"image_url\": {\n                    \"url\": \"https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/RealWorld/RealWorld-04.png\"\n                }\n            },\n            {\n                \"type\": \"text\",\n                \"text\": \"Where is this?\"\n            }\n        ]\n    }\n]\n\nchat_response = client.chat.completions.create(\n    model=\"Qwen/Qwen3.8-Flash-Next\",\n    messages=messages,\n    temperature=0.7,\n    top_p=0.8,\n    presence_penalty=1.5,\n    extra_body={\n        \"top_k\": 20,\n        \"chat_template_kwargs\": {\"enable_thinking\": False},\n    }, \n)\nprint(\"Chat response:\", chat_response)\n```\n\n> [!Note]\n> If you are using APIs from Qwen Cloud, in addition to changing `model`, please use `\"enable_thinking\": False` instead of `\"chat_template_kwargs\": {\"enable_thinking\": False}`.\n\n\n##### Disable Preserved Thinking\n\nBy default, Qwen3.8-Flash-Next retains thinking blocks from all historical messages, maintaining a complete reasoning trace across the conversation. This behavior, known as preserved thinking, ensures full context continuity and is especially beneficial for agent scenarios where decision consistency and reduced redundant reasoning are critical. It also improves KV cache utilization, optimizing inference efficiency in both thinking and non-thinking modes.\n\nIf you prefer to retain only the thinking blocks from the latest user message, you can disable this behavior by setting `preserve_thinking` to `False`:\n\n```python\nfrom openai import OpenAI\n\n# Configured by environment variables\nclient = OpenAI()\nmessages = [...]\nchat_response = client.chat.completions.create(\n    model=\"Qwen/Qwen3.8-Flash-Next\",\n    messages=messages,\n    extra_body={\n        \"chat_template_kwargs\": {\"preserve_thinking\": False},\n    },\n)\nprint(\"Chat response:\", chat_response)\n```\n\n> [!Note]\n> If you are using APIs from Qwen Cloud, in addition to changing `model`, please use `\"preserve_thinking\": False` directly instead of wrapping it in `chat_template_kwargs`.\n\n\n\n## Best Practices\n\nTo achieve optimal performance, we recommend the following settings:\n\n1. **Sampling Parameters**: We suggest using the following sets of sampling parameters:  \n    \n    - Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`\n    - Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`\n    \n    For supported frameworks, you can adjust the `presence_penalty` parameter between 0 and 2 to reduce endless repetition. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.\n\n2. **Adequate Output Length**: To optimize performance on agentic tasks, we recommend allocating sufficient output length to allow the model to generate detailed and comprehensive responses. For frameworks that support separate token limits for internal reasoning and final outputs, we suggest the following configuration within the 1M context length:\n    \n    - Reasoning Content: Set the maximum output length to 262,144 tokens.\n    - Final Response: Set the maximum output length to 131,072 tokens.\n\n    These settings provide the necessary capacity for complex reasoning while ensuring ample space for high-quality final deliverables.\n\n3. **Processing Ultra-Long Texts**: Qwen3.8-Flash-Next natively supports context lengths of up to 262,144 tokens. For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.\n\n    YaRN is currently supported by several inference frameworks, e.g., vLLM, SGLang, and TokenSpeed. \n    In general, there are two approaches to enabling YaRN for supported frameworks:\n\n    - Modifying the model configuration file:\n        \n        In the `config.json` file, change the `rope_parameters` fields in `text_config` to:\n        ```json\n        {\n            \"mrope_interleaved\": true,\n            \"mrope_section\": [\n                11,\n                11,\n                10\n            ],\n            \"rope_type\": \"yarn\",\n            \"rope_theta\": 10000000,\n            \"partial_rotary_factor\": 0.25,\n            \"factor\": 4.0,\n            \"original_max_position_embeddings\": 262144\n        }\n        ```\n    \n    > [!NOTE]\n    > All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.**\n    > We advise modifying the `rope_parameters` configuration only when processing long contexts is required. \n    > It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set `factor` as 2.0. \n\n\n4. **Long Video Understanding**: To optimize inference efficiency for plain text and images, the `size` parameter in the released `video_preprocessor_config.json` is conservatively configured. It is recommended to set the `longest_edge` parameter in the video_preprocessor_config file to 469,762,048 (corresponding to 224k video tokens) to enable higher frame-rate sampling for hour-scale videos and thereby achieve superior performance. For example,\n    ```json\n    {\"longest_edge\": 469762048, \"shortest_edge\": 4096}\n    ```\n\n    Alternatively, override the default values via engine startup parameters. For implementation details, refer to: [vLLM](https://github.com/vllm-project/vllm/pull/34330) / [SGLang](https://github.com/sgl-project/sglang/pull/18467).\n\n\n### Citation\n\nIf you find our work helpful, feel free to give us a cite.\n\n```bibtex\n@techreport{qwen2026design,\n    title       = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},\n    author      = {{Qwen Team}},\n    institution = {Alibaba Group},\n    month       = {August},\n    year        = {2026}\n}\n\n@misc{qwen3.8flashnext,\n    title  = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency},\n    author = {{Qwen Team}},\n    month  = {August},\n    year   = {2026},\n    url    = {https://qwen.ai/blog?id=qwen3.8-flash-next}\n}\n```\n\n---\n\n## Operation 2\n\n# hf_fs ls\n\nURI: `hf://models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS`\n\n| Type | Path | URI | Target | Details |\n|---|---|---|---|---|\n| file | UD-IQ4\\_XS/Qwen3.8-Flash-Next-UD-IQ4\\_XS-00001-of-00003.gguf |  |  | lfs, size=10.9 MB |\n| file | UD-IQ4\\_XS/Qwen3.8-Flash-Next-UD-IQ4\\_XS-00002-of-00003.gguf |  |  | lfs, size=49.8 GB |\n| file | UD-IQ4\\_XS/Qwen3.8-Flash-Next-UD-IQ4\\_XS-00003-of-00003.gguf |  |  | lfs, size=43.8 GB |\n\n---\n\n## Operation 3\n\n# hf_fs cat\n\nURI: `hf://models/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF/README.md`\nPath: `README.md`\nBytes: 13461\n\n---\nlicense: other\nlicense_name: qwen-community-1.0\nlicense_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/f5d08274bafd880402bd16f5e3e6c514136ec06c/LICENSE\nbase_model: Qwen/Qwen3.8-Flash-Next\nbase_model_relation: quantized\npipeline_tag: image-text-to-text\ntags:\n  - gguf\n  - mixed-quant\n  - qwen4exp\n  - qwen3.8-flash-next\n  - dgx-spark\n  - ds4\n  - ssd-offload\n---\n\n# Qwen3.8-Flash-Next Mixed-Quant SSD-PLE GGUF\n\n> **Single-DGX-Spark ds4 serving at a configured 262,144-token context is verified; performance tuning remains.** The complete [`MQ-Q6-SSD-PLE-BF16`](https://huggingface.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF/tree/main/MQ-Q6-SSD-PLE-BF16) set is published: four Q5/Q6 main-GGUF shards plus four BF16 PLE sidecars. Local structural/full-file checks and remote LFS hashes match. The bounded `O_DIRECT` cache, exact row hashing, CUDA UVA gather, complete PLE injection path, 48-layer prefill/decode, native tokenizer/final logits, and OpenAI-compatible API paths have now run together on an NVIDIA GB10. This is a functional serving result at the configured 262K capacity, not a claim that a full 262K-token prompt, model quality, or final performance has been validated.\n\nThis is a memory-hierarchy-aware mixed-precision conversion of [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), pinned to revision [`f5d08274bafd880402bd16f5e3e6c514136ec06c`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/tree/f5d08274bafd880402bd16f5e3e6c514136ec06c).\n\nThe design moves the 51.2B-parameter predictive latent embedding (PLE) table out of the accelerator-resident GGUF weight set and stores it as BF16 SSD-backed sparse lookup memory. Nearly all of the released residency budget is then reassigned to the 128.8B-parameter compute backbone. Compatibility with upstream llama.cpp, vLLM, SGLang, or other GGUF runtimes is **not** implied; a dedicated ds4 SSD-PLE loader is required.\n\n## Target recipe\n\n| Model region | Storage / quantization | Residency policy |\n|---|---:|---|\n| **51.2B-parameter PLE n-gram table (128 shards)** | BF16 SSD sidecar | non-resident; bounded pinned page cache only |\n| **Routed expert gate/up, interior layers 2–45** | `Q5_K` | accelerator-resident |\n| **Routed expert gate/up, edge layers 0, 1, 46, 47** | `Q6_K` | accelerator-resident |\n| **Routed expert down, main 512 columns** | `Q6_K` | accelerator-resident |\n| **Routed expert down, 128-column tail** | `Q5_0` | accelerator-resident |\n| **MTP routed experts and most always-active matrices** | primarily `Q8_0` | accelerator-resident |\n| **Hyper-connection, non-quantizable convolution/vision tensors** | BF16 | accelerator-resident |\n| **Norms, gates, recurrent/control state** | F32 where required | accelerator-resident |\n| **Integer PLE controls** | I64 | accelerator-resident |\n\nThe requested 128-column tail target was `Q5_K`, but standard GGUF `Q5_K` uses 256-value blocks and therefore cannot encode a tensor whose row width (`ne[0]`) is 128. The first artifact uses `Q5_0`, which is valid at width 128 and has the same 5.5-bit nominal storage cost. It is not mislabeled as `Q5_K`. A paired-row custom `Q5_K` layout would require a nonstandard MMQ kernel and is deferred.\n\n## Separate memory metrics\n\n`Overall model BPW` is not the useful metric for this artifact. The audited Phase-1 census is:\n\n| Metric | Exact value |\n|---|---:|\n| Main GGUF logical parameters | 128,799,735,699 |\n| Main GGUF tensor count | 1,628 |\n| **Resident backbone payload** | **97,660,877,400 bytes / 90.9538 GiB** |\n| **Resident backbone effective BPW** | **6.0659** |\n| Actual four-shard main GGUF files | 97,671,932,960 bytes / 90.9641 GiB |\n| PLE parameters | 51,200,245,760 |\n| **PLE storage dtype** | **BF16 (16 bpw)** |\n| **PLE payload on SSD** | **102,400,491,520 bytes / 95.3679 GiB** |\n| Planned PLE files including 4 KiB alignment | 102,400,786,432 bytes / 95.3682 GiB |\n| Total stored payload, before GGUF metadata | 200,061,368,920 bytes / 186.3217 GiB |\n| Actual eight weight files | 200,072,719,392 bytes / 186.3322 GiB |\n\nThe published resident-PLE baseline spends 38,400,184,320 bytes / 35.7630 GiB on a Q5_1 PLE table. The new compute backbone costs 37,570,478,080 bytes / 34.9902 GiB more, so the new resident payload is **829,706,240 bytes / 0.7727 GiB smaller** than that baseline. These are exact tensor payload calculations using GGUF block sizes; final file size, CUDA allocations, page cache, KV state, and runtime workspace are separate.\n\n## Published artifact layout\n\n```text\nMQ-Q6-SSD-PLE-BF16/\n├── Qwen3.8-Flash-Next-MQ-Q6-SSD-PLE-BF16-*.gguf\n├── SHA256SUMS\n├── quant-recipe.yaml\n├── artifact-manifest.json\n└── ple/\n    ├── ple-bf16-00001-of-00004.bin\n    ├── ple-bf16-00002-of-00004.bin\n    ├── ple-bf16-00003-of-00004.bin\n    ├── ple-bf16-00004-of-00004.bin\n    └── ple-manifest.json\n```\n\nThe 128 logical checkpoint shards remain individually addressable through `ple-manifest.json`; four larger physical files reduce descriptor and Hub-transfer overhead. Every logical-shard start is 4 KiB aligned. The manifest records the source tensor, global row range, physical file and offset, row stride, per-head prime vocabulary sizes and offsets, SplitMix64-derived layer multipliers, total bytes, and checksums.\n\n## SSD runtime contract\n\nThe full 95.37 GiB PLE sidecar must never become resident in the 128 GB unified-memory pool. The planned ds4 path uses exact CPU-side Qwen n-gram row derivation, asynchronous SSD prefetch overlapped with decoder layer 0, a bounded 512 MiB / 1 GiB / 2 GiB pinned mapped page cache, and a separate CUDA UVA gather kernel. `O_DIRECT` plus aligned asynchronous reads is preferred; a buffered fallback must evict sidecar pages with an explicit `POSIX_FADV_DONTNEED`-class policy.\n\nRegistering the whole file with CUDA, relying on unbounded `mmap`, or allowing Linux page cache to retain the full sidecar fails the residency objective. A successful load of the main GGUF alone is not proof that SSD-PLE works.\n\nThe current ds4 handoff implementation covers the correctness-critical core of\nthis contract: exact EOS-bounded CPU hashing, strict manifest validation,\nbounded four-way page caching, asynchronous worker prefetch, `O_DIRECT` with a\npage-cache-dropping fallback, mapped-cache UVA gathering, and cache-slot leases\nheld until the consuming CUDA stream has completed. Decoder-layer-0 overlap\nand cache-size tuning are target-hardware integration work, not claims made by\nthis artifact.\n\n## DGX Spark ds4 serving verification\n\nOn 2026-08-27, [`Baekpica/ds4@407ddf9`](https://github.com/Baekpica/ds4/commit/407ddf9)\non branch [`dfm`](https://github.com/Baekpica/ds4/tree/dfm) was verified on one\nDGX Spark (NVIDIA GB10) with the server configured for a 262,144-token context.\n\n- The worker imported a content-verified 91.05 GiB shared VMM weight cache and\n  allocated a 14.33 GiB 262K session graph. The ds4 memory census reported\n  105.43 GiB device-live memory.\n- The four PLE sidecars opened with `O_DIRECT` (`4/4`) behind a bounded 512 MiB\n  cache. A bounded `mincore` check after inference found 0.00 MiB of the\n  95.37 GiB sidecar set resident in Linux page cache; the sidecars were not\n  present in the server's memory map.\n- Chat Completions and Responses returned coherent deterministic output.\n  Buffered and streaming Responses, a function call, and its\n  `function_call_output` continuation completed successfully.\n- Two simultaneous requests completed through the current serial lane with no\n  request, graph-fit, memory-census, or memory-governor failures.\n- A 3,069-token marker-retrieval prompt returned the exact marker at 83.0\n  prefill tok/s. This is the longest prompt used in this smoke test.\n\nThe host run used a 115 GiB hard memory cap and a 6 GiB admission floor; no\ncgroup high/max/OOM event occurred. These observations establish functional\nserving with the runtime configured for 262,144 tokens. They do **not** establish\na completed 262,144-token prompt run, full-model quality, sustained throughput,\nor SSD latency percentiles.\n\n## Resident BF16 versus SSD-offload validation\n\nThe handoff gate compares the production offload path against the 128 resident\nBF16 PLE tensors in the public BF16 GGUF. It does not use the SSD cache as its\nown reference.\n\n- Three tokens, including an EOS boundary, produced 48 row lookups and 7,680\n  BF16 values.\n- A deliberately tiny 16 KiB/four-page cache was churned with 257 unrelated\n  reads, producing 366 evictions; gathering the original rows then required 48\n  new physical reads.\n- Both the first and post-eviction SSD gathers matched the resident BF16 rows\n  byte-for-byte.\n- Key/value projections, transformed gate, gated value, convolution input,\n  BF16 dilated convolution, final 122,880-byte PLE injection, and 368,640-byte\n  persistent convolution state were all byte-exact between the two paths.\n- `O_DIRECT` was active on 4/4 sidecars. Repeated ordinary and\n  `CUDA_LAUNCH_BLOCKING=1` runs stayed around 402 MiB peak process RSS; the\n  95.3682 GiB sidecar was never mapped or registered in full.\n\nBroader rerun gates also passed: 595,616 row IDs against the pinned\nTransformers implementation, 1,023 BF16 rows across 33 original safetensor\nshards, an eight-reader cache stress, and two concurrent CUDA gathers of 4,112\nrows each (including 253 page-crossing rows). These are correctness and bounded\nresidency observations on the H200 development host, not DGX Spark throughput\nmeasurements.\n\nThe corresponding implementation and tests are checkpointed in\n[`Baekpica/ds4@ebc3b72`](https://github.com/Baekpica/ds4/commit/ebc3b72)\non branch\n[`feature/qwen38-ssd-ple-handoff`](https://github.com/Baekpica/ds4/tree/feature/qwen38-ssd-ple-handoff).\n\n## Current verification status\n\n- **Passed:** exact source topology and revision pin.\n- **Passed:** Q5_K/Q6_K block-writer round-trip, finite-output, RMSE, and thread-determinism unit gates.\n- **Passed:** main-GGUF template audit: 1,628 tensors, expected type/shape/byte census, zero verifier errors.\n- **Passed:** bit-exact BF16 sidecar extraction, 128 logical-part hashes, reread whole-file checksums, and four-of-four remote LFS SHA-256/byte-count matches.\n- **Passed:** complete four-shard Q5/Q6 main-GGUF conversion, 1,628-tensor structural audit with zero errors, independent full-file SHA-256 checks, and four-of-four remote LFS SHA-256/byte-count matches.\n- **Passed:** exact PLE row-ID derivation, source-safetensor row checks, bounded-cache concurrency, CUDA gather, forced-eviction reload, resident-BF16 A/B, complete PLE forward, and persistent-state comparisons.\n- **Passed in isolated production-shape/real-weight paths:** PLE chunk/decode state, all 36 GDN blocks' required primitives, all 12 QSA blocks' required primitives, four-stream hyper-connections, and Q5/Q6/Q5_0 text-MoE kernels.\n- **Passed on DGX Spark:** integrated 48-layer prefill/decode, native tokenizer/final logits, Chat and Responses serving, streaming, and tool-output continuation with a server configured for a 262,144-token context.\n- **Passed on DGX Spark:** bounded-residency smoke test with 105.43 GiB device-live census, 512 MiB PLE cache, `O_DIRECT` on 4/4 sidecars, and 0.00 MiB sidecar page-cache residency after inference.\n- **Pending:** true multi-sequence batching, slot reuse, MTP, full-model quality comparisons, and a full-length 262,144-token prompt run.\n- **Pending:** repeatable nsys/ncu optimization results and SSD latency-distribution measurements.\n\nNo full-model quality, full-length 262K prompt, sustained-throughput, or\nSSD-stall-distribution claim is made before those remaining gates pass.\n\n## Required performance reporting\n\nThe final report will separate model resident bytes, bounded PLE-cache bytes, CUDA allocations, system used/available memory, KV/cache allocations, and peak prefill/decode memory. SSD counters include unique pages per token, cache hit/miss ratio, logical/physical bytes per token, IOPS/bandwidth, read latency p50/p95/p99, prefetch overlap, and—most importantly—PLE wait-at-layer mean/p50/p95/p99. Serving metrics include TTFT, TPOT, prefill/decode tok/s, and MTP draft/accepted-token statistics.\n\n## Runtime implementation references\n\nThe ds4 implementation will consult the public serving paths where they provide relevant Qwen3.8-Flash-Next architecture and scheduling evidence:\n\n- [SGLang Qwen3.8-Flash-Next cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next)\n- [vLLM Qwen3.8-Flash-Next recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next)\n- [TokenSpeed Qwen3.8-Flash-Next recipe](https://lightseek.org/tokenspeed/recipes/models#qwen3-8-flash-next)\n- [Qwen3.8-Flash-Next official repository](https://github.com/QwenLM/Qwen3.8-Flash-Next)\n\nThese engines do not establish compatibility with this external-PLE GGUF layout.\n\n## Public mixed-quant references\n\nThe workflow is informed by the publicly accessible model releases in [Baekpica/ds4-mixed-quant-for-spark](https://hf.co/collections/Baekpica/ds4-mixed-quant-for-spark). They are referenced as public precedents for region-aware quantization, artifact auditing, and Spark memory budgeting; no private handoff bucket is required to understand this model card.\n\n## License\n\nThe converted weights remain governed by the original **Qwen Community License 1.0**. The exact pinned upstream license is included with the artifact. No Apache-2.0 license is claimed.\n","convId":"incyjf36ghq","role":"tool","timestamp":1787790397381,"toolCallId":"uLhMU7SfoX3QIIvOAYaZqajICyJyshHp","type":"text","id":"rdyk4h1avxo","parent":"uvhsx10tlkn"},"type":"message"}
{"message":{"children":["ay6axiw8owv"],"content":"Title: llama.cpp/tools/cli/README.md at master · ggml- ...\nURL: https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK) |\n...\n| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (env: LLAMA_ARG_MMAP) |\n...\nndio, --\n...\n-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available (env: LLAMA_\n...\n_DIO) |\n...\n| `-lm, --load-mode MODE` | model loading mode (default: auto) - auto: mmap, unless a device does not support it - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) |\n...\n| `-fit, --fit [on\\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |\n...\n| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |\n\n---\n\nTitle: llama.cpp/tools/server/README.md at master · ggml-org ...\nURL: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK) |\n...\n| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (env: LLAMA_ARG_MMAP) |\n...\nload-mode\n...\n| `-lm, --load-mode MODE` | model loading mode (default: auto) - auto: mmap, unless a device does not support it - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) |\n...\n| `-fit, --fit [on\\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |\n...\n| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |\n...\nFor string options like `--load-mode`, the environment variable is handled as shown in this example:\n...\n- `LLAMA_ARG_LOAD_MODE=auto` sets the loading mode to auto (default)\n- `LLAMA_ARG_LOAD_MODE=none` disables special loading\n- `LLAMA_ARG_LOAD_MODE=mmap` enables memory-mapping\n- `LLAMA_ARG_LOAD_MODE=mlock` locks the model in RAM\n- `LLAMA_ARG_LOAD_MODE=mmap+mlock` enables memory-mapping and locks in RAM\n- `LLAMA_ARG_LOAD_MODE=dio` uses DirectIO if available\n\n---\n\nTitle: tools/cli/README.md at e1a9a6dcbefccb4b864d9385ce8494f2a7f2ffcd · ggml-org/llama.cpp\nURL: https://github.com/ggml-org/llama.cpp/blob/e1a9a6dcbefccb4b864d9385ce8494f2a7f2ffcd/tools/cli/README.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n| `--mmap, --no-mmap` | whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled) (env: LLAMA_ARG_MMAP) |\n...\n| `-fit, --fit [on\\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |\n...\n| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |\n\n---\n\nTitle: llama-server(1) — llama.cpp-tools — Debian unstable\nURL: https://manpages.debian.org/unstable/llama.cpp-tools/llama-server.1.en.html\nPublished: 2026-08-15T00:00:00.000Z\nAuthor: N/A\nHighlights:\n(env: LLAMA_ARG_DEFRAG_THOLD) --mlock DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK) --mmap, --no-mmap DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (env: LLAMA_ARG_MMAP)\n...\n-lm, --load-mode MODE model loading mode (default: auto)\n...\n- auto: mmap, unless a device does not support it - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) --numa TYPE attempt optimizations that help on some NUMA systems - distribute: spread execution evenly over all nodes - isolate: only spawn threads on CPUs on the node that execution started on - numactl: use the CPU map provided by numactl if run without this previously, it is recommended to drop the system page cache before using this see https://github.com/ggml-org/llama.cpp/issues/1437(env: LLAMA_ARG_NUMA)\n...\n-fit, --fit [on|off] whether to adjust unset arguments to fit in device memory ('on' or\n...\n'off', default: 'on') (env: LLAMA_ARG_FIT)\n...\n-fitt, --fit-target MiB0,MiB1,MiB2,...\n...\ntarget margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET)\n\n---\n\nTitle: tools/server/README.md at 873c825611d9cb76427931b5e74642bade4853dd · ggml-org/llama.cpp\nURL: https://github.com/ggml-org/llama.cpp/blob/873c825611d9cb76427931b5e74642bade4853dd/tools/server/README.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n| `--mmap, --no-mmap` | whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled) (env: LLAMA_ARG_MMAP) |\n...\n| `-fit, --fit [on\\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |\n...\n| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |\n...\nFor boolean options like `--mmap` or `--kv-offload`, the environment variable is handled as shown in this example:\n...\n- `LLAMA_ARG_MMAP=true` means enabled, other accepted values are: `1`, `on`, `enabled`\n- `LLAMA_ARG_MMAP=false` means disabled, other accepted values are: `0`, `off`, `disabled`\n- If `LLAMA_ARG_NO_MMAP` is present (no matter the value), it means disabling mmap\n\n---\n\nTitle: docs/multi-gpu.md\nURL: https://github.com/ggerganov/llama.cpp/blob/master/docs/multi-gpu.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n## Command-line arguments reference\n...\n| Short | Long | Value | Default | Notes |\n|---|---|---|---|---|\n| `-sm` | `--split-mode` | `none` \\| `layer` \\| `tensor` | `layer` | See modes above. |\n...\n| `-ts` | `--tensor-split` | comma-separated proportions, e.g. `3,1` | mode-dependent | How much of the model goes to each GPU. If omitted, `layer`/`row` use automatic splitting proportional to memory, while `tensor` splits tensor segments evenly. With `3,1` on two GPUs, GPU 0 gets 75 %, GPU 1 gets 25 %. The values follow the order in `--device`. |\n...\nsplit-mode\n...\n| `-fa` | `--flash-attn` | `on` \\| `off` \\| `auto` | `auto` | Required when using `--split-mode tensor` and/or quantized V cache. Supported (and therefore enabled by default) for most combinations\n...\n` | `--\n...\n-type-v` | same as `-\n...\ntype for V. |\n...\n| `-fit` | `--fit` | `on` \\| `off` | `on` | Auto-fit unset args to device memory. **Not supported with `tensor`. You may need to manually set the `--ctx-size` to make the model fit.** |\n...\nEasiest configuration. KV cache spreads across the GPUs along with the layers. `--fit` (on by default) sizes things automatically.\n...\n| CUDA OOM at startup or during prefill in `--split-mode tensor` | Auto-fit is disabled in this mode, so reduce memory pressure yourself. In order from least to most disruptive: lower `--ctx-size` (`-c`) (KV cache is roughly proportional to `n_ctx`); for `llama-server`, lower `--parallel` (`-np`) (a slot KV cache is allocated per concurrent sequence); as a last resort, reduce `--n-gpu-layers` (`-ngl`) (the remaining layers run on CPU and inference will be much slower). |","convId":"incyjf36ghq","role":"tool","timestamp":1787790398389,"toolCallId":"hioCqA5SR5Lte4BGEbcEtwpsiQ15HRKl","type":"text","id":"ckd32slblr","parent":"uvhsx10tlkn"},"type":"message"}
{"message":{"children":["4aqhi7cht7v","jkspost8exp"],"content":"Now the two remaining unknowns: exact llama-server flag names for tool-calling/reasoning kwargs, and whether a vision projector exists (your UD-IQ4_XS dir has no mmproj).\n\n","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790398395,"type":"text","id":"ay6axiw8owv","parent":"ckd32slblr","reasoningContent":"Now I have a lot. Let me get the remaining pieces:\n\n1. **Exact GGUF tensor names** for the PLE/ngram table. The Baekpica README says \"Integer PLE controls\" as I64, \"128 logical checkpoint shards\", \"the longcat-flash-lite impl split the engram table semantically while qwen is one big tensor\". ngxson said \"concat them into one big tensor is still necessary\". So in the unsloth GGUF the PLE table is one big tensor. What's its name? The PR mentions `per_layer_token_embd` explicitly. So `per_layer_token_embd` is right. But is `ple_ngram_embd` real? Probably not. Need to verify.\n\n   Hmm, but also shard 00001 is only 10.9 MB and 00002 is 49.8 GB, 00003 is 43.8 GB. So shard 2 (49.8GB) likely contains the big PLE tensor (26GB?) — hmm, actually the split is by size. Let's think: 49.8 + 43.8 = 93.6 GB total. PLE at 4-bit from 51.2G elements ≈ 25.6GB. So PLE is somewhere in there.\n\n   Let me verify tensor names. I could use `llama-quant` info... no local. Maybe check the GGUF metadata via... I could cat the first bytes of the GGUF header? The metadata is at the start of shard 1. hf_fs cat with --max-bytes on a binary — it's an LFS pointer for git-lfs actually (10.9MB is the actual file? \"lfs, size=10.9 MB\" means it's an LFS file whose real size is 10.9MB). So shard 00001 is 10.9 MB real. GGUF metadata for the whole model is in shard 1's header. I could try to cat and grep... but cat on binary probably fails or returns garbage. Let me instead search for a \"tensor names\" list. \n\n   Alternatively check the AtomicChat metrics dataset or the model's `tokenizer.chat_template` / `generation_config.json` for the base repo. Hmm.\n\n   Actually — a cleaner path: the PR review mentioned \"ngxson: I have only reviewed the engrams code path\". And llama.cpp's Gemma 3n implementation uses `per_layer_token_embd.weight` — that's a real llama.cpp tensor name in `LLM_TENSOR_PER_LAYER_TOKEN_EMBEDDING`. And the PR says: \"`--tensor-type` could not name `per_layer_token_embd` because `--token-embedding-type` returned first\". That confirms `per_layer_token_embd` is the tensor name in the GGUF. \n\n   So `ple_ngram_embd` is almost certainly NOT a real tensor name → that part of the user's `--override-tensor` is a no-op (llama.cpp warns \"unknown tensor\" and ignores). And the real table name is `per_layer_token_embd`. So they're already offloading the right one, and the second is dead. Good — need to verify though; maybe unsloth made two tensors. Let me check whether there's an mmproj too — the UD-IQ4_XS dir has NO mmproj file! So vision is not available from this repo. Hermes vision would need a separate mmproj. Let me note: no mmproj in unsloth repo → check if there's a separate vision gguf. Let me search HF for mmproj for this model.\n\n2. **Multi-GPU docs**: `--fit` not supported with `--split-mode tensor`. layer is fine.\n\n3. Need: `--parallel` bug (keep 1). Need: `--reasoning-preserve` for Hermes (preserved thinking beneficial for agents per Qwen README: \"especially beneficial for agent scenarios\" and \"improves KV cache utilization\"). Big one!\n\n4. Need: tool call parser flags for llama-server. Let me check the server README for `--tool-call-parser` / `--chat-template-parser` / `--jinja`. And `--chat-template-kwargs` vs `--default-chat-template-kwargs`. Important: `--chat-template-kwargs` may not exist; the real llama.cpp flag is `--default-chat-template-kwargs` (and `--chat-template-kwargs` mentioned by unsloth doc). Need to verify.\n\n5. `--spec-type` for MTP — PR says MTP WIP/not supported in this PR (\"What are the limitations? Is MTP supported?\" and later danielhanchen \"I'm adding MTP - still WIP\"). Also issue #1993 says \"llama.cpp's leading PR ships without it, but the fitting GGUF has no MTP weights\". Hmm — \"the fitting GGUF has no MTP weights\". Interesting. So MTP may not be available. Worth mentioning as \"watch this\".\n\n6. Also `--deterministic`, `--flash-attn` with QSA.\n\n7. `--defrag-thold`, `--keep`, `--cache-ram` / `--prompt-cache`. Let me get the server README for the relevant sections.\n\nLet me fetch the llama.cpp server README in full (it's big, but I need specific flags) and search for mmproj.\n\nLet me be strategic: fetch server/README.md with maxCharacters large, and search HF for mmproj.\n\nActually, the key facts I still need:\n- exact flag names: `--default-chat-template-kwargs`, `--reasoning-preserve`, `--tool-call-parser` or `--chat-template-parser`, `--parser-*`\n- `--keep`, `--cache-idle-buckets`? \n- `--no-mmproj`, `--mmproj`\n\nLet me fetch the raw README from github raw to get full text.\n\nAlso search for whether an mmproj exists.\n\nAnd check for MTP/speculation support status + any recommended 4x3090 configs.\n\nLet me batch.\n","timings":{"cache_n":43743,"prompt_n":28044,"prompt_ms":128858.13,"prompt_per_token_ms":4.594855584082157,"prompt_per_second":217.63469639051877,"predicted_n":1497,"predicted_ms":65763.511,"predicted_per_token_ms":43.95956617647059,"predicted_per_second":22.7481771768542},"toolCalls":[{"function":{"arguments":"{\"urls\":[\"https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md\"],\"maxCharacters\":120000}","name":"web_fetch_exa"},"id":"xMB1YdzO01VwI2GY4XRbSxzrE957o8kh","type":"function"},{"function":{"arguments":"{\"limit\":15,\"query\":\"Qwen3.8-Flash-Next mmproj vision gguf\",\"repo_types\":[\"model\"]}","name":"hub_repo_search"},"id":"jdT0sYKTzkr205KUPliEZTCdbVE8n4b4","type":"function"}]},"type":"message"}
{"message":{"children":[],"content":"# README.md\nURL: https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md\n\n# LLaMA.cpp HTTP Server\n\nFast, lightweight, pure C/C++ HTTP server based on [httplib](https://github.com/yhirose/cpp-httplib), [nlohmann::json](https://github.com/nlohmann/json) and **llama.cpp**.\n\nSet of LLM REST APIs and a web UI to interact with llama.cpp.\n\n**Features:**\n * LLM inference of F16 and quantized models on GPU and CPU\n * [OpenAI API](https://github.com/openai/openai-openapi) compatible chat completions, responses, and embeddings routes\n * [Anthropic Messages API](https://docs.anthropic.com/en/api/messages) compatible chat completions\n * Reranking endpoint (https://github.com/ggml-org/llama.cpp/pull/9510)\n * Parallel decoding with multi-user support\n * Continuous batching\n * Multimodal ([documentation](../../docs/multimodal.md)) / with OpenAI-compatible API support\n * Monitoring endpoints\n * Schema-constrained JSON response format\n * Prefilling of assistant messages similar to the Claude API\n * [Function calling](../../docs/function-calling.md) / tool use for ~any model\n * Speculative decoding\n * Easy-to-use web UI\n\nFor the full list of features, please refer to [server's changelog](https://github.com/ggml-org/llama.cpp/issues/9291)\n\n## Usage\n\n \n\n \n\n### Common params\n\n| Argument | Explanation |\n| -------- | ----------- |\n| `-h, --help, --usage` | print usage and exit |\n| `--version` | show version and build info |\n| `-cl, --cache-list` | show list of models in cache |\n| `--completion-bash` | print source-able bash completion script for llama.cpp |\n| `-t, --threads N` | number of CPU threads to use during generation (default: -1) (env: LLAMA_ARG_THREADS) |\n| `-tb, --threads-batch N` | number of threads to use during batch and prompt processing (default: same as --threads) |\n| `-C, --cpu-mask M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: \"\") |\n| `-Cr, --cpu-range lo-hi` | range of CPUs for affinity. Complements --cpu-mask |\n| `--cpu-strict <0\\|1>` | use strict CPU placement (default: 0) |\n| `--prio N` | set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0) |\n| `--poll <0...100>` | use polling level to wait for work (0 - no polling, default: 50) |\n| `-Cb, --cpu-mask-batch M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask) |\n| `-Crb, --cpu-range-batch lo-hi` | ranges of CPUs for affinity. Complements --cpu-mask-batch |\n| `--cpu-strict-batch <0\\|1>` | use strict CPU placement (default: same as --cpu-strict) |\n| `--prio-batch N` | set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |\n| `--poll-batch <0\\|1>` | use polling to wait for work (default: same as --poll) |\n| `-c, --ctx-size N` | size of the prompt context (default: 0, 0 = loaded from model) (env: LLAMA_ARG_CTX_SIZE) |\n| `-n, --predict, --n-predict N` | number of tokens to predict (default: -1, -1 = infinity) (env: LLAMA_ARG_N_PREDICT) |\n| `-b, --batch-size N` | logical maximum batch size (default: 2048) (env: LLAMA_ARG_BATCH) |\n| `-ub, --ubatch-size N` | physical maximum batch size (default: 512) (env: LLAMA_ARG_UBATCH) |\n| `--keep N` | number of tokens to keep from the initial prompt (default: 0, -1 = all) |\n| `--swa-full` | use full-size SWA cache (default: false) [(more info)](https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055) (env: LLAMA_ARG_SWA_FULL) |\n| `-fa, --flash-attn [on\\|off\\|auto]` | set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: LLAMA_ARG_FLASH_ATTN) |\n| `--perf, --no-perf` | whether to enable internal libllama performance timings (default: false) (env: LLAMA_ARG_PERF) |\n| `-e, --escape, --no-escape` | whether to process escapes sequences (\\n, \\r, \\t, \\', \\\", \\\\) (default: true) |\n| `--rope-scaling {none,linear,yarn}` | RoPE frequency scaling method, defaults to linear unless specified by the model (env: LLAMA_ARG_ROPE_SCALING_TYPE) |\n| `--rope-scale N` | RoPE context scaling factor, expands context by a factor of N (env: LLAMA_ARG_ROPE_SCALE) |\n| `--rope-freq-base N` | RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: LLAMA_ARG_ROPE_FREQ_BASE) |\n| `--rope-freq-scale N` | RoPE frequency scaling factor, expands context by a factor of 1/N (env: LLAMA_ARG_ROPE_FREQ_SCALE) |\n| `--yarn-orig-ctx N` | YaRN: original context size of model (default: 0 = model training context size) (env: LLAMA_ARG_YARN_ORIG_CTX) |\n| `--yarn-ext-factor N` | YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: LLAMA_ARG_YARN_EXT_FACTOR) |\n| `--yarn-attn-factor N` | YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: LLAMA_ARG_YARN_ATTN_FACTOR) |\n| `--yarn-beta-slow N` | YaRN: high correction dim or alpha (default: -1.00) (env: LLAMA_ARG_YARN_BETA_SLOW) |\n| `--yarn-beta-fast N` | YaRN: low correction dim or beta (default: -1.00) (env: LLAMA_ARG_YARN_BETA_FAST) |\n| `-kvo, --kv-offload, -nkvo, --no-kv-offload` | whether to enable KV cache offloading (default: enabled) (env: LLAMA_ARG_KV_OFFLOAD) |\n| `--repack, -nr, --no-repack` | whether to enable weight repacking (default: enabled) (env: LLAMA_ARG_REPACK) |\n| `--no-host` | bypass host buffer allowing extra buffers to be used (env: LLAMA_ARG_NO_HOST) |\n| `-ctk, --cache-type-k TYPE` | KV cache data type for K allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K) |\n| `-ctv, --cache-type-v TYPE` | KV cache data type for V allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V) |\n| `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED) (env: LLAMA_ARG_DEFRAG_THOLD) |\n| `--rpc SERVERS` | comma-separated list of RPC servers (host:port) (env: LLAMA_ARG_RPC) |\n| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK) |\n| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (env: LLAMA_ARG_MMAP) |\n| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available (env: LLAMA_ARG_DIO) |\n| `-lm, --load-mode MODE` | model loading mode (default: auto) - auto: mmap, unless a device does not support it - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) |\n| `--numa TYPE` | attempt optimizations that help on some NUMA systems - distribute: spread execution evenly over all nodes - isolate: only spawn threads on CPUs on the node that execution started on - numactl: use the CPU map provided by numactl if run without this previously, it is recommended to drop the system page cache before using this see https://github.com/ggml-org/llama.cpp/issues/1437 (env: LLAMA_ARG_NUMA) |\n| `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload) use --list-devices to see a list of available devices (env: LLAMA_ARG_DEVICE) |\n| `--list-devices` | print list of available devices and exit |\n| `-ot, --override-tensor =,...` | override tensor buffer type (env: LLAMA_ARG_OVERRIDE_TENSOR) |\n| `-cmoe, --cpu-moe` | keep all Mixture of Experts (MoE) weights in the CPU (env: LLAMA_ARG_CPU_MOE) |\n| `-ncmoe, --n-cpu-moe N` | keep the Mixture of Experts (MoE) weights of the first N layers in the CPU (env: LLAMA_ARG_N_CPU_MOE) |\n| `-ngl, --gpu-layers, --n-gpu-layers N` | max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS) |\n| `-sm, --split-mode {none,layer,row,tensor}` | how to split the model across multiple GPUs, one of: - none: use one GPU only - layer (default): split layers and KV across GPUs (pipelined) - row: split weight across GPUs by rows (parallelized) - tensor: split weights and KV across GPUs (parallelized, EXPERIMENTAL) (env: LLAMA_ARG_SPLIT_MODE) |\n| `-ts, --tensor-split N0,N1,N2,...` | fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1 (env: LLAMA_ARG_TENSOR_SPLIT) |\n| `-mg, --main-gpu INDEX` | the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row) (default: 0) (env: LLAMA_ARG_MAIN_GPU) |\n| `-fit, --fit [on\\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |\n| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |\n| `-fitc, --fit-ctx N` | minimum ctx size that can be set by --fit option, default: 4096 (env: LLAMA_ARG_FIT_CTX) |\n| `--check-tensors` | check model tensor data for invalid values (default: false) |\n| `--override-kv KEY=TYPE:VALUE,...` | advanced option to override model metadata by key. to specify multiple overrides, either use comma-separated values. types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false,tokenizer.ggml.add_eos_token=bool:false |\n| `--op-offload, --no-op-offload` | whether to offload host tensor operations to device (default: true) |\n| `--lora FNAME` | path to LoRA adapter (use comma-separated values to load multiple adapters) |\n| `--lora-scaled FNAME:SCALE,...` | path to LoRA adapter with user defined scaling (format: FNAME:SCALE,...) note: use comma-separated values |\n| `--control-vector FNAME` | add a control vector note: use comma-separated values to add multiple control vectors |\n| `--control-vector-scaled FNAME:SCALE,...` | add a control vector with user defined scaling SCALE note: use comma-separated values (format: FNAME:SCALE,...) |\n| `--control-vector-layer-range START END` | layer range to apply the control vector(s) to, start and end inclusive |\n| `-m, --model FNAME` | model path to load (env: LLAMA_ARG_MODEL) |\n| `-mu, --model-url MODEL_URL` | model download url (default: unused) (env: LLAMA_ARG_MODEL_URL) |\n| `-dr, --docker-repo [ /] [:quant]` | Docker Hub model repository. repo is optional, default to ai/. quant is optional, default to :latest. example: gemma3 (default: unused) (env: LLAMA_ARG_DOCKER_REPO) |\n| `-hf, -hfr, --hf-repo / [:quant]` | Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist. mmproj is also downloaded automatically if available. to disable, add --no-mmproj example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M (default: unused) (env: LLAMA_ARG_HF_REPO) |\n| `-hff, --hf-file FILE` | Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused) (env: LLAMA_ARG_HF_FILE) |\n| `-hft, --hf-token TOKEN` | Hugging Face access token (default: value from HF_TOKEN environment variable) (env: HF_TOKEN) |\n| `--log-disable` | Log disable |\n| `--log-file FNAME` | Log to file (env: LLAMA_ARG_LOG_FILE) |\n| `--log-colors [on\\|off\\|auto]` | Set colored logging ('on', 'off', or 'auto', default: 'auto') 'auto' enables colors when output is to a terminal (env: LLAMA_ARG_LOG_COLORS) |\n| `-v, --verbose, --log-verbose` | Set verbosity level to infinity (i.e. log all messages, useful for debugging) |\n| `--offline` | Offline mode: forces use of cache, prevents network access (env: LLAMA_ARG_OFFLINE) |\n| `-lv, --verbosity, --log-verbosity N` | Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values: - 0: generic output - 1: error - 2: warning - 3: info - 4: trace (more info) - 5: debug (default: 3) (env: LLAMA_ARG_LOG_VERBOSITY) |\n| `--log-prefix, --no-log-prefix` | Enable prefix in log messages (env: LLAMA_ARG_LOG_PREFIX) |\n| `--log-timestamps, --no-log-timestamps` | Enable timestamps in log messages (env: LLAMA_ARG_LOG_TIMESTAMPS) |\n| `--spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE` | KV cache data type for K for the draft model allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K) |\n| `--spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE` | KV cache data type for V for the draft model allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) |\n\n### Sampling params\n\n| Argument | Explanation |\n| -------- | ----------- |\n| `--samplers SAMPLERS` | samplers that will be used for generation in the order, separated by ';' (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature) |\n| `-s, --seed SEED` | RNG seed (default: -1, use random seed for -1) |\n| `--sampler-seq, --sampling-seq SEQUENCE` | simplified sequence for samplers that will be used (default: edskypmxt) |\n| `--ignore-eos` | ignore end of stream token and continue generating (implies --logit-bias EOS-inf) |\n| `--temp, --temperature N` | temperature (default: 0.80) |\n| `--top-k N` | top-k sampling (default: 40, 0 = disabled) (env: LLAMA_ARG_TOP_K) |\n| `--top-p N` | top-p sampling (default: 0.95, 1.0 = disabled) |\n| `--min-p N` | min-p sampling (default: 0.05, 0.0 = disabled) |\n| `--top-nsigma, --top-n-sigma N` | top-n-sigma sampling (default: -1.00, -1.0 = disabled) |\n| `--xtc-probability N` | xtc probability (default: 0.00, 0.0 = disabled) |\n| `--xtc-threshold N` | xtc threshold (default: 0.10, 1.0 = disabled) |\n| `--typical, --typical-p N` | locally typical sampling, parameter p (default: 1.00, 1.0 = disabled) |\n| `--repeat-last-n N` | last n tokens to consider for penalize (default: 64, 0 = disabled) |\n| `--repeat-penalty N` | penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled) |\n| `--presence-penalty N` | repeat alpha presence penalty (default: 0.00, 0.0 = disabled) |\n| `--frequency-penalty N` | repeat alpha frequency penalty (default: 0.00, 0.0 = disabled) |\n| `--dry-multiplier N` | set DRY sampling multiplier (default: 0.00, 0.0 = disabled) |\n| `--dry-base N` | set DRY sampling base value (default: 1.75) |\n| `--dry-allowed-length N` | set allowed length for DRY sampling (default: 2) |\n| `--dry-penalty-last-n N` | set DRY penalty for the last n tokens (default: 64, 0 = disable) |\n| `--dry-sequence-breaker STRING` | add sequence breaker for DRY sampling, clearing out default breakers ('\\n', ':', '\"', '*') in the process; use \"none\" to not use any sequence breakers |\n| `--adaptive-target N` | adaptive-p: select tokens near this probability (valid range 0.0 to 1.0; negative = disabled) (default: -1.00) [(more info)](https://github.com/ggml-org/llama.cpp/pull/17927) |\n| `--adaptive-decay N` | adaptive-p: decay rate for target adaptation over time. lower values are more reactive, higher values are more stable. (valid range 0.0 to 0.99) (default: 0.90) |\n| `--dynatemp-range N` | dynamic temperature range (default: 0.00, 0.0 = disabled) |\n| `--dynatemp-exp N` | dynamic temperature exponent (default: 1.00) |\n| `--mirostat N` | use Mirostat sampling. Top K, Nucleus and Locally Typical samplers are ignored if used. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) |\n| `--mirostat-lr N` | Mirostat learning rate, parameter eta (default: 0.10) |\n| `--mirostat-ent N` | Mirostat target entropy, parameter tau (default: 5.00) |\n| `-l, --logit-bias TOKEN_ID(+/-)BIAS` | modifies the likelihood of token appearing in the completion, i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello', or `--logit-bias 15043-1` to decrease likelihood of token ' Hello' |\n| `--grammar GRAMMAR` | BNF-like grammar to constrain generations (see samples in grammars/ dir) |\n| `--grammar-file FNAME` | file to read grammar from |\n| `-j, --json-schema SCHEMA` | JSON schema to constrain generations (https://json-schema.org/), e.g. `{}` for any JSON object For schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead |\n| `-jf, --json-schema-file FILE` | File containing a JSON schema to constrain generations (https://json-schema.org/), e.g. `{}` for any JSON object For schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead |\n| `-bs, --backend-sampling` | enable backend sampling (experimental) (default: disabled) (env: LLAMA_ARG_BACKEND_SAMPLING) |\n\n### Server-specific params\n\n| Argument | Explanation |\n| -------- | ----------- |\n| `-lcs, --lookup-cache-static FNAME` | path to static lookup cache to use for lookup decoding (not updated by generation) |\n| `-lcd, --lookup-cache-dynamic FNAME` | path to dynamic lookup cache to use for lookup decoding (updated by generation) |\n| `-ctxcp, --ctx-checkpoints, --swa-checkpoints N` | max number of context checkpoints to create per slot (default: 32)[(more info)](https://github.com/ggml-org/llama.cpp/pull/15293) (env: LLAMA_ARG_CTX_CHECKPOINTS) |\n| `-cms, --checkpoint-min-step N` | minimum spacing between context checkpoints in tokens (default: 8192, 0 = no minimum) (env: LLAMA_ARG_CHECKPOINT_MIN_SPACING_NT) |\n| `-cram, --cache-ram N` | set the maximum cache size in MiB (default: 8192, -1 - no limit, 0 - disable)[(more info)](https://github.com/ggml-org/llama.cpp/pull/16391) (env: LLAMA_ARG_CACHE_RAM) |\n| `-kvu, --kv-unified, -no-kvu, --no-kv-unified` | use single unified KV buffer shared across all sequences (default: enabled if number of slots is auto) (env: LLAMA_ARG_KV_UNIFIED) |\n| `--cache-idle-slots, --no-cache-idle-slots` | save idle slots to the prompt cache on new task, and clear them when using unified KV (default: enabled, requires cache-ram) (env: LLAMA_ARG_CACHE_IDLE_SLOTS) |\n| `--context-shift, --no-context-shift` | whether to use context shift on infinite text generation (default: disabled) (env: LLAMA_ARG_CONTEXT_SHIFT) |\n| `-r, --reverse-prompt PROMPT` | halt generation at PROMPT, return control in interactive mode |\n| `-sp, --special` | special tokens output enabled (default: false) |\n| `--warmup, --no-warmup` | whether to perform warmup with an empty run (default: enabled) |\n| `--spm-infill` | use Suffix/Prefix/Middle pattern for infill (instead of Prefix/Suffix/Middle) as some models prefer this. (default: disabled) |\n| `--pooling {none,mean,cls,last,rank}` | pooling type for embeddings, use model default if unspecified (env: LLAMA_ARG_POOLING) |\n| `-np, --parallel N` | number of server slots (default: -1, -1 = auto) (env: LLAMA_ARG_N_PARALLEL) |\n| `-cb, --cont-batching, -nocb, --no-cont-batching` | whether to enable continuous batching (a.k.a dynamic batching) (default: enabled) (env: LLAMA_ARG_CONT_BATCHING) |\n| `-mm, --mmproj FILE` | path to a multimodal projector file. see tools/mtmd/README.md note: if -hf is used, this argument can be omitted (env: LLAMA_ARG_MMPROJ) |\n| `-mmu, --mmproj-url URL` | URL to a multimodal projector file. see tools/mtmd/README.md (env: LLAMA_ARG_MMPROJ_URL) |\n| `--mmproj-auto, --no-mmproj, --no-mmproj-auto` | whether to use multimodal projector file (if available), useful when using -hf (default: enabled) (env: LLAMA_ARG_MMPROJ_AUTO) |\n| `--mmproj-offload, --no-mmproj-offload` | whether to enable GPU offloading for multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_OFFLOAD) |\n| `-mmdev, --mmproj-device DEVICE` | device to use for multimodal projector (none = don't offload, default: auto) use --list-devices to see a list of available devices (env: MTMD_BACKEND_DEVICE) |\n| `--image-min-tokens N` | minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MIN_TOKENS) |\n| `--image-max-tokens N` | maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MAX_TOKENS) |\n| `--mtmd-batch-max-tokens N` | maximum number of image tokens per batch when encoding images (default: 1024) (env: LLAMA_ARG_MTMD_BATCH_MAX_TOKENS) |\n| `-a, --alias STRING` | set model name aliases, comma-separated (to be used by API) (env: LLAMA_ARG_ALIAS) |\n| `--tags STRING` | set model tags, comma-separated (informational, not used for routing) (env: LLAMA_ARG_TAGS) |\n| `--embd-normalize N` | normalisation for embeddings (default: 2) (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm) |\n| `--host HOST` | ip address to listen, or bind to an UNIX socket if the address ends with .sock (default: 127.0.0.1) (env: LLAMA_ARG_HOST) |\n| `--port PORT` | port to listen (default: 8080) (env: LLAMA_ARG_PORT) |\n| `--reuse-port` | allow multiple sockets to bind to the same port (default: disabled) (env: LLAMA_ARG_REUSE_PORT) |\n| `--path PATH` | path to serve static files from (default: ) (env: LLAMA_ARG_STATIC_PATH) |\n| `--cors-origins ORIGINS` | comma-separated list of allowed origins for CORS (default: *) if set to special value 'localhost', reflect the Origin header only if it is localhost (env: LLAMA_ARG_CORS_ORIGINS) |\n| `--cors-methods METHODS` | comma-separated list of allowed methods for CORS (default: GET, POST, DELETE, OPTIONS) (env: LLAMA_ARG_CORS_METHODS) |\n| `--cors-headers HEADERS` | comma-separated list of allowed headers for CORS (default: *) (env: LLAMA_ARG_CORS_HEADERS) |\n| `--cors-credentials, --no-cors-credentials` | whether to allow credentials for CORS (default: enabled) note: if this is enabled and --cors-origins is set to * (default), the Origin header will be echoed back, and credentials will always be allowed (env: LLAMA_ARG_CORS_CREDENTIALS) |\n| `--api-prefix PREFIX` | prefix path the server serves from, without the trailing slash (default: ) (env: LLAMA_ARG_API_PREFIX) |\n| `--ui-config, --webui-config JSON` | JSON that provides default UI settings (overrides UI defaults) (env: LLAMA_ARG_UI_CONFIG) |\n| `--ui-config-file, --webui-config-file PATH` | JSON file that provides default UI settings (overrides UI defaults) (env: LLAMA_ARG_UI_CONFIG_FILE) |\n| `--ui-mcp-proxy, --webui-mcp-proxy, --no-ui-mcp-proxy, --no-webui-mcp-proxy` | experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled) (env: LLAMA_ARG_UI_MCP_PROXY) |\n| `--tools TOOL1,TOOL2,...` | experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools) specify \"all\" to enable all tools available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info note: for security reasons, this will limit --cors-origins to localhost by default (env: LLAMA_ARG_TOOLS) |\n| `--tools-runtime OPTION` | experimental: run tools in a separate runtime environment (default: none, use host environment) available options: 'docker: ', 'podman: ': spin up a new container and reuse it for all invocations, clean up on server exit 'docker-container: ', 'podman-container: ': use an existing container by ID, won't stop on server exit 'ssh: ': run tools on a remote POSIX host over SSH, key-based auth and a trusted host key are required (env: LLAMA_ARG_TOOLS_RUNTIME) |\n| `--mcp-servers-config PATH` | experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none) note: for security reasons, this will limit --cors-origins to localhost by default (env: LLAMA_ARG_MCP_SERVERS_CONFIG) |\n| `--mcp-servers-json JSON` | experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none) note: for security reasons, this will limit --cors-origins to localhost by default (env: LLAMA_ARG_MCP_SERVERS_JSON) |\n| `-ag, --agent, -no-ag, --no-agent` | whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled) note: for security reasons, this will limit --cors-origins to localhost by default (env: LLAMA_ARG_AGENT) |\n| `--ui, --webui, --no-ui, --no-webui` | whether to enable the Web UI (default: enabled) (env: LLAMA_ARG_UI) |\n| `--embedding, --embeddings` | restrict to only support embedding use case; use only with dedicated embedding models (default: disabled) (env: LLAMA_ARG_EMBEDDINGS) |\n| `--rerank, --reranking` | enable reranking endpoint on server (default: disabled) (env: LLAMA_ARG_RERANKING) |\n| `--api-key KEY` | API key to use for authentication, multiple keys can be provided as a comma-separated list (default: none) (env: LLAMA_API_KEY) |\n| `--api-key-file FNAME` | path to file containing API keys, one per line; lines starting with a hash are treated as comments (default: none) (env: LLAMA_ARG_API_KEY_FILE) |\n| `--ssl-key-file FNAME` | path to file a PEM-encoded SSL private key (env: LLAMA_ARG_SSL_KEY_FILE) |\n| `--ssl-cert-file FNAME` | path to file a PEM-encoded SSL certificate (env: LLAMA_ARG_SSL_CERT_FILE) |\n| `--chat-template-kwargs STRING` | sets additional params for the json template parser, must be a valid json object string, e.g. '{\"key1\":\"value1\",\"key2\":\"value2\"}' (env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS) |\n| `-to, --timeout N` | server read/write timeout in seconds (default: 3600) (env: LLAMA_ARG_TIMEOUT) |\n| `--sse-ping-interval N` | server SSE ping interval in seconds (-1 = disabled, default: 30) (env: LLAMA_ARG_SSE_PING_INTERVAL) |\n| `--threads-http N` | number of threads used to process HTTP requests (default: -1) (env: LLAMA_ARG_THREADS_HTTP) |\n| `--cache-prompt, --no-cache-prompt` | whether to enable prompt caching (default: enabled) (env: LLAMA_ARG_CACHE_PROMPT) |\n| `--cache-reuse N` | min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled (default: 0) [(card)](https://ggml.ai/f0.png) (env: LLAMA_ARG_CACHE_REUSE) |\n| `--metrics` | enable prometheus compatible metrics endpoint (default: disabled) (env: LLAMA_ARG_ENDPOINT_METRICS) |\n| `--props` | enable changing global properties via POST /props (default: disabled) (env: LLAMA_ARG_ENDPOINT_PROPS) |\n| `--slots, --no-slots` | expose slots monitoring endpoint (default: enabled) (env: LLAMA_ARG_ENDPOINT_SLOTS) |\n| `--slot-save-path PATH` | path to save slot kv cache (default: disabled) |\n| `--media-path PATH` | directory for loading local media files; files can be accessed via file:// URLs using relative paths (default: disabled) |\n| `--models-dir PATH` | directory containing models for the router server (default: disabled) (env: LLAMA_ARG_MODELS_DIR) |\n| `--models-preset PATH` | path to INI file containing model presets for the router server (default: disabled) (env: LLAMA_ARG_MODELS_PRESET) |\n| `--models-max N` | for router server, maximum number of models to load simultaneously (default: 4, 0 = unlimited) (env: LLAMA_ARG_MODELS_MAX) |\n| `--models-autoload, --no-models-autoload` | for router server, whether to automatically load models (default: enabled) (env: LLAMA_ARG_MODELS_AUTOLOAD) |\n| `--jinja, --no-jinja` | whether to use jinja template engine for chat (default: enabled) (env: LLAMA_ARG_JINJA) |\n| `--reasoning-format FORMAT` | controls whether thought tags are allowed and/or extracted from the response, and in which format they're returned; one of: - none: leaves thoughts unparsed in `message.content` - deepseek: puts thoughts in `message.reasoning_content` - deepseek-legacy: keeps ` ` tags in `message.content` while also populating `message.reasoning_content` (default: auto) (env: LLAMA_ARG_THINK) |\n| `-rea, --reasoning [on\\|off\\|auto]` | Use reasoning/thinking in the chat ('on', 'off', or 'auto', default: 'auto' (detect from template)) (env: LLAMA_ARG_REASONING) |\n| `--reasoning-effort LEVEL` | reasoning effort level given to the chat template: 'default' to keep the template default, or a level such as 'minimal', 'low', 'medium', 'high', 'xhigh' or 'max' (default: default) (env: LLAMA_ARG_REASONING_EFFORT) |\n| `--reasoning-budget N` | token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1) (env: LLAMA_ARG_THINK_BUDGET) |\n| `--reasoning-budget-message MESSAGE` | message injected before the end-of-thinking tag when reasoning budget is exhausted (default: none) (env: LLAMA_ARG_THINK_BUDGET_MESSAGE) |\n| `--reasoning-preserve, --no-reasoning-preserve` | preserve reasoning trace in the full history, not just the last assistant message (default: template default) compatible with certain templates having 'supports_preserve_reasoning' capability example: https://docs.z.ai/guides/capabilities/thinking-mode#preserved-thinking (env: LLAMA_ARG_REASONING_PRESERVE) |\n| `--chat-template JINJA_TEMPLATE` | set custom jinja chat template (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE) |\n| `--chat-template-file JINJA_TEMPLATE_FILE` | set custom jinja chat template file (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE_FILE) |\n| `--skip-chat-parsing, --no-skip-chat-parsing` | force a pure content parser, even if a Jinja template is specified; model will output everything in the content section, including any reasoning and/or tool calls (default: disabled) (env: LLAMA_ARG_SKIP_CHAT_PARSING) |\n| `--prefill-assistant, --no-prefill-assistant` | whether to prefill the assistant's response if the last message is an assistant message (default: prefill enabled) when this flag is set, if the last message is an assistant message then it will be treated as a full message and not prefilled (env: LLAMA_ARG_PREFILL_ASSISTANT) |\n| `-sps, --slot-prompt-similarity SIMILARITY` | how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.10, 0.0 = disabled) |\n| `--lora-init-without-apply` | load LoRA adapters without applying them (apply later via POST /lora-adapters) (default: disabled) |\n| `--sleep-idle-seconds SECONDS` | number of seconds of idleness after which the server will sleep (default: -1; -1 = disabled) |\n| `--log-prompts-dir PATH` | Log prompts to directory (auto-created if not present; only used for debugging, default: disabled) |\n| `--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft / [:quant]` | Same as --hf-repo, but for the draft model (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_HF_REPO) |\n| `--spec-draft-threads, -td, --threads-draft N` | number of threads to use during generation (default: same as --threads) |\n| `--spec-draft-threads-batch, -tbd, --threads-batch-draft N` | number of threads to use during batch and prompt processing (default: same as --threads-draft) |\n| `--spec-draft-cpu-mask, -Cd, --cpu-mask-draft M` | Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |\n| `--spec-draft-cpu-range, -Crd, --cpu-range-draft lo-hi` | Ranges of CPUs for affinity. Complements --cpu-mask-draft |\n| `--spec-draft-cpu-strict, --cpu-strict-draft <0\\|1>` | Use strict CPU placement for draft model (default: same as --cpu-strict) |\n| `--spec-draft-prio, --prio-draft N` | set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |\n| `--spec-draft-poll, --poll-draft <0\\|1>` | Use polling to wait for draft model work (default: same as --poll) |\n| `--spec-draft-cpu-mask-batch, -Cbd, --cpu-mask-batch-draft M` | Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |\n| `--spec-draft-cpu-strict-batch, --cpu-strict-batch-draft <0\\|1>` | Use strict CPU placement for draft model (default: --cpu-strict-draft) |\n| `--spec-draft-prio-batch, --prio-batch-draft N` | set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |\n| `--spec-draft-poll-batch, --poll-batch-draft <0\\|1>` | Use polling to wait for draft model work (default: --poll-draft) |\n| `--spec-draft-override-tensor, -otd, --override-tensor-draft =,...` | override tensor buffer type for draft model |\n| `--spec-draft-cpu-moe, -cmoed, --cpu-moe-draft` | keep all Mixture of Experts (MoE) weights in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_CPU_MOE) |\n| `--spec-draft-n-cpu-moe, --spec-draft-ncmoe, -ncmoed, --n-cpu-moe-draft N` | keep the Mixture of Experts (MoE) weights of the first N layers in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_N_CPU_MOE) |\n| `--spec-draft-n-max N` | number of tokens to draft for speculative decoding (default: 3) (env: LLAMA_ARG_SPEC_DRAFT_N_MAX) |\n| `--spec-draft-n-min N` | minimum number of draft tokens to use for speculative decoding (default: 0) (env: LLAMA_ARG_SPEC_DRAFT_N_MIN) |\n| `--spec-draft-p-split, --draft-p-split P` | speculative decoding split probability (default: 0.10) (env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT) |\n| `--spec-draft-p-min, --draft-p-min P` | minimum speculative decoding probability (greedy) (default: 0.00) (env: LLAMA_ARG_SPEC_DRAFT_P_MIN) |\n| `--spec-draft-backend-sampling, --no-spec-draft-backend-sampling` | offload draft sampling to the backend (default: enabled) (env: LLAMA_ARG_SPEC_DRAFT_BACKEND_SAMPLING) |\n| `--spec-draft-device, -devd, --device-draft <dev1,dev2,..>` | comma-separated list of devices to use for offloading the draft model (none = don't offload) use --list-devices to see a list of available devices |\n| `--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N` | max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT) |\n| `--spec-draft-model, -md, --model-draft FNAME` | draft model for speculative decoding (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_MODEL) |\n| `--spec-type none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache` | comma-separated list of types of speculative decoding to use (default: none) (env: LLAMA_ARG_SPEC_TYPE) |\n| `--spec-ngram-mod-n-min N` | minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48) |\n| `--spec-ngram-mod-n-max N` | maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64) |\n| `--spec-ngram-mod-n-match N` | ngram-mod lookup length (default: 24) |\n| `--spec-ngram-simple-size-n N` | ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12) |\n| `--spec-ngram-simple-size-m N` | ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48) |\n| `--spec-ngram-simple-min-hits N` | minimum hits for ngram-simple speculative decoding (default: 1) |\n| `--spec-ngram-map-k-size-n N` | ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12) |\n| `--spec-ngram-map-k-size-m N` | ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48) |\n| `--spec-ngram-map-k-min-hits N` | minimum hits for ngram-map-k speculative decoding (default: 1) |\n| `--spec-ngram-map-k4v-size-n N` | ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12) |\n| `--spec-ngram-map-k4v-size-m N` | ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48) |\n| `--spec-ngram-map-k4v-min-hits N` | minimum hits for ngram-map-k4v speculative decoding (default: 1) |\n| `--draft, --draft-n, --draft-max N` | the argument has been removed. use --spec-draft-n-max or --spec-ngram-mod-n-max (env: LLAMA_ARG_DRAFT_MAX) |\n| `--draft-min, --draft-n-min N` | the argument has been removed. use --spec-draft-n-min or --spec-ngram-mod-n-min (env: LLAMA_ARG_DRAFT_MIN) |\n| `--spec-ngram-size-n N` | the argument has been removed. use the respective --spec-ngram-*-size-n or --spec-ngram-mod-n-match |\n| `--spec-ngram-size-m N` | the argument has been removed. use the respective --spec-ngram-*-size-m |\n| `--spec-ngram-min-hits N` | the argument has been removed. use the respective --spec-ngram-*-min-hits |\n| `--embd-gemma-default` | use default EmbeddingGemma model (note: can download weights from the internet) |\n| `--fim-qwen-1.5b-default` | use default Qwen 2.5 Coder 1.5B (note: can download weights from the internet) |\n| `--fim-qwen-3b-default` | use default Qwen 2.5 Coder 3B (note: can download weights from the internet) |\n| `--fim-qwen-7b-default` | use default Qwen 2.5 Coder 7B (note: can download weights from the internet) |\n| `--fim-qwen-7b-spec` | use Qwen 2.5 Coder 7B + 0.5B draft for speculative decoding (note: can download weights from the internet) |\n| `--fim-qwen-14b-spec` | use Qwen 2.5 Coder 14B + 0.5B draft for speculative decoding (note: can download weights from the internet) |\n| `--fim-qwen-30b-default` | use default Qwen 3 Coder 30B A3B Instruct (note: can download weights from the internet) |\n| `--gpt-oss-20b-default` | use gpt-oss-20b (note: can download weights from the internet) |\n| `--gpt-oss-120b-default` | use gpt-oss-120b (note: can download weights from the internet) |\n| `--vision-gemma-4b-default` | use Gemma 3 4B QAT (note: can download weights from the internet) |\n| `--vision-gemma-12b-default` | use Gemma 3 12B QAT (note: can download weights from the internet) |\n| `--spec-default` | enable default speculative decoding config |\n\n \n\nNote: If both command line argument and environment variable are both set for the same param, the argument will take precedence over env var.\n\nFor string options like `--load-mode`, the environment variable is handled as shown in this example:\n- `LLAMA_ARG_LOAD_MODE=auto` sets the loading mode to auto (default)\n- `LLAMA_ARG_LOAD_MODE=none` disables special loading\n- `LLAMA_ARG_LOAD_MODE=mmap` enables memory-mapping\n- `LLAMA_ARG_LOAD_MODE=mlock` locks the model in RAM\n- `LLAMA_ARG_LOAD_MODE=mmap+mlock` enables memory-mapping and locks in RAM\n- `LLAMA_ARG_LOAD_MODE=dio` uses DirectIO if available\n\nFor boolean options like `--kv-offload`:\n- `LLAMA_ARG_KV_OFFLOAD=true` means enabled, other accepted values are: `1`, `on`, `enabled`\n- `LLAMA_ARG_KV_OFFLOAD=false` means disabled, other accepted values are: `0`, `off`, `disabled`\n\nExample usage of docker compose with environment variables:\n\n```yml\nservices:\n  llamacpp-server:\n    image: ghcr.io/ggml-org/llama.cpp:server\n    ports:\n      - 8080:8080\n    volumes:\n      - ./models:/models\n    environment:\n      # alternatively, you can use \"LLAMA_ARG_MODEL_URL\" to download the model\n      LLAMA_ARG_MODEL: /models/my_model.gguf\n      LLAMA_ARG_CTX_SIZE: 4096\n      LLAMA_ARG_N_PARALLEL: 2\n      LLAMA_ARG_ENDPOINT_METRICS: 1\n      LLAMA_ARG_PORT: 8080\n```\n\n### Multimodal support\n\nMultimodal support was added in [#12898](https://github.com/ggml-org/llama.cpp/pull/12898) and is currently an experimental feature.\nIt is currently available in the following endpoints:\n- The OAI-compatible chat endpoint.\n- The non-OAI-compatible completions endpoint.\n- The non-OAI-compatible embeddings endpoint.\n\nFor more details, please refer to [multimodal documentation](../../docs/multimodal.md)\n\n### Server tools support\n\nThe server includes a set of server tools that enable the LLM to access the local file system directly from the Web UI.\n\nTo use this feature, start the server with `--tools all`. You can also enable only specific tools by passing a comma-separated list: `--tools name1,name2,...`. Run `--help` for the full list of available tool names.\n\n### MCP servers\n\nBesides the built-in tools, the server can expose tools coming from MCP servers, added in [#26062](https://github.com/ggml-org/llama.cpp/pull/26062). Only the stdio transport is supported: such a server is a child process reading JSON-RPC messages on its stdin and writing replies on its stdout, so nothing has to be started or maintained outside `llama-server`.\n\nServers are declared in a Cursor-compatible JSON file:\n\n```json\n{\n  \"mcpServers\": {\n    \"example\": { \"command\": \"/path/to/server\", \"args\": [] }\n  }\n}\n```\n\n```sh\nllama-server -m model.gguf --mcp-servers-config mcp.json\n```\n\nThe same JSON can be passed inline with `--mcp-servers-json`. Each entry under `mcpServers` accepts:\n\n| Key | Explanation |\n| --- | ----------- |\n| `command` | executable to spawn, required, entries without it are skipped |\n| `args` | array of arguments |\n| `env` | object merged over the parent environment |\n| `cwd` | working directory of the child process |\n| `timeout_ms` | per-tool-call timeout (default: 30000) |\n\nEvery server is spawned once at startup to list its tools, then stopped, and respawned on demand when one of its tools is called. Tools are exposed as ` _ ` alongside the built-in ones: they show up in the Web UI and in `GET /tools`, and the model calls them like any other tool. A name colliding with an already registered tool is skipped. This is independent of `--tools`, MCP servers can be the only tools available.\n\nThe child process runs with the same privileges as the server, so only declare commands you trust. As with `--tools`, `--cors-origins` then defaults to `localhost`.\n\nNote: `--ui-mcp-proxy` is unrelated, it only lets the Web UI reach remote MCP servers from the browser.\n\nAny server written against the [MCP specification](https://modelcontextprotocol.io) works as is, whether it uses an official SDK or not: the transport is one JSON-RPC message per line on stdio, so a script wrapping an existing program is a valid server too.\n\n### CORS\n\nBy default the server reflects any `Origin` header back with credentials allowed. This matches the old, always-on `*` behavior and is fine as long as the server only exposes stateless, read-only endpoints.\n\nEnabling `--tools` or `--agent` exposes file read/write over the API, so in that case `--cors-origins` defaults to `localhost` instead: only pages served from localhost can reach the server. Pass `--cors-origins` explicitly to override either default.\n\nRecommended `--cors-origins` setting, depending on where the server runs:\n\n| Deployment | Recommendation |\n| ---------- | --------------- |\n| Public | set an API key, put the server behind a reverse proxy, `--cors-origins` optional |\n| Local network | set `--cors-origins` to your frontend's origin |\n| Same machine | `--cors-origins localhost` (default once `--agent` is set) |\n\nRelated flags: `--cors-origins`, `--cors-methods`, `--cors-headers`, `--cors-credentials` / `--no-cors-credentials`. Background and rationale: [#25655](https://github.com/ggml-org/llama.cpp/pull/25655).\n\n## Build\n\n`llama-server` is built alongside everything else from the root of the project\n\n- Using `CMake`:\n\n  ```bash\n  cmake -B build\n  cmake --build build --config Release -t llama-server\n  ```\n\n Binary is at `./build/bin/llama-server`\n\n## Build with SSL\n\n`llama-server` can also be built with SSL support using OpenSSL 3\n\n- Using `CMake`:\n\n  ```bash\n  cmake -B build -DLLAMA_OPENSSL=ON\n  cmake --build build --config Release -t llama-server\n  ```\n\n## Quick Start\n\nTo get started right away, run the following command, making sure to use the correct path for the model you have:\n\n### Unix-based systems (Linux, macOS, etc.)\n\n```bash\n./llama-server -m models/7B/ggml-model.gguf -c 2048\n```\n\n### Windows\n\n```powershell\nllama-server.exe -m models\\7B\\ggml-model.gguf -c 2048\n```\n\nThe above command will start a server that by default listens on `127.0.0.1:8080`.\nYou can consume the endpoints with Postman or NodeJS with axios library. You can visit the web front end at the same url.\n\n### Docker\n\n```bash\ndocker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080\n\n# or, with CUDA:\ndocker run -p 8080:8080 -v /path/to/models:/models --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99\n```\n\n## Using with CURL\n\nUsing [curl](https://curl.se/). On Windows, `curl.exe` should be available in the base OS.\n\n```sh\ncurl --request POST \\\n    --url http://localhost:8080/completion \\\n    --header \"Content-Type: application/json\" \\\n    --data '{\"prompt\": \"Building a website can be done in 10 simple steps:\",\"n_predict\": 128}'\n```\n\n## API Endpoints\n\n### GET `/health`: Returns health check result\n\nThis endpoint is public (no API key check). `/v1/health` also works.\n\n**Response format**\n\n- HTTP status code 503\n - Body: `{\"error\": {\"code\": 503, \"message\": \"Loading model\", \"type\": \"unavailable_error\"}}`\n - Explanation: the model is still being loaded.\n- HTTP status code 200\n - Body: `{\"status\": \"ok\" }`\n - Explanation: the model is successfully loaded and the server is ready.\n\n### POST `/completion`: Given a `prompt`, it returns the predicted completion.\n\n> [!IMPORTANT]\n>\n> This endpoint is **not** OAI-compatible. For OAI-compatible client, use `/v1/completions` instead.\n\n*Options:*\n\n`prompt`: Provide the prompt for this completion as a string or as an array of strings or numbers representing tokens. Internally, if `cache_prompt` is `true`, the prompt is compared to the previous completion and only the \"unseen\" suffix is evaluated. A `BOS` token is inserted at the start, if all of the following conditions are true:\n\n - The prompt is a string or an array with the first element given as a string\n - The model's `tokenizer.ggml.add_bos_token` metadata is `true`\n\nThese input shapes and data type are allowed for `prompt`:\n\n - Single string: `\"string\"`\n - Single sequence of tokens: `[12, 34, 56]`\n - Mixed tokens and strings: `[12, 34, \"string\", 56, 78]`\n - A JSON object which optionally contains multimodal data: `{ \"prompt_string\": \"string\", \"multimodal_data\": [\"base64\"] }`\n\nMultiple prompts are also supported. In this case, the completion result will be an array.\n\n - Only strings: `[\"string1\", \"string2\"]`\n - Strings, JSON objects, and sequences of tokens: `[\"string1\", [12, 34, 56], { \"prompt_string\": \"string\", \"multimodal_data\": [\"base64\"]}]`\n - Mixed types: `[[12, 34, \"string\", 56, 78], [12, 34, 56], \"string\", { \"prompt_string\": \"string\" }]`\n\nNote for `multimodal_data` in JSON object prompts. This should be an array of strings, containing base64 encoded multimodal data such as images and audio. There must be an identical number of MTMD media markers in the string prompt element which act as placeholders for the data provided to this parameter. The multimodal data files will be substituted in order. The marker string (e.g. `<__media__>`) can be found by calling `mtmd_default_marker()` defined in [the MTMD C API](https://github.com/ggml-org/llama.cpp/blob/5fd160bbd9d70b94b5b11b0001fd7f477005e4a0/tools/mtmd/mtmd.h#L87). A client *must not* specify this field unless the server has the multimodal capability. Clients should check `/models` or `/v1/models` for the `multimodal` capability before a multimodal request.\n\n`temperature`: Adjust the randomness of the generated text. Default: `0.8`\n\n`dynatemp_range`: Dynamic temperature range. The final temperature will be in the range of `[temperature - dynatemp_range; temperature + dynatemp_range]` Default: `0.0`, which is disabled.\n\n`dynatemp_exponent`: Dynamic temperature exponent. Default: `1.0`\n\n`top_k`: Limit the next token selection to the K most probable tokens. Default: `40`\n\n`top_p`: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: `0.95`\n\n`min_p`: The minimum probability for a token to be considered, relative to the probability of the most likely token. Default: `0.05`\n\n`n_predict`: Set the maximum number of tokens to predict when generating text. **Note:** May exceed the set limit slightly if the last token is a partial multibyte character. When 0, no tokens will be generated but the prompt is evaluated into the cache. Default: `-1`, where `-1` is infinity.\n\n`n_indent`: Specify the minimum line indentation for the generated text in number of whitespace characters. Useful for code completion tasks. Default: `0`\n\n`n_keep`: Specify the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be discarded. The number excludes the BOS token.\nBy default, this value is set to `0`, meaning no tokens are kept. Use `-1` to retain all tokens from the prompt.\n\n`n_cmpl`: Number of completions to generate from the current prompt. If input has multiple prompts, the output will have N prompts times `n_cmpl` entries.\n\n`n_cache_reuse`: Min chunk size to attempt reusing from the cache via KV shifting. For more info, see `--cache-reuse` arg. Default: `0`, which is disabled.\n\n`stream`: Allows receiving each predicted token in real-time instead of waiting for the completion to finish (uses a different response format). To enable this, set to `true`.\n\n`stop`: Specify a JSON array of stopping strings.\nThese words will not be included in the completion, so make sure to add them to the prompt for the next iteration. Default: `[]`\n\n`typical_p`: Enable locally typical sampling with parameter p. Default: `1.0`, which is disabled.\n\n`repeat_penalty`: Control the repetition of token sequences in the generated text. Default: `1.1`\n\n`repeat_last_n`: Last n tokens to consider for penalizing repetition. Default: `64`, where `0` is disabled.\n\n`presence_penalty`: Repeat alpha presence penalty. Default: `0.0`, which is disabled.\n\n`frequency_penalty`: Repeat alpha frequency penalty. Default: `0.0`, which is disabled.\n\n`dry_multiplier`: Set the DRY (Don't Repeat Yourself) repetition penalty multiplier. Default: `0.0`, which is disabled.\n\n`dry_base`: Set the DRY repetition penalty base value. Default: `1.75`\n\n`dry_allowed_length`: Tokens that extend repetition beyond this receive exponentially increasing penalty: multiplier * base ^ (length of repeating sequence before token - allowed length). Default: `2`\n\n`dry_penalty_last_n`: How many tokens to scan for repetitions. Default: `64`, where `0` is disabled.\n\n`dry_sequence_breakers`: Specify an array of sequence breakers for DRY sampling. Only a JSON array of strings is accepted. Default: `['\\n', ':', '\"', '*']`\n\n`xtc_probability`: Set the chance for token removal via XTC sampler. Default: `0.0`, which is disabled.\n\n`xtc_threshold`: Set a minimum probability threshold for tokens to be removed via XTC sampler. Default: `0.1` (> `0.5` disables XTC)\n\n`mirostat`: Enable Mirostat sampling, controlling perplexity during text generation. Default: `0`, where `0` is disabled, `1` is Mirostat, and `2` is Mirostat 2.0.\n\n`mirostat_tau`: Set the Mirostat target entropy, parameter tau. Default: `5.0`\n\n`mirostat_eta`: Set the Mirostat learning rate, parameter eta. Default: `0.1`\n\n`grammar`: Set grammar for grammar-based sampling. Default: no grammar\n\n`json_schema`: Set a JSON schema for grammar-based sampling (e.g. `{\"items\": {\"type\": \"string\"}, \"minItems\": 10, \"maxItems\": 100}` of a list of strings, or `{}` for any JSON). See [tests](../../tests/test-json-schema-to-grammar.cpp) for supported features. Default: no JSON schema.\n\n`seed`: Set the random number generator (RNG) seed. Default: `-1`, which is a random seed.\n\n`ignore_eos`: Ignore end of stream token and continue generating. Default: `false`\n\n`logit_bias`: Modify the likelihood of a token appearing in the generated text completion. For example, use `\"logit_bias\": [[15043,1.0]]` to increase the likelihood of the token 'Hello', or `\"logit_bias\": [[15043,-1.0]]` to decrease its likelihood. Setting the value to false, `\"logit_bias\": [[15043,false]]` ensures that the token `Hello` is never produced. The tokens can also be represented as strings, e.g. `[[\"Hello, World!\",-0.5]]` will reduce the likelihood of all the individual tokens that represent the string `Hello, World!`, just like the `presence_penalty` does. For compatibility with the OpenAI API, a JSON object {\" \": bias, ...} can also be passed. Default: `[]`\n\n`n_probs`: If greater than 0, the response also contains the probabilities of top N tokens for each generated token given the sampling settings. Note that for temperature < 0 the tokens are sampled greedily but token probabilities are still being calculated via a simple softmax of the logits without considering any other sampler settings. Default: `0`\n\n`min_keep`: If greater than 0, force samplers to return N possible tokens at minimum. Default: `0`\n\n`t_max_predict_ms`: Set a time limit in milliseconds for the prediction (a.k.a. text-generation) phase. The timeout will trigger if the generation takes more than the specified time (measured since the first token was generated) and if a new-line character has already been generated. Useful for FIM applications. Default: `0`, which is disabled.\n\n`id_slot`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot. Default: `-1`\n\n`cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `true`\n\n`return_tokens`: Return the raw generated token ids in the `tokens` field. Otherwise `tokens` remains empty. Default: `false`\n\n`samplers`: The order the samplers should be applied in. An array of strings representing sampler type names. If a sampler is not set, it will not be used. If a sampler is specified more than once, it will be applied multiple times. Default: `[\"dry\", \"top_k\", \"typ_p\", \"top_p\", \"min_p\", \"xtc\", \"temperature\"]` - these are all the available values.\n\n`timings_per_token`: Include prompt processing and text generation speed information in each response. Default: `false`\n\n`return_progress`: Include prompt processing progress in `stream` mode. The progress will be contained inside `prompt_progress` with 4 values: `total`, `cache`, `processed`, and `time_ms`. The overall progress is `processed/total`, while the actual timed progress is `(processed-cache)/(total-cache)`. The `time_ms` field contains the elapsed time in milliseconds since prompt processing started. Default: `false`\n\n`sse_ping_interval`: Interval in seconds between SSE comment pings emitted while the stream stays silent, keeping the connection observable during long prompt processing. Overrides the server `--sse-ping-interval` setting for this request, `-1` disables pings. Default: server setting\n\n`post_sampling_probs`: Returns the probabilities of top `n_probs` tokens after applying sampling chain.\n\n`response_fields`: A list of response fields, for example: `\"response_fields\": [\"content\", \"generation_settings/n_predict\"]`. If the specified field is missing, it will simply be omitted from the response without triggering an error. Note that fields with a slash will be unnested; for example, `generation_settings/n_predict` will move the field `n_predict` from the `generation_settings` object to the root of the response and give it a new name.\n\n`lora`: A list of LoRA adapters to be applied to this specific request. Each object in the list must contain `id` and `scale` fields. For example: `[{\"id\": 0, \"scale\": 0.5}, {\"id\": 1, \"scale\": 1.1}]`. If a LoRA adapter is not specified in the list, its scale will default to `0.0`. Please note that requests with different LoRA configurations will not be batched together, which may result in performance degradation.\n\n**Response format**\n\n- Note: In streaming mode (`stream`), only `content`, `tokens` and `stop` will be returned until end of completion. Responses are sent using the [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) standard. Note: the browser's `EventSource` interface cannot be used due to its lack of `POST` request support.\n\n- `completion_probabilities`: An array of token probabilities for each completion. The array's length is `n_predict`. Each item in the array has a nested array `top_logprobs`. It contains at **maximum** `n_probs` elements:\n  ```\n  {\n    \"content\": \"<the generated completion text>\",\n    \"tokens\": [ generated token ids if requested ],\n    ...\n    \"probs\": [\n      {\n        \"id\": <token id>,\n        \"logprob\": float,\n        \"token\": \"<most likely token>\",\n        \"bytes\": [int, int, ...],\n        \"top_logprobs\": [\n          {\n            \"id\": <token id>,\n            \"logprob\": float,\n            \"token\": \"<token text>\",\n            \"bytes\": [int, int, ...],\n          },\n          {\n            \"id\": <token id>,\n            \"logprob\": float,\n            \"token\": \"<token text>\",\n            \"bytes\": [int, int, ...],\n          },\n          ...\n        ]\n      },\n      {\n        \"id\": <token id>,\n        \"logprob\": float,\n        \"token\": \"<most likely token>\",\n        \"bytes\": [int, int, ...],\n        \"top_logprobs\": [\n          ...\n        ]\n      },\n      ...\n    ]\n  },\n  ```\n Please note that if `post_sampling_probs` is set to `true`:\n - `logprob` will be replaced with `prob`, with the value between 0.0 and 1.0\n - `top_logprobs` will be replaced with `top_probs`. Each element contains:\n - `id`: token ID\n - `token`: token in string\n - `bytes`: token in bytes\n - `prob`: token probability, with the value between 0.0 and 1.0\n - Number of elements in `top_probs` may be less than `n_probs`\n\n- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.\n- `tokens`: Same as `content` but represented as raw token ids. Only populated if `\"return_tokens\": true` or `\"stream\": true` in the request.\n- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)\n- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`. These options may differ from the original ones in some way (e.g. bad values filtered out, strings converted to tokens, etc.).\n- `model`: The model alias (for model path, please use `/props` endpoint)\n- `prompt`: The processed `prompt` (special tokens may be added)\n- `stop_type`: Indicating whether the completion has stopped. Possible values are:\n - `none`: Generating (not stopped)\n - `eos`: Stopped because it encountered the EOS token\n - `limit`: Stopped because `n_predict` tokens were generated before stop words or EOS was encountered\n - `word`: Stopped due to encountering a stopping word from `stop` JSON array provided\n- `stopping_word`: The stopping word encountered which stopped the generation (or \"\" if not stopped due to a stopping word)\n- `timings`: Hash of timing information about the completion such as the number of tokens `predicted_per_second`\n- `tokens_cached`: Number of tokens from the prompt which could be re-used from previous completion\n- `tokens_evaluated`: Number of tokens evaluated in total from the prompt\n- `truncated`: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (`tokens_evaluated`) plus tokens generated (`tokens predicted`) exceeded the context size (`n_ctx`)\n\n### POST `/tokenize`: Tokenize a given text\n\n*Options:*\n\n`content`: (Required) The text to tokenize.\n\n`add_special`: (Optional) Boolean indicating if special tokens, i.e. `BOS`, should be inserted. Default: `false`\n\n`parse_special`: (Optional) Boolean indicating if special tokens should be tokenized. When `false` special tokens are treated as plaintext. Default: `true`\n\n`with_pieces`: (Optional) Boolean indicating whether to return token pieces along with IDs. Default: `false`\n\n**Response:**\n\nReturns a JSON object with a `tokens` field containing the tokenization result. The `tokens` array contains either just token IDs or objects with `id` and `piece` fields, depending on the `with_pieces` parameter. The piece field is a string if the piece is valid unicode or a list of bytes otherwise.\n\nIf `with_pieces` is `false`:\n```json\n{\n  \"tokens\": [123, 456, 789]\n}\n```\n\nIf `with_pieces` is `true`:\n```json\n{\n  \"tokens\": [\n    {\"id\": 123, \"piece\": \"Hello\"},\n    {\"id\": 456, \"piece\": \" world\"},\n    {\"id\": 789, \"piece\": \"!\"}\n  ]\n}\n```\n\nWith input 'á' (utf8 hex: C3 A1) on tinyllama/stories260k\n```\n{\n  \"tokens\": [\n    {\"id\": 198, \"piece\": [195]}, // hex C3\n    {\"id\": 164, \"piece\": [161]} // hex A1\n  ]\n}\n```\n\n### POST `/detokenize`: Convert tokens to text\n\n*Options:*\n\n`tokens`: Set the tokens to detokenize.\n\n### POST `/apply-template`: Apply chat template to a conversation\n\nUses the server's prompt template formatting functionality to convert chat messages to a single string expected by a chat model as input, but does not perform inference. Instead, the prompt string is returned in the `prompt` field of the JSON response. The prompt can then be modified as desired (for example, to insert \"Sure!\" at the beginning of the model's response) before sending to `/completion` to generate the chat response.\n\n*Options:*\n\n`messages`: (Required) Chat turns in the same format as `/v1/chat/completions`.\n\n**Response format**\n\nReturns a JSON object with a field `prompt` containing a string of the input messages formatted according to the model's chat template format.\n\n### POST `/embedding`: Generate embedding of a given text\n\n> [!IMPORTANT]\n>\n> This endpoint is **not** OAI-compatible. For OAI-compatible client, use `/v1/embeddings` instead.\n\nThe same as [the embedding example](../embedding) does.\n\nThis endpoint also supports multimodal embeddings. See the documentation for the `/completions` endpoint for details on how to send a multimodal prompt.\n\n*Options:*\n\n`content`: Set the text to process.\n\n`embd_normalize`: Normalization for pooled embeddings. Can be one of the following values:\n```\n  -1: No normalization\n   0: Max absolute\n   1: Taxicab\n   2: Euclidean/L2\n  >2: P-Norm\n```\n\n### POST `/reranking`: Rerank documents according to a given query\n\nSimilar to https://jina.ai/reranker/ but might change in the future.\nRequires a reranker model (such as [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3)) and the `--embedding --pooling rank` options.\n\n*Options:*\n\n`query`: The query against which the documents will be ranked.\n\n`documents`: An array strings representing the documents to be ranked.\n\n*Aliases:*\n - `/rerank`\n - `/v1/rerank`\n - `/v1/reranking`\n\n*Examples:*\n\n```shell\ncurl http://127.0.0.1:8012/v1/rerank \\\n    -H \"Content-Type: application/json\" \\\n    -d '{\n        \"model\": \"some-model\",\n            \"query\": \"What is panda?\",\n            \"top_n\": 3,\n            \"documents\": [\n                \"hi\",\n            \"it is a bear\",\n            \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n            ]\n    }' | jq\n```\n\n### POST `/infill`: For code infilling.\n\nTakes a prefix and a suffix and returns the predicted completion as stream.\n\n*Options:*\n\n- `input_prefix`: Set the prefix of the code to infill.\n- `input_suffix`: Set the suffix of the code to infill.\n- `input_extra`: Additional context inserted before the FIM prefix.\n- `prompt`: Added after the `FIM_MID` token\n\n`input_extra` is array of `{\"filename\": string, \"text\": string}` objects.\n\nThe endpoint also accepts all the options of `/completion`.\n\nIf the model has `FIM_REPO` and `FIM_FILE_SEP` tokens, the [repo-level pattern](https://arxiv.org/pdf/2409.12186) is used:\n\n```txt\n<FIM_REP>myproject\n<FIM_SEP>{chunk 0 filename}\n{chunk 0 text}\n<FIM_SEP>{chunk 1 filename}\n{chunk 1 text}\n...\n<FIM_SEP>filename\n<FIM_PRE>[input_prefix]<FIM_SUF>[input_suffix]<FIM_MID>[prompt]\n```\n\nIf the tokens are missing, then the extra context is simply prefixed at the start:\n\n```txt\n[input_extra]<FIM_PRE>[input_prefix]<FIM_SUF>[input_suffix]<FIM_MID>[prompt]\n```\n\n### **GET** `/props`: Get server global properties.\n\nBy default, it is read-only. To make POST request to change global properties, you need to start server with `--props`\n\n**Response format**\n\n```json\n{\n  \"default_generation_settings\": {\n    \"id\": 0,\n    \"id_task\": -1,\n    \"n_ctx\": 1024,\n    \"speculative\": false,\n    \"is_processing\": false,\n    \"params\": {\n      \"n_predict\": -1,\n      \"seed\": 4294967295,\n      \"temperature\": 0.800000011920929,\n      \"dynatemp_range\": 0.0,\n      \"dynatemp_exponent\": 1.0,\n      \"top_k\": 40,\n      \"top_p\": 0.949999988079071,\n      \"min_p\": 0.05000000074505806,\n      \"xtc_probability\": 0.0,\n      \"xtc_threshold\": 0.10000000149011612,\n      \"typical_p\": 1.0,\n      \"repeat_last_n\": 64,\n      \"repeat_penalty\": 1.0,\n      \"presence_penalty\": 0.0,\n      \"frequency_penalty\": 0.0,\n      \"dry_multiplier\": 0.0,\n      \"dry_base\": 1.75,\n      \"dry_allowed_length\": 2,\n      \"dry_penalty_last_n\": 64,\n      \"dry_sequence_breakers\": [\n        \"\\n\",\n        \":\",\n        \"\\\"\",\n        \"*\"\n      ],\n      \"mirostat\": 0,\n      \"mirostat_tau\": 5.0,\n      \"mirostat_eta\": 0.10000000149011612,\n      \"stop\": [],\n      \"max_tokens\": -1,\n      \"n_keep\": 0,\n      \"n_discard\": 0,\n      \"ignore_eos\": false,\n      \"stream\": true,\n      \"n_probs\": 0,\n      \"min_keep\": 0,\n      \"grammar\": \"\",\n      \"samplers\": [\n        \"dry\",\n        \"top_k\",\n        \"typ_p\",\n        \"top_p\",\n        \"min_p\",\n        \"xtc\",\n        \"temperature\"\n      ],\n      \"speculative.n_max\": 16,\n      \"speculative.n_min\": 5,\n      \"speculative.p_min\": 0.8999999761581421,\n      \"timings_per_token\": false\n    },\n    \"prompt\": \"\",\n    \"next_token\": {\n      \"has_next_token\": true,\n      \"has_new_line\": false,\n      \"n_remain\": -1,\n      \"n_decoded\": 0,\n      \"stopping_word\": \"\"\n    }\n  },\n  \"total_slots\": 1,\n  \"model_path\": \"../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf\",\n  \"chat_template\": \"...\",\n  \"chat_template_caps\": {},\n  \"modalities\": {\n    \"vision\": false\n  },\n  \"media_marker\": \"<__media_YoNhud46VdDqbuFmKYEO9PY7A4ARzRfg__>\",\n  \"build_info\": \"b(build number)-(build commit hash)\",\n  \"is_sleeping\": false\n}\n```\n\n- `default_generation_settings` - the default generation settings for the `/completion` endpoint, which has the same fields as the `generation_settings` response object from the `/completion` endpoint.\n- `total_slots` - the total number of slots for process requests (defined by `--parallel` option)\n- `model_path` - the path to model file (same with `-m` argument)\n- `chat_template` - the model's original Jinja2 prompt template\n- `chat_template_caps` - capabilities of the chat template (see `common/jinja/caps.h` for more info)\n- `modalities` - the list of supported modalities\n- `is_sleeping` - sleeping status, see [Sleeping on idle](#sleeping-on-idle)\n\n### POST `/props`: Change server global properties.\n\nTo use this endpoint with POST method, you need to start server with `--props`\n\n*Options:*\n\n- None yet\n\n### POST `/embeddings`: non-OpenAI-compatible embeddings API\n\nThis endpoint supports all poolings, including `--pooling none`. When the pooling is `none`, the responses will contain the *unnormalized* embeddings for *all* input tokens. For all other pooling types, only the pooled embeddings are returned, normalized using Euclidean norm.\n\nNote that the response format of this endpoint is different from `/v1/embeddings`.\n\n*Options:*\n\nSame as the `/v1/embeddings` endpoint.\n\n*Examples:*\n\nSame as the `/v1/embeddings` endpoint.\n\n**Response format**\n\n```\n[\n  {\n    \"index\": 0,\n    \"embedding\": [\n      [ ... embeddings for token 0   ... ],\n      [ ... embeddings for token 1   ... ],\n      [ ... ]\n      [ ... embeddings for token N-1 ... ],\n    ]\n  },\n  ...\n  {\n    \"index\": P,\n    \"embedding\": [\n      [ ... embeddings for token 0   ... ],\n      [ ... embeddings for token 1   ... ],\n      [ ... ]\n      [ ... embeddings for token N-1 ... ],\n    ]\n  }\n]\n```\n\n### GET `/slots`: Returns the current slots processing state\n\nThis endpoint is enabled by default and can be disabled with `--no-slots`. It can be used to query various per-slot metrics, such as speed, processed tokens, sampling parameters, etc.\n\nIf query param `?fail_on_no_slot=1` is set, this endpoint will respond with status code 503 if there is no available slots.\n\n**Response format**\n\n \n Example with 2 slots \n\n```json\n[\n  {\n    \"id\": 0,\n    \"id_task\": 135,\n    \"n_ctx\": 65536,\n    \"speculative\": false,\n    \"is_processing\": true,\n    \"params\": {\n      \"n_predict\": -1,\n      \"seed\": 4294967295,\n      \"temperature\": 0.800000011920929,\n      \"dynatemp_range\": 0.0,\n      \"dynatemp_exponent\": 1.0,\n      \"top_k\": 40,\n      \"top_p\": 0.949999988079071,\n      \"min_p\": 0.05000000074505806,\n      \"top_n_sigma\": -1.0,\n      \"xtc_probability\": 0.0,\n      \"xtc_threshold\": 0.10000000149011612,\n      \"typical_p\": 1.0,\n      \"repeat_last_n\": 64,\n      \"repeat_penalty\": 1.0,\n      \"presence_penalty\": 0.0,\n      \"frequency_penalty\": 0.0,\n      \"dry_multiplier\": 0.0,\n      \"dry_base\": 1.75,\n      \"dry_allowed_length\": 2,\n      \"dry_penalty_last_n\": 131072,\n      \"mirostat\": 0,\n      \"mirostat_tau\": 5.0,\n      \"mirostat_eta\": 0.10000000149011612,\n      \"max_tokens\": -1,\n      \"n_keep\": 0,\n      \"n_discard\": 0,\n      \"ignore_eos\": false,\n      \"stream\": true,\n      \"n_probs\": 0,\n      \"min_keep\": 0,\n      \"chat_format\": \"GPT-OSS\",\n      \"reasoning_format\": \"none\",\n      \"reasoning_in_content\": false,\n      \"generation_prompt\": \"\",\n      \"samplers\": [\n        \"penalties\",\n        \"dry\",\n        \"top_k\",\n        \"typ_p\",\n        \"top_p\",\n        \"min_p\",\n        \"xtc\",\n        \"temperature\"\n      ],\n      \"speculative.n_max\": 16,\n      \"speculative.n_min\": 0,\n      \"speculative.p_min\": 0.75,\n      \"timings_per_token\": false,\n      \"post_sampling_probs\": false,\n      \"lora\": []\n    },\n    \"next_token\": {\n      \"has_next_token\": true,\n      \"has_new_line\": false,\n      \"n_remain\": -1,\n      \"n_decoded\": 0\n    }\n  },\n  {\n    \"id\": 1,\n    \"id_task\": 0,\n    \"n_ctx\": 65536,\n    \"speculative\": false,\n    \"is_processing\": true,\n    \"params\": {\n      \"n_predict\": -1,\n      \"seed\": 4294967295,\n      \"temperature\": 0.800000011920929,\n      \"dynatemp_range\": 0.0,\n      \"dynatemp_exponent\": 1.0,\n      \"top_k\": 40,\n      \"top_p\": 0.949999988079071,\n      \"min_p\": 0.05000000074505806,\n      \"top_n_sigma\": -1.0,\n      \"xtc_probability\": 0.0,\n      \"xtc_threshold\": 0.10000000149011612,\n      \"typical_p\": 1.0,\n      \"repeat_last_n\": 64,\n      \"repeat_penalty\": 1.0,\n      \"presence_penalty\": 0.0,\n      \"frequency_penalty\": 0.0,\n      \"dry_multiplier\": 0.0,\n      \"dry_base\": 1.75,\n      \"dry_allowed_length\": 2,\n      \"dry_penalty_last_n\": 131072,\n      \"mirostat\": 0,\n      \"mirostat_tau\": 5.0,\n      \"mirostat_eta\": 0.10000000149011612,\n      \"max_tokens\": -1,\n      \"n_keep\": 0,\n      \"n_discard\": 0,\n      \"ignore_eos\": false,\n      \"stream\": true,\n      \"n_probs\": 0,\n      \"min_keep\": 0,\n      \"chat_format\": \"GPT-OSS\",\n      \"reasoning_format\": \"none\",\n      \"reasoning_in_content\": false,\n      \"generation_prompt\": \"\",\n      \"samplers\": [\n        \"penalties\",\n        \"dry\",\n        \"top_k\",\n        \"typ_p\",\n        \"top_p\",\n        \"min_p\",\n        \"xtc\",\n        \"temperature\"\n      ],\n      \"speculative.n_max\": 16,\n      \"speculative.n_min\": 0,\n      \"speculative.p_min\": 0.75,\n      \"timings_per_token\": false,\n      \"post_sampling_probs\": false,\n      \"lora\": []\n    },\n    \"next_token\": {\n      \"has_next_token\": true,\n      \"has_new_line\": true,\n      \"n_remain\": -1,\n      \"n_decoded\": 136\n    }\n  }\n]\n```\n\n \n\n### GET `/metrics`: Prometheus compatible metrics exporter\n\nThis endpoint is only accessible if `--metrics` is set.\n\nIn *router mode* the query param `?model={model_id}` has to be set. This endpoint will respond with status code 400 `model name is missing from the request` if not set.\n\n#### Available metrics\n\n| Metric | Type | Description |\n| ------ | ---------------------- | ----------- |\n| `llamacpp:prompt_tokens_total` | Counter | Number of prompt tokens processed. |\n| `llamacpp:prompt_seconds_total` | Counter | Prompt process time in seconds. |\n| `llamacpp:prompt_tokens_seconds` | Gauge | Average prompt throughput in tokens/s. |\n| `llamacpp:tokens_predicted_total` | Counter | Number of generation tokens processed. |\n| `llamacpp:tokens_predicted_seconds_total` | Counter | Predict process time in seconds. |\n| `llamacpp:predicted_tokens_seconds` | Gauge | Average generation throughput in tokens/s. |\n| `llamacpp:requests_processing` | Gauge | Number of requests processing. |\n| `llamacpp:requests_deferred` | Gauge | Number of requests deferred. |\n| `llamacpp:n_tokens_max` | Counter | High watermark of the context size observed. |\n| `llamacpp:n_decode_total` | Counter | Total Number of llama_decode() calls. |\n| `llamacpp:n_busy_slots_per_decode` | Gauge | Average number of busy slots per llama_decode() call. |\n| `llamacpp:spec_decode_num_draft_tokens_total` | Counter | Total draft tokens generated (0 when spec-decode is off). |\n| `llamacpp:spec_decode_num_accepted_tokens_total` | Counter | Total draft tokens accepted by the target model (0 when spec-decode is off). |\n| `llamacpp:spec_decode_num_drafts_total` | Counter | Total speculative decoding verification steps (0 when spec-decode is off). |\n| `llamacpp:spec_decode_num_accepted_tokens_per_pos_total` | Counter | Accepted tokens per draft position (labeled `position=\"N\"`; absent when spec-decode is off or before the first completed speculative request). |\n\n### POST `/slots/{id_slot}?action=save`: Save the prompt cache of the specified slot to a file.\n\n*Options:*\n\n`filename`: Name of the file to save the slot's prompt cache. The file will be saved in the directory specified by the `--slot-save-path` server parameter.\n\n**Response format**\n\n```json\n{\n    \"id_slot\": 0,\n    \"filename\": \"slot_save_file.bin\",\n    \"n_saved\": 1745,\n    \"n_written\": 14309796,\n    \"timings\": {\n        \"save_ms\": 49.865\n    }\n}\n```\n\n### POST `/slots/{id_slot}?action=restore`: Restore the prompt cache of the specified slot from a file.\n\n*Options:*\n\n`filename`: Name of the file to restore the slot's prompt cache from. The file should be located in the directory specified by the `--slot-save-path` server parameter.\n\n**Response format**\n\n```json\n{\n    \"id_slot\": 0,\n    \"filename\": \"slot_save_file.bin\",\n    \"n_restored\": 1745,\n    \"n_read\": 14309796,\n    \"timings\": {\n        \"restore_ms\": 42.937\n    }\n}\n```\n\n### POST `/slots/{id_slot}?action=erase`: Erase the prompt cache of the specified slot.\n\n**Response format**\n\n```json\n{\n    \"id_slot\": 0,\n    \"n_erased\": 1745\n}\n```\n\n### GET `/lora-adapters`: Get list of all LoRA adapters\n\nThis endpoint returns the loaded LoRA adapters. You can add adapters using `--lora` when starting the server, for example: `--lora my_adapter_1.gguf --lora my_adapter_2.gguf ...`\n\nBy default, all adapters will be loaded with scale set to 1. To initialize all adapters scale to 0, add `--lora-init-without-apply`\n\nPlease note that this value will be overwritten by the `lora` field for each request.\n\nIf an adapter is disabled, the scale will be set to 0.\n\n**Response format**\n\n```json\n[\n    {\n        \"id\": 0,\n        \"path\": \"my_adapter_1.gguf\",\n        \"scale\": 0.0\n    },\n    {\n        \"id\": 1,\n        \"path\": \"my_adapter_2.gguf\",\n        \"scale\": 0.0\n    }\n]\n```\n\n### POST `/lora-adapters`: Set list of LoRA adapters\n\nThis sets the global scale for LoRA adapters. Please note that this value will be overwritten by the `lora` field for each request.\n\nTo disable an adapter, either remove it from the list below, or set scale to 0.\n\n**Request format**\n\nTo know the `id` of the adapter, use GET `/lora-adapters`\n\n```json\n[\n  {\"id\": 0, \"scale\": 0.2},\n  {\"id\": 1, \"scale\": 0.8}\n]\n```\n\n## OpenAI-compatible API Endpoints\n\n### GET `/v1/models`: OpenAI-compatible Model Info API\n\nReturns information about the loaded model. See [OpenAI Models API documentation](https://platform.openai.com/docs/api-reference/models).\n\nThe returned list always has one single element. The `meta` field can be `null` (for example, while the model is still loading).\n\nBy default, model `id` field is the path to model file, specified via `-m`. You can set a custom value for model `id` field via `--alias` argument. For example, `--alias gpt-4o-mini`.\n\nExample:\n\n```json\n{\n    \"object\": \"list\",\n    \"data\": [\n        {\n            \"id\": \"../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf\",\n            \"object\": \"model\",\n            \"created\": 1735142223,\n            \"owned_by\": \"llamacpp\",\n            \"meta\": {\n                \"vocab_type\": 2,\n                \"n_vocab\": 128256,\n                \"n_ctx_train\": 131072,\n                \"n_embd\": 4096,\n                \"n_params\": 8030261312,\n                \"size\": 4912898304\n            }\n        }\n    ]\n}\n```\n\n### POST `/v1/completions`: OpenAI-compatible Completions API\n\nGiven an input `prompt`, it returns the predicted completion. Streaming mode is also supported. While no strong claims of compatibility with OpenAI API spec is being made, in our experience it suffices to support many apps.\n\n*Options:*\n\nSee [OpenAI Completions API documentation](https://platform.openai.com/docs/api-reference/completions).\n\nllama.cpp `/completion`-specific features such as `mirostat` are supported.\n\n*Examples:*\n\nExample usage with `openai` python library:\n\n```python\nimport openai\n\nclient = openai.OpenAI(\n    base_url=\"http://localhost:8080/v1\", # \"http://<Your api-server IP>:port\"\n    api_key = \"sk-no-key-required\"\n)\n\ncompletion = client.completions.create(\n  model=\"davinci-002\",\n  prompt=\"I believe the meaning of life is\",\n  max_tokens=8\n)\n\nprint(completion.choices[0].text)\n```\n\n### POST `/v1/chat/completions`: OpenAI-compatible Chat Completions API\n\nGiven a ChatML-formatted json description in `messages`, it returns the predicted completion. Both synchronous and streaming mode are supported, so scripted and interactive applications work fine. While no strong claims of compatibility with OpenAI API spec is being made, in our experience it suffices to support many apps. Only models with a [supported chat template](https://github.com/ggml-org/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template) can be used optimally with this endpoint. By default, the ChatML template will be used.\n\n*Options:*\n\nSee [OpenAI Chat Completions API documentation](https://platform.openai.com/docs/api-reference/chat). llama.cpp `/completion`-specific features such as `mirostat` are also supported.\n\nThe `response_format` parameter supports both plain JSON output (e.g. `{\"type\": \"json_object\"}`) and schema-constrained JSON (e.g. `{\"type\": \"json_object\", \"schema\": {\"type\": \"string\", \"minLength\": 10, \"maxLength\": 100}}` or `{\"type\": \"json_schema\", \"schema\": {\"properties\": { \"name\": { \"title\": \"Name\", \"type\": \"string\" }, \"date\": { \"title\": \"Date\", \"type\": \"string\" }, \"participants\": { \"items\": {\"type: \"string\" }, \"title\": \"Participants\", \"type\": \"string\" } } } }`), similar to other OpenAI-inspired API providers.\n\n`chat_template_kwargs`: Allows sending additional parameters to the json templating system. For example: `{\"enable_thinking\": false}`\n\n`reasoning_effort`: If `none`, reasoning/thinking is disabled. Otherwise, the value is made available to the jinja template.\n\n`reasoning_format`: The reasoning format to be parsed. If set to `none`, it will output the raw generated text.\n\n`reasoning_control`: Arms realtime reasoning control for this completion so it can be ended early via `/v1/chat/completions/control`. Defaults to `false`.\n\n`generation_prompt`: The generation prompt that was prefilled in by the template. Prepended to model output before parsing.\n\n`parse_tool_calls`: Whether to parse the generated tool call.\n\n`parallel_tool_calls` : Whether to enable parallel/multiple tool calls (only supported on some models, verification is based on jinja template).\n\nFor multimodal input (typed content, `messages[i].content[j]`):\n- If `type == \"image_url\"`:\n - `image_url.url` can be a remote URL, base64 (raw or URI-encoded via `data:image/...;base64`) or path to local file\n - Accepts formats supported by `stb_image` (jpeg, png, tga, bmp, gif, ...)\n- If `type == \"input_audio\"`:\n - Either `input_audio.data` or `input_audio.url` can be specified, can be a remote URL, raw base64 or path to local file\n - Accepts formats supported by `miniaudio` (mp3, wav, flac)\n - `input_audio.format` will be ignored, the file format will be determined automatically\n- If `type == \"input_video\"`:\n - Either `input_video.data` or `input_video.url` can be specified, can be a remote URL, raw base64 or path to local file\n - Accepts formats supported by `ffmpeg`\n- Note: for local file, make sure to set `--media-path`. File path must be prefixed by `file://`\n\n*Examples:*\n\nYou can use either Python `openai` library with appropriate checkpoints:\n\n```python\nimport openai\n\nclient = openai.OpenAI(\n    base_url=\"http://localhost:8080/v1\", # \"http://<Your api-server IP>:port\"\n    api_key = \"sk-no-key-required\"\n)\n\ncompletion = client.chat.completions.create(\n  model=\"gpt-3.5-turbo\",\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"},\n    {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n  ]\n)\n\nprint(completion.choices[0].message)\n```\n\n... or raw HTTP requests:\n\n```shell\ncurl http://localhost:8080/v1/chat/completions \\\n-H \"Content-Type: application/json\" \\\n-H \"Authorization: Bearer no-key\" \\\n-d '{\n\"model\": \"gpt-3.5-turbo\",\n\"messages\": [\n{\n    \"role\": \"system\",\n    \"content\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"\n},\n{\n    \"role\": \"user\",\n    \"content\": \"Write a limerick about python exceptions\"\n}\n]\n}'\n```\n\n*Tool call support*\n\n[OpenAI-style function calling](https://platform.openai.com/docs/guides/function-calling) is supported with the `--jinja` flag (and may require a `--chat-template-file` override to get the right tool-use compatible Jinja template; worst case, `--chat-template chatml` may also work).\n\n**See our [Function calling](../../docs/function-calling.md) docs** for more details, supported native tool call styles (generic tool call style is used as fallback) / examples of use.\n\n*Timings and context usage*\n\nThe response contains a `timings` object, for example:\n\n```js\n{\n  \"choices\": [],\n  \"created\": 1757141666,\n  \"id\": \"chatcmpl-ecQULm0WqPrftUqjPZO1CFYeDjGZNbDu\",\n  // ...\n  \"timings\": {\n    \"cache_n\": 236, // number of prompt tokens reused from cache\n    \"prompt_n\": 1, // number of prompt tokens being processed\n    \"prompt_ms\": 30.958,\n    \"prompt_per_token_ms\": 30.958,\n    \"prompt_per_second\": 32.301828283480845,\n    \"predicted_n\": 35, // number of predicted tokens\n    \"predicted_ms\": 661.064,\n    \"predicted_per_token_ms\": 18.887542857142858,\n    \"predicted_per_second\": 52.94494935437416\n  }\n}\n```\n\nThis provides information on the performance of the server. It also allows calculating the current context usage.\n\nThe total number of tokens in context is equal to `prompt_n + cache_n + predicted_n`\n\nThe response also includes a standard `usage` object:\n\n```js\n{\n    // ...\n    \"usage\": {\n        \"completion_tokens\": 48,\n        \"prompt_tokens\": 44,\n        \"total_tokens\": 92,\n        \"prompt_tokens_details\": {\n            \"cached_tokens\": 0\n        }\n    }\n}\n```\n\n*Reasoning support*\n\nThe server supports parsing and returning reasoning via the `reasoning_content` field, similar to Deepseek API.\n\nReasoning input (preserve reasoning in history) is also supported by some specific templates. For more details, please refer to [PR#18994](https://github.com/ggml-org/llama.cpp/pull/18994).\n\n### POST `/v1/chat/completions/control`: Control a running chat completion in real time\n\nActs on an in-flight completion identified by its `id` (the `id` field streamed back by `/v1/chat/completions`). The request is processed in parallel with the SSE stream, so the client sends it while still reading tokens.\n\n*Options:*\n\n`id`: (Required) The chat completion id to act on. A completion that has already finished matches nothing and the call is a no-op.\n\n`action`: (Required) The control action to perform. Currently the only supported value is `reasoning_end`, which forces the end of the current reasoning block so the model moves on to the final answer. Requires `reasoning_control: true` on the original completion request.\n\n`model`: (Required in router mode) The model name, used to route the request to the right instance. Ignored in single model mode.\n\n**Response format**\n\nReturns a JSON object with a boolean `success` field, and an optional `message` field describing the reason when `success` is `false`.\n\n### POST `/v1/responses`: OpenAI-compatible Responses API\n\n*Options:*\n\nSee [OpenAI Responses API documentation](https://platform.openai.com/docs/api-reference/responses).\n\n*Examples:*\n\nYou can use either Python `openai` library with appropriate checkpoints:\n\n```python\nimport openai\n\nclient = openai.OpenAI(\n    base_url=\"http://localhost:8080/v1\", # \"http://<Your api-server IP>:port\"\n    api_key = \"sk-no-key-required\"\n)\n\nresponse = client.responses.create(\n  model=\"gpt-4.1\",\n  instructions=\"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\",\n  input=\"Write a limerick about python exceptions\"\n)\n\nprint(response.output_text)\n```\n\n... or raw HTTP requests:\n\n```shell\ncurl http://localhost:8080/v1/responses \\\n-H \"Content-Type: application/json\" \\\n-H \"Authorization: Bearer no-key\" \\\n-d '{\n\"model\": \"gpt-4.1\",\n\"instructions\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\",\n\"input\": \"Write a limerick about python exceptions\"\n}'\n```\n\nThis endpoint works by converting Responses request into Chat Completions request.\n\n### POST `/v1/embeddings`: OpenAI-compatible embeddings API\n\nThis endpoint requires that the model uses a pooling different than type `none`. The embeddings are normalized using the Eucledian norm.\n\n*Options:*\n\nSee [OpenAI Embeddings API documentation](https://platform.openai.com/docs/api-reference/embeddings).\n\n*Examples:*\n\n- input as string\n\n  ```shell\n  curl http://localhost:8080/v1/embeddings \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer no-key\" \\\n  -d '{\n          \"input\": \"hello\",\n          \"model\":\"GPT-4\",\n          \"encoding_format\": \"float\"\n  }'\n  ```\n\n- `input` as string array\n\n  ```shell\n  curl http://localhost:8080/v1/embeddings \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer no-key\" \\\n  -d '{\n          \"input\": [\"hello\", \"world\"],\n          \"model\":\"GPT-4\",\n          \"encoding_format\": \"float\"\n  }'\n  ```\n\n### POST `/v1/responses/input_tokens`: Token Counting\n\nSimilar to [Response input token counts API](https://developers.openai.com/api/reference/python/resources/responses/subresources/input_tokens/methods/count).\n\nExample response:\n\n```json\n{\n  \"object\": \"response.input_tokens\",\n  \"input_tokens\": 11\n}\n```\n\n### POST `/v1/chat/completions/input_tokens`: Token Counting\n\nSimilar to [Response input token counts API](https://developers.openai.com/api/reference/python/resources/responses/subresources/input_tokens/methods/count), but accepts a chat completion body as input.\n\nNote: This is not an official OAI endpoint, but is added for completeness and convenience.\n\nExample response:\n\n```json\n{\n  \"object\": \"response.input_tokens\",\n  \"input_tokens\": 11\n}\n```\n\n## Anthropic-compatible API Endpoints\n\n### POST `/v1/messages`: Anthropic-compatible Messages API\n\nGiven a list of `messages`, returns the assistant's response. Streaming is supported via Server-Sent Events. While no strong claims of compatibility with the Anthropic API spec are made, in our experience it suffices to support many apps.\n\n*Options:*\n\nSee [Anthropic Messages API documentation](https://docs.anthropic.com/en/api/messages). Tool use requires `--jinja` flag.\n\n`model`: Model identifier (required)\n\n`messages`: Array of message objects with `role` and `content` (required)\n\n`max_tokens`: Maximum tokens to generate (default: 4096)\n\n`system`: System prompt as string or array of content blocks\n\n`temperature`: Sampling temperature 0-1 (default: 1.0)\n\n`top_p`: Nucleus sampling (default: 1.0)\n\n`top_k`: Top-k sampling\n\n`stop_sequences`: Array of stop sequences\n\n`stream`: Enable streaming (default: false)\n\n`tools`: Array of tool definitions (requires `--jinja`)\n\n`tool_choice`: Tool selection mode (`{\"type\": \"auto\"}`, `{\"type\": \"any\"}`, or `{\"type\": \"tool\", \"name\": \"...\"}`)\n\n*Examples:*\n\n```shell\ncurl http://localhost:8080/v1/messages \\\n  -H \"Content-Type: application/json\" \\\n  -H \"x-api-key: your-api-key\" \\\n  -d '{\n    \"model\": \"gpt-4\",\n    \"max_tokens\": 1024,\n    \"system\": \"You are a helpful assistant.\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"Hello!\"}\n    ]\n  }'\n```\n\n### POST `/v1/messages/count_tokens`: Token Counting\n\nCounts the number of tokens in a request without generating a response.\n\nAccepts the same parameters as `/v1/messages`. The `max_tokens` parameter is not required.\n\n*Example:*\n\n```shell\ncurl http://localhost:8080/v1/messages/count_tokens \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"gpt-4\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"Hello!\"}\n    ]\n  }'\n```\n\n*Response:*\n\n```json\n{\"input_tokens\": 10}\n```\n\n## Server tools\n\nThe server exposes a REST API under `/tools` that allows the Web UI to call server tools. This endpoint is intended to be used internally by the Web UI and subject to change or to be removed in the future.\n\n**Please do NOT use this endpoint in a downstream application**\n\nFor further documentation about this endpoint, please refer to [server internal documentation](./README-dev.md)\n\n## Using multiple models\n\n`llama-server` can be launched in a **router mode** that exposes an API for dynamically loading and unloading models. The main process (the \"router\") automatically forwards each request to the appropriate model instance.\n\nTo start in router mode, launch `llama-server` **without specifying any model**:\n\n```sh\nllama-server\n```\n\n### Model sources\n\nThere are 3 possible sources for model files:\n1. Cached models (controlled by the `LLAMA_CACHE` environment variable)\n2. Custom model directory (set via the `--models-dir` argument)\n3. Custom preset (set via the `--models-preset` argument)\n\nBy default, the router looks for models in the cache. You can add Hugging Face models to the cache with:\n\n```sh\nllama-server -hf <user>/<model>:<tag>\n```\n\n*The server must be restarted after adding a new model.*\n\nAlternatively, you can point the router to a local directory containing your GGUF files using `--models-dir`. Example command:\n\n```sh\nllama-server --models-dir ./models_directory\n```\n\nIf the model contains multiple GGUF (for multimodal or multi-shard), files should be put into a subdirectory. The directory structure should look like this:\n\n```sh\nmodels_directory\n │\n │  # single file\n ├─ llama-3.2-1b-Q4_K_M.gguf\n ├─ Qwen3-8B-Q4_K_M.gguf\n │\n │  # multimodal\n ├─ gemma-3-4b-it-Q8_0\n │    ├─ gemma-3-4b-it-Q8_0.gguf\n │    └─ mmproj-F16.gguf   # file name must start with \"mmproj\"\n │\n │  # multi-shard\n ├─ Kimi-K2-Thinking-UD-IQ1_S\n │    ├─ Kimi-K2-Thinking-UD-IQ1_S-00001-of-00006.gguf\n │    ├─ Kimi-K2-Thinking-UD-IQ1_S-00002-of-00006.gguf\n │    ├─ ...\n │    └─ Kimi-K2-Thinking-UD-IQ1_S-00006-of-00006.gguf\n```\n\nYou may also specify default arguments that will be passed to every model instance:\n\n```sh\nllama-server -ctx 8192 -n 1024 -np 2\n```\n\nNote: model instances inherit both command line arguments and environment variables from the router server.\n\nAlternatively, you can also add GGUF based preset (see next section)\n\n### Model presets\n\nModel presets allow advanced users to define custom configurations using an `.ini` file:\n\n```sh\nllama-server --models-preset ./my-models.ini\n```\n\nEach section in the file defines a new preset. Keys within a section correspond to command-line arguments (without leading dashes). For example, the argument `--n-gpu-layers 123` is written as `n-gpu-layers = 123`.\n\nShort argument forms (e.g., `c`, `ngl`) and environment variable names (e.g., `LLAMA_ARG_N_GPU_LAYERS`) are also supported as keys.\n\nExample:\n\n```ini\nversion = 1\n\n; (Optional) This section provides global settings shared across all presets.\n; If the same key is defined in a specific preset, it will override the value in this global section.\n[*]\nc = 8192\nn-gpu-layers = 8\n\n; If the key corresponds to an existing model on the server,\n; this will be used as the default config for that model\n[ggml-org/MY-MODEL-GGUF:Q8_0]\n; string value\nchat-template = chatml\n; numeric value\nn-gpu-layers = 123\n; flag value (for certain flags, you need to use the \"no-\" prefix for negation)\njinja = true\n; shorthand argument (for example, context size)\nc = 4096\n; environment variable name\nLLAMA_ARG_CACHE_RAM = 0\n; file paths are relative to server's CWD\nmodel-draft = ./my-models/draft.gguf\n; but it's RECOMMENDED to use absolute path\nmodel-draft = /Users/abc/my-models/draft.gguf\n\n; If the key does NOT correspond to an existing model,\n; you need to specify at least the model path or HF repo\n[custom_model]\nmodel = /Users/abc/my-awesome-model-Q4_K_M.gguf\n```\n\nNote: some arguments are controlled by router (e.g., host, port, API key, HF repo, model alias). They will be removed or overwritten upon loading.\n\nThe precedence rule for preset options is as follows:\n1. **Command-line arguments** passed to `llama-server` (highest priority)\n2. **Model-specific options** defined in the preset file (e.g. `[ggml-org/MY-MODEL...]`)\n3. **Global options** defined in the preset file (`[*]`)\n\nWe also offer additional options that are exclusive to presets (these aren't treated as command-line arguments):\n- `load-on-startup` (boolean): Controls whether the model loads automatically when the server starts. Only applies at startup: if the model list is reloaded later (for example after editing the preset file), a newly added model is listed but not loaded\n- `stop-timeout` (int, seconds): After requested unload, wait for this many seconds before forcing termination (default: 10)\n- `dedup-cache-models` (boolean): When the preset uses `hf-repo` pointing to a model that is already downloaded, hide the corresponding cached model entry from `GET /models` (the preset entry remains visible). Set it in the `[*]` section to apply to all presets.\n\n### Routing requests\n\nRequests are routed according to the requested model name.\n\nFor **POST** endpoints (`/v1/chat/completions`, `/v1/completions`, `/infill`, etc.) The router uses the `\"model\"` field in the JSON body:\n\n```json\n{\n  \"model\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"hello\"\n    }\n  ]\n}\n```\n\nFor **GET** endpoints (`/props`, `/metrics`, etc.) The router uses the `model` query parameter (URL-encoded):\n\n```\nGET /props?model=ggml-org%2Fgemma-3-4b-it-GGUF%3AQ4_K_M\n```\n\nBy default, the model will be loaded automatically if it's not loaded. To disable this, add `--no-models-autoload` when starting the server. Additionally, you can include `?autoload=true|false` in the query param to control this behavior per-request.\n\n### GET `/models`: List available models\n\nListing all models in cache. The model metadata will also include a field to indicate the status of the model:\n\n```json\n{\n  \"data\": [{\n    \"id\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\",\n    \"path\": \"/Users/REDACTED/Library/Caches/llama.cpp/ggml-org_gemma-3-4b-it-GGUF_gemma-3-4b-it-Q4_K_M.gguf\",\n    \"status\": {\n      \"value\": \"loaded\",\n      \"args\": [\"llama-server\", \"-ctx\", \"4096\"]\n    },\n    \"architecture\": {\n      \"input_modalities\": [\n        \"text\",\n        \"image\"\n      ],\n      \"output_modalities\": [\n        \"text\"\n      ]\n    },\n    ...\n  }]\n}\n```\n\nNote:\n1. Adding `?reload=1` to the query params will refresh the list of models. The behavior is as follow:\n - If a model is running but updated or removed from the source, it will be unloaded\n - If a model is not running, it will be added or updated according to the source\n2. When the model is loaded, the info from `/v1/models` is forwarded to router's `/v1/models`. This includes metadata about the model and the runtime instance.\n\nThe `status` object can be:\n\n```json\n\"status\": {\n  \"value\": \"unloaded\"\n}\n```\n\n```json\n\"status\": {\n  \"value\": \"loading\",\n  \"args\": [\"llama-server\", \"-ctx\", \"4096\"]\n}\n```\n\n```json\n\"status\": {\n  \"value\": \"unloaded\",\n  \"args\": [\"llama-server\", \"-ctx\", \"4096\"],\n  \"failed\": true,\n  \"exit_code\": 1\n}\n```\n\n```json\n\"status\": {\n  \"value\": \"loaded\",\n  \"args\": [\"llama-server\", \"-ctx\", \"4096\"]\n}\n```\n\n```json\n\"status\": {\n  \"value\": \"sleeping\",\n  \"args\": [\"llama-server\", \"-ctx\", \"4096\"]\n}\n```\n\nNote: for \"downloading\" state, there can be multiple files be downloading in parallel\n\n```json\n\"status\": {\n  \"value\": \"downloading\",\n  \"progress\": {\n    \"https://...model.gguf\": {\n      \"done\": 195963406,\n      \"total\": 219307424\n    }\n  }\n}\n```\n\n### POST `/models/load`: Load a model\n\nLoad a model\n\nPayload:\n- `model`: name of the model to be loaded.\n\n```json\n{\n  \"model\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\"\n}\n```\n\nResponse:\n\n```json\n{\n  \"success\": true\n}\n```\n\n### POST `/models/unload`: Unload a model\n\nUnload a model\n\nPayload:\n\n```json\n{\n  \"model\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\",\n}\n```\n\nResponse:\n\n```json\n{\n  \"success\": true\n}\n```\n\n### GET `/models/sse`: Real-time events\n\nExample events:\n\n```js\n{\n  \"model\": \"...\",\n  \"event\": \"model_status\",\n  \"data\": {\n    \"status\": \"loading\"\n  }\n}\n\n{\n  \"model\": \"...\",\n  \"event\": \"download_progress\",\n  \"data\": {\n    // note: there can be multiple files being downloaded in parallel\n    \"https://...model.gguf\": {\n      \"done\": 195963406,\n      \"total\": 219307424\n    }\n  }\n}\n\n{\n  \"model\": \"...\",\n  \"event\": \"model_status\",\n  \"data\": {\n    \"status\": \"loading\",\n    \"progress\": {\n      \"stages\": [\"text_model\", \"spec_model\", \"mmproj_model\"],\n      \"current\": \"text_model\",\n      \"value\": 0.5\n    }\n  }\n}\n// note for \"loading\" status:\n// - subsequent events will follow the same order of \"stages\" list\n// - mmap may report incorrect progress on some platforms; if you need exact progress, use --load-mode none\n\n{\n  \"model\": \"...\",\n  \"event\": \"model_status\",\n  \"data\": {\n    \"status\": \"loaded\",\n    \"info\": {\n      // note: only include info on first load\n      // waking up from sleep doesn't have this\n    }\n  }\n}\n\n{\n  \"model\": \"...\",\n  \"event\": \"model_status\",\n  \"data\": {\n    \"status\": \"sleeping\"\n  }\n}\n\n{\n  \"model\": \"...\",\n  \"event\": \"model_remove\"\n}\n\n// special event: reload of the list of all models\n{\n  \"model\": \"*\",\n  \"event\": \"models_reload\"\n}\n```\n\n### POST `/models`: Download new model\n\nTrigger a new download (non-blocking), the progress can be tracked via SSE endpoint `/models/sse`\n\nTo cancel model downloading, send an event to `/models/unload`\n\nDownload procedure:\n- Send POST request to `/models`\n- Subscribe to `/models/sse` for updates\n- On downloading completed, you will receive either `download_finished` or `download_failed` event\n- Call GET `/models` to trigger model list update. If the download success, you should see the new model in the list\n\nPayload:\n\n```json\n{\n  \"model\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\",\n}\n```\n\nResponse (download is started in the background):\n\n```json\n{\n  \"success\": true\n}\n```\n\nResponse (error, cannot start the download):\n\n```json\n{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"model validation failed, unable to download\",\n    \"type\": \"invalid_request_error\"\n  }\n}\n```\n\n### DELETE `/models`: Delete a model from cache\n\nIMPORTANT: only model stored in cache can be deleted. You cannot delete models in a preset.\n\nModel name must be passed via query param: `?model={name}`\n\nIf delete success, it will send an SSE event of type `model_remove`\n\nResponse:\n\n```json\n{\n  \"success\": true\n}\n```\n\n## API errors\n\n`llama-server` returns errors in the same format as OAI: https://github.com/openai/openai-openapi\n\nExample of an error:\n\n```json\n{\n    \"error\": {\n        \"code\": 401,\n        \"message\": \"Invalid API Key\",\n        \"type\": \"authentication_error\"\n    }\n}\n```\n\n## Sleeping on Idle\n\nThe server supports an automatic sleep mode that activates after a specified period of inactivity (no incoming tasks). This feature, introduced in [PR #18228](https://github.com/ggml-org/llama.cpp/pull/18228), can be enabled using the `--sleep-idle-seconds` command-line argument. It works seamlessly in both single-model and multi-model configurations.\n\nWhen the server enters sleep mode, the model and its associated memory (including the KV cache) are unloaded from RAM to conserve resources. Any new incoming task will automatically trigger the model to reload.\n\nThe sleeping status can be retrieved from the `GET /props` endpoint (or `/props?model=(model_name)` in router mode).\n\nNote that the following endpoints are exempt from being considered as incoming tasks. They do not trigger model reloading and do not reset the idle timer:\n- `GET /health`\n- `GET /props`\n- `GET /models`\n- `GET /metrics`\n\n## More examples\n\n### Interactive mode\n\nCheck the sample in [chat.mjs](chat.mjs).\nRun with NodeJS version 16 or later:\n\n```sh\nnode chat.mjs\n```\n\nAnother sample in [chat.sh](chat.sh).\nRequires [bash](https://www.gnu.org/software/bash/), [curl](https://curl.se) and [jq](https://jqlang.github.io/jq/).\nRun with bash:\n\n```sh\nbash chat.sh\n```\n\nApart from error types supported by OAI, we also have custom types that are specific to functionalities of llama.cpp:\n\n**When /metrics or /slots endpoint is disabled**\n\n```json\n{\n    \"error\": {\n        \"code\": 501,\n        \"message\": \"This server does not support metrics endpoint.\",\n        \"type\": \"not_supported_error\"\n    }\n}\n```\n\n**When the server receives invalid grammar via */completions endpoint**\n\n```json\n{\n    \"error\": {\n        \"code\": 400,\n        \"message\": \"Failed to parse grammar\",\n        \"type\": \"invalid_request_error\"\n    }\n}\n```\n\n### Custom default Web UI preferences\n\nYou can specify default preferences for the web UI using `--ui-config ` or `--ui-config-file `. For example, you can disable pasting long text as attachments and enable rendering Markdown in user messages with this command:\n\n```bash\n./llama-server -m model.gguf --ui-config '{\"pasteLongTextToFileLen\": 0, \"renderUserContentAsMarkdown\": true}'\n```\n\n> **Note:** The old flags `--webui-config` and `--webui-config-file` are deprecated but still work as aliases.\n\nYou may find available preferences in [settings-keys.ts](../ui/src/lib/constants/settings-keys.ts).","convId":"incyjf36ghq","role":"tool","timestamp":1787790593585,"toolCallId":"xMB1YdzO01VwI2GY4XRbSxzrE957o8kh","type":"text","id":"4aqhi7cht7v","parent":"ay6axiw8owv"},"type":"message"}
{"message":{"children":["1lt953q214g"],"content":"No repositories found for the given criteria.","convId":"incyjf36ghq","role":"tool","timestamp":1787790593731,"toolCallId":"jdT0sYKTzkr205KUPliEZTCdbVE8n4b4","type":"text","id":"jkspost8exp","parent":"ay6axiw8owv"},"type":"message"}
{"message":{"children":["tdyyfaqb86e","e1sosznazua"],"content":"Last verification round — exact PLE tensor name, MTP/vision availability, and what other 3090/4090 users converged on.\n\n","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790593738,"type":"text","id":"1lt953q214g","parent":"jkspost8exp","reasoningContent":"Now I have essentially everything. Let me also quickly confirm the exact tensor name for the PLE table so the `--override-tensor` is right, and check `-ot` regex support. phhusson used `-ot 'blk\\.(([0123][0123456789]|40|41|42|43|44))\\.ffn_.*=CPU'` — so `-ot` supports regex. And the PR explicitly names `per_layer_token_embd`. The user's `ple_ngram_embd` is likely bogus.\n\nLet me do one more verification attempt on the actual GGUF tensor names. Options: the AtomicChat metrics dataset (probably lists quant metrics), or the base model's `model.safetensors.index.json` which would give HF tensor names — the GGUF names are derived from conversion maps. Let me check the base model index for the ngram tensor name, and also check if there's an mmproj anywhere (e.g., a separate repo like `ultraiers` or `mradermacher`). Also worth checking AtomicChat's GGUF (text-generation, might list tensors) and whether the PR README mentions vision mmproj.\n\nActually let me check the safetensors index for Qwen/Qwen3.8-Flash-Next to get HF names, which will confirm \"ngram\"/\"ple\" naming, plus confirm whether MTP weights are present.\n\nAlso: I should double check the RAM question. User has 8 DIMMs at 2133 on WRX80. WRX80 = 4-channel DDR4-3200 supported, 8 DIMMs = still 4 channels (2DPC). At 2133 MT/s: 2133 × 8 bytes × 4 ch = 68.3 GB/s peak. That's the constraint for PLE-on-CPU. Capacity unknown; need at least ~26GB for the PLE + page cache headroom. I'll flag that I need their RAM size and give decision branches.\n\nLet me get the safetensors index (may be large but it's a JSON listing). Let me use hf_fs cat with a byte limit and grep-ish. Actually the index for a 180B model could be huge (thousands of entries). Let me instead check the AtomicChat GGUF README which might list tensor info, or just accept PR evidence.\n\nBetter: use hf_fs search? No. Let me try cat of the config-adjacent files: `generation_config.json`, and check for mmproj via a broader hub search for \"qwen4exp mmproj\".\n\nLet me do:\n1. cat Qwen/Qwen3.8-Flash-Next model.safetensors.index.json limited (offset to find ngram keys?) — hmm, cat with offset. The keys would appear early-ish? JSON keys sorted by... typically in the index it's \"weight_map\" object with keys sorted alphabetically. \"model.language_model.layers...\" and ngram would be something like \"model.language_model.layers.0.ngram...\" or \"ngram_embeddings\". Alphabetically it'd be found somewhere in the middle. Not great.\n2. Search HF for mmproj repos differently: search \"qwen4exp mmproj\".\n\nLet me just do a hub search for the AtomicChat repo details + a targeted web search on tensor names / mmproj availability, and get the AtomicChat GGUF README (they may document `-ot per_layer_token_embd=CPU` usage for their 4090/3090 users — very relevant since AtomicChat targets consumer GPUs).\n\nI'll do a final verification batch, then write the answer. I've used a fair number of calls; let me be efficient and get it right.\n","timings":{"cache_n":73283,"prompt_n":30391,"prompt_ms":200638.664,"prompt_per_token_ms":6.601910565627982,"prompt_per_second":151.4713036566073,"predicted_n":1009,"predicted_ms":54503.748,"predicted_per_token_ms":54.07117857142857,"predicted_per_second":18.49414099008384},"toolCalls":[{"function":{"arguments":"{\"operations\":[{\"cmd\": \"cat\", \"args\": [\"hf://models/AtomicChat/Qwen3.8-Flash-Next-GGUF/README.md\", \"--max-bytes\", \"18000\"]}, {\"cmd\": \"ls\", \"args\": [\"hf://models/AtomicChat/Qwen3.8-Flash-Next-GGUF\"]}]}","name":"hf_fs"},"id":"GPuVO0vrpgRspmSMXlPSXmoNJS8rDn5O","type":"function"},{"function":{"arguments":"{\"query\":\"Qwen3.8-Flash-Next llama.cpp per_layer_token_embd override-tensor CPU 3090 ngram offload recommended flags\",\"numResults\":8}","name":"web_search_exa"},"id":"Mlk0a8I1drztkdTsXAfU3LzOMlXVU0Zh","type":"function"}]},"type":"message"}
{"message":{"children":[],"content":"## Operation 1\n\n# hf_fs cat\n\nURI: `hf://models/AtomicChat/Qwen3.8-Flash-Next-GGUF/README.md`\nPath: `README.md`\nBytes: 7511\n\n---\nlicense: other\nlicense_name: qwen-community-1.0\nlicense_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE\nbase_model:\n- Qwen/Qwen3.8-Flash-Next\nbase_model_relation: quantized\nquantized_by: AtomicChat\npipeline_tag: text-generation\nlibrary_name: gguf\ntags:\n- atomic-chat\n- qwen\n- qwen3.8\n- flash-next\n- moe\n- multimodal\n- gguf\n- imatrix\n- quantized\n- llama.cpp\n---\n\n# How to Run Qwen3.8-Flash-Next Locally\n<p style=\"margin-top: 0; margin-bottom: 0;\">\n  <em>Built from Qwen's original weights with our own importance matrix. The <a href=\"https://huggingface.co/datasets/AtomicChat/calib-corpora\">calibration corpora</a> behind our builds are public.</em>\n</p>\n<div style=\"display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;\">\n  <a href=\"https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_qwen3_8_flash_next&utm_content=btn_atomic\"><img src=\"https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/resolve/main/btn_atomic.png\" width=\"162\" alt=\"Atomic Chat\"></a>\n  <a href=\"https://discord.gg/8wGSsvmg4V\"><img src=\"https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/resolve/main/btn_discord.png\" width=\"119\" alt=\"Discord\"></a>\n  <a href=\"https://github.com/AtomicBot-ai/Atomic-Chat\"><img src=\"https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/resolve/main/btn_github.png\" width=\"115\" alt=\"GitHub\"></a>\n</div>\n<ul style=\"margin: 0 0 12px 0;\">\n  <li>Qwen3.8-Flash-Next is the first open-weight release of the architecture behind Qwen4.</li>\n  <li>These GGUFs are self-quantized from Qwen's original weights with our own importance matrix, published alongside the quants.</li>\n  <li>The quants are still uploading and need a llama.cpp build with Qwen3.8-Flash-Next support; Atomic Chat runs it as support ships.</li>\n</ul>\n<hr style=\"margin: 0 0 16px 0;\">\n\n<img src=\"https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/resolve/main/hero.png\" alt=\"Qwen3.8-Flash-Next architecture\" style=\"width:560px; max-width:100%; height:auto;\"/>\n\n*Qwen3.8-Flash-Next architecture (Qwen).*\n\n## Highlights\n\n- **125B total with 6B active** sparse MoE (512 experts, 10 routed + 1 shared), plus a 51B n-gram embedding and a 4B MTP layer. An experimental preview of the architecture behind Qwen4.\n- **Hybrid attention with QSA**: Gated DeltaNet paired with Qwen Sparse Attention, which operates at the micro-block level rather than per token to cut long-context latency for agentic workloads.\n- **Gated Residual**: a data-dependent read gate plus a per-branch scalar write gate over widened residual streams, for finer expressiveness at low inference overhead.\n- **N-gram Embedding**: 20M bigram/trigram embeddings indexed at layer 2, a compute-light axis for parameter scaling that offloads well on memory-constrained accelerators.\n- **262,144-token context**, extensible up to 1,000,000 tokens with RoPE scaling.\n- **Natively multimodal** (causal language model with a vision encoder, image-text-to-text). These GGUF quants cover the text path.\n- **Frontier coding and agentic scores** (Qwen-reported): LiveCodeBench v6 91.9, GPQA Diamond 91.7, SWE-bench Multilingual 81.0, CoWorkBench 73.9.\n- **Full imatrix quantization** with our public [calibration corpora](https://huggingface.co/datasets/AtomicChat/calib-corpora).\n\n> [!NOTE]\n> These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.\n\n> [!IMPORTANT]\n> Always pass `--jinja` so the **Qwen3.8-Flash-Next chat template** is applied. Without it the model can emit malformed turns.\n\n## Model Overview\n\n| Property | Value |\n|---|---|\n| Base model | `Qwen/Qwen3.8-Flash-Next` |\n| Type | Causal language model with a vision encoder (image-text-to-text) |\n| Total / active parameters | 125B total / 6B active, plus 51B n-gram embedding and a 4B MTP layer |\n| Layers | 48. Hidden layout: 12 x (3 x (Gated DeltaNet then MoE) then 1 x (Qwen Sparse Attention then MoE)) |\n| Experts | 512 experts, 10 routed + 1 shared activated |\n| Attention | Hybrid: Gated DeltaNet (linear) and Qwen Sparse Attention (micro-block sparse); Gated Residual over widened residual streams |\n| Context length | 262,144 native, extensible up to 1,000,000 |\n| This repo | GGUF quants (imatrix), text path. The importance matrix we built is published here too. |\n\n<img src=\"https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/resolve/main/benchmark.png\" alt=\"Qwen3.8-Flash-Next benchmark scores\" style=\"width:100%; max-width:900px;\"/>\n\nScores are Qwen's published results for the base `Qwen/Qwen3.8-Flash-Next`. Quantization preserves the large majority of this; `Q4_K_M` and up sit within a point or two of full precision.\n\n\n## Choosing a quant\n\n| Quant | Size | Notes |\n|---|---|---|\n| `IQ2_M` | — | Smallest usable. Aggressive low-bit for memory-constrained boxes. |\n| `IQ3_M` | — | Beats Q3 at similar size thanks to imatrix. Best low-RAM pick. |\n| **`Q4_K_M`** | — | **Recommended default. Best balance of size, speed and quality.** |\n| **`UD-Q4_K_XL`** | — | **Dynamic. Embeddings and output kept at Q8_0 for higher quality at a Q4 footprint.** |\n| `Q6_K` | — | Near lossless. |\n| `Q8_0` | — | Effectively lossless, reference quality. |\n\n> [!TIP]\n> Sizes fill in once the quants finish uploading. Pick the largest file that fits your (V)RAM with room for context.\n\n## Get started\n\n> [!NOTE]\n> Qwen3.8-Flash-Next is a brand-new Qwen4-preview architecture (Gated DeltaNet, Qwen Sparse Attention, n-gram embedding). The quants in this repo are still uploading, and running them needs a `llama.cpp` build that has landed Qwen3.8-Flash-Next support. Until then, [Atomic Chat](https://atomic.chat) is the easiest way to run it as support ships.\n\nRun Qwen3.8-Flash-Next locally with:\n\n- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/Qwen3.8-Flash-Next-GGUF`, pick a quant, hit **Use this model**.\n- **llama.cpp:** `llama-server -hf AtomicChat/Qwen3.8-Flash-Next-GGUF:Q4_K_M --jinja -c 8192`\n- **Ollama:** `ollama run hf.co/AtomicChat/Qwen3.8-Flash-Next-GGUF:Q4_K_M`\n- **LM Studio / Jan:** search the repo id, download any quant.\n\n## Best practices\n\n| Parameter | Value |\n|---|---|\n| temperature | 1.0 |\n| top_p | 0.95 |\n| top_k | 20 |\n| min_p | 0.0 |\n\nQwen's recommended thinking-mode settings. For non-thinking (instruct) use `temperature=0.7`, `top_p=0.80`, `top_k=20`, `presence_penalty=1.5`. Allocate generous output length for agentic tasks.\n\n## Run in llama.cpp\n\n```bash\ngit clone https://github.com/ggerganov/llama.cpp\ncmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON\ncmake --build llama.cpp/build --config Release -j --target llama-cli llama-server\n```\n\n```bash\n./llama.cpp/build/bin/llama-server \\\n    -hf AtomicChat/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \\\n    --jinja -ngl 99 -c 8192 -fa on\n```\n\n## How these were made\n\n1. Download `Qwen/Qwen3.8-Flash-Next` (original weights).\n2. Convert to GGUF with a [llama.cpp](https://github.com/ggerganov/llama.cpp) build that supports the Qwen3.8-Flash-Next architecture (Gated DeltaNet, Qwen Sparse Attention, n-gram embedding).\n3. Build an importance matrix over our public [calibration corpora](https://huggingface.co/datasets/AtomicChat/calib-corpora).\n4. Quantize the ladder with `--imatrix`; `UD-Q4_K_XL` additionally pins the token-embedding and output tensors to `Q8_0`.\n\n## License\n\nReleased by Qwen under the Qwen Community License 1.0. Quantized by Atomic Chat.\n\n\n\n---\n\n## Operation 2\n\n# hf_fs ls\n\nURI: `hf://models/AtomicChat/Qwen3.8-Flash-Next-GGUF`\n\n| Type | Path | URI | Target | Details |\n|---|---|---|---|---|\n| dir | Qwen3.8-Flash-Next-AD-4.27bpw-M64 |  |  |  |\n| dir | Qwen3.8-Flash-Next-AD-5.00bpw-M64 |  |  |  |\n| dir | Qwen3.8-Flash-Next-AD-IQ1\\_M-M64-split |  |  |  |\n| file | .gitattributes |  |  | size=13.7 KB |\n| file | README.md |  |  | size=7.5 KB |\n| file | benchmark.png |  |  | size=73.1 KB |\n| file | btn\\_atomic.png |  |  | size=22.0 KB |\n| file | btn\\_discord.png |  |  | size=20.5 KB |\n| file | btn\\_github.png |  |  | size=14.4 KB |\n| file | hero.png |  |  | lfs, size=400.8 KB |\n| file | imatrix.gguf |  |  | lfs, size=580.0 MB |\n| file | mmproj-Qwen3.8-Flash-Next-BF16.gguf |  |  | lfs, size=907.5 MB |\n| file | mmproj-Qwen3.8-Flash-Next-F16.gguf |  |  | lfs, size=904.0 MB |","convId":"incyjf36ghq","role":"tool","timestamp":1787790849361,"toolCallId":"GPuVO0vrpgRspmSMXlPSXmoNJS8rDn5O","type":"text","id":"tdyyfaqb86e","parent":"1lt953q214g"},"type":"message"}
{"message":{"children":["s56zjtjc8ok"],"content":"Title: docs/qwen38-2x3090.md\nURL: https://github.com/VeroFess/llama.cpp_3090x2_qwen3.8_q8_opt/blob/main/docs/qwen38-2x3090.md\nPublished: N/A\nAuthor: N/A\nHighlights:\n1024 \\\n...\nThe preset compiles CUDA only for SM86 and enables FlashAttention, CUDA graphs, NCCL, and `GGML_QWEN38_2X3090`.\n...\nThe server requires exactly two visible RTX 3090 devices, 24 GB class VRAM, and compute capability 8.6. CUDA peer access is detected at startup. Tensor mode uses NCCL over the enabled peer path; layer mode retains bounded pinned host staging as its fallback.\n...\n- Tensor split with an enforced 1:1 split is the recommended P2P fast path. Layer split remains available as the compatibility path.\n- The Qwen3.8 output head is mirrored in tensor mode so target and MTP draft sampling stay on the GPUs. Model trunk and paged KV tensors remain sharded.\n- Single-sequence contiguous page runs use the Q8 tensor-core FlashAttention path for both prefill and generation.\n- Native context limit of 262144 tokens.\n- Up to eight resident sequences in one unified token pool.\n- Q8_0 K and V cache with no automatic precision fallback.\n- Tensor-sharded Q8 paged attention, FlashAttention, continuous batching, prompt reuse, exact recurrent checkpoints, and MTP enabled.\n- MTP drafts at most two tokens.\n- At most two exact recurrent checkpoints per resident slot with a 2048 MiB host cache budget.\n- Context shift, YaRN, row split, CPU KV, image input, and video input are rejected.\n...\n`--max-model\n...\nThe production profile always uses Q8_0 KV. TurboQuant and TCQ remain available outside the profile as explicit capacity modes. They are never selected after an allocation failure.\n...\n`llama-qwen38-tune` writes a schema-v2 cache keyed by the model and GGUF inventory SHA-256 values, source commit, fork manifest, CUDA toolkit, driver, GPU UUIDs, power limits, PCIe topology, Q8 KV codec, and resident sequence limit. Placement ratios include the two MTP layers, so a 36/28 target split is encoded as `36/30` for the 66 placed layers.\n...\n- `requires exactly two visible CUDA devices`: expose only GPU 0 and GPU 1 to the container.\n- Unexpected `cuda_peer` or `pinned_host_staged` mode: check PCIe topology, ACS, IOMMU policy, and Resizable BAR configuration.\n- `core matrix below the Q8 quality floor`: use the pinned Q8_0 GGUF.\n- `runtime context does not match the profile limits`: remove conflicting context or parallel options.\n- MTP initialization failure: verify that the GGUF contains its `nextn` tensors.\n\n---\n\nTitle: Insufficient documentation for -ot and --override-tensor flag ...\nURL: https://github.com/ggml-org/llama.cpp/discussions/13154\nPublished: N/A\nAuthor: ggml-org\nHighlights:\n| I tentatively think that -ot and --override-tensor should be listed as options under Llama-server, llama-cli, and llama-bench README.md files, as the option exists, and people should probably know about it. A basic explanation of the behavior (that it overrides the default assignment of tensors), would be welcome, as well as a note that tensor override is applied last would be helpful. I've spent quite a bit of time in the past week or so helping people troubleshoot custom tensor override setups to optimize their rigs for -ot to get the most tokens per second out of recent MoE models, and it's really difficult to manage. The issue is that if you have [Layer1, attn, shexp, exp] [Layer 2, attn, shexp, exp] ... [Layer 46, attn, shexp, exp] And so on. If you tensor override just the experts onto CPU, so... Layer1: GPU[attn, shexp] CPU[exp] Layer2: GPU[attn, shexp, exp] Layer20+: GPU2[attn, shexp, exp] You end up in a situation where --tensor-split actually assigns layers before the -ot flag. This intuitively makes sense if you understand what the flag is, but the issue is, if you're a casual user, you might not necessarily understand in a multi-GPU setup why your tensor-split suddenly is not evenly balanced anymore (I'm not necessarily saying this is an issue with tensor split, itself, just that due to -ot being effectively completely undocumented, it's confusing to a lot of users). I don't really think that this being a low level feature is necessarily a great reason for there not to be a note that it exists, the use case for it, an example usage, and a link to the PR the implemented it in the major binaries that support it as a flag. I think that a full guide explaining its usage probably is out of scope of the main documentation, and probably should be a community driven effort, but I do think that as it becomes a more popular feature, it's worth at least an acknowledgement that it exists. I think that general awareness about the flag and functionality is spurring a lot of interesting choices in computer builds and expanding what people can and can't do on a given hardware budget, and I think given sufficient usage of it we may see more ergonomic flags and options that operate in a more intuitive way, like a dedicated --conditional-expert-offload=\"CPU\" flag or something like that, but I think to get that done we probably need people to play with -ot first and figure out the best strategies for a variety of systems, and the way we get people playing with it is by putting a note in the documentation. |\n...\n| Concur. It should at least be listed here and here. As for specifics, depending on the level of user, it could cover: how to discover the tensor names for a given model what regex syntax it uses how to get the necessary ggml struct to use it via the public interface maybe even what the tensors of certain names generally do For the record, I actually got a 10x performance difference from using this feature to move just four layers of Qwen3-30B-A3B-Q6_K to my CPU so I wasn't using a tiny sliver of \"shared GPU memory\" anymore:`-ot \"blk\\.[3-4][0-9].*=CPU\"` eval time = 5892776.34 ms / 7560 tokens ( 779.47 ms per token, 1.28 tokens per second)`-ot \"blk\\.(2[6-9]|[3-4][0-9]).*=CPU\"` eval time = 754064.63 ms / 9580 tokens ( 78.71 ms per token, 12.70 tokens per second) (Full command line was more like:`llama-server -m Qwen_Qwen3-30B-A3B-Q6_K.gguf --port 7861 -c 32768 -b 2048 --gpu-layers 99 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn`+ one of those; I also tried`-ot exps=CPU`) The public API expecting us to pass in a`ggml_backend_buffer_type_t` seems pretty odd, too. I was briefly looking at exposing this in LlamaSharp, but there are very few references to ggml across the whole wrapper library. |\n...\n| @steampunque The advantage of the Scout, Maverick, and Deepseek architecture is in my opinion not that the shared expert may be sent to the CPU, but in fact, that it may not. The thing about MoE, is that it requires a large number of parameters, but a small number of active parameters. This lends itself rather well to CPU,\n...\n-ot \"\\d+.ffn_.*_exps.=CPU\"\n...\n` Which moves all conditional experts to CPU, but leaves all other parameters, notably the shared expert on GPU, as well as Attention. At 16k context, this results in around 7.8GB and 5.4GB of VRAM usage on either of my GPUs respectively, using the Unsloth q4_k_xl quantization. I will note that it appears your first command may have left the attention on CPU, as well. In the case that you have significantly more GPU (such as a build meant to run around 70B - 100B models on GPU), it can be necessary to specify specific layers to offload to CPU until you hit maximum VRAM usage (as in, to only offload the layers that you need to to fit the model) In practice, I get around 10 tokens per second on Maverick. Ryzen 9950X (compiled with BLAS) DDR5 4400MHZ dual channel memory (192GB) RTX 4000 SFF RTX 2000 ADA Layer split (pipeline parallel) Model stored on a gen 5 NVMe (relevant because I don't have sufficient RAM to load the model, meaning that conditional experts are streamed). You will, to reiterate, be limited to roughly the speeds that you would get running around a 9B parameter LLM on its own, on CPU, to the best of my observation. |\n...\n| I took another swing at this override stuff with Llama scout and think I finally made some progress. My first couple tries I was not selecting the right tensors to send to CPU. To find the set of tensor names do a dummy server run with ` NGL=99 OT='-ot .*=CPU' and also pass in the -v flag (verbose). \n` This will override every tensor in the model to the CPU so you scan see the names. For Scout you should get a list something like this : `tensor token_embd.weight buffer type overriden to CPU\n...\n` There are 3 tensors every layer for experts and 3 tensors every layer for shared expert. These expert tensors need to be offloaded to CPU for all layers: `tensor blk.2.ffn_gate_exps.weight buffer type overriden to CPU\ntensor blk.2.ffn_down_exps.weight buffer type overriden to CPU\n...\ntensor blk.2.ffn_up_exps.weight buffer type overriden to CPU\n...\n` These experts contain the majority of the parameters for the models. Scout has about 6G/expert * 16 experts = 96G parameters. The shared experts are `tensor blk.2.ffn_gate_shexp.weight buffer type overriden to CPU\n...\n` These shared experts run every token and need to be all on GPU for efficiency. For the entire model the shared experts take 6G of space and can easily fit in GPU. So experts (96G) + shared expert (6G) brings model parameters to 102G. There are another 6G total parameters in the rest of the tensors for the model to bring the whole thing to 108G. Thus shared + rest = 12G of parameters. This 12G of parameters can offload easily to a 12G GPU with a lot of room left over when quantizing to <4b/param. This command will pattern match all the experts and send them to CPU: `OT='-ot blk.*_exps*=CPU'\n...\n` Now load model with NGL=99 and the 6G params of shared experts and 6G params of other stuff should easily fit into single GPU. On my 4070 with q8_0 KV there is enough GPU VRAM left over for a 45k token KV cache (using my Q2_K_H hybrid quant https://huggingface.co/steampunque/Llama-4-Scout-17B-16E-Instruct-GGUF/resolve/main/Llama-4-Scout-17B-16E-Instruct.Q2_K_H.gguf to really push down the parameters of the model itself). Using this I get 8.9ts with a single GPU, about 1 to 2t/s slower than if I offload layers to 3 RPC machines. The huge advantage with this approach is I don't need to use RPC at all, I have a 108B parameter model running on one commodity 4070 and supporting 45k KV. The bottleneck in gen speed for me is I only have a 9900k CPU with DDR4 memory so it doesn't have a lot of mem BW, but it only needs to effectively evaluate 6G params per token on CPU which is not too bad. So I think this is actually the best approach to run the model instead of trying to offload it to a bunch of GPUs with RPC or otherwise. |\n\n---\n\nTitle: VeroFess/llama.cpp_3090x2_qwen3.8_q8_opt\nURL: https://github.com/VeroFess/llama.cpp_3090x2_qwen3.8_q8_opt\nPublished: N/A\nAuthor: N/A\nHighlights:\n# llama.cpp for Qwen3.8 27B Q8 on two RTX 3090s\n...\nThis fork is tuned for one deployment: `Qwen3.8-27B-Q8_0.gguf` on two 24 GB RTX 3090 cards. The fast path uses a 1:1 tensor split, CUDA P2P, NCCL, Q8_0 paged KV, FlashAttention, MTP, and up to eight resident sequences. The normal llama.cpp codebase is still here, but this profile is deliberately specific to this model and hardware.\n...\n## Recommended deployment\n...\nRun one Linux `llama-server` process with both GPUs visible. Enable Resizable BAR, verify CUDA P2P read and write access in both directions, and install NCCL before configuring the build. Cooling matters on long prompts. Both cards need enough airflow to hold their normal clocks instead of sitting at the thermal limit.\n...\nUse the supplied SM86 release preset:\n...\nStart the server with tensor split and P2P enabled:\n...\n```bash\nCUDA_VISIBLE_DEVICES=0,1 \\\nGGML_CUDA_P2P=1 \\\nGGML_CUDA_ALLREDUCE=nccl \\\n./build-qwen38-3090/bin/llama-server \\\n  --model /models/Qwen3.8-27B-Q8_0.gguf \\\n  --alias Qwen3.8-27B-Q8_0 \\\n  --profile qwen38-27b-q8-2x3090 \\\n  --split-mode tensor \\\n  --ubatch-size 1024 \\\n  --reasoning off \\\n  --reasoning-preserve \\\n  --host 0.0.0.0 \\\n  --port 8080 \\\n  --metrics\n```\n...\nThe profile fixes the context limit at 262,144 tokens, allows eight resident sequences, keeps K and V cache in Q8_0, enables two-token MTP drafts, and rejects settings that fall outside the tested configuration. Use the pinned model revision and SHA-256 listed in [the deployment notes](docs/qwen38-2x3090.md). The included Docker Compose file is convenient for packaging, but the native command above is the reference deployment and makes driver, NCCL, P2P, and clock checks easier.\n...\n- AVX\n...\nX512 and AMX support for x86 architectures\n-\n...\nVFH,\n...\nICBOP and\n...\nIHINTPAUSE support for RISC-V architectures\n- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-\n...\n, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use\n- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)\n-\n...\nkan and SYCL backend support\n- CPU+GPU hybrid inference to partially accelerate models larger\n...\nthe total VRAM capacity\n\n---\n\nTitle: How to run big MoE models like Qwen-3–235B-A22B in ...\nURL: https://medium.com/@david.sanftenberg/gpu-poor-how-to-configure-offloading-for-the-qwen-3-235b-a22b-moe-model-using-llama-cpp-13dc15287bed\nPublished: 2025-06-06T22:08:16.000Z\nAuthor: David Sanftenberg\nHighlights:\nTo work with the above, Llama-cpp has two different command-line options we’re interested in right now:\n...\n```\n--gpu-layers--override-tensor\n```\n...\nOne allows us to tell Llama.cpp how many model layers (aka Blocks) to assign to the GPU.\n...\nThe other allows us to exempt some tensors in those layers from GPU assignment and instead offload them to the CPU.\n...\n- Keep as many attention tensors on our GPUs as possible (ideally all)\n- Offload just enough experts ffn tensors to CPU to keep us from going OOM when loading the full model into our GPU(s).\n...\nIn my home lab I have 8 GPUs, all RTX 3090s with 24GB VRAM. This means I have a total of 192GB of VRAM to play with, for a model of size ~200GB.\n...\nModels also need some room for the Key-Value cache in addition to this, so it’s clear that if I want to use this model, I need to offload some bits of it to system RAM and the CPU.\n...\n```\n/usr/bin/docker run -d \\  --gpus all \\  --restart always \\   --network host \\  --health-cmd=\"curl -f http://localhost:8000/health || exit 1;\" \\  --health-interval=30s \\  --health-timeout=10s \\  --health-retries=3 \\  --health-start-period=60s \\  --name qwen3-235b \\  -v \"/home/dbsanfte/.cache/llama.cpp:/models\" \\  ghcr.io/ggml-org/llama.cpp:server-cuda \\      -v \\      -m /models/Qwen3-235B-A22B-UD-Q6_K_XL.gguf \\      --port 8000 \\      --host 0.0.0.0 \\      --ctx-size 32768 \\      --n-predict 32768 \\      --no-context-shift \\      -fa \\      --cache-reuse 256 \\      --cache-type-k q8_0 \\      --cache-type-v q8_0 \\      --threads 13 \\      --threads-http 13 \\      --gpu-layers 999 \\      --override-tensor \"blk\\.(1|2|13)\\.ffn_.*_exps\\.weight=CPU\" \\      --override-tensor \"blk\\.(25|37|49|61|73)\\.(ffn_gate_exps|ffn_up_exps)\\.weight=CPU\" \\      --temp 0.6 \\      --top-k 20 \\      --top-p 0.95 \\      --min-p 0 \\      --repeat-penalty 1.05 \\      --presence-penalty 1.0 \\      --main-gpu 0 \\      --numa distribute\n...\n```\n/usr/bin/docker run -d \\  --gpus all \\  --restart always \\   --network host \\  -v \"/home/dbsanfte/.cache/llama.cpp:/models\" \\  ghcr.io/ggml-org/llama.cpp:server-cuda \\      -v \\      -m /models/Qwen3-235B-A22B-UD-Q6_K_XL.gguf \\      --port 8000 \\      --host 0.0.0.0 \\      --ctx-size 32768 \\      -fa \\      --cache-type-k q8_0 \\      --cache-type-v q8_0 \\      --main-gpu 0 \\      --gpu-layers 999 \\      --override-tensor \"blk\\.(1|2|13)\\.ffn_.*_exps\\.weight=CPU\" \\      --override-tensor \"blk\\.(25|37|49|61|73)\\.(ffn_gate_exps|ffn_up_exps)\\.weight=CPU\" \n```\n...\n- `— gpu-layers` tells Llama.cpp to assign every possible layer (block) to the GPU. Just out of habit I put 999 as shorthand for “all”, but I could have put 95.\n- The first`--override-tensor` is a Regex selector for tensors. It tells Llama.cpp to send every Tensor that matches that regex to the CPU.\n- The second`--override-tensor` is another Regex selector for tensors. It also tells Llama.cpp to send every Tensor that matches that regex to the CPU.\n...\nYou will notice that I am choosing`ffn` tensors to offload to CPU in both cases. This is because of their performance characteristics as noted earlier in the post: they are the biggest tensors and the least impactful to offload to CPU in an MoE model.\n...\nMy regexes specifically select these for CPU offloading:\n...\n- All`ffn` tensors in blocks 1, 2, and 13\n- All`ffn_gate` and`ffn_up` tensors in blocks 25, 37, 49, 61, and 73.\n- Nothing at all for any blocks higher than 73.\n...\nThis means that the last GPU will only have 11 layers assigned to it instead of 12. So we don’t actually need to offload any of its tensors. Hence, I don’t offload any blocks higher than 73 (GPU 6). GPU 7 can fit all 11 layers and all their tensors.\n...\nIf you follow the Llama.cpp Docker container log, with`-v` verbosity set, it will tell you what it’s doing and where’s it’s sending each layer and which tensor overrides it applies:\n...\n= 0load_tensors: layer  85 assigned to device CUDA7, is_swa = 0load_tensors: layer  86 assigned to device CUDA7, is_swa = 0load_tensors: layer  87 assigned to device CUDA7, is_swa = 0load_tensors: layer  88 assigned to device CUDA7, is_swa = 0load_tensors: layer  89 assigned to device CUDA7, is_swa = 0load_tensors: layer  90 assigned to device CUDA7, is_swa = 0load_tensors: layer  91 assigned to device CUDA7, is_swa = 0load_tensors: layer  92 assigned to device CUDA7, is_swa = 0load_tensors: layer  93 assigned to device CUDA7, is_swa = 0load_tensors: layer  94 assigned to device CUDA7, is_swa = 0tensor blk.1.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.1.ffn_down_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.1.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.2.ffn_gate_exps.weight (816 MiB q8_0) buffer type overridden to CPUtensor blk.2.ffn_down_exps.weight (816 MiB q8_0) buffer type overridden to CPUtensor blk.2.ffn_up_exps.weight (816 MiB q8_0) buffer type overridden to CPUtensor blk.13.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.13.ffn_down_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.13.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.25.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.25.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.37.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.37.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.49.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.49.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.61.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.61.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.73.ffn_gate_exps.weight (630 MiB q6_K) buffer type overridden to CPUtensor blk.73.ffn_up_exps.weight (630 MiB q6_K) buffer type overridden to CPUload_tensors: offloading 94 repeating layers to GPUload_tensors: offloading output layer to GPUload_tensors: offloaded 95/95 layers to GPUload_tensors:        CUDA0 model buffer size = 20256.39 MiBload_tensors:        CUDA1 model buffer size = 22704.39 MiBload_tensors:        CUDA2 model buffer size = 22218.39 MiBload_tensors:        CUDA3 model buffer size = 22218.39 MiBload_tensors:        CUDA4 model buffer size = 22218.39 MiBload_tensors:        CUDA5 model buffer size = 22218.39 MiBload_tensors:        CUDA6 model buffer size = 22218.39 MiBload_tensors:        CUDA7 model buffer size =\n...\nLlama.cpp also tells us how big each of the`ffn` layers is: 630MiB. This means that for each of those`ffn` layers we offload, we free up 630MiB of VRAM.\n...\nIf you look again at my regexps up above, you will see I offload all`ffn` tensors in blocks 1 and 2, which are assigned to GPU 0, but I only offload`ffn` tensors from one block for the rest of the GPUs. This allows me to free up extra VRAM on card 0, because it starts off with less space available to begin with.\n...\nIf you only have one GPU, you will want a broad ffn regex to offload every single experts ffn tensor in every layer, and only keep the attention tensors on the GPU:\n...\n```\n# Offload every expert to CPU (for single-GPU systems with lots of RAM):--override-tensor \"\\.ffn_.*_exps\\.weight=CPU\"\n```\n...\nThrough trial and error I’ve modified these regexps to ensure that I’ve offloaded just enough to system RAM while maxing out my utilised VRAM. But your own regexes will vary depending on your exact system specs and offloading needs.\n...\nThat’s all there is to it really. Construct your`— override-tensor` regexes to maximize the used VRAM while taking into account the way llama-cpp assigns blocks to GPUs.\n...\n```\nThere's a typo in the regexp for single GPU:--override-tensor \"\\.ffn_.*_exps\\.weight=CPU\"The above expects tensors to start with \".ffn\" and matches nothing. It should be:--override-tensor \".*\\.ffn_.*_exps\\.weight=CPU\"\n```\n\n---\n\nTitle: Qwen3.8-27B best llama.cpp config on RTX 4090 24GB (BeeLlama, UD-Q4_K_XL v3.0, kvarn6 + kv-tail 2048 @ 165K, MTP n-max 3, fit off)\nURL: https://gist.github.com/ryan4yin/19db9fa44972c5735c1d181e8888d4fe\nPublished: N/A\nAuthor: N/A\nHighlights:\nUse the BeeLlama fork image — mainline llama.cpp silently falls back to CPU for\nnon-q4 KV caches on Qwen3.x hybrid architecture (no error is reported).\n...\n## Recommended config\n...\n```bash\ndocker run -d --name llama-server \\\n  --gpus all --shm-size=1gb -p 8001:8001 \\\n  -v \"$PWD/models:/models\" \\\n  ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.3 \\\n  --model /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \\\n  --mmproj /models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf \\\n  --alias Qwen3.8-27B \\\n  --ctx-size 165000 \\\n  --cache-type-k kvarn6 --cache-type-v kvarn6 \\\n  --kv-tail-tokens 2048 \\\n  --fit off \\\n  --flash-attn on \\\n  --n-gpu-layers auto \\\n  --load-mode mmap \\\n  --host 0.0.0.0 --port 8001 \\\n  --spec-type draft-mtp --spec-draft-n-max 3 \\\n  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \\\n  --image-max-tokens 4000 --image-min-tokens 1024 \\\n  --reasoning-budget 4000 \\\n  --chat-template-kwargs '{\"reasoning_effort\": \"low\"}' \\\n  --reasoning-preserve \\\n  --reasoning-budget-message \"... reasoning budget exceeded, need to answer.\\n\"\n...\n## Why these flags\n...\n| Flag | Reason |\n|---|---|\n| UD-Q4_K_XL (v3.0) | dynamic mixed-precision quant; v3.0 update shrank it ~364MB, buying +35K ctx at the same VRAM |\n| kvarn6/kvarn6 | KVarN supersedes stock quants: better quality at the same VRAM |\n| `--kv-tail-tokens 2048` | last 2K tokens kept full-precision F16 (~55 MiB); fixes \"attention drift / missing details\" in long contexts |\n| `--ctx-size 165000` | 150K/160K/165K pass restorable-continuation tests (fill to ~97%); 170K crashes during prefill (flash-attn kvarn segfault) |\n| `--fit off` | CUDA graphs conflict with MTP draft; fit on randomly drops speed to 9–20 t/s |\n| MTP `n-max 3` | measured best: n=1 → 60 t/s, n=2 → 37 t/s, n=3 → best |\n| `--flash-attn on` | required for quantized V, loader refuses otherwise |\n| `mmap` / `n-gpu-layers auto` | BeeLlama has no `auto` load mode |\n| temp/top_p/top_k/min_p | thinking-mode sampling recommended by unsloth |\n| reasoning budget/preserve | effort `low` keeps thinking short; preserve keeps multi-turn thinking context |\n...\n- **BeeLlama required**: non-q4 KV on mainline = silent CPU fallback (GPU 0–30%).\n- f16/bf16 KV + 27B on 24GB OOMs outright.\n- OOM depends on request batch shape, not just ctx depth: a restorable-prefix\n multi-turn continuation (KV only 66.8K) can OOM even when a 150K prefill passes.\n Validate ctx ceilings with restorable-continuation traffic, not single-shot prefill.\n...\n- Throughput:\n...\npredicted_ms\n...\ncompletion_tokens / total time\n...\n(dragged\n...\nby prefill\n\n---\n\nTitle: How to Fix Slow Qwen 3.6 27B on RTX 3090 (10-80 tok/s) | InsiderLLM\nURL: https://insiderllm.com/guides/fix-slow-qwen-3-6-27b-rtx-3090/\nPublished: 2026-05-01T00:00:00.000Z\nAuthor: · by Mark Bartlett\nHighlights:\n| Symptom | Likely cause | Fix |\n| --- | --- | --- |\n| 10-15 tok/s, partial offload | Layers on CPU | `-ngl 99` explicit |\n| 15-20 tok/s, full offload | Wrong quant or template | UD-Q4_K_XL + fixed template |\n| 25-30 tok/s sustained | Backend tradeoff | Try ik_llama.cpp |\n| ~38 fresh, ~35 at 32K | Normal prefill→decode shift (re-benched June\n...\nMiu,\n...\n.cpp f9cd456ea)\n...\nWorking as expected\n...\nIf the first number is anything less than the second, you have layers on CPU and you’ll never hit the speeds you want. Add`-ngl 99`(or`--n-gpu-layers 99`) explicitly. The default in some llama.cpp builds is 0 if you didn’t specify, which silently puts everything on CPU. Multiple users in the thread reported “I’m at full GPU offload, why so slow” and the answer was that they weren’t.\n...\n```\n./build/bin/llama-server \\\n  --model qwen3.6-27b-UD-Q4_K_XL.gguf \\\n  -ngl 99 \\\n  --flash-attn on \\\n  ...\n\n```\n...\n`-fa on` is also worth confirming. Flash attention shaves a measurable amount off generation time on the 3090 and is on by default in recent builds, but old configs sometimes carry it as off.\n...\nFix: pull the current Jinja template from the Qwen 3.6-27B model card and pass it explicitly with`--jinja --chat-template-file qwen3.6.jinja`. Or grab a known-good one from the Unsloth GGUF repo’s tokenizer config. Update as of June 10: the major distributors (Unsloth, bartowski, ggml-org, LM Studio community) now all ship the correct template; the broken-template problem mostly hit early-May GGUFs and custom fine-tunes. If you re-pulled from a major distributor in the last few weeks, you’re probably fine — but if your output is gibberish, the template is still the first thing to check.\n...\n. Backend choice: llama.cpp\n...\n.cpp + M\n...\nllama.cpp baseline. UD-Q4_K_XL on a 3090 lands at 38 tok/s fresh and 35 tok/s sustained at 32K — re-benched June 10 on Miu against llama.cpp f9cd456ea. The 30-40 baseline I cited at publication holds up: I re-ran this on current main and the number is still right. If you’re seeing 80+, you’re either measuring fresh-context burst on a lighter quant or running speculative decoding. If you’re below the 30-40 floor, the problem is in steps 1-6. Measured on Miu (RTX 3090), June 10, 2026, llama.cpp f9cd456ea.\n...\nllama.cpp + MTP via the llama-mtp fork. PR #22673 merged MTP head support into mainline llama.cpp the week after this article first published. One important caveat surfaced in the re-bench: mainline’s speculative-decoding path is draft-model based and does NOT auto-consume the embedded MTP head from an MTP-tagged GGUF. To use the MTP head as the draft source, you need the llama-mtp fork with`--spec-type draft-mtp`. Mainline’s PR #22673 means the MTP head loads; using it for specdec needs the fork.\n...\nik_llama.cpp. A llama.cpp fork with custom kernels for hybrid SSM models. Running sokann/Qwen3.6-27B-GGUF-5.076bpw(a 5-bit quant tuned for the fork) at full GPU offload, users in the thread report 31-39 tok/s decode at 128K context — comparable to llama.cpp at 4-bit but at 5-bit quality. If you’re VRAM-comfortable and want better output without the speed hit, this is the path. The fork also supports MTP heads.\n...\nvLLM\n...\nproduction-serving\n...\nOpenAI-compatible\n...\ncontinuous batching\n...\nThe deepest cause, and the one most users do not have. Qwen 3.6 is a hybrid SSM architecture. Three Gated DeltaNet layers per Gated Attention layer. The DeltaNet recurrence step uses a small CPU-side compute buffer (around 552 MiB labeled`CUDA_Host buffer` in the loader output). On older CPUs without AVX-VNNI or AVX-512 (i9-9900K, i7-6700K, anything pre-2019 Intel), this bookkeeping path becomes a real bottleneck on a fast GPU.\n...\nYou’ll see`graph splits = 2` in the llama.cpp startup log and a small per-token CPU cost that matters when the GPU is doing 30+ tok/s. Newer CPUs (i7-12700K and up, Ryzen 5000+) close this gap and the SSM cost falls into the noise.\n...\nThere’s no software fix. CPU upgrade is the answer. If that’s not on the table, switch to vLLM, which handles the hybrid path differently and is less sensitive to CPU vintage.\n...\n1. Layers on CPU because`-ngl` wasn’t set. One flag fix.\n2. Bad quant or bad template. Switch to UD-Q4_K_XL and update the Jinja template.\n3. Wrong expectations. 80 tok/s is a fresh-context burst, not your sustained speed at 32K.\n\n---\n\nTitle: Qwen3.8 - How to Run Locally\nURL: https://unsloth.ai/docs/models/qwen3.8\nPublished: 2026-08-26T15:34:10.281Z\nAuthor: N/A\nHighlights:\nIf the model fits, you will get \\~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.\n...\nQwen3.8 can run in [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:\n...\nsampling, networking,\n...\n:llama: Run Q\n...\n3.8 in llama.cpp\n...\nWe need to use the specific IQ1\\_XXXS branch [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.\n...\n```bash\napt-get update\napt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y\ngit clone --branch iq1-narrow https://github.com/unslothai/llama.cpp\ncmake llama.cpp -B llama.cpp/build \\\n    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON\ncmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split\ncp llama.cpp/build/bin/llama-* llama.cpp\n```\n\n{% endstep %}\n...\n{% step %}\nIf you just want to run the standard `IQ1_S` and other quants, then compile llama.cpp normally:\n\n{% code overflow=\"wrap\" %}\n\n```bash\napt-get update\napt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y\ngit clone https://github.com/ggml-org/llama.cpp\ncmake llama.cpp -B llama.cpp/build \\\n    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON\ncmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split\ncp llama.cpp/build/bin/llama-* llama.cpp\n```\n\n{% endcode %}\n{% endstep %}\n...\nQwen3.8\n...\n27B\n...\n{% code overflow=\"wrap\" %}\n...\n./llama.cpp/llama-cli \\\n    --model unsloth/Qwen3.8-27B-GGUF/Qwen3.8-2\n...\nB-UD-Q4_K_XL.gguf \\\n    --temp 1.0 \\\n    --top-p 0.95 \\\n    --top-k 20 \\\n    --min-p 0.0\n...\n{% endcode %}\n\n---\n\nTitle: Don't Offload GGUF Layers, Offload Tensors! 200%+ Gen ...\nURL: https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/\nPublished: 2025-08-27T00:34:10.281Z\nAuthor: N/A\nHighlights:\nIdea: With llama.cpp and derivatives like koboldcpp, you offload entire LAYERS typically. Layers are comprised of various attention tensors, feed forward network (FFN) tensors, gates and outputs. Within each transformer layer, from what I gather, attention tensors are GPU heavy and smaller benefiting from parallelization, while FFN tensors are VERY LARGE tensors that use more basic matrix multiplication that can be done on CPU. You can use the --overridetensors flag in koboldcpp or -ot in llama.cpp to selectively keep certain TENSORS on the cpu.\n...\n```\npython ~/koboldcpp/koboldcpp.py --threads 10 --usecublas --contextsize 40960 --flashattention --port 5000 --model ~/Downloads/MODELNAME.gguf --gpulayers 65 --quantkv 1 --overridetensors \"\\.[13579]\\.ffn_up|\\.[1-3][13579]\\.ffn_up=CPU\"\n...\n...\nUse regex to\n...\nIn this example, overriding tensors ffn_down at a higher Q5 to CPU would save more space on your GPU that fnn_up or fnn_gate at Q3. My regex from above only targeted ffn_up on","convId":"incyjf36ghq","role":"tool","timestamp":1787790850515,"toolCallId":"Mlk0a8I1drztkdTsXAfU3LzOMlXVU0Zh","type":"text","id":"e1sosznazua","parent":"1lt953q214g"},"type":"message"}
{"message":{"children":[],"content":"Verified before reasoning, as you asked. Everything below is grounded in the actual model files, the PR, and this build's real flag list.\n\n## ✅ Existence check (you were right to make me do this)\n\n| Thing | Status |\n|---|---|\n| [`Qwen/Qwen3.8-Flash-Next`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) | Real. `qwen4_exp`, 125B total / 6B active + **51B n-gram embedding** + 4B MTP, 48 layers, 262,144 ctx. Created 24 Aug 2026. |\n| [`unsloth/Qwen3.8-Flash-Next-GGUF`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) | Real. Your exact file `UD-IQ4_XS/...-00001-of-00003.gguf` exists (10.9 MB + 49.8 GB + 43.8 GB ≈ **93.6 GB decimal / 87.2 GiB**). |\n| [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) | Real, **still open/draft** as of now. Merge commit `1d85467`. Your engine is ~3 days old as an architecture. |\n| [Unsloth writeup](https://unsloth.ai/docs/models/qwen3.8-next) | Real. UD-IQ4_XS = 93.7 GB, KLD 0.079162, same-top 91.089%. Matches your number. |\n\nYour `GGFF` directory typo is verbatim from Unsloth's own `hf download --local-dir` example, so you followed the doc exactly. It works, it's just funny.\n\n---\n\n## 🚨 Three landmines in your current block\n\nYour block is the signature of someone who hit all three known bugs of a 3-day-old architecture and papered over them with the workarounds from the PR thread. Let me be direct about which of those workarounds are now costing you real quality.\n\n### 1. `--cache-type-k q8_0 --cache-type-v q8_0` is buying you nothing and forcing a ~15% quality tax\n\nThis is the big one. Read this arithmetic before anything else:\n\n```\nQSA (full_attention) layers:        12 of 48   (layer_types = 3× linear, 1× full, ×12)\nnum_key_value_heads:                 2\nhead_dim:                            256\nKV bytes / token = 2 × 256 × 2 (K,V) × 2B = 2,048 B / layer\nKV bytes / token = 2,048 × 12 layers     = 24,576 B ≈ 24 KiB / token\n\n@ 131,072 ctx  →  3.0 GiB   (f16)   →  1.6 GiB (q8_0)\n@ 262,144 ctx  →  6.0 GiB   (f16)   →  3.1 GiB (q8_0)\n```\n\nThe hybrid Gated DeltaNet design means **your KV cache is nearly free**. The 36 linear layers hold a *constant-size recurrent state* (~110 MB/sequence, not per-token), not a growing cache. You are saving 1.4 GiB.\n\nWhat that 1.4 GiB costs you: quantized KV caches store K/V **Hadamard-rotated**. `qwen4exp`'s QSA path (`build_attn_qsa`, copied from the MLA/DSA overload) never expected rotated K/V and just asserted they","convId":"incyjf36ghq","model":"qwen3.8-flash-next","role":"assistant","timestamp":1787790850521,"type":"text","id":"s56zjtjc8ok","parent":"e1sosznazua","reasoningContent":"Now I have everything I need, including a critical find: **AtomicChat's repo has `mmproj-Qwen3.8-Flash-Next-BF16.gguf` (907.5 MB)** — so vision IS available, just not in the unsloth repo. And the user's UD-IQ4_XS dir has no mmproj, so they need to download that mmproj (or unsloth's if it exists elsewhere). Actually unsloth repo doesn't have mmproj listed at root. So point them at AtomicChat's mmproj.\n\nAlso important caveats from search:\n- VeroFess fork notes: \"MTP initialization failure: verify that the GGUF contains its `nextn` tensors\" — and issue #1993 said \"the fitting GGUF has no MTP weights\". So `--spec-type draft-mtp` likely won't work with unsloth GGUFs. And PR author said MTP still WIP in PR #27742. So: don't rely on MTP; test it, and note the ngram-based spec decode options (`--spec-type ngram-*`) are a separate thing — those look up the *context* ngrams, unrelated to the PLE table. Actually ngram speculative decoding could be genuinely useful for agentic/coding (repetitive boilerplate). Worth suggesting as an experiment since it costs no VRAM.\n- The BeeLlama gist: `--fit off` recommended there because of CUDA graphs + MTP conflict on Qwen3.8-27B, and kvarn KV. Different model (Qwen3.8-27B, not Flash-Next), but a data point on `--fit` quirks. Also mainline \"silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture.\"\n- insiderllm: hybrid SSM CPU-side `CUDA_Host buffer` bookkeeping ~552 MiB; `graph splits = 2`; on older CPUs without AVX-VNNI/AVX-512 the DeltaNet recurrence path becomes a bottleneck. **3945WX = Zen2 Threadripper → has AVX2 but NO AVX-512, no AVX-VNNI.** That's directly relevant! Zen 2 gen 3 = Skylake-era AVX-512 (Cascade Lake has AVX-512 + VNNI). Wait: 3945WX is Castle Peak/Zen 2. Zen 2 does NOT have AVX-512. Cascade Lake (Xeon) does. So 3945WX = AVX2 only, no AVX-512, no VNNI. That means the DeltaNet linear-attention CPU bookkeeping is a real bottleneck risk, AND the PLE gather on CPU will be slower. Important for thread config: on Zen2 without AVX-512, more threads than physical cores rarely helps; 12 physical is right. Also this argues strongly for keeping everything except the PLE on GPU.\n\nNow let me do the memory math carefully.\n\n**Model size breakdown (UD-IQ4_XS = 93.7 GB = 87.3 GiB... careful with GB vs GiB).**\nFiles: 10.9 MB + 49.8 GB + 43.8 GB = 93.6 GB (decimal) = 87.2 GiB.\n\nHmm, HF reports in decimal GB typically. 93.6 GB = 87.2 GiB.\n\nVRAM: 4 × 3090 = 4 × 24 GiB = 96 GiB nominal, but actual usable per 3090 is ~23.7 GiB (some 3090s 23.5). llama.cpp sees ~23.4-23.7 GiB usable per card. Total ≈ 94 GiB usable, minus what the display/other procs use.\n\nSo model = 87.2 GiB, VRAM total ≈ 93.6 GiB usable. Leaves ~6.4 GiB across 4 cards for KV + compute buffers + CUDA context (each CUDA context ~300-500 MB, so ~1.6 GiB just for contexts). Very tight. This is exactly why they need the PLE on CPU.\n\n**PLE table size**: 51.2B params. PR says the GGUF table is quantized at 4-bit minimum. From Baekpica: \"The published resident-PLE baseline spends 38,400,184,320 bytes / 35.7630 GiB on a Q5_1 PLE table.\" Wait — that says the published baseline (unsloth's? or the BF16 GGUF?) uses Q5_1 for PLE at 35.76 GiB. 51.2e9 params at 5.5 bpw = 35.2 GB. Yes! So in the *published GGUFs*, PLE is at Q5_1 (not 4-bit). Hmm — Baekpica says \"The published resident-PLE baseline spends 38,400,184,320 bytes on a Q5_1 PLE table\" — that's probably referring to *their* earlier artifact (\"resident-PLE baseline\" = their previous MQ-Q5 release). Hmm ambiguous. Unsloth docs say PLE \"are not quantized that heavily (4-bit minimum)\". Either way ~26-36 GiB.\n\nLet me sanity-check against total: if PLE = 35.76 GiB, main = 87.2 - 35.8 = 51.4 GiB. Main is 128.8B params → 51.4 GiB = 55.1 GB → 3.42 bpw. IQ4_XS is 4.25 bpw; dynamic (UD) mixes in lower bits for MoE experts, so 3.4 bpw for the backbone is plausible for a Dynamic quant where experts get IQ3/Q3 and PLE stays at Q5_1. Actually plausible.\n\nAlternatively if PLE at 4-bit = 25.6 GB = 23.8 GiB → main = 63.4 GiB → 128.8B at 3.72bpw. Also plausible.\n\nI can't pin it exactly, so I should present the arithmetic parametrically and tell them how to read the actual numbers from the loader log. That's more honest and more useful. I'll give them the exact log lines to look for.\n\nKey decision: **whether to keep the PLE on CPU or GPU.**\n\nOption A (their current): PLE→CPU. GPU holds ~51-63 GiB of backbone + KV. Fits comfortably. Per-token cost: one sparse gather from host RAM (~a few rows × 2560 × 2 bytes; small), plus the host→device copy. Latency-bound. At 2133 MT/s this adds latency per token, but the transfer is tiny (a few KB–tens of KB per token), so it's latency (~µs) not bandwidth. Should be OK, and this is what the community converged on.\n\nOption B: everything on GPU — 87.2 GiB into 93.6 GiB usable. Extremely tight, and with `--fit on` llama.cpp would then *auto-shrink ctx* (that's exactly what `-fit` does: it adjusts unset args; but `-c` was explicitly set so fit won't reduce it — fit only adjusts *unset* args, so if ctx-size is explicitly set it won't shrink it, it'll try ngl instead, or fail). Important nuance!\n\nActually this is a key subtlety worth verifying: `--fit` \"adjust unset arguments to fit in device memory\". Since they set `--ctx-size 131072` explicitly, fit will not reduce ctx. It reduces `-ngl`. So with fit on and an explicit ctx, llama.cpp will drop layers to CPU until it fits. That means: **with fit on + explicit ctx, the \"offloading\" happens silently at the layer level.** They should watch the log for \"reducing ngl\". And `--fit-target` gives margin. And `--fit-ctx` is the minimum ctx fit can set.\n\nHmm, and note ServeurpersoCom found that on a 96GB card the model fits so `--fit` never probes the offload path — so `--fit on` passed. And phhusson's `--fit on` crash on 3090 was the `graph_max_nodes`/`obj_new` assert, fixed by that commit. So: **the user MUST include commit c30f6685 (or a build after it) for `--fit on` to work at all on their 4×3090 box.** That's a headline finding, since they explicitly said they're running fit on.\n\nAlso: does the user's build have the fix? Their runblock has `--fit off` and `LLAMA_ATTN_ROT_DISABLE=1` — that's the classic \"I hit all three bugs and worked around them\" signature. So their build is likely pre-fix. I should give them the three commits to cherry-pick / the commit to update to, and tell them how to verify.\n\nLet me also reconsider: is `--fit on` even desirable here? With PLE on CPU and everything else on GPU, you know the fit. fit on is nice for robustness but it also probes by temporarily allocating — and with this arch it crashed. My recommendation: update the build, then `--fit on` is fine and gives auto-shrink safety; but explicitly set `-ngl all` and `-c` and add `--fit-target` margin. Or run `--fit off` deterministically. I'll present: once the patch is in, prefer `--fit on` for resilience + `-fit-target` per-GPU margin; and verify in the log that it did not drop layers.\n\n**KV cache math** — let me redo carefully.\nQSA (full attention) layers: 12. kv_heads=2, head_dim=256.\nPer token per layer: K = 2×256 = 512 elements; V = 512 elements → 1024 elements.\nf16 → 2048 bytes/layer/token. × 12 layers = 24,576 bytes ≈ 24 KiB/token.\n131072 tokens × 24 KiB = 3,145,728 KiB = 3.0 GiB.\n262144 tokens → 6.0 GiB.\n\nIndexer cache: indexer_kv_heads=1, indexer_head_dim=128, compress_ratio 4. Probably 128 elems per block; blocks of 4 → per token ≈ 32 elements × 12 layers = 384 elements ≈ 768 B/token f16 → ~0.1 GiB at 131K. Small. (Actually QSA typically keeps the indexer cache at f32 or f8; either way small.)\n\nLinear attention (GDN) state: constant per slot, not per token. conv state: linear_conv_kernel_dim 4 × conv dim (16 heads×128 key head dim ×2 for k... roughly 8192?) plus recurrent state 16×128×128 per layer × 36 layers. That's ~262k elements/layer → ~1 MB/layer f32 → ~36-40 MB per slot. Tiny. Note the VeroFess fork mentions \"368,640-byte persistent convolution state\" and \"2048 MiB host cache budget\" for recurrent checkpoints. So recurrent state checkpointing can be memory-heavy with many checkpoints — `--ctx-checkpoints` default 32! With 36 GDN layers, 32 checkpoints could be large. Hmm, VeroFess limited to \"at most two exact recurrent checkpoints per resident slot with a 2048 MiB host cache budget\". So the default `-ctxcp 32` might blow up memory or cost time on this arch. Worth flagging: consider lowering `--ctx-checkpoints` — that's a real, non-obvious tuning lever for GDN/hybrid models in llama.cpp. Let me flag it as \"watch this / test\", not as a hard claim. Actually the checkpoints are created for SWA/hybrid caches to enable prompt-cache reuse. With `--parallel 1` and one long-lived session, 32 checkpoints × ~1MB×36 layers = ~1.1 GB. Manageable but non-trivial. I'll mention `-ctxcp 8` as a candidate and `-cram` sizing.\n\nAlso `--swa-full` — for hybrid models, llama.cpp may drop linear-attention states for cached-away regions; not applicable directly.\n\nSo: **KV at f16 for 131K = ~3 GiB. That is trivially affordable, which means the q8_0 KV cache is buying nothing but is triggering a correctness bug.** Killer insight. Even 262K at f16 = 6 GiB.\n\nSo the recommendation:\n- Drop `--cache-type-k q8_0 --cache-type-v q8_0` → f16 (default). Remove `LLAMA_ATTN_ROT_DISABLE=1`. This removes the ~15% quality hit and the crash risk, at a cost of ~1.5 GiB extra VRAM. Free.\n\n**Threads**: `-t 6` is too low for this box. The DeltaNet/GDN CPU bookkeeping and the PLE gather both run on `-t`. Zen2 12c/24t. Recommend `-t 12` (physical core count), `-tb 24`? For memory-bandwidth-bound CPU work, SMT siblings contend; but llama.cpp's PLE lookup (`set_input` host-side row index computation + get_rows) and the mamba/GDN state update on `CUDA_Host` buffers are latency-tolerant and parallelizable. I'll recommend `-t 12` and `-tb 24`, and explain how to A/B: `-t 12` vs `-t 16` vs `-t 24`, measuring decode t/s. Also `--cpu-strict 1` + `--prio 2` to stop scheduler migration; and importantly **`--poll 50`** default is fine.\n\nHmm, careful: `-tb` default = `-t`. Setting `-tb` higher than physical helps prefill on CPU-heavy models; here prefill is GPU-bound, so `-tb` mainly affects the CPU-side ops. 4096 batch on GPU with ubatch 2048 — is ubatch 2048 safe for this arch? The QSA/indexer path with `indexer_budget 2048` and `indexer_compress_ratio 4` — large ubatch means large intermediate tensors for the hyper-connection (`hc_count 4`) which multiplies activation buffers by 4! That's a real memory consideration: hyper-connections widen the residual stream 4×. `build_hc_combine` crashed in `ggml_scale` during `graph_reserve` — that's a compute-buffer sizing issue at large n_tokens. So **compute buffers scale with ubatch, and this arch has 4× residual widening**. Reducing `--ubatch-size` from 2048 → 1024/512 meaningfully reduces VRAM for compute buffers at the cost of a little prefill throughput. Given they're tight on VRAM, `-ub 1024` (or 512) is a smart lever. And `-b 4096` ≥ `-ub` is fine.\n\nNote: default `-b` 2048, `-ub` 512. They set 4096/2048 — that's 4× the default ubatch → 4× compute buffer. On 4 cards with 24GB each and 87GiB of weights... this is likely the reason they're on the edge. Strong recommendation: `-ub 1024`, `-b 4096`. Test prefill t/s.\n\n`--op-offload`: default true. For hybrid/CPU-mixed graphs, `--no-op-offload` sometimes helps when host ops are involved. Mention as an experiment. Also `--kv-offload` default enabled, fine. `--no-host` \"bypass host buffer allowing extra buffers to be used\" — with a PLE-on-CPU + `get_rows` graph, the host buffer is needed. Do NOT add `--no-host`. Good call to mention as a trap.\n\n**split-mode**: `layer` is correct and default. `--tensor-split 1,1,1,1` — with equal cards this is what you want, but since `per_layer_token_embd` is pulled out to CPU, the remaining tensors are evenly distributed, so 1,1,1,1 is right. However: with layer split, `--main-gpu 0` puts... main-gpu matters for `row` split; for `layer` split it determines the device for the output/sampling logits. Card 0 then carries extra load. Because card 0 also holds the token embedding + output head (vocab 248320 × 2560, not tied → embed 248320×2560 + output 248320×2560; at IQ4_XS-ish ~0.3GB each). Fine. But note the output head is huge-ish and logits for a 248320 vocab need a f32 buffer of ~1 MB/token. With ubatch 2048 that's 2 GB! Actually logits are only computed for the last token in decode, but during prefill llama.cpp only computes logits for the last token of the batch too. OK.\n\nHmm — but `--parallel 1` means no batching across slots.\n\nOne more important thing: **`--fit on` + `-ot ...=CPU` interaction**. `-fit` probes offload paths; ServeurpersoCom's crash was in the \"MoE-to-system-memory probe\" triggered by `--fit-target 70000`, fixed by the `LLM_ARCH_QWEN4EXP` addition to `graph_max_nodes`. So fit's probing path is exactly where this arch crashes. If they can't patch, keep `--fit off` and manage ngl manually. I'll give both paths.\n\n**Prompt caching for Hermes** — this is THE most important perf lever for an agent harness:\n- `--cache-prompt` already on (it's the default).\n- `--cache-ram` default 8192 MiB — this is the llama-pack/keep-cache. For long agentic sessions with a big system prompt + tool schemas, bump it: `-cram 32768` or `-1`.\n- `--cache-idle-slots` default enabled (requires cache-ram) — good.\n- `--cache-reuse 256` — enables KV shifting reuse for prompts that share a prefix but diverge. Hermes rewrites history (summarization, memory injection) so pure longest-prefix match breaks; `--cache-reuse` helps a lot. Caveat: it interacts with the QSA/indexer cache and the hybrid memory — the indexer cache must track attention cache cell-for-cell (that assert!). So `--cache-reuse` could trip that assert. Flag as \"test carefully, may hit the indexer desync assert\".\n- `--keep -1` or `--keep N`: protects the system prompt + tool defs from being evicted. Very relevant for Hermes long tasks. But with `--parallel 1` and no context-shift, eviction only happens at ctx overflow. Since Hermes does its own context management/compaction, `--keep 2048` is cheap insurance.\n- `-sps/--slot-prompt-similarity`: with 1 slot, set `--slot-prompt-similarity 0` ? No — with one slot the LRU always picks it; the default 0.10 similarity gate means a request that doesn't match well enough... With a single slot, if similarity is below threshold llama.cpp may still reuse the slot but clear it. Setting `-sps 0` disables the check. Since they have 1 slot and 1 session, `-sps 0` is a mild win (avoids spurious clears). Actually careful: `-sps 0` = disabled means it always uses the slot. Fine, recommend.\n- `--defer-polling` — not in this list. Skip.\n- `--timeout`: Hermes long tasks with big prefills → default 3600s fine.\n- `--threads-http 4` fine.\n\n**Reasoning/tool-calling for Hermes (quality/correctness):**\n- `--jinja` ✓ (essential)\n- Hermes needs tool calls → `--jinja` + template handles it; may need `--chat-template-file`. Also `parse_tool_calls` is per-request.\n- **`--reasoning-preserve`** — the Qwen README explicitly says preserved thinking is \"especially beneficial for agent scenarios where decision consistency and reduced redundant reasoning are critical\" and \"improves KV cache utilization\". llama.cpp flag: `--reasoning-preserve`. Server log even hints: \"chat template supports preserving reasoning, consider enabling it via --reasoning-preserve\". This is a real, high-value addition for Hermes. BUT: it bloats the context. Hermes + preserve thinking = longer prompts = more prefill. Trade-off. Since Hermes benefits and Qwen recommends it for agents, recommend enabling, and measure.\n- `--reasoning-effort` is a first-class llama.cpp flag now! `--reasoning-effort xhigh|high|medium|low`. Better than `--chat-template-kwargs '{\"reasoning_effort\":\"medium\"}'`. Note unsloth doc said use `--chat-template-kwargs`; both work but the native flag is cleaner and Hermes can also pass `reasoning_effort` per request.\n- `--reasoning-budget N` — cap thinking tokens. For a long-horizon agent you may not want to cap. But `--reasoning-budget-message` helps avoid runaway. Mention as a latency control knob.\n- Hermes may send `presence_penalty`? Their defaults are correct for thinking mode (temp 1.0/top_p .95/top_k 20/min_p 0/pres 0/rep 1.0). ✓ Note that `--min-p 0.0` explicitly disables min_p which is what Qwen wants; llama.cpp default is 0.05, so explicitly setting 0 matters. ✓ They did it right.\n- `--temp 1.0`: llama.cpp applies `top_k`/`top_p` before `temperature` per the sampler chain — fine.\n- Endless repetition in agentic loops: Qwen says presence_penalty 0–2 helps but hurts; better to leave 0 and use DRY only if needed.\n- `--no-ctx-shift`/`--context-shift` default disabled ✓.\n- `--warmup` default enabled ✓ keep (it warms the graph; with QSA sparse path it also reserves graph memory correctly).\n- `--check-tensors` for first run.\n- `--metrics` ✓ plus `--slots` (default on) — useful.\n- `--alias` ✓\n- Weak api key on LAN: mention. Also `--host 192.168.1.61` binding to a specific LAN IP — fine but if Hermes runs on the same host use 127.0.0.1, or bind 0.0.0.0 with the key. Also note: newer builds warn default port will change to 9931.\n- `--ui` — can disable to save a bit / reduce attack surface.\n\n**Vision**: their GGUF dir has no mmproj. Hermes has vision (clipboard images) and document extraction. If they want vision: download `mmproj-Qwen3.8-Flash-Next-F16.gguf` (904 MB) from AtomicChat's repo and pass `--mmproj`. Cost: ViT ~0.9 GB + it will be offloaded to GPU (mmproj-offload default). Also images inject many tokens → more KV. And `--image-max-tokens`. Note Hermes' vision tool sends images. Recommend adding it (cheap) since Hermes supports it, or explicitly `--no-mmproj` if they don't want it.\n\n**MTP / speculative**:\n- config has MTP (1 layer, 4B params per README). PR says MTP WIP; issue #1993 says \"llama.cpp's leading PR ships without it, but the fitting GGUF has no MTP weights\". So `--spec-type draft-mtp --spec-draft-n-max 3` probably won't work today on the unsloth GGUF. Recommend: check with `llama-quant info` / grep loader log for `nextn` / `mtp` tensors. If present and support lands → huge win for agentic decode (MTP is the single biggest decode speedup available for this model, ~1.5-2×).\n- Free-ish alternative to try now: `--spec-type ngram-mod` or `ngram-map-k` (context ngram drafting, no VRAM, no draft model). For coding/agent loops with repetitive text, can be a real win. Risk: extra prefill on mispredicts.\n- Caution: the QSA/indexer + spec decode interaction is untested territory on a 3-day-old arch. Flag it.\n\n**Ngram offload to RAM/Optane (their explicit ask)**:\nStatus:\n- danielhanchen: \"I copied Gemma-3N's PLE approach so it's 1 large table - it can be offloaded to RAM or Disk via mmap.\" So today: `-ot per_layer_token_embd=CPU` + mmap = RAM/page-cache backed; that's exactly the Gemma-3n PLE mechanism, already merged upstream.\n- `-lm none` currently *disables* mmap, which forces the whole 87 GiB into anonymous RAM at load. For the PLE specifically you want the opposite: `--load-mode mmap` (or `auto`) so the PLE pages stay file-backed in the page cache. Downside reported: mmap load is 10× slower to *load* on DGX Spark, but \"once loaded, doesn't hurt performance much\".\n- FlorinAndrei: \"Maybe everything could just obey --load-mode\" → currently PLE offload has its own mmap path that ignores it. coder543: \"Everything seems to obey --load-mode already, but forcing the PLEs into pinned memory wastes a ton of memory. It would be nice to have the PLEs streamed from disk.\"\n- ngxson: qwen's engram is \"one big tensor from the get-go, so it cannot split across different storages\"; concat into one big tensor is necessary; wants conversion refactor so manual mmap isn't needed.\n- Real SSD-offload work exists but is NOT in llama.cpp: Baekpica/ds4 `feature/qwen38-ssd-ple-handoff`, with BF16 sidecars, O_DIRECT, bounded 512MiB–2GiB pinned page cache, CUDA UVA gather, verified functional on DGX Spark at 262K (83 tok/s prefill on a 3,069-token prompt). Requires a dedicated loader — \"Compatibility with upstream llama.cpp ... is not implied.\"\n- Community consensus on feasibility (lingyezhixing/PmNz8): random 4K reads; a top-tier PCIe 4.0 SSD ~50k+ tokens/s query rate; single-user inference doesn't need more; page cache helps a lot.\n\nOptane specifically: Optane P5801X/P4801X is the right class of drive — ~10-20 µs random 4K read latency at depth vs ~70-100 µs for TLC NVMe, and ~550-1000k IOPS. If rows are ~5.5KB (Q5_1 of 2560 elems ≈ 1.76 KB; 51.2e9/... let me compute row size: table = 20,000,000 ngram vocab... wait `ngram_vocab_size_base: 20000000`, `ple_embed_dim: 2560`. 20M × 2560 = 51.2e9. ✓ Row = 2560 elements. At Q5_1 (5.5 bpw) = 1760 B/row + block overhead. Hmm GGUF Q5_1 block is 32 columns → row of 2560 = 80 blocks × 144 B = 11520 B. Hmm, quantized layout is column-major blocked so a \"row\" read touches 2560/32 = 80 blocks of 144 bytes = 11,520 bytes, spread contiguously across 11.5 KB — that's ~3 contiguous 4K pages. Good: so per-token SSD reads are a handful of ~4KB page reads at semi-random offsets. With ngram_size 3 and `heads_per_ngram 8` and `split_ngram_parts 128`... per-token lookups maybe ~4-12 rows. So ~50-150 KB/token random → at 30 tok/s = ~4.5 MB/s but IOPS-bound: ~12-40 page reads/token = ~1200 IOPS/token×s. Trivial for Optane. So Optane would work extremely well.\n- Also Optane as a RAM tier: you could put the whole table on Optane and mmap it → behaves like slow RAM with the page cache holding hot rows. Their box: 3945WX/WRX80 has plenty of PCIe lanes free (128 lanes, 4×16 = 64 used).\n- Practical recommendation today: 96 GB of RAM? They didn't say. Ask. If ≥96 GB: keep PLE in RAM (mmap, warm the page cache, and disable swap / `vm.swappiness=1`). If they want bigger quants (UD-Q4_K_XL 111.3 GB, or Q5/Q6 backbone like Baekpica's MQ-Q6), then put the PLE on Optane.\n\nAlso: **`--load-mode mlock` / `mmap+mlock`** — `mmap+mlock` would pin the PLE in RAM (no eviction) which is what you want if RAM is ample, to avoid page-cache eviction stalls mid-session. But it pins the WHOLE file (87 GiB) not just the PLE. Hmm. So `-ot per_layer_token_embd=CPU` + `--load-mode mmap` (page-cache, evictable) vs `--load-mode mlock` (all resident, no mmap). Given the model is 87 GiB and they likely have 128–256 GB, `mlock` or `none` both avoid swap; `none` is actually a defensible choice — it reads everything into private anonymous memory (no page cache double-count). Their `--load-mode none` is not wrong; it just costs RAM equal to the full file and a slow-ish load. If they have ≥128 GB, `none` is the most predictable. Good — I should validate their choice conditionally rather than \"fix\" it.\n\nWait — one more: does `--load-mode none` break the PLE-to-CPU offload benefit? No, but it means you can't benefit from the page cache being shared across restarts (llama.cpp's mmap gives instant restart if pages are warm; with `none` you re-read 93 GB every start). On NVMe that's ~1-2 min. Fine.\n\n**CPU RAM bandwidth note**: 2133 MT/s quad-channel = 68.3 GB/s theoretical, ~50-55 GB/s achievable. If they can push the DIMMs to 3200 (WRX80 supports DDR4-3200, and 8 DIMMs at 3200 is often limited to 2933), that's 68 GB/s → biggest single lever if any CPU work matters. But since only the PLE gather + GDN host bookkeeping is on CPU, bandwidth is mostly latency-bound → 2133 is tolerable. Mention it as a real but secondary lever, and note that 8 DIMMs at 2133 on WRX80 is leaving ~15-20% memory bandwidth on the table vs 3200. Also: Zen2 Infinity Fabric on Threadripper PRO — FCLK 1:1 with MEMCLK up to 1600 typically; 2133 → FCLK 1066.\n\nActually — I should double check 3945WX: 12-core Threadripper PRO 3945WX, WRX80, 4-channel DDR4-3200, 128 lanes PCIe 4.0. Yes. AVX-512: Zen2 does not implement AVX-512. Correct.\n\n**Other things to flag in their block:**\n- `--fit off` vs fit on: discussed.\n- `--parallel 1`: MUST keep (multi-slot indexer assert). Do not raise. Consequence: only one Hermes session at a time; if Hermes' `delegate_task` spawns parallel children or a gateway + CLI concurrently, requests queue/defer. Hermes-specific mitigation: run subagents serially, or set Hermes `max_concurrent` / use queueing. Also `--no-cont-batching`? No — keep cont batching (irrelevant at np=1).\n- `--cache-prompt` redundant (default on) but harmless/explicit ✓.\n- `--no-context-shift` redundant (default disabled) ✓ harmless; important it's OFF given QSA + hybrid cache; keep it.\n- `--override-tensor \"per_layer_token_embd=CPU,ple_ngram_embd=CPU\"` → `per_layer_token_embd` is correct; `ple_ngram_embd` is almost certainly nonexistent → drop it or it's a silent no-op. And note the comma syntax is correct. Verify with `-lv 4` / `-v` and grep \"overridden to\".\n- `--n-gpu-layers all` — valid value ('auto' or 'all'). ✓ With fit on it may be reduced.\n- `--split-mode layer` ✓ default, keep; don't use `tensor` (fit unsupported + no P2P on 3090 without NVLink, and NCCL allreduce over PCIe would hurt for a 6B-active MoE).\n  Actually interesting: the VeroFess fork used tensor split + P2P + NCCL successfully on 2×3090 with Resizable BAR + peer access. But that's a custom fork for Qwen3.8-27B. Not applicable.\n- `--main-gpu 0`: with `layer` split, `--main-gpu` is documented as for `row` split; harmless. Consider `-mg 3` — no, keep 0. Actually with layer split, card 0 holds output head + logits + sampling; keep in mind.\n- CUDA_DEVICE_ORDER=PCI_BUS_ID ✓ good.\n- Missing `CUDA_VISIBLE_ORDER`... fine.\n- Add `--poll`, `--prio`? Optional.\n- Consider `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1`? Not needed.\n- `--flash-attn on` ✓ required for any quantized KV; with f16 KV it's still beneficial and required for the QSA top-k mask path most likely. Keep on.\n- `--perf` flag off is fine.\n- `--defrag-thold` deprecated, they don't use it ✓.\n- Model path: `Qwen3.8-Flash-Next-GGFF` (typo'd dir name, matches unsloth's own doc typo) — just note it's fine but they should confirm all 3 shards are present in that dir, since llama.cpp resolves shards by sibling-file naming. Their `--model` points at `-00001-of-00003.gguf`. That works if all 3 are in the same dir.\n- `set -euo pipefail` + exec: add `exec` so signals propagate (systemd stop). And add `--log-file` or capture to journald; add `--no-warmup`? No.\n- Suggest `--api-key` stronger + note the port-change warning.\n\nLet me also reconsider whether to bump ctx to 262144. Hermes \"long ranging tasks\". KV at f16: 24 KiB/token → 262144 × 24 KiB = 6.0 GiB. With PLE off GPU, GPU holds ~51.4 GiB weights + 6 GiB KV + compute buffers. 93.6 - 51.4 = 42 GiB headroom. Wait that's a LOT of headroom. Hmm, that suggests even with the PLE on GPU there'd be ~6 GiB headroom, and moving it to CPU gives ~27-32 GiB. So context is not the binding constraint at all.\n\nHold on — if PLE is on CPU, GPU only needs 51 GiB of 93.6 GiB. That means they're wasting ~30 GiB of GPU. Two options:\n(a) Move PLE back to GPU and use the freed... no, you can't use VRAM for anything but KV/compute, and KV is cheap. So the extra VRAM is genuinely idle.\n(b) Better: run a **bigger quant** — UD-Q4_K_XL (111.3 GB ≈ 103.6 GiB). With PLE on CPU: main backbone ≈ 103.6 - 35.8(PLE at Q5_1... but Q4_K_XL may keep PLE at Q6/Q8) ... Let's estimate: UD-Q4_K_XL total 111.3 GB. If PLE is ~38.4 GB (Q5_1) → backbone 72.9 GB = 67.9 GiB. GPU total usable 93.6 GiB → fits with 25 GiB spare for KV+compute. And UD-Q4_K_XL KLD 0.0447 vs IQ4_XS 0.0792, same-top 93.5% vs 91.1%. That's a **meaningful quality upgrade** for agentic work, and their box can likely take it!\n\nThat's a genuinely valuable recommendation: their 4×3090 + PLE-to-CPU setup is under-utilized at IQ4_XS; they should test UD-Q4_K_XL. Their CPU RAM becomes the new constraint (~36-38 GiB for PLE + OS + page cache + KV spills) → need ≥64 GB, comfortable at 128 GB.\n\nAnd the Optane idea scales that further: SSD-backed BF16 PLE (95.4 GiB) + Q5/Q6 backbone (91 GiB) = Baekpica's MQ-Q6 recipe, i.e., \"6.07 bpw backbone\" instead of 3.4.\n\nI should also flag: UD-Q4_K_XL ships as 3 shards too? The repo listing didn't show. Also IQ4_XS KLD is 91.09% same-top vs Q4_K_XL 93.48% — for tool-calling, top-1 agreement matters a lot; that ~2.4pp gap is the difference between reliable and flaky tool call formatting.\n\nLet me also reconsider `-b 4096 -ub 2048` → for QSA the indexer operates on blocks of 4 and budget 2048; the PR said \"QSA is dense by construction under indexer_top_k + compress_ratio - 1 cached tokens (2051)\". So below ~2K cached tokens it's dense. Their `--ubatch-size 2048` aligns... coincidence. Reduce ubatch for VRAM, and `-b` should be a multiple of `-ub`.\n\nLet me now also double-check the claim that KV cache is only 24 KiB/token. num_key_value_heads=2, head_dim=256 → 512 elements per K per layer. K+V = 1024 elements. f16 = 2048 B. Only 12 full-attention layers of 48 → 24,576 B/token. Yes 24 KiB/token. At 131072 → 3.0 GiB. Confirmed. (For comparison a dense 48-layer model with the same shape would be 98 KiB/token.)\n\nBut wait — does llama.cpp allocate a full-size KV for the linear_attention layers too? No — llama_memory_hybrid allocates attention cache only for the full-attention layers, plus a separate recurrent/SSM state per GDN layer. The GDN recurrent state: linear_num_key_heads 16, linear_key_head_dim 128, linear_num_value_heads 48, linear_value_head_dim 128. Recurrent state ≈ n_v_heads × d_k × d_v = 48 × 128 × 128 = 786,432 elements per layer. × 36 layers = 28.3M elements. In f32 (mamba_ssm_dtype float32) = 113 MB per slot! Plus conv state: kernel 4 × (d_conv_dim) where conv dim ~ key heads × key dim ×2 + value part ≈ ... let's say ~12,288 × 4 ≈ 49k elements/layer → 7 MB total. So ~120 MB per resident sequence. Not huge, but it means `--ctx-checkpoints` of 32 could be 32 × 120 MB = 3.8 GB!! That's a real concern. VeroFess capped at \"two exact recurrent checkpoints per resident slot with a 2048 MiB host cache budget\". So default `-ctxcp 32` × ~120 MB = ~3.8 GB of host (or device) memory for checkpoints. I should recommend `--ctx-checkpoints 4` (or 2-8) as a concrete memory/stability lever, and `-cram` to bound it.\n\nHmm, careful — I'm not 100% sure checkpoints are made eagerly (they're made on demand when sequences are evicted/reused). But it's clearly worth bounding. I'll present it as \"budget this; llama.cpp creates up to 32 checkpoints/slot by default and each has to snapshot the GDN recurrent state (~110 MB/seq/layer-set)\". Mark as estimate with the arithmetic shown, and tell them to verify with `-lv 4`.\n\nAlso `m_ssm_dtype: float32` and the delta-net state on the GPU: `graph splits = 2` phenomenon (CPU bookkeeping for GDN). On Zen2 (no AVX-512/VNNI) this is the insiderllm-documented bottleneck → so `-t 12` matters, and if they see `graph splits` in the log with a per-token CPU cost, that's why decode might sit below what the 3090s could do.\n\nAlso worth mentioning: **CUDA graphs / `-cq`**? The server README doesn't list `--cuda-graphs` in this build (BeeLlama gist mentions CUDA graphs conflicting with MTP). Not listed → skip, don't invent. Actually there's `GGML_CUDA_GRAPH=1` env in some builds. I won't claim it since it's absent from the docs I read. I'll avoid.\n\nLet me also decide on the final config. I'll produce:\n1. A verdict/summary table of what's wrong in their block.\n2. Memory math.\n3. The corrected runblock (annotated).\n4. A \"test matrix\" of things to A/B.\n5. The ngram/Optane section.\n6. Open questions for them.\n\nLet me compute a couple more numbers for the write-up:\n- 4×3090 = 96 GB nominal (89.4 GiB? no). 24 GiB × 4 = 96 GiB = 103 GB decimal. Usable ~23.5 GiB × 4 = 94 GiB = 101 GB decimal.\n- Model 93.6 GB decimal = 87.2 GiB.\n- PLE on GPU leaves ~7 GiB → too tight once you add 4 CUDA contexts (~0.4 GiB each = 1.6), compute buffers at ub 2048 with 4× hyper-connection widening (multiple GiB), KV 3 GiB. → Confirms PLE-to-CPU is necessary, not optional. Good.\n- With PLE on CPU (~36 GiB at Q5_1, or ~26 GiB if truly 4-bit): GPU weights ≈ 51–61 GiB → ~13–15 GiB/card. Card has 23.5 → ~8-10 GiB/card free = 32-40 GiB. KV needs 3 GiB (131K) or 6 GiB (262K). So enormous headroom → run UD-Q4_K_XL.\n\nAlso `--tensor-split 1,1,1,1`: with layer split, llama.cpp assigns whole layers; tensor-split weights the distribution. Since the PLE is pulled out, the remaining 128.8B across 48 layers + output head is fairly uniform → 1,1,1,1 fine. But card 0 also gets the token embedding and output head and `output`. Actually with `--main-gpu 0` and layer split, the output head goes to... Let me not over-claim; suggest checking `llama.cpp` log \"CUDA0 model buffer size\" per card and re-weighting if one card is hot (e.g., `1.05,1,1,0.95`). That's the standard trick.\n\nAlso note: they should NOT use `--cpu-moe` (would put 512-expert weights on the slow 2133 MT/s memory — disastrous here since it fits).\n\nAnd `--no-host`: warn against.\nAnd `--op-offload`: keep default.\n\nOne more Hermes-specific: Hermes is \"OpenAI-compatible\"; llama.cpp also has an Anthropic-compatible `/v1/messages` endpoint — useful if they point Claude-Code-style harnesses at it. And Hermes supports `/v1/responses`. Fine.\n\nHermes-specific config: they'd set `provider: custom`/`openai` base_url `http://192.168.1.61:9876/v1`, model `qwen3.8-flash-next`. Key: `nerdtastic`. Also: because `--parallel 1`, set Hermes to avoid parallel tool/agent fanout. And Hermes' context management: set its max context so it compacts before llama.cpp hard-truncates — leave ~10% headroom below 131072 (e.g., Hermes context limit ~120K). Because llama.cpp truncating mid-agent-loop is one of the Hermes \"my agent feels dumber\" failure modes (their docs literally have a troubleshooting page on \"context pressure, wrong context detection\"). Good tie-in: Hermes reads the model's max context; if llama.cpp reports 131072 as n_ctx but the harness also counts its own overhead, set explicitly.\n\nAlso worth noting: `--slots` endpoint to monitor, and `/metrics` for Prometheus, plus the `timings` object → `cache_n` to verify prompt caching is actually hitting. That's how they'll know whether the Hermes integration is efficient. Very actionable: `timings.cache_n / prompt_n` should be high (>80%) for a well-behaved agent loop.\n\nLet me write it up. Keep it organized and scannable but comprehensive (they asked for comprehensive and I have long ctx). Use headers, tables, code blocks.\n\nI should be honest about uncertainty: exact PLE tensor size, whether MTP weights are in the file, whether the user's build has the fixes. Give commands to verify each.\n\nOne more: the user wrote `--model ... /unsloth/Qwen3.8-Flash-Next-GGFF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf`. Note \"GGFF\" — matches unsloth's own doc `--local-dir unsloth/Qwen3.8-Flash-Next-GGFF`. Ha. Fine, and it confirms they followed unsloth's doc verbatim (including the typo) — worth a light note. Actually — that's a useful signal: it means they copied unsloth's `hf download ... --local-dir unsloth/Qwen3.8-Flash-Next-GGFF` command. Which also means: they may have only downloaded the IQ4_XS dir. Fine.\n\nVerification command ideas:\n- `llama-llama-server --version`? use `--version`.\n- Grep their build for the fix: `git -C /root/qwen3-next/llama.cpp log --oneline | grep -iE 'fit|qwen4exp'` and check for the three commits: `7044fe16fcb334164d03d20108ffac0b504353e1` (KV hadamard), `c30f6685ddde935a1fa90868343dfd59ba7e52aa` (--fit), `d885ad824b204ecddcbdeffe852edd91bf9d9fbd` (quantize low-mem). And `git merge-base --is-ancestor <sha> HEAD`.\n- Tensor name discovery: `llama-server -m ... -ot '.*=CPU' -lv 4 2>&1 | grep -oE 'tensor [^ ]+'` — too heavy (loads everything to CPU). Better: `gguf-dump` or `llama-quant info`. There's `gguf-py`: `python -m gguf_dump` or `gguf-cli`. Simplest robust: `python -c \"from gguf import GGUFReader; ...\"` on shard 1, print tensor names > 1 GB. I'll give a small snippet using `gguf-py` from the llama.cpp repo, noting metadata lives in shard 00001.\n\nThat's a great actionable verification for the `ple_ngram_embd` question. Let me include:\n```\npython3 - <<'PY'\nfrom gguf import GGUFReader   # pip install gguf  (or use llama.cpp's gguf-py)\nimport sys\nr = GGUFReader(\"/root/qwen3-next/unsloth/Qwen3.8-Flash-Next-GGFF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf\")\ntot=0\nfor t in r.tensors.values():\n    b = 1\n    for d in t.shape: b*=int(d)\n    b *= (t.dtype.itemsize if hasattr(t.dtype,'itemsize') else 2)\n    if b > 2**30: print(f\"{b/2**30:8.2f} GiB  {t.name}\")\n    tot+=b\nprint(f\"total {tot/2**30:.1f} GiB\")\nPY\n```\nCareful — GGUFReader on a shard: shape/dtype are in metadata so fine; byte estimate approximate for quantized types (itemsize wrong for block quants). Better to just print names + shape + dtype and let them eyeball, and note that block-quant sizes need n_elems × bpw/8. I'll keep it simple: print the big tensor names and note that sizes are indicative. Actually GGUFReader does expose `t.n_bytes` maybe. I'll write it defensively: print name, shape, dtype, and n_elems. And mention `-v` log grep as the ground truth.\n\nAlright, and note: llama.cpp ships `gguf-py` at `<repo>/gguf-py`, so `PYTHONPATH=/root/qwen3-next/llama.cpp/gguf-py python3 ...`.\n\nNow write.\n\nFor the final config I need to be careful to only use flags that exist. Let me verify each flag I use against the README I fetched:\n- --model ✓, --alias ✓, --host ✓, --port ✓, --api-key ✓, --jinja ✓\n- --flash-attn on ✓ (auto|on|off)\n- --fit on ✓, --fit-target ✓, --fit-ctx ✓\n- --n-gpu-layers all ✓ ('auto' or 'all')\n- --split-mode layer ✓\n- --tensor-split ✓\n- --main-gpu ✓\n- --override-tensor ✓\n- --load-mode {auto,none,mmap,mlock,mmap+mlock,dio} ✓\n- --ctx-size ✓, --parallel ✓, --batch-size ✓, --ubatch-size ✓\n- --cache-type-k/v: f16 ✓ (and note kvarn6 NOT in the allowed list of stock master — the BeeLlama gist is a fork; don't recommend kvarn6 for stock. Good catch — I'll explicitly warn that `kvarn6`/`--kv-tail-tokens` seen in other guides are fork-only, not in stock master's allowed list.)\n- --cache-prompt ✓\n- --no-context-shift ✓ (--context-shift/--no-context-shift)\n- --threads ✓, --threads-batch ✓, --threads-http ✓\n- --reasoning on ✓, --reasoning-effort ✓, --reasoning-preserve ✓, --reasoning-budget ✓\n- --chat-template-kwargs ✓\n- --temp/--top-p/--top-k/--min-p/--presence-penalty/--repeat-penalty ✓\n- --metrics ✓, --slots ✓ (default enabled)\n- --keep ✓\n- --cache-ram ✓\n- --cache-reuse ✓\n- --slot-prompt-similarity ✓\n- --ctx-checkpoints ✓, --checkpoint-min-step ✓\n- --cache-idle-slots ✓\n- --mmproj ✓\n- --image-max-tokens ✓\n- --cpu-strict ✓, --prio ✓, --poll ✓\n- --timeout ✓\n- --warmup ✓\n- --check-tensors ✓\n- --spec-type ✓, --spec-draft-n-max ✓\n- --log-prefix / --log-timestamps ✓\n- --no-ui / --ui off: `--ui, --webui, --no-ui, --no-webui` ✓\n- --no-jinja (no), --no-mmproj ✓\n- -lv ✓\n- --force-params? not present, skip.\n\nGood. All verified.\n\nNote `--reasoning-effort` and `--reasoning on` are both present. Unsloth recommends xhigh default. For Hermes long-horizon, xhigh is probably right but slow; Qwen warns lower effort isn't always faster overall. I'll recommend leaving xhigh as default and exposing per-request control.\n\nSampling: they left server defaults as thinking-mode which is right for Hermes (Hermes may pass its own params per request though — Hermes config can set temperature). Note: any Hermes request that specifies temperature will override the server default. So they should set it in Hermes' model config too, or the server defaults won't apply. That's a valuable practical gotcha.\n\nLength: This needs to be comprehensive but let me keep it tight enough to be usable. I'll use tables and code.\n\nLet me write the final answer.\n\nStructure:\n1. ✅ Model verified (with links)\n2. The 3 landmines in your block (the important part)\n3. Memory math for your box\n4. Corrected runblock (annotated)\n5. Hermes-specific tuning\n6. Things to A/B + how to measure\n7. Ngram → RAM/Optane/SSD status\n8. What I need from you\n\nLet me be careful with the claim about LLAMA_ATTN_ROT_DISABLE cost — it was measured on IQ1_S by jamespa16: \"disabling rotation via LLAMA_ATTN_ROT_DISABLE=1 costs ~15% on the chunk where the sparse path is most active\". So it's ~15% perplexity on the sparse path, one tester, IQ1_S. I'll qualify.\n\nAlso note: if they switch to f16 KV, they no longer need LLAMA_ATTN_ROT_DISABLE — but I should double check: is the env var needed for ANY other reason? The assert fires only when the cache is quantized. With f16, self_k_rot/self_v_rot are null → no assert → no env var needed. ✓\n\nLet me write it.\n","timings":{"agentic":{"llm":{"predicted_ms":1212948.124,"predicted_n":24529,"prompt_ms":507272.075,"prompt_n":106548},"perTurn":[{"llm":{"predicted_ms":7800.728,"predicted_n":346,"prompt_ms":4815.282,"prompt_n":4610},"toolCalls":[{"duration_ms":66,"name":"hf_whoami","success":true},{"duration_ms":87,"name":"hub_repo_search","success":true},{"duration_ms":272,"name":"web_fetch_exa","success":true}],"toolsMs":425,"turn":1},{"llm":{"predicted_ms":56661.296,"predicted_n":2025,"prompt_ms":25561.565,"prompt_n":14969},"toolCalls":[{"duration_ms":182,"name":"hub_repo_details","success":true},{"duration_ms":187,"name":"hf_fs","success":true},{"duration_ms":314,"name":"web_fetch_exa","success":true}],"toolsMs":683,"turn":2},{"llm":{"predicted_ms":205822.536,"predicted_n":6038,"prompt_ms":43443.077,"prompt_n":15758},"toolCalls":[{"duration_ms":522,"name":"hf_fs","success":true},{"duration_ms":999,"name":"web_search_exa","success":true}],"toolsMs":1521,"turn":3},{"llm":{"predicted_ms":65763.511,"predicted_n":1497,"prompt_ms":128858.13,"prompt_n":28044},"toolCalls":[{"duration_ms":435,"name":"web_fetch_exa","success":true},{"duration_ms":135,"name":"hub_repo_search","success":true}],"toolsMs":570,"turn":4},{"llm":{"predicted_ms":54503.748,"predicted_n":1009,"prompt_ms":200638.664,"prompt_n":30391},"toolCalls":[{"duration_ms":290,"name":"hf_fs","success":true},{"duration_ms":1148,"name":"web_search_exa","success":true}],"toolsMs":1438,"turn":5},{"llm":{"predicted_ms":822396.305,"predicted_n":13614,"prompt_ms":103955.357,"prompt_n":12776},"toolCalls":[],"toolsMs":0,"turn":6}],"toolCalls":[{"duration_ms":66,"name":"hf_whoami","success":true},{"duration_ms":87,"name":"hub_repo_search","success":true},{"duration_ms":272,"name":"web_fetch_exa","success":true},{"duration_ms":182,"name":"hub_repo_details","success":true},{"duration_ms":187,"name":"hf_fs","success":true},{"duration_ms":314,"name":"web_fetch_exa","success":true},{"duration_ms":522,"name":"hf_fs","success":true},{"duration_ms":999,"name":"web_search_exa","success":true},{"duration_ms":435,"name":"web_fetch_exa","success":true},{"duration_ms":135,"name":"hub_repo_search","success":true},{"duration_ms":290,"name":"hf_fs","success":true},{"duration_ms":1148,"name":"web_search_exa","success":true}],"toolCallsCount":12,"toolsMs":4637,"turns":6},"cache_n":104682,"predicted_ms":822396.305,"predicted_n":13614,"prompt_ms":103955.357,"prompt_n":12776}},"type":"message"}