Qwen3.8 Flash Next Notes

When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network. As an Amazon Associate I earn from qualifying purchases. #ad #promotions

Qwen 3.8 Flash Next Local Ai Guide

Qwen is like the gift that just keeps giving, and this time we are getting a new technology preview wrapped up Qwen 3.8 Flash Next which is capable of running surprisingly well in a Q4 thanks to the new N-GRAM offloading capability that sends this MoE’s performance up a step. Checkout the video if you have not yet we will be doing another now that I have this LLM ready for my hermes agent harness which will look at the quality of the model in deeper detail. It is running the runblock enhancer prompt now and I decided to test it out at 256K ctx to just see what that performance might look like and hopefully it can finish. Will update this article when I get some additional results on that also or good comments on the video with good tips and tricks!

Here is a decent starting point for a runblock that can fit in a 64GB RAM + Quad 24GB GPU VRAM/RAM footprint and stay with ya at least out to 128K context depth. Once you get this tuned up, drop your improvements to the videos comments section! Check below for some notes on things you want to check out and probably disable once there is llama.cpp main support in a day or so.

Qwen 3.8 Flash Next Llama.cpp runblock

Copy Runblock 


#!/bin/bash
set -euo pipefail

export CUDA_VISIBLE_DEVICES=0,1,2,3
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export LLAMA_ATTN_ROT_DISABLE=1

MODEL=/root/qwen3-next/unsloth/Qwen3.8-Flash-Next-GGFF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-I>

/root/qwen3-next/llama.cpp/build/bin/llama-server \
  --model "$MODEL" \
  --alias qwen3.8-flash-next \
  --host 192.168.1.61 \
  --port 9876 \
  --api-key nerdtastic \
  --jinja \
  --flash-attn on \
  --fit off \
  --n-gpu-layers all \
  --split-mode layer \
  --tensor-split 1,1,1,1 \
  --main-gpu 0 \
  --override-tensor "per_layer_token_embd=CPU,ple_ngram_embd=CPU" \
  --load-mode none \
  --ctx-size 131072 \
  --parallel 1 \
  --batch-size 4096 \
  --ubatch-size 2048 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cache-prompt \
  --no-context-shift \
  --threads 6 \
  --threads-batch 12 \
  --threads-http 4 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --metrics

Llama.cpp Optimization Trace

Qwen LOVES to overthink and if you watched the video you know how that ended up, sooo close. Here is a link to that big research trace you can feed to your bot.

Runblock Optimization Block

I have been using a variation of this script to get some pretty good insights into the various engine flags for a while and I think you might want to give this a try also. This has worked very well for a few recent models and does especially help when there is a day-0 launch that just has something new or different and you cant get it running right. Do update the docs links of course to be relevant to whatever LLM and engine it is your are trying to get your local ai rig running optimally. No need to hunt down recipes, your bot is pretty good at it for you!

Copy Prompt 


Hi you are qwen3.8-next-flash running as an unsloth UD-IQ4_XS quant go and checkout info on that around huggingface, unsloths writeup here https://unsloth.ai/docs/models/qwen3.8-next (You MUST check that this model exists before you start reasoning about flags and configs, you have a prior cutoff date before it was launched. Do that FIRST)

Then check llama.cpp which is your engine here https://github.com/ggml-org/llama.cpp

Lets see if you can optimize my runblock and especially PP for hermes agent https://hermes-agent.nousresearch.com/docs/assets/files/llms-7240021af84660c2a79f9fdaf65e9e8f.txt and here is my current runblock:

XXX_INSERT_YOUR_RUNBLOCK_HERE_XXX

This is your current runblock and I am no llama.cpp expert I ususally use vllm so I need help tuning this up! I do expect there to be likely some offloading as well potentially, that might not have a huge impact due to your n-gram additional hash table. I am running on quad 3090s on a 3945wx (12c/24t) with 8 dimms 2133 off a wrx80 mobo that has 128 lanes of pcie and all GPUs are negotiated on that full gen4 x16 wide. The work we are optimizing for is the hermes agentic harness to run smoothly with this configuration for long ranging tasks. That is the currently running runblock also so you have a large ctx so please be comprehensive. I am adding this note the UD-IQ4_XS is 93.7GB. Llama.cpp is going to be using fit on (the default) Please check into ngram offloading to RAM/OPTANE as an option to run larger quants as well.
JOIN AS A YOUTUBE MEMBER

Table of Contents

Hardware Review

Local Ai Setup Guides