Qwen3.8-Flash-Next Local Inference Tuning Recipe

Qwen3.8-Flash-Next local setup with RTX 3090 x2 (48GB VRAM) + 128GB system RAM: 1. Quantization: Choose Q4_K_XL or UD-IQ1_S depending on memory vs quality tradeoff. 2. KV cache: Use q8_0 KV for balanced memory and decode performance at larger contexts. 3. MoE layers: Use --n-cpu-moe and --moe-stream to offload MoE workload to system RAM. 4. Layer splitting: Leave several GB VRAM headroom to avoid page-faults during inference. 5. Batching: Set --batch-size and --ubatch-size to 2048-4096 for prefill throughput. 6. NUMA: Bind workload properly to reduce CPU-GPU memory traffic. 7. Environment: Prefer Linux or WSL2 for best performance. Expected: 300-600 tok/s prefill, 40-50 tok/s decode.