abcdqfr@abcdqfr@lemmy.world
edited

Replying to @⁨Chee_Koala@lemmy.world⁩

There is a way. There was a post yesterday on exactly this, let me find it… lemmy.world/post/51283416

lemmy.worldQwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU - Lemmy.World* Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller [https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller] is a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget. * Jinja chat template [https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates] helps use fewer thinking tokens without noticeably affecting quality, which is great for speed. * And the final ingridient is beellama.cpp [https://github.com/Anbeeld/beellama.cpp] engine which supports the kvarn KV cache types needed for this optimization. Here’s a command to start the server, the magic is in the kvarn cache settings and the tail precision: bash "$LLAMA_DIR"/llama-server \ -m "$MODEL_PATH" \ -a "$MODEL_NAME" \ --port 11434 \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --parallel 1 \ --n-gpu-layers 99 \ --batch-size 1024 \ --ubatch-size 256 \ --flash-attn on \ --spec-type draft-mtp \ --spec-