floofloof@floofloof@lemmy.ca
edited

Replying to @⁨Darkaga@lemmy.world⁩

There just aren’t any good deals any more. Prices for everything have gone crazy in the last few months. For coding LLMs the cloud services may now be the least worst value, by design, until they hike the prices.

That said, I still just paid way too much for a used graphics card so I could do many things locally, because I just don’t want to give the likes of Sam Altman a single penny.

deleted@deleted@lemmy.world
edited

Replying to @⁨Chee_Koala@lemmy.world⁩

For your hardware, the VRam is not enough to run 27b but, I’d recommend Qwen 3.5 9b for image / text to text.

And I’m planning to experiment with Qwen 3.8 9b for text to text.

4_k_m quantization is the sweet spot for performance and ram usage.

Also, I find Llama cpp is better than Ollama in terms of performance.

huggingface.coDavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
abcdqfr@abcdqfr@lemmy.world
edited

Replying to @⁨Chee_Koala@lemmy.world⁩

There is a way. There was a post yesterday on exactly this, let me find it… lemmy.world/post/51283416

lemmy.worldQwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU - Lemmy.World* Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller [https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller] is a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget. * Jinja chat template [https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates] helps use fewer thinking tokens without noticeably affecting quality, which is great for speed. * And the final ingridient is beellama.cpp [https://github.com/Anbeeld/beellama.cpp] engine which supports the kvarn KV cache types needed for this optimization. Here’s a command to start the server, the magic is in the kvarn cache settings and the tail precision: bash "$LLAMA_DIR"/llama-server \ -m "$MODEL_PATH" \ -a "$MODEL_NAME" \ --port 11434 \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --parallel 1 \ --n-gpu-layers 99 \ --batch-size 1024 \ --ubatch-size 256 \ --flash-attn on \ --spec-type draft-mtp \ --spec-