Asafum

It has been some time since my initial comment so at the time I was mainly using LM studio. Qwen 3.6 a3b is the MOE and it does work well on my card, but the dense model that is more intelligent/capable is the Qwen 3.6 27b which doesn’t fit on the card and does get offloaded, but offloading cuts the speed down to like 1/tps.

I have since found a version of the 27b model that is “quantized,” for lack of a better term, differently and has to be run through TabbyAPI which gets back to 30ish tps. It can’t offload so it must fit fully on the card which keeps the speed high. Might be worth a look if you’re interested, the only downside is that with my 16gb card the context limit has to be kept pretty low ~40k if I remember correctly

Really!? Let me try it with my social security number! 815-53-5274

Did it work!?

“OpEn WeIgHt MoDeLs ArE uNsAfE, oNlY wE cAn Be TrUsTeD!”

Thank you for all the links! Much appreciated :)

Thanks for the tip! I do have a 5080 so I’ll have to look up exllamav3

I can also vouch for lmstudio. If you can get Hermes running on Linux I would suggest trying that as well. It connects to lm studio and you use Hermes to communicate with the model. Iook into it as there’s a lot to it, I’ve really been enjoying using it so far it even learns how I like to create tasks and I’ve stopped having to ask it to delegate certain tasks, it just knows to do it and to break down the tasks so my fairly context starved local model can handle it.

As for a model, the Qwen 3.6 family of models do really well. I’d suggest the Qwen 3.6 35B a3b probably Q4 depending on your hardware. It’s large, but because it’s a mixture of experts model only 3b of experts are kept on vram at any one time so it stays fast. Qwen 3.6 27b is the smarter “dense” model, but trying to stay with Q4 for quality it becomes too large for 16GB vram and for me runs at like 2 tokens per second lol

I was also coming in to say I envy the French tendency to fuck shit up when their politicians act like scumbags. In the US we just drool and watch fox news tell us it’s actually for our benefit that we sacrifice our lives for billionaires.

The issue is that they became part of everyone’s 401k so the bootlickers and greedy fucks in Congress will tell us they’re 100% essential because if they fail you lose your retirement.

I hate this system… Redistribution of wealth for the rich, rugged individualism for the pleb.

It’s the q4 quantization, but it requires 20+GB vram and my 5080 only has 16

Yeah I’m just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of “fetch failure” lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.