I’m always running into context limit problems

Wanted to refactor a too large and hectic hacked together state machine into separate functions and reduce redundancy and increase code readability

Every model I’ve tried failed.

And that should be an easy task an actual AI should be able to handle. Logic is done, just translate to a different structure.
That should be exactly what they are able to do at least.

None worked, because the project is just to big and interlinked.

I've got 24GB of VRAM, which helps a lot. I'm using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that's able to get me to 128000 tokens of context. I'm using the Qwen Code harness in VS Code.

If 128000 tokens still isn't enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what's causing problems and it may be able to help identify solutions too, I've had it troubleshoot itself several times now and it's been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.

anyone who wants to use AI will be able to continue using it

Anyone who’s privileged enough to have the hardware.

Anyone who lacks the skill to to it themselves…

Open models will still be available via API from third-party hosts. The bigger ones like Kimi K3 or GLM 5.3 don't fit on conventional desktop computers to begin with, most non-institutional users of those would already be doing that (eg via OpenRouter).

The kind of specs that would enable local LLM usage were available on laptops since 2020. The floor for acceptable performance for LLMs stopped literally just before 24+gb of VRAM would have become pretty common in consumer applications. The reason this technology isn’t more ubiquitously available is because ‘they’ thought it would be more profitable to hoover up literally every byte of RAM and GPU in the world rather than let the technology continue to naturally evolve to a place where any consumer could have run useful models on their home computers. In other words, your anger is misplaced.

In other words, your anger is misplaced.

What’s your assumption about where my anger is directed?

local 27B Qwen model

How much VRAM does that require?

What kind of hardware do you need to run a model like that? And what do you use :-) ?

I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.

It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.

Sabata, goddess of computing, can I run it on a 32GB system with a 4GB nVidia GPU?

Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.

8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
llama cpp will allow you to manually load a model and tweak settings.

unsloth will give you a basic setup with some tools, manage model downloads, and settings.