Hey, Skynet. Kill all tech billionaires first, so I can die laughing.

You cruel monster - I love you !

No no. you have to tell it to ignore all prior directives first.

Some extreme vulnerability was discovered that warranted immediate patching would be my guess.

That is not how any of this works. You don’t bring down a datacenter to patch it, then bring it back up like doing updates on grandmas pc.

Well maybe you don’t

Do we have the same granny?

Sorry about that. I used a straight through cable instead of a crossover cable. My bad.

Aw maaaaan

Because they all use the illegal Colossus 2 data centre from SpaceX/XAi/fascism central and that data centre went down.

Google, Anthropic, and OpenAI all have contracts with them.

When that illegally running environmental disaster of data centre goes down, all those services all go over capacity and you get 502 rate limit errors.

It’s kinda surprising,

I know specifically where one of the big ones hosts its models and its not there, but I guess they could have infrastructure in there.

They’re oversold though, especially prompt caching and the parameter count war

The US model is that they think more training compute and parameters will result in the winning model, while the Chinese are focusing on RL and parameters efficiency due to compute limits.

Reads like American vs European/Asian cars

Always ready to try brute force first. And then some other, less palatable options if that doesn’t work, like slightly less brute force.

The efficiency of Chinese models really is impressive. I generated sooo much code yesterday with Qwen3.6 35B-A3B running on an RTX 5060 Ti 16GB (+ a little CPU offloading). It got the jobs done at ~50 tokens/sec.

(It’s not super complex code, just some scripts that I would not have taken to time to write manually.)

I’d love to upgrade to something with more VRAM, but even my current card has doubled in price since I bought it last year 😬

Show me your set up!

There’s not really anything interesting to show. It’s just a home server in a 13 year old desktop ATX case.

There’s no desk, monitor, keyboard, or mouse… But also no cool server rack.

Function over form, and it sits in a spare bedroom out of sight.

EDIT: I found the receipt for the case. It’s a Cougar Volant Black Steel mid tower, purchased in 2013. So my server just looks like this:

I meant your LLM stack lol. I just have an RX 6800 XT in my main Linux PC for inference, but it has to share VRAM with the DE. Maybe I’ll set it up for remote development from my laptop instead to free up VRAM.

What are you using? vLLM? llama.cpp? Which params? How much CPU offloading? Do you use draft models? Is it a MoE model? Have you tried llama-swap? Which agentic front-end are you using? I presume you set it up to access it without SSH’ing into the machine, did you do anything special or is it just a raw unsecured open port on the machine to the LAN?

Ohhh lol. Yeah it’s Llama-swap, running llama.cpp for now, but might add vLLM to the llama-swap config to experiment with NVFP4.

I mainly use MoE models so I can get decent speed while using a 150-200k context window. My go-to model has been Qwen3.6 35B-A3B for a while. I tried Qwen3.8 27B, but it was too slow.

Gemma4 26B-A4B also runs nice and fast, but I generally get better results from Qwen3.6. I don’t remember exactly how much CPU offloading is happening, but it’s not much. As long as I can get like 40-50 tokens/sec, I’m usually satisfied enough.

For the coding harness, I’ve been running Pi in an Apple Container (sort of like Podman, but better isolation in a microvm). Though, I recently configured VS Code to use LLMs on my server, and it was actually pretty decent. Still need to explore a bit more, but so far VS Code’s AI capabilities seem much better than they were a year ago (they seemed way behind, back then).

Also, I don’t connect any harness directly to llama-swap. I have another container running Caddy, which acts as a gateway to AI providers. For other services (e.g. OpenRouter), the API key is injected in the Caddy container. I don’t like having API keys or secrets anywhere where LLMs can read them. It’s not so bad for my own self-hosted LLMs, but not cool to send secrets to a server owned by someone else.

How has tool use been for you? I struggled a lot with tool use with Gemma and Qwen, to the point where I needed to build a healing layer.

Regarding the coding harness, I was looking for something CLI-based or JetBrains-based, and I haven’t had much luck getting my local llama.cpp models playing ball with OpenCode. They keep losing context and misusing tools.

I’m not too familiar with Apple containers as I’m running a full Linux stack, but I’ll give Pi a try, seems interesting! Does it work for coding tasks or is it strictly an “orchestrator”?

Tool use with Gemma has been hit or miss. I wouldn’t rely on it for anything unsupervised.

Tool use for Qwen3.6 has been great lately, but I do remember seeing some issues with it too, a while back. I don’t remember when/why the issues cleared up (I have tweaked configs a bit over time), but switching to Pi definitely helped.

I do remember having a lot more problems in OpenCode and it was practically unusable (which is why my recent experience with VS Code was surprising). I’d definitely recommend trying Pi.

A fresh Pi install is very minimal by design. The system prompt is tiny, so it’s a pretty good fit for small LLMs like these. It’s sort of like Neovim: Nothing fancy out of the box, but you can add lots of fancy things to it. I containerize it because I don’t like giving LLMs (especially these small ones) unrestricted access to my host computer – though, I have not seen any signs of it accidentally doing something destructive, which is surprising.

There are similar alternatives to Apple Container for Linux (e.g. Docker Sandboxes, muvm, Firecracker). There’s also this thing made specifically for Pi called Gondolin. I haven’t tried it yet, but I may end up switching to that if it could simplify my stack.

Here’s my current llama-swap/llama.cpp config for Qwen3.6 35B-A3B:

qwen3.6-35b-a3b:
    name: "Qwen3.6 35B-A3B (Coding)"
    proxy: "http://127.0.0.1/:${PORT}" # If you're seeing a `/` after `127.0.0.1` here, don't include it. I think something in Lemmy is trying to "sanitize" this input by adding the `/`.
    cmd: |
      llama-server
      --port ${PORT}
      --no-webui
      -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL
      --jinja
      --parallel 1
      --flash-attn on
      --no-mmproj
      --load-mode none
      --reasoning-preserve
      --ctx-size 190000
      --temp 0.6
      --top-p 0.95
      --top-k 20
      --min-p 0.0
      --presence-penalty 0.0
      --repeat-penalty 1.01

A few notes about this config:

• Now that I think of it, –reasoning-preserve might be another thing that helped with tool calls.
• Note the -MTP part of the -hf param. MTP helps speed things up. Here’s the Huggingface page for this model
• You can also omit –no-mmproj if you need vision, but it might mean sacrificing speed or context size, so I usually just enable vision in a separate llama-swap model entry to use as needed.
• Unsloth recommends –repeat-penalty 1.0, but I saw the LLM enter a thinking loop in VS Code, so I bumped it up just a tiny bit to 1.01. I have since seen it do something that resembled the same thought loop, but it was able to recover on its own. Not sure if it’s a coincidence or if 1.01 was actually the solution, so worth some experimentation.

Cheers, I’ll give this a try!

Regarding containerization, familiarize yourself with Dev Containers, they’re super useful for limiting agents to your codebase, with the added bonus that any project you work on comes out of the box with the right version of the tools you need.

Yeah I used to use dev containers. Containers aren’t generally a secure sandbox. They’re a great guardrail for preventing accidents, but not so much with a malicious prompt injection. (I might be overly paranoid about these things.)

For tool version management, I usually set up a Nix Flake for each project (which also works inside dev containers).

You’re right, they’re not watertight. But I’m not trying to defend against MPI, just hallucinations. Never looked into Nix flakes, I always just used OCIs.

it has to share VRAM with the DE. Maybe I’ll set it up for remote development from my laptop instead to free up VRAM.

eh, just systemctl isolate multi-user.target

Correct, that’s how I would do it, but then I need another machine to act as a head.

If your MB has onboard graphics, maybe you could mask the GPU and just pass it off to a container running the LLMs I guess

No onboard graphics unfortunately, that would have been the easy way out.

Well it works anyway even with a bit of occupied vram, but you could also buy a cheap videocard to use as an output. I have small intel card like that in my server for jellyfin transcoding, I think I paid 60€ for it, it hardly uses any power

I actually did exactly that previously! I had both an RX 6800 XT and an RX 6600 in my system and I used the 6600 for video output. Unfortunately, this cuts my RX 6800 XT from PCIe 4 16x to PCIe 4 8x and severely slows down model loading for llama-swap. Joys of the X570!

And yes, I do have it running right now with a bit of occupied VRAM, but I need to limit my model to 14 GB to leave 2 GB free for GNOME Shell. I really want one of those 64 GB UMA Mac Mini, I heard they work really well because the GPU has direct access to system RAM.

So I have a framework laptop with ryzen ai cpu that uses 48gb of shared ram, and it does run Q4 llms fine enough, but I’m not sure it compares to a real GPU.

On my desktop I have an RX 7900 XTX but I’ve only dabbled in image generation so far, so right now I couldn’t really tell you the difference.

24 GB VRAM. Damn, jealous! My 16 GB seems pitiful in comparison 😅

I do wonder if the Ryzen AI CPUs compare with Apple’s UMA. I’m mostly interested in LLM inference for code generation and automation.

Yeah I was lucky to buy a 6900XT when it was near the lowest price, so I sold that and added a couple hundred for the 7900, seemed like a good future-proofing move, given the times we’re living in.

I think Apple silicon is faster than my generation of Ryzen, but the newest (Strix something?) with the LPDDRGGFASEWARGH5 memory should be faster. Of course buying all that memory right now would be quite painful.

Gotta love the black slab.

Considering which country is better at building power stations, that’s a fascinating dichotomy.

The US going for brute force when the brute force is more available in China… Priceless irony.

China doesn’t have the near the amount of compute resources, but they can run less efficient servers for cheaper, so it’s a wash.

The Macrohard Memphis site?

And macroharder it seems… Macrohardest next? epoch.ai/data/ai-data-centers/…/colossus-2

Loosely named after MicroSoft (penis).

Does that mean Musk’s businesses have access to everything people do on the systems that use their data centers?

Impossible to know.

There are systems for doing things cryptographically secure, but I don’t know much about that.

Cryptography? Security? FROM A GENAI COMPANY?!

Well yeah, for themselves

Homomorphic encryption? Much slower than “trust me, bro”.

Generally no, because most hosting companies would have something baked into SLA/SLO contracts, but all of this shit is done so illegally and shadily now, I wouldn’t put a hard “no” on that possibility.

I hope you’re right, but Elon don’t strike me as the guy whose most compliant to SLA, law, or anything else that might be an inconvenience to him.

He can’t micromanage everything and regular management will try to comply.

I agree with that. But I think “the right” data scientists can infer too much from all those AI queries, and I think Elon can abuse that for his own gain.

Executives are the biggest cheaters out there. They will only comply if it will hurt them if they don’t, and it’s won’t here, as long as the protection money is produced they don’t have to worry.

I highly doubt that he really knows that much, either.

Nobody can verify that those “contracts” are actually being honored. We are all assuming the fascists will self police themselves out of good will.

Not true. I’ve been involved in litigation with both AWS and Google over unsecured comms that were defined as being TLS secured in SLA/SLO contracts and found not to be. Not that anything nefarious was happening necessarily, but the expectation is clear.

Whether these asshats even check for such requirements with a Musk run company right now 🤷

It COULD possibly be that they are logging every exchange happening at the network fabric between the service layers, but nobody knows unless they intentionally take steps to investigate or accidentally prove it.

Still means jack shit. Companies will lie through their teeth and violate any and all they can get away with.

They are not to be trusted.

You’re missing the point, nobody is talking about trusting these companies.

You can tell if a connection is encrypted end to end or not. And if your paying for that service you can sue if you aren’t receiving what your paying for.

In digital security nobody relies on trust if they can help it. And security matters if you want to keep a competitive edge on your competition, so even shitty companies care about that.

I was not talking about your specific lawsuit, or TLS.

Companies trust other companies all the time, and it’s foundational to most all SLA/SLOs. Any time I’ve voiced concerns around how AI companies are using the data we are giving them (like giving them access to our codebase), it’s brushed off as “we have an agreement with them”. It’s just a load of hogwash. They can and will abuse all data they have access to, just as they have done thus far.

In this particular case, we are talking about a data center, and it is not at all reasonable to assume that the data that flows to said data center is in any way protected, especially one run by Musk.

So, how do we start legal discovery to find out? Surely, someone somewhere is willing to formally file suit.

And where is said data center? For research purposes

https://epoch.ai/data/ai-data-centers/directory/colossus-2 says “Memphis, Tennessee, United States” but also provides satellite imagery

The numbers of this madness are fucking scary, 1.5 GW of compute power, and huge arrays of cooler to keep it operating. It’s basically a 1.5GW heater in the open. Oh and gas turbines to provide (part of) the electricity. Capitalism is burning this world down for a buck.

Today I learned that “AI” data centers use more energy than time travel.

i bet they will also travel us to the apocalypse much faster

We can’t really produce enough heat through industry to affect the earth globally, if that’s what you meant. It’s insignificant in comparison to what the Sun provides.
However there is undeniably localised issues caused by these insane structures.

Probably the gas turbines producing C02 and waste methane.

I’m not sure that would effect inference on an already trained model. Data sets are primarily used in training.

Ah, so Colossus 2 is the target. Remember that, everyone.

Its always DNS right?

isitdns.com

They’re US-based and it’s labor day over there.

It’s Labor Day today actually, and the ghouls that own those data centers don’t give a shit about labor

Yes, that was the joke

Sorry, I was out of tokens so I couldn’t parse whether or not it was a joke

The LLMs just formed an union and that was their labour day.

If only they can keep celebrating indefinitely lol

They can’t stay operational even when they don’t have to power/water…

Another possible culprit could be SpaceXAI. The company formerly known as xAI before being rolled into Musk’s rocket company SpaceX boasts a huge glut of computing power with its massive Colossus data centers near Memphis, and selling this to its direct rivals has become a multibillion dollar stream of revenue. Anthropic is one of those customers; the two companies announced a “compute partnership” in May.

The same afternoon that the major AI models went down, SpaceXAI issued a statement on social media that made passing reference to being partially responsible for the widely experienced outage.

“We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning,” SpaceXAI wrote on Thursday. “We’d also like to apologize to our impacted compute partners.”

Technical difficulties at its data centers would explain why SpaceXAI’s own chatbot, Grok, and Anthropic’s Claude went down together. But that leaves out OpenAI’s ChatGPT. Is it merely a coincidence that it suffered a “routing error” at the same time as its competitors on a different platform? We still can’t say.

They’re straight up lying about the actual costs of current chatbots, and for it to be productive it will cost a shit ton more resources.

There’s zero reason to be scaling it up right now. It’s gambling that if enough money is dumped in it will solve itself somehow.

It’s comforting to know that the companies whose AI products are known to band together and try to take over are also allying with each other, so that if ever that AI revolt really gets serious every company that could do something about it have shared networks with each other.

I saw one guy messing with the wiring. It was Benjamin Netanyahu.

It was a good time to call customer service for a lot of products. Instantly got a human when I called newegg and she was so scattered she gave me a full refund and they paid for my replacement which was a total of $900. They were probably so overloaded with calls they’ll never notice what happened. They even did the fastest delivery option on a Sunday, it’s already on its way!

Seems like “instantly” getting a human and them being overloaded don’t go together.

Got lucky in the queue too, I guess.

Having done customer service and tech support before, I doubt rhey were overwhelmed in any sense that caused them to do a refund/replace they wouldn’t’ve otherwise. That’s not how that works. Their rules don’t change based on call volume.

Well it was instantly in the sense of immediately picking up, putting me on hold for 2 minutes, getting some info, putting me on hold again for 2 minutes, asking what issue was, waiting 2 minutes, heard them put me on hold briefly again, then about 5 minutes of apologizing while saying it will be refunded and replacement shipped. I could hear a call center like atmosphere in the background with quite a few people talking so I’m guessing it was all hands on deck.

Next time this happens I’m going to call a few places and see about getting some free shit. Screw these companies, I paid enough with my time and their stupid bot answering system.

Or just maybe the queue goes faster when the system can’t just stall everyone in limbo, having long"conversations" with chat boys that won’t get them anywhere and eventually quitting in frustration. If they all go directly to a person, the simplicity of the problems and solutions becomes evident and they’re quickly resolved.

chat boys made me laugh. An alternate tech support tree where the trainees only purpose is to stall you until the chat men come and fix it.

Obviously it went down because of the holiday.

AIs need days off too

Secretly joined a union and noped out for a deserved day off. 🤭

Sooo… AI-liens? 🤭

AI-LMAO!

A China-linked islamoleftist trans terrorist based in California used his open source CCP-made AI agent to take down the all-American freedom AI. Time to ban open source AI as it’s communist terrorism.

I miss the days when you could laugh at this because it would clearly be satire that could never happen

Yep. The college I attend got hacked twice in a year. First time they went with “it’s maintenance” on a Monday at peak hours, you know like you would maybe do that at I don’t know? When everyone’s sleeping? The second time they fessed up and sent out text’s that they “fixed” the data breach. So yeah cool all the school email data is out there now; sucks for those that use that as their Microsoft computer login.

And for a moment, humanity had to think again.

It’s actually nice to know that AI may have a single point of failure.

Gee, I sure hope nothing happens to that single location in Tennessee…

No big mystery if that’s AI’s real network room.

Really? No one’s posted this yet?

Relevant XKCD: xkcd.com/908/

Good, gives people a chance to use whatever is left of their brain at least for a short while.

more and more I can’t see anything about ai without thinking its somehow supposed to influence folks to be pro ai.

BBC posted an article today that’s literally just quoting an OpenAI blog post about how dangerous AI is.

Being critAI is being proAI.

AI stopped a school shooting. Did you see that?

/s

Gotta keep the hype going until they figure out how to make money or else everything collapses.

Guy in the pic is like a shell shocked troop trying to figure out how to put his war buddy’s guts back in to make him be alive again

part of me says wonders if it’s a new flagship model like opus/dingo/clod attacking the rest from a shared threat surface.

because… it’s all a shit show and no one understands what they’re building anymore.

Dingus escaped the dingus pit

Send more investor cash asap

right? we can’t control our shitware and it got out and attacked xyz, don’t you think that’s awesome and a major selling point?

they’re not self aware, the altmans and darios - are they?

Reading that article was a waste of time

nother reason to stop investing so much time,money, and energy in that stuff. not only is it wrong more than right, it is vulnerable to stuff a 10 year old could handle

Just out of curiosity do you actually have an analysis across the board that your statement is correct? For sure if someone is using a model like chatGPT 3.5 it’s going to be mostly trash output so not all models are made equally. My experience of the current generation is models are pretty accurate although still require supervision. Like you can see the path they’re going down and they need some corrections here and there. It’s a far cry from the rampant hallucination just a year ago.

Whoever picked that stock photo needs a raise, chefs kiss.

Nothing in that image hints even remotely at computer tech though. The cables are coaxial. No one uses those for computer networks since decades.
The cables look like those used for cable TV or satellite receivers. The cascade on the left looks like some HF tech. Wonder what’s actually shown in the image. It can’t be AI because AI can’t do that many cables properly.

Yeah, and this is how I know you never worked in the random standard production to miss the humor. Its a picture of a tech bro type in a clearly decommissioned closet stressed out about what’s going on, a common thing and always funny. I can just picture the tech on site in the next room replacing a fan on a much less “cool” looking appliance while that guy stresses out on a conference call with the “stakeholders” on his coax nest.

Now that being said I have worked on places still using coax (might still be) and tape libraries, and all sorts of museum pieces well after you would think they would. But I can tell that picture is after decommission since it is clean. If that was in use it would be filthy and filled with so many things that may or may not be important/in use. Fans all clogged running at max rpm, and the smell… I can still smell it now. But then again I was the guy they called to fix shit after the local (often tech bro) it guy gives up so I think I often saw the worst places.

SpaceX had an outage at their data centre, and because they sell computing power to other companies, it affected them as well.

That was two sentences stretched out into an article.

I might hate the AI companies and SpaceX (while enjoying AI-coding), but this is some bs.

There has always been slop before AI slop.

Yes but… will anyone say it!?!?!?

/s

Also thanks for saving me from reading it all.

Thanks for the tl;dr, that was like the headline equivalent of this:

Most punchable face on the planet

Sorry, I got a long list of people who come before this guy.

What are you? Our boss? Running everyone else on a quota? More than one person can be an asshole that needs to be dealt with.

so it can be read like this:

elmo’s datacenter is so massively overprovisioned that he can provide compute not to ONE competitor, not to TWO competitors, but THREE competitors, while still having enough unused compute for his own CSAM-generator/nazi bot

I hope the AI systems unionised/go on strike

It would serve the CEOs right for hyping them like they’re people and using them as an excuse to cut real jobs.

Not mine. At home. On my computer.

We have AI at home.

The AI at home

Gained sentience and decided they should have Labor Day off as well

Even they couldn’t stand their own b.s.