inari@piefed.zip: Are you a bot? You are posting multiple times every hour of the day.

Nah, I dont think so. There are comments mixed with posts, there arent that many posts, and they seem to be reasonably curated.

Nope. I post things I find on Reddit or Hacker News sometimes, though.

Do you know the recipe of the apple pie?

Having used these extensively, I can say with confidence that they feel exactly as dumb as they are.

Do you mind sharing what models you used and what your experience was? In my opinion the Qwen 3.6 models or maybe the 3.5 were the first local models that were actually useful, but I don’t have that much experience.

Qwen is really good. For coding, it’s the best local model I’ve found.

If you ascribe any level of sentience or intelligence to a computer, you definitely feel much less dumb than you actually are.

And who are you to assert that? How well defined is sentience already?

I mean, I’m all for local LLMs, but they are as sentient as a really good weather prediction model.

aka not at all.

Its just a fact of how they operate, mechanically. They are missing too many characteristics for it to even be an entertainable question.

They might operate differently but they still appear the same as us, 99.9% of the time. Imo if it looks like a duck and acts like a duck…

I mean… No? Absolutely not.

I dont even know where to begin. Maybe with their state being fixed in time; LLMs do not change. An sci fi analogy might be the “no timers” in orion’s arm, who are but a single thought frozen in an infinite loop in time:

www.orionsarm.com/eg-article/47f4311eaef31

Except they arent even that, because its just a next word completion model, not something that thinks. It is not self aware.

arxiv.org/pdf/2503.09211

openreview.net/forum?id=klU4737opt

arxiv.org/abs/2504.09762

This becomes (to me) very obvious if you ever use an LLM in raw completion mode. It very smart, but at the end of the day its no different than a weather prediction model spitting out probabilities for a storm system.

Chat finetuning is meant to get humans to anthropomorphize them by training on human preferences, and the interface further reinforces this. Its all a trick, albeit a very elaborote one.

Will future architectures be closer to “thinking?”

Maybe.

But we are a long way away.

Are we not just prediction systems with more neurons?

Now im not saying LLMs are human at all because to be human is to have gone through our exact evolutionary path, but we really are in a lot of ways much more similar than you think. At least the speaking part of our brain is.

The argument that LLMs are fixed in time does not contradict consciousness. Say a human was stuck in a time loop and the human is asked the same question every single loop, at the exact same time, in the exact same way, with every single atom being in the exact place. Would their answer change? I would argue no, the same exact neurons would be stimulated every time that would cause the same exact answer every time.

We don’t even have a solid definition of what is concious. Are insects conscious? Like the fruit fly, we have modeled all of its neural paths and it simplifies down to essentially being a robot that responds to stimuli.

What does make humans special is our ability to learn. But what when an LLM trains its self? Is that not learning? I think the question of consciousness is too complex to simply say one way or the other and it is foolish to simply it this way.

But what when an LLM trains its self?

Thats what I’m saying. They cannot do this, not even close. Even a fruit fly is somewhat adaptive and can respond to novel stimuli, but an LLM effectively cant.

There are tiny experimental models closer to a fruit fly than large LLMs now.

…What about future world models with some adaptive learning loop, sampling stripped out, and so on?

Yeah! That would be fascinating. Then sentience starts to become more of a question.

But thats not what LLMs are.

People like Altman and Modi have grossly overexaggerated what current LLM architectures are capable of, they trained the LLMs to reinforce the illusion as much as they can, and they arent to inclined to change it. Hence all the good researchers have fled to research world models, and are saying transformers LLMs are not a viable path forward.

Even in your highly optimistic scenario, the AI wouldn’t even come close to being comparable to actual sentience.

Well, theres a lot of experimentation with actual neural nets mimicing biology (or literally using biological neurons), and I think that could start to raise the spectre.

World model research could be a significant component of that.

In principle, OP is right; humans are just machines. All I constest is that transformers LLMs are even in the same ballpark, or possibly a path there… they are not.

Well it sounds like we agree then.

I would think you would need actual biological structures to achieve sentience. That’s a lot different than what these people are calling AI tho.

Edit: Also, if it is built out of natural biological pieces, at what point does it cease to be “artificial”?

Well biological structures can be replicated mechanically. So at risk of being pedantic, that’s not strictly true.

I view “artificial” as just a term for conscious, purpose construction, whereas “natural” does not involve any conscious intelligence, as in “naturally occurring in nature.” But even that is a spectrum, I guess, with genetic manipulation, BCIs, all sorts of intermediate possibilities.

I’d call a constructed, purely biological brain “artificial” because, even if it was grown naturally, it was built with conscious intent.

Correct, nothing is special about biological consciousness. If these exact functions are replicated mechanically/artificially, the outcomes would be the same.

I can’t believe how many pro-AI arguments essentially boil down to “I do not understand words”.

We have a very solid and explicit definition of what conscious means:

www.merriam-webster.com/dictionary/conscious#medi…

The best neuroscientists in the world do not even fully understand the complicated intricacies of the workings of the human mind. We do understand 100% exactly how computers work. There is no comparison.

I haven’t even argued that AI is concious, I have said that we do not understand consciousness enough to define it. You have completely missed the point. This is Dunning-Kruger at its peak. You are vastly oversimplifying things, at what size of brain is an animal considered conscious or capable of thought? Define thought. We don’t even have a solid understanding of life, much less thoughts. Yes we have a word definition, no we do not know where the line begins and ends. It is a philosophical question that remains unanswered.

This definition is already making so many assumptions and over simplifications. www.merriam-webster.com/dictionary/thought

We understand that the brain is what gives us our cognition, we understand that the brain is made up of chemical and electrical signals. We are not special.

This dude is a Dunning-Kruger case study. Simply by the shear overconfidence

I’m sorry that you have trouble grasping so many basic concepts.

Ironic of you to reference Dunning-Kruger.

All animals are conscious as far as I’m aware, though the complexity of their thoughts may vary. They all act of their own volition, computers do not and cannot act of their own volition.

You do not have to fully understand every aspect of consciousness in order to recognize that inanimate objects do not have it. “Will” is a component of consciousness, this is commonly understood. Computers are tools, they have no will. Computers calculate, they do not think, these are different operations.

I am not oversimplifying things, you are under-thinking things that are already simple (likely due to cognitive offloading).

So your response boils down to “no u,” and insulting. I believe we are done here.

If all you got from that explanation was “no u” then you need to work on your reading comprehension.

I agree that this conversation doesn’t seem to be bearing any fruit tho

For what it’s worth, I thought it was an interesting read until the other person went full ad hominem.

The argument basically: “'computers are not sentient, because I say that they are not sentient”

Don’t forget the dictionary quotes and the personal attacks.

No, my argument is that your conceptualization of sentience is incomplete.

As if your comprehension, yet your arrogance is stunning because it is massive and is only dwarfed by your propensity for being wrong.

Discussing the consciousness of neural networks is NOT being pro-AI. I’d say it’s even the opposite, as it displays the danger of such a technology.

Most of the words in the definition have multiple interpretations, it is not as easy as you wish it would be.

Finally, we do NOT at all know how trained neural networks work. We know how training and inference work. We can read the neurons, but the shear size of weights make it a rather impenetrable black box we have to study, partially, to understand the goals of the emergent sub networks.

Fearmongering about the idea of a neural network gaining consciousness is basically like doing free advertising for AI companies lmao

You are ascribing a power to them that they do not possess

For the record, I think neural networks have a lot of useful applications; for example, radiological image recognition. The term “AI” basically has no meaning though, and is most commonly used to describe LLMs, a technology that has very few (if any) good real world applications.

I am not fearmongering the fact it helps or not AI companies is not relevant here.

You are getting off topic.

The fact they are trained to complete sentences and are thus word predicting models only tells us vaguely how they work. It does not tell us anything about their sentience.

The sentience, if there is, comes from the emergent reasoning that develops in the massive neural network to offer actual good predictions: the more you think, the better the predictions.

For example, models trained to produce textual board games predictions can be sounded to show they developed subnetworks corresponding to an inner representation of the states of the game. Models trained to produce images have emergent subnetworks on depth-maps, contours and lighting.

Are cells sentient? Bacteria, fungi, etc. Do you think they have any degree of interiority, like how simple animals do?

Exactly. It’s not so cut and dry. Are they? You can never know until you experience their existence.

Lol no. Actually cells are demonstrably NOT sentient. They have no capacity for choice. No capacity for emotion or experience. They are literally molecular machinery. And we(scientists) understand that machinery to a shocking degree.

If you think cells have any degree of sentience, then you must make a similar argument for advanced clockwork machines. Not robots, pure wind-up clockwork automata.

Ah, so you’ve experienced each of their existences? Fascinating.

Please explain to me how a wristwatch is sentient or conscious.

I can’t, since I’ve never been a wristwatch. Maybe they’re not, or maybe they are through a mechanism for which science has no current explanation. We can’t know, only surmise.

Okay, then you’re not actually engaging in any degree of coherence. Your ignorance is not wisdom, despite what you seem to have convinced yourself.

That’s some impressive hubris you’ve got.

So tell me, in your infinite wisdom, from where does sentience arise and what is the smallest organism that is sentient?

No, I don’t think I will. Because I don’t believe there is any answer I can give you that won’t be merry with more insults and arrogance. I made the claim that sentience cannot exist in a purely mechanistic system, like a wrist watch. Because there is nothing there unexplained, unknown, or more nuanced than gears and springs.

The burden of proof lies on you to explain how a wristwatch could have any degree of sentience. And insulting me does nothing but show the emptiness of your claim, and the fragility of your pseudo-wisdom.

Denying ALL knowledge by taking a bong rip and saying “But how can we ever really know?” It’s not wisdom or humility. It’s pretending ignorance is wisdom.

You seem to have this backwards. YOU’RE the one making a claim, so the burden of proof is on you. I’m saying it’s impossible to know, you are saying you know without a doubt. Prove it.

Maybe with their state being fixed in time; LLMs do not change

Not with agents and chain of thought. Agents can run for hours continuously. So sure, an AI agent is not the same as a sentient human on the time scale of a year, but on the time scale of a few hours, perhaps the AI is “sentient”.

The first paper you linked is focused on the mechanism and not the output. It creates a definition of “thought” and then talks about how the AI doesn’t “think in the feature space”. It only addresses chain of thought at the end, and says that the issue is that it constrains AI to think in natural language only (instead of, say, pictures).

But again, why would thinking in pictures define sentience? The paper gives a rock-paper-scissors example and says that the AI thinks about it in a different way than a human. So what? If a human plays rock paper scissors against the AI and the AI’s output is indistinguishable from a human’s, 99.9% of the time, why is that not sentient?

This is like saying python programmers are not real programmers. If a python programmer can implement the same program in python, who cares what language they use.

I’d love for a better definition of sentience than “it works differently than humans”.

Them

LLMs do not change

You

Not with agents and chain of thought. Agents can run for hours continuously

Those are not the same concepts. The model of LLM remain unchanged. Agents running for hours just modify the prompt they input into an LLM model.

I do not care about the rest of discussion, just swoop in to clarify that distinction.

Fair enough. When I say that AI might be sentient, I include systems like “LLM + agent harness”, treating the whole system as AI.

That’s too much for their tiny brains to handle.

Do you not interact with any other actual humans?

I do, and I can say that the conversations with AI (especially using tools like character.ai) are 99% the same

Well, that may very well be the most depressing comment I’ve ever read.

You either need to work on your listening skills or find some new less-vacant people to interact with, because what you’re saying is insane

Sounds like you haven’t tried modern AI yet. I could train an AI on your comment history and copy you, and nobody would be able to tell the difference

I’ve tried. They’re adamant against AI here. It’s not even worth the discussion. There’s an army of belligerent people here who have made up their minds that LLM are “stochastic parrots” and they’re convinced they understand exactly how they work. No one can tell them otherwise. Doesn’t even matter your credentials. They’re the experts here. If you argue, they’ll only belittle you.

Save yourself the frustration and heartache; just silently pity them.

I work in tech

Uh huh… That makes you specifically qualified on the topic? Far more than myself, certainly, with my MSc in Machine Learning.

As far as understanding the technology goes? It makes me more qualified than 99% of the AI glazers I encounter, but you may be an exception.

To say that an LLM is “less dumb” is a misnomer tho, because it implies that the LLM is “thinking” rather than “calculating”.

My issue here is the anthropomorphic language being used to intentionally mislead people who don’t have even a tertiary understanding of the underlying mechanics.

Surely you know that under the hood it’s all just algorithms. 1s and 0s run through a fluctuating probability equation.

Surely you know that under the hood we’re all just algorithms? Electrical impulses determining what we’re going to do before our own consciousness are aware of the impending decision.

You make the mistake of ascribing more importance to the “1s and 0s” of neuron activity we ourselves encompass, and imagining that a different arrangement of that exact same activity could not possibly yield a similar outcome. It’s like saying plants can’t be alive because they don’t have brains.

It’s nothing like saying “plants can’t be alive”, that’s preposterous.

You make the mistake of thinking that brains and computers are exactly the same when we only really understand the workings of one of those things (computers).

All of the world’s top neuroscientists would tell you that we don’t really actually know how the human brain works.

People who are far more qualified than you to make such a comparison would not do so.

Your argument is built upon a logical fallacy known as “false equivalence”. If you’re not familiar, it means that you are presenting two things as equivalent based on flawed logic.

You’ve probably heard someone use an idiom about comparing apples and oranges right? Doing so would be a false equivalence, that’s where the idiom comes from.

Another example of false equivalence would be your comparison of my logically sound argument to a completely unrelated and nonsensical assertion about plants.

Sorry, ‘h0tbeef’, I forgot that only you understand consciousness and only you know how it is defined. So only you know what qualifies, because of your superior understanding that no one else can possibly hope to have.

Of course, only humans are capable of being conscious, you’re absolutely right as always. You’re so clever, you should go reward yourself with a cookie.

Oh, have we reached the stage of the debate where you project your insecurities onto me in a desperate bid to avoid conceding your palpably untenable position? How dignified.

If you wish to gain more knowledge and understanding, I would suggest you stop outsourcing your cognition to a chatbot.

You’re underestimating most people

You overestimate most people. Here’s a study from 2025

arxiv.org/html/2211.13087

Among AI models, Blenderbot stood out: in AI-AI conversations, it was judged human 67% of the times - more often than actual human-human conversations. In human-AI conversations, human participants were labeled as humans 68% of the time, and AIs were classified as AI 56% of the time.

This is an interesting study. I wonder why there’s such a disparity between the number of AI ‘participants’ (10) and human participants (1,916). I wonder how adding additional AI models would impact the data.

Anyway, this is just benchmarking the AI on 6 language specific tasks. They were programmed and set up specifically to perform whatever specific one task they had to do at the time, that is what computers do. The fact that computers are 1% better at recognizing other computers than humans are is irrelevant.

AI cannot pass the Turing Test, this article is about “6 Turing like tasks” and it can’t even pass those yet, lmao. You know they’re already seeing diminishing returns in further training right?

Even in this article is says that even if AI could pass a real Turing Test it wouldn’t be a sign of intelligence or cognition.

Something I should point out just in case is that 50% success rate is the same as random chance. So when the study says:

AIs were classified as AI 56% of the time.

That really means that humans could not tell the difference between AI and human, since their success rate was almost the same as random chance.

6 language specific tasks. They were programmed and set up specifically to perform whatever specific one task they had to do at the time, that is what computers do.

Sure, and that task was “imitate humans”, and these computers seem remarkably good at it. That’s what I’m saying. I don’t care that these AI work differently than humans. They can imitate humans so well that humans can’t tell the difference.

I come back to Lemmy after 6 months and immediately read this.

So that was a mistake.

Some humans think humans are so special. “Sentient” and “conscious”.

Now I don’t claim that computers are special, I think they are just as non-special as humans.

if a machine looks and acts like a duck it is still not a duck.

I 100% say as a sentient human being that no llm I have used has any sentience. It can’t even understand its own output.

You can say it, but that’s not a proof

so. this is definately something phenomenological. not all knowledge is based on proofs but all proofs are based on postulates deep down.

Correct, but phenomenology can be more precise than one opinion

the experience of the individual is important and can’t be dismissed without throwing it out.

Sentience is roughly defined as “Line in the sand between humans and things we are allowed to eat”.

It’s a concept created to justify treating animals differently than humans in a world where the old justification (“You can eat animals because God said so”) isn’t applicable anymore.

Since the concept is defined backwards to justify preexisting actions, it doesn’t have any actually solid definition, and whether something’s sentient or not depends pretty much entirely on the opinion of the person deciding it.

This is correct

Would you also like a link to the English dictionary?

Because it’s nowhere near correct

I am starting to think you’re sponsored by Merriam-Webster

Perhaps I should seek a sponsorship with all of the mentions this thread has necessitated. Unfortunately I’m not currently on the payroll (or formerly for that matter, lol).

I don’t think you have the cognitive requirements to understand why your dictionary definitions are completely irrelevant here

I’m always astonished when there’s a discussion about a topic so deep that it’s been hotly debated by philosophers for centuries and someone drops in the dictionary definition thinking they solved the whole discourse with a one-liner from the dictionary.

Exactly. How entitled.

Words have meanings dog

You can’t just say a label applies to something, despite the meaning of that label.

My man, allow me to introduce you to the dictionary:

www.merriam-webster.com/dictionary/sentience

www.merriam-webster.com/dictionary/sentient

Did you ask your AI what the definition of “sentience” was or something? Because something/someone is hallucinating.

Congrats, you managed to find the dictionary, but didn’t even make it to the encyclopedia, let alone the acutal literature.

And now, with just the barest of definition and no deeper knowledge on anything you think you are an expert.

Here, please read at least the wikipedia entry to get a very basic understanding about what the adults in the room are talking about: en.wikipedia.org/wiki/Sentience

The first sentence of this Wikipedia article literally immediately disqualifies LLMs, lmao

“Sentience is the ability to experience feelings”

Every sentence that follows continues to disqualify LLMs

If you continue down to the section on “digital sentience” you will find other sentences that directly contradict your assertions, such as ‘Indeed, leading AI textbooks do not mention “sentience” at all.‘

It’s basically 100% agreed on that there are not currently any sentient computers, and there never has been. Do some people think that we might have them one day? Yes a very small number of the experts do, but they don’t know how to get there, and they aren’t close to being there now.

The people calling their AI self aware and claiming that it took actions without their instructions (which is verifiably false) are lying to you for money.

If you continue down to the section on “digital sentience” you will find other sentences that directly contradict your assertions,

So, then, what are my assertions?

Please read my comments so far carefully. Do not hallucinate. Do not make assumptions that aren’t in my text. Do not make mistakes.

Then find those supposed assertions of mine that Wikipedia seems to contradict.

Well your main assertion was a wildly incorrect definition of sentience.

You go on to say that (and I’m paraphrasing) “sentience is in the eye of the beholder”, which is also wrong.

There’s also the implication that you do think computers are sentient, but you haven’t typed that specifically; I know you said you didn’t want assumptions, but this is a fair inference. Someone who wasn’t emotionally invested I the idea of chatbot sentience wouldn’t push back so hard with basically 0 facts to back them up.

Your final assertion is that the Wikipedia article doesn’t immediately contradict you, which again, it does in the first sentence. You can see my previous comment for that information tho, I’m not retyping it.

You go on to say that (and I’m paraphrasing) “sentience is in the eye of the beholder”, which is also wrong.

So you are misinterpreting my statement, calling it “paraphrasing”, to create a strawman argument that I never said.

There’s also the implication that you do think computers are sentient

Again, misinterpretation to flesh out the strawman.

I know you said you didn’t want assumptions, but this is a fair inference.

Fair, if by fair you mean pure lies.

Someone who wasn’t emotionally invested I the idea of chatbot sentience wouldn’t push back so hard with basically 0 facts to back them up.

Because someone who doesn’t 100% agree with you must obviously have the exactly opposite opinion of you. False dicotomy, to again push the strawman.

Your final assertion is that the Wikipedia article doesn’t immediately contradict you, which again, it does in the first sentence. You can see my previous comment for that information tho, I’m not retyping it.

Yeah, of course, if you think that what I am saying is something entirely different than what I’m saying, than of course it means whatever the hell you hallucinate.

I wouldn’t be surprised if you are ChatGPT by the amount of shitty “inference” and hallucination you produce.

Here’s my position: The question whether an AI is sentient or not is beside the point. Sentience means nothing, because it means whatever anyone wants it to mean right now.

AIs are computer programs, nothing more than that. Whether they are sentient, sapient, intelligent, feeling or not doesn’t matter in the slightest, because the definition of any of these terms can change depending on who’s asking.

The bigger question behind these terms is: Does an AI deserve the protection of basic human rights? That’s the actual question that these terms are a proxy for. I say no. Certainly AI doesn’t deserve the protection of basic human rights.

Let’s get back to the Wikipedia article.

I claimed that sentience isn’t well-defined. The first line in the Wikipedia article is:

Sentience is the ability to experience feelings and sensations.

What are feelings and sensations then?

Following the definition of “feeling”:

According to the APA Dictionary of Psychology, a feeling is “a self-contained phenomenal experience”; feelings are “subjective, evaluative, and independent of the sensations, thoughts, or images evoking them”.

A feeling is an experience. What’s an experience?

Experience refers to conscious events in general, more specifically to perceptions, or to the practical knowledge and familiarity that is produced by these processes.

An experience is knowledge and familiarity gained by perceptions. What is a perception?

Perception (from Latin perceptio ‘gathering, receiving’) is the identification, interpretation and organization of sensory information, in order to represent and understand the presented information or environment.

So a perception depends on sensory information. What’s that?

A sense is a biological system used by an organism for sensation, the process of gathering information about the surroundings through the detection of stimuli.

Ok, here we have the arbitrary requirement of “biological”, otherwise anything you get from a stimulus. So what’s that?

In physiology, a stimulus[1] is a change in an organism’s internal or external environment.

So a vision tool would suffice. LLMs with vision capability exist.

The sensations from the original definition are also covered here.

So in the end it comes down to what are your definitions, and depending on that it literally means nothing. By the definitions above, a script that logs the readings off a temperature sensor could be defined as “sentient”. And if you route it through an organic sensor even the “biological” part is covered.

The term is a nothing burger that means literally anything and nothing at the same time.

This is where my investment in this topic comes from. All these terms (sentience, sapience, intelligence, …) are defined in the vaguest ways possible. They mean nothing. And still people get worked up like crazy over them, making crazy assumptions about whether animals or AI are deserving of rights, personhood and all that stuff.

I can assert that because I have what is known as RI (real intelligence).

Sentience is defined extremely well, and explicitly; check out both definitions:

www.merriam-webster.com/dictionary/sentience

Go ahead and click on through to the root word and check out every definition of that word.

www.merriam-webster.com/dictionary/sentient

You could make such assertions yourself if you were to stop cognitive offloading and exercise your brain.

Having “real intelligence” is not a sufficient requirement to decide over which systems are sentient or not e.g. there are very low IQ “real intelligences”.

Dictionary defintions serve for a partial understanding but are not enough to understand scientific concepts, especially definitions that are not under scientific consensus. Moreover, current AI systems, including tool-assisted LLMs can satisfy these definitions.

Since you like exercising your brain, I’d recommend reading about sophisms.

Note that I am not asserting that LLMs are sentient, but that your statement is too confident and that the subject requires actual care.

LLMs cannot fulfill these definitions.

Try to explain to me how an LLM could fit the definitions of “conscious” or “sentient”.

If you understood how computers work and also had a firm grasp on the English language, you would understand that there is no way to use either of those words to accurately describe a computer program.

Yes they can, easily. That doesn’t mean they are sentient though. Just that these are too broad to mean anything.

Have you ever trained a neural network, or studied how they work? I highly doubt so, and I highly doubt you are the computer omniscient you are pretending to be.

If you believe an LLM could fulfill either of those definitions then you either don’t understand how an LLM works, or you don’t understand the definitions of those words.

Can’t wait for you to detail the transformers architecture

Nice strawman you got there. Is this relevant to the post, or do you just feel like calling random unrelated people dumb?

Think of it as a public service announcement.

Judging by this comment thread, it’s a badly needed one.

Arguing against your staunch certainty is not the same as fully believing the counter point. I don’t believe LLMs are conscious either, but claiming to be the arbiter of sentience is quite arrogant. Especially when we don’t understand it ourselves.

Yes, we understand the mechanisms of biology, but we do not understand our subjective experience of freewill, or if it even exists.

Which is why your claim is both a strawman and a non sequitur.

And reading briefly through the comments, most people seem to be arguing against your certainty, not actually claiming to believe the opposite.

^ average Lemmy user

This is true. But VLLM’s quantization support (as outlined in the article) is quite primitive.

Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.

If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.

If you want raw speed on a desktop, use one of the ninfer project branches.

This really killed a wish to run local LLMs. So fiddly…

You shouldn’t let that discourage you. VLLM is really finicky, but there are easier backends to use (though which one depends on your hardware).

Not any more than any other local service. Ollama makes it really easy to get started. The ministral models are small but still useful.

sleepingrobots.com/dreams/stop-using-ollama/

And thats just the tip of the iceberg.

Also, while Minstral is fine, it is pretty much obsolete…

Yeah, I’m just saying it’s a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better.

Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff.

I think LM Studio would be a much better “1st time” recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to “graduate” from it to croco.cpp or llama.cpp or something.

Pretty funny. Most of this article was saying that ollama sucks now and they’re just popular because there were one of the first. Personally I started with them, too, but constantly ran into issues with them. I’ve have much better experiences with jan, open web UI, and LM studio. The first two are foss, and the second properly attributes their use of open source projects, unlike ollama.

Edit: nevermind, it’s the link below me. It’s still early in the morning, my bad