80% of developers find AI coding more addictive than helpful

A new Coddy Developer Survey found that four in five developers, 80%, say their use of AI has felt more like a dependence than an advantage.

www.zdnet.com/article/80-of-developers-find-ai-coding-more-addictive-than-helpful/

Maybe it’s just because Anthropic is so many more miles ahead than OpenAI, but I have not had the “it doesn’t work” experience that people are talking about here. I’ve set up agentic pipelines with code architecture guidelines, the orchestrator calls the implementer, tester, reviewer when needed. If the developed code isn’t up to standard or the work item isn’t fully captured, it’ll reject it and send it back to implementation. It invokes the architect if it runs into something unanticipated. The pipeline can run a feature with high accuracy and very little input.

I’ve even used it to run batteries of frozen experiments on self synthesized data using its own simulation harnesses to discover novel techniques that work in the real world. If it isn’t helpful, I feel like you are doing it wrong.

I use Claude all the time at work. It is good. But it makes massive mistakes, it misses tests, it confidently says something it screwed up will be fixed by something that certainly isn’t the right way to fix the problem.

I recently explained to a colleague: if you can use 1 AIU (arbitrary quantity of ai usage) and get 10% productivity bump, that doesn’t mean 5 AIU gets you 50% and 10 doubles your speed. The AI will do and say promising things, make you believe it’s on the verge of solving the problems, but it never quite arrives. There’s always one more problem and if you’re very lucky the AI will find it itself, but most likely it will be found when you pass it on to another person and it’s completely useless.

Let me put it this way: in addition to development, I use Claude to help with production support issues. It wrote some scripts I didn’t have time to and it pulls logs and data from multiple systems — honestly it works great and has saved me so much time. But I’m constantly in meetings and so I set Claude to investigate an incident so I can focus on my meeting and return when I have time, and it gets RCA wrong well over 50% of the time.

If it is so bad at RCA, how do you imagine it is fixing the bugs in the code it finds? Badly. It misunderstands the cause of problems, and so it fixes the wrong things until it has cobbled together the creakiest of code that passes the test. In fact I think AI is far worse at fixing code than it is at writing it in the first place.

I’m not anti AI. I’m trying to find ways to make it effective. And my teams are seeing 20-30% productivity gains - I think because they are skeptical about AI rather than trusting. But it has to be used appropriately, and everywhere I look, even within my own company, people are trying to do too much with it and creating huge problems I have to sort through.

It is increasing activity, not necessarily productivity. I see the issue that LLMs do not recognize and cannot communicate when they are missing necessary information to push the inference in the direction that is needed.

It’s a bit like when you go into a diner and order a chicken sandwich, and the cook has read every book containing a chicken sandwich recipe in the world, no common sense, no budget / time limits on getting ingredients and an attention deficit disorder. You will (most of the time) get something that resembles a chicken sandwich, but it will probably never be to your taste. A normal cook has a lot of real world context to make sure that the sandwich is tasty - he knows what area he works in and what a chicken sandwich in this area normally contains, he knows how different ingredients interact with each other, and he is limited to what is available in his kitchen. If he knows you personally, he will be able to fine-tune the sandwich to your taste.

LLM output does not have ANY of that context. Even things we do not actively think about - because it’s just the way how things are - are simply not available to an LLM. Only way to make sure our hypothetical ADD-ridden cook with eidetic memory makes a chicken sandwich worth a shit is to provide him with as much clear, non-ambiguous information as possible; and since the connection to the real world is missing, there can never be enough information - or rather, we are in the dark about what pieces of information would be needed to shift the probability towards the thing we actually want.

Great reply. I agree with a lot what you said, but I want to make two points:

First, based on a human breakdown and pointing of stories as if AI weren’t being used, using the methodology we have used for years, we are seeing real delivery velocity increases. Now measurement is tricky; story points are arbitrary for a team. Some of those gains are in stuff like “wrote AI tool” or “created AI documentation” which means we’re being more productive at things we never used to need to do. Still, when you factor all of that in as best you can, we are seeing a measurable 20% increase. (The analysis does point at 30%, but I see holes in how those numbers are built.)

Second:

we are in the dark about what pieces of information would be needed to shift the probability towards the thing we actually want

I don’t think we are completely in the dark. I think we are working on figuring out how to improve the context. The problem I see with the current approach is everyone is doing it independently, their methodology is poor because we can’t afford the tokens for exhaustive tests that are invalidated with the next model, and the results are entirely subjective (and frequently written by AI to sound like absolute success).

So I come up with something that is amazing for my teams, and you try it, but because you’re in a different domain my technique isn’t quite right, and because our lives have provided us with different context, you don’t implement it quite the way I would have, and so your results differ and is it the domain, or the implementation, or did I just get some lucky rolls?

One of the things I do in my spare time because I’ve been fascinated by AI for years, is I use AI to write stories. I come up with a premise, maybe do some world building, write an opening, and just let AI go and see what happens, steering it when necessary.

Now there are two ways to steer: you can “reroll” or you can just edit the text directly. And what I’ve seen over years of doing this is the more human text you inject in, the better the results are. Even if you reroll to steer in the direction you want, the AI will eventually start producing utter gibberish. The quality starts to go down and then falls off a cliff.

Human input slows that or even stops it altogether. Human input is an essential element in getting good results out of AI. Because bullshit built upon bullshit is multiplicative. An AI that is 99% good falls to 50% good very quickly.

My point is that humans need to interact with the AI to provide that context you correctly point out the AI can never have, and it has to be frequent because you get to a point of saturation where a human is overwhelmed by the volume of text.

Asking the AI to write stories for you == “using AI to write stories”