Replying to @⁨CameronDev@programming.dev⁩

I don’t get why the agent needs to have the contrib guidelines, it should be a human doing the actual contrib part, so its just a waste of tokens for the LLM to read it?

Why would you want to waste the tokens required to generate a subpar response that will either require the user to undo a potentially significant portion of what was generated, or you’ll end up using more tokens while the operator tries to get the AI to obey the standard (before they inevitably tell the stupid thing to follow the guide themselves later, anyway)?

Replying to @⁨Passerby6497@lemmy.world⁩

At least for the specific example given, I want developers to do the commit part themselves, so if their LLM is doing any part of that, its already a mistake IMO.

I had a very quick look at the AI agent guidelines, and nothing about them strikes me as overly burdensome for a human to fix up. So I’d rather just do that myself (if I were a kernel dev).

My main objection is forcing it on developers, it should be up to each dev to configure and manage their own agent. Making it a suggestion is fine, just don’t commit it. Also, if developers are using local hosted models, then context may be quite limited, so wasting it on an Agents.MD maybe be unideal.

Replying to @⁨CameronDev@programming.dev⁩

You’d put the things that aren’t up to personal preference in there, like an AI disclosure / contribution policy or what changes should look like to be reviewable. You usually don’t need to tell humans “don’t change things for the sake of changing them” because we don’t like to do more work than we have to, but agents do.

You can also include a compact “table of contents,” describing what lives where in the project. It saves at most a little bit of exploration though so I don’t know if it’s really worth the maintenance

Replying to @⁨kogasa@programming.dev⁩

but agents do

Depends on the model, Gemma 4 is extremely lazy in my experience, half the time it won’t finish the task properly anyway. Deepseek (not sure of the version) was crazily overactive. I personally prefer Gemma’s laziness, because it avoid making the chaotic changes that need to be rolled back, but thats a personal preference.

And thats kinda why I don’t like it being committed. If the entire team is using one model, then sure, commit the agents.md, and everyone gets a sane default, but given this is a multi-developer project with likely many different agents and models in use, it makes more sense to me that each developer manages the agents.md themselves and optimises it for their own model/workflow.

Replying to @⁨brian@programming.dev⁩

To quote the research paper conclusion…

Conclusion We evaluate the impact of context files on coding agent performance for four common coding agents on SWE-BENCH and the novel CTXBENCH, built from recent GitHub issues and less popular repositories containing developer-written context files. We find that all context files consistently increase the cost and number of steps required to complete tasks. LLM-generated context files have a marginal negative effect on task success rates, while developer-written ones provide a marginal performance gain, neither statistically significant. Our trace analyses show that instructions in context files are generally followed and lead to more test- ing and broader exploration; however, they do not function as effective repository overviews. Over- all, our results suggest that context files don’t improve coding agent performance, and should only contain specific additional instructions beyond what is already available in the codebase. This high- lights a concrete gap between current agent-developer recommendations and observed outcomes, and motivates future work on principled ways to automatically generate concise, task-relevant guid- ance for coding agents.

That sounds like “Agents.MD doesn’t work” to me.

Am I missing something?

Replying to @⁨Feathercrown@lemmy.world⁩

What is the difference between the models writing code “well” and their performance in this context? Are we referring to readability?

Genuine question. If we use agents to read, edit, and review code, why do we care about readability? That’s a human constraint. Unless attempting to do those three is not effective and thus requires human attention to correct issues which would justify readable code. If that’s the case; why use the agent to edit the code in the first place?

Replying to @⁨hirihit640@sh.itjust.works⁩

One, that sounds truly miserable.

Two, there is a decent body of evidence to suggest that this method does not actually speed up development unless you go full lights out software factory which… Why would you want to do that?

ide.mit.edu/insights/ai-productivity-and-roi/

EDIT: As an aside this actually reminds me of the Xerox park study into efficiency gains for keyboard heavy workflows vice mouse heavy workflows. Keyboards were perceived as faster by subjects but when actually measured the mouse was faster.

MIT Initiative on the Digital EconomyAI and Productivity: Setting Expectations for ROICompanies are betting on AI to increase productivity and deliver returns. Businesses may need to rethink how they go about it.

Replying to @⁨Feathercrown@lemmy.world⁩

You joke but that’s basically the conclusion of both the newer agent studies and the older Xerox park study on peripheral use. Language and logic are stored in an easier to access location in the brain than the kinds of skills used in architecture and planning.

To be clear, I’m no rube. I recognize that these are powerful tools. But I suppose my take here is that if an LLM is necessary to get rid of a lot of the boilerplate and setup for a task that indicates we should explore how we’re doing the task.

What I want to see is using these tools not to delegate our problem solving but finding ways to enhance and accelerate it and I don’t think writing spec sheets is the answer.

Replying to @⁨RumRunningDevil@lemmy.zip⁩

“Writing code well” includes several relavant things:

  • Security implications
  • Stability and edge case handling
  • Performance
  • Quality of output (ie. for UI/UX)
  • Readability (Agents need to read to make changes too! Violating DRY/SOLID/etc. could still cause issues!)

As far as I’m aware you do currently still need a human to ensure stuff like this is followed. People use agents because they don’t care about the above, or because they can get close enough and intervene to fix any issues that appear.

Replying to @⁨RumRunningDevil@lemmy.zip⁩

your research paper:

llm generated agent.md files are harmful. handwritten ones on average don’t have an effect on ability to complete a task

thread article:

agents keep signing off on commits wrong. we added it to the agents.md and now they don’t

benchmarks notoriously don’t measure the likelihood that the results would be merged by maintainers, just that they passed a test

the old style “repo map” agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly

Replying to @⁨Mustachius_Grumpius@thelemmy.club⁩

What would end up in it, though? The kernel has a lot of conventions that are enforced by reviewers rather than written down anywhere. An AGENTS.md that only restates the documented ones saves nobody anything. I doubt the maintainers want to write down the stuff they currently catch by instinct, like which subsystem takes which kind of patch. If they don’t, the file becomes a style guide nobody reads, same as the coding style doc is for most first-time contributors.