Do agents.md files help coding agents?(twitter.com) |
Do agents.md files help coding agents?(twitter.com) |
I think the problem some people fall into, and especially LLM authored ones (which is where they see the documents not helping here) is instead describing the code, or the structure of the code. Which I don't think helps much - the agent can already see you have 4 modules called a b c and d, and can read the readmes inside of them just fine if it has questions.
One more marginal thing I find helpful but im less sure has positive impact is describing the right terminology for the agent so it can be smarter at communicating with the developer. Things like different names for the product, products it interfaces with, resource names in infra, terms from the customer and product team. I don't think it helps the agent code (much) but it does help communication if it knows what we mean when we speak (and naming things is, as we know, one of the hard problems in CS)
Overall, most of my agents.md now are a list of useful bash commands for working and testing with the project & tests. (heres how to spin up docker services, heres how to update the libraries, heres how to run a command against the local db, heres how to insert a document to be run etc)
and then at the end a terminology blob, which I find myself referencing too.
I don't think this is bad research, but you have to understand how far it generalizes. I'm not saying that evals are useless, we need to do our best to produce good benchmarks. But benchmarks are always going to lag pretty far behind real world applications.
Paper was discussed here 4 months ago, and the linked tweet on this post doesn't add any insights and completely misses the huge caveats that come with the found result: the main benefits of using AGENTS.md files are inherently opposed to the characteristics of _median_ "public github project that has an AGENTS.md file".
I think it's useful for people using agent harness to regularly evaluate your skills, agent rules, and memory implementation to ensure there's no conflict across them all. Also, best to rely as little as possible on the agent to write its own agents.md.
It can be be tedious, but that's why agentic coding can still be considered a "skill".
while even a mildly bad agents.md can be _very_ bad for the agent. they rot very quickly which is why human curation is essential.
same with memory - a lot of the self-learning tools that are becoming popular now degrade agents over time - which is why you end up being able to run an eval with no context and it performs better
> but that's why agentic coding can still be considered a "skill".
yes - far too many cases of throwing a kitchen sink of prompts, skills, tools etc. thinking the llm will sort it out. you need to constantly prune, eval, tweak, observe, update etc. in a loop
A culture of really long preambles certainly benefits cloud AI companies though!
On a new project I usually set up the context of the model (language to use, reason of the product/prototype, etc.) and then I tell the LLM to write a AGENT.md, STATE.md and ROADMAP.md. I don't tell the LLM what's in there because the model has it's own directive and flavor what should be in these files. The models already know the purpose of these files by themself! On a new session, I let the agent read the markdown files in order to continue with the work. Before a session ends, I let the LLM update the markdown files. Maybe one word of caution: don't switch models - it's like putting another person on a working station and ask them to continue the work of others.
Easy setup, really good outcome!
I use AGENTS.md to make sure my agents loop effectively (tests, quality, etc). Not to describe the code / architecture.
To me it is! I thought if an AGENTS.md is researched with much effort once, next time reading the brief conclusion would save on effort and time and cost. Would have seemed logical to me. Like a glossary, looking up a manual, getting briefed before starting.
But I accept the research findings, and I have even had bad experiences with an outdated AGENTS.md derailing the agent.
So now I'm trying to keep to the basics, e.g. just a sentence or two saying "use bun instead of npm" only because I've had to manually prompt that too many times.
It's remarkable how good agents are at exploring, and at treating relatively sparse code reads and get a good picture nevertheless.
This way it can refer to the docs only when relevant instead of dragging around the full knowledge of the front-end structure while it's working on the backend.
Why not just try and see? The fast feedback loop allow testing all kind of weird theories in a matter of 30m-1h during normal working sessions, most results are obvious
There is a difference between trying something and performing a test whose results are meant to be representative of most setups. A lot of people (in tech surprisingly) who see themselves as practitioners of applied science, apply flawed methodologies and try to generalise the results.
Just because testing is fast, it does not mean that it is free or cheap. Plus time is money and verifying every LLM setup every koolaid dev is proclaiming on bluesky could easily be a full time job.
I think shorter the better.
also a strange finding from my own experiences: specific empirical formats seem to yield much better results. For example people often say "get this done to 100%" but I say "get this to 88.47%".
Some of it is cosmic ordering, some of it is cargo culting, some of it is clearly FUD.
The trick presumably is to end up on the words that actively shift the “culture” of the subsequent interpretation of words.
I don’t know how you do that, really, but I do think it’s extremely instructive to see what a model can do with an empty prompt.
I'm adding pointers to specification documents, and it saves me from the /new dumb coding agent that sees your code base for the first time and knows nothing about architecture, concepts, code organisation, etc...
I'm using no cookie cutter directives though (except maybe "do not attempt to deploy, we're using CI CD to deploy" to avoid an automatic "wrangler deploy" to Cloudflare)
"Overall, our results suggest that context files have only marginal effect on agent behavior, and are likely only desirable when manually written."
/s
Dunno if there's a way to test that AGENTS.md help the human more than the machine.