Before LLMs, I found "mostly works" systems like this to be incredibly sketchy and not worth using. The main thing I've had to learn in this past year is that what I thought was an ironclad rule turned out to be only a heuristic that was useful before but not always helpful, because empircally as much as I might find the lack of determinism jarring, in practice these tools are genuinely good enough at what they do to be worthwhile to use, as long as you're making sure not to use them in ways that the occasional failure costs more than just some wasted time.

I have not even really needed a formal memory system. If I see an error happen more than once, I just say "hey add a note on this to agents.md". Tends to be verbose but overall works quite well for the projects I am doing.

For Python, I have the agent use `uv` (even for direct script invocations) so that the agent doesn't need to burn tokens concerning itself with the details that `uv` is managing behind the scenes.

Someone betting—sorry, investing in a prediction market—means you can have even more confidence in their convictions.

But someone having a Patreon account for people to voluntarily donate money to them means they can't be trusted.

I work on several projects on the side, and they all have agents files that give the context about what the project is, what the elements of it are, and what things we're typically working on. This allows my initial prompt to reference things that would otherwise not be in the context at all.

I mean, they're not magic, they're just some automatic context that's supplied. I feel like the study is trying to say that context with an LLM doesn't matter, which is obviously not a tractable position to hold.

The fact that they generated all the agents files instead of curating them with a human is probably part of the problem.

All the scripts are designed to bail quickly. They all create a log in the background instead of blocking. They're all annotated with the necessary comments to keep it on path and locate files, track pratfalls, etc.

So any entrypoint to whatever I'm doing typically starts with one of these scripts. It's heavy handed but orientating your agent for the specific task is better than just dumping a whole set of context that will be ignored if it has nothing to do with your very next command.

Telling it how to do a pull request isn't going to help if you're trying to debug a technical issue.

- Don't vibe you agents.md file, it won't capture any intuition about the project that the model doesn't already have.

- Keep your agents.md file short. Long ones mostly bloat context for minimal difference in behavior.

- Writing for a human audience is probably better. Any LLM can read docs made for humans anyways.