Grep beats LSP? Why coding agents ignore your fancier tools(agentconnect.md) |
Grep beats LSP? Why coding agents ignore your fancier tools(agentconnect.md) |
Just painful to read.
Hope this clarifies things.
As a non-native speaker, I really hate how people use this as an excuse. LLMs are good at fixing grammatical errors, so I use them daily as spell checkers. Using LLMs for rewriting is just lazy.
Try: I'm learning English as a second language. I will provide you with a text, and you have to print the corrected text in a code block so that I can copy it. Under it, there should be two lists: one containing the errors and brief descriptions of them, and another containing tips for improving the text. Do not give your opinion about the text. Focus on grammar.
Format: [TEXT] C: [CONTEXT]
> Avoid generic tangents.
> Please don't post shallow dismissals
> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage.
See: Hacker News Guidelines
I’ll point out you also violate this guideline.
Do you seriously read the article and not see that it is extremely low signal-to-noise? And full of non-sequiturs and strange unnecessary clarifications? And passed off as a research project.
“Claude, write an article about why agents use grep instead of LSP”
This would have saved everyone the pain of reading this.
As another user pointed out: “Training support is a hypothesis consistent with these results, not something this study proves.” is not a sentence a human would write, nor is it a sentence that a human should ever be made to read. I apologize for reproducing it; the article is chock full of these “gems”.
Try to remain civil, even when you have Big Feelings.
Too bad. Because this is sort of an interesting subject.
But also that sentence is entirely fine as it is to me, it’s pretty simple and clear isn’t it?
We’ve also collaborated on some Python tooling that takes a rather slow data format that I often have to process and analyze, indexed the whole corpus, and for analysis I can do (or Claude Code can) a single-pass conversion to Parquet which is then queryable with DuckDB. That tool has dramatically improved my turnaround time on one-off analysis tasks and as a Python CLI tool using Typer, the interface is also nicely discoverable for LLM harnesses to work with.
For example, I was handling a lot of infra operations, many of which follow fixed workflows like locating a specific log in certain time range, identify the issue, and apply the appropriate mitigation steps. Since these workflows are entirely command-line based, I added a hook to Claude Code that monitors the commands I repeatedly run, including those invoked by Claude Code itself, and compiles them into reusable scripts(with skills as entries). It can work well.
The takeaway is that we should spend more time reviewing our own work patterns and workflows.
What's interesting is observing how much work this takes. (it gives me much more empathy toward my past self; how was a clumsy human supposed to know and reason about these things!?, especially when I hadn't touched the configs since a few months prior and had forgotten them almost entirely).
Most often there's 10-20 very small programs all working together to give the desired experience. The amount of minutes and tokens required to solve these seemingly simple problems like "My LSP isn't working" is sometimes much more than expected.
I feel like a very large percentage of my Claude usage ends up having it automate configuration shit, because historically that has been the part of software engineering I have always hated.
I also have been using the same LSP config for approximately 7 years.
I don't get the pain.
https://github.com/theduke/smartedit
With the skill installed GPT 5.6 usually automatically uses it, and it reduces code exploration time and token usage significantly, for example by just printing the types and functions in a file without bodies, and only expanding when needed.
(note: it also has editing functionality, which doesn't work so well, since the models are heavily tilted towards common editing tools in post training)
Another tricky part is that how to evaluate customized toolsets properly and ensure they continue to work perfectly as the underlying model/harness evolves.
Also some sample commands runs and output, would be useful in the README, for anyone passing by.
(a) The LSP's tools being competently built & consistent, and
(b) the LLM using it having been properly trained to use LSPs in general.
Using a native tool like grep has the same 2 assumptions, but
(a) is satisfied due to ossification of grep's core features (a good thing), and
(b) is extra-satisfied because the LLM can be trained to properly use grep specifically, and not "20th variation of grep wrapped behind an LSP, but just different enough to throw curveballs".
On top of that, grep is almost always present in default Linux environments, so its presence is assumed & can be relied upon when needed.
What is "lexical noise", which appears to be a key term in the argument, and why is it not defined? Does it mean that an identifier could show up in, say, a docstring in a non-structured way where a semantic tool would not identify it?
Makes sense to me that LLMs, which are language based, would match better with a tool that searches at a language lever rather than at a deeper structured level.
LSP works best when using dependencies that are already compiled locally, but if all source is available, yeah... I still don't have a solid answer on which one is best.
But again, for already compiled dependencies (think Java bytecode), without LSP configs, the agent is likely going to attempt to extract binaries from JAR files, use grep and javap, and potentially attempt to decompile the .class files.
"Training support is a hypothesis consistent with these results, not something this study proves."
I understand it, but I find the wording very unnatural. To improve readability I would have written it in the active form:
"We cannot prove training explains this result, but this fits the data best."
1. Use bash to stream code into a new file
2. Run file
3. Check the change step 2 did.
4. Delete file from step 1
https://github.com/agentconnect-md/lsp-vs-grep-token-study
The author uses their own harness, I'd want to see if what they are seeing is related to the harness, they should at the minimum also use CC which has direct LSP support.
Their metric is token economy, many of us don't pay per token and capability in terms of difficulty and quality are a more important metrics for the kinds of work I do. I'd want those tested as well.
There still seems to be a lot of issues with lsp/pyright, and might contribute to why grep at least gives a more stable and predictable performance (https://github.com/anthropics/claude-code/issues?q=is%3Aissu...)
Also, I hope we do experiments for other agents like codex.
I use lsp when it is integrated in the IDE - but I haven't seen yet any usable command line tools - even if theoretically it would be very useful to type 'lsp-python -replace-function-name foo bar'.
Do you recommend any resources explaining the workflow?
There’s no reason why you couldn’t write a search tool that e.g combines LSP and grep. Or ast-grep, for that matter. It feels like one of those things we haven’t spent much time investigating because grep is good enough
that's also my doubt, it's much easier to train with grep while only a fraction of project can setup LSP properly.
This lack of representation may also indirectly limit the effectiveness of subsequently generated synthetic data.