Study finds coding agents choose grep over LSP except when exhaustive references matter

My LSP setup breaks every few months, and I now point an LLM at my dotfiles until it works. What surprises me is how many tiny programs are involved and how many tokens a supposedly simple fix consumes.

I suspect my past self used much less effort than the LLM appears to use. These models are often bad at taking a direct path to the fix.

I see this too. A large share of my Claude usage is automating configuration work I have always hated, and I wonder whether a small local model could handle this niche.

My Neovim config is a single 500-line init.lua and LSP works fine. This sounds like the LLM is making the problem much more complicated than it needs to be.

The article shows why grep may work better, but it does not really explain why agents choose it. We could combine LSP and grep in one search tool, or use ast-grep, but grep may simply have been good enough that nobody studied the alternatives much.

I have the same question about training. Grep is much easier to train on, while only a fraction of projects can set up LSP properly.

Claude Code may have collected many real CLI coding trajectories, but trajectories involving LSP, MCP, or AST-grep were probably scarce. That gap would also limit later synthetic training data.

I have had good results with a SKILL that tells the agent to use LSP more often. For compiled Java dependencies, LSP can avoid the agent extracting JARs, running grep and javap, then trying to decompile class files.

I watch Claude Code use find, grep, and ripgrep, then ask what tool would have made the task easier. That has led me to fzf, notmuch, and a Python CLI that converts a slow format to Parquet for DuckDB queries.

I think there is room to optimize token use for fixed command-line workflows. I added a Claude Code hook that records repeated commands and compiles them into reusable scripts and skills, which works well for recurring infrastructure tasks.

This does not match my experience. On a Python project with string and cross-repository references, Aider's LSP lookups were fast, while grep was more complete despite its token cost.

I do not see why an LLM needs to act as an LSP when code is structured data.

The article's claim that training support is only a hypothesis is fair, but the wording is awkward. I would write: "We cannot prove training explains this result, but this fits the data best."