Are LLMs better suited for PR reviews than full codebases?

4 points by aaa_2006 254 days ago | 3 comments

Semgrep recently published an analysis of how LLMs perform at spotting vulnerabilities in code: https://semgrep.dev/blog/2025/finding-vulnerabilities-in-modern-web-apps-using-claude-code-and-openai-codex/

I’ve been thinking about this problem and wanted to share a perspective.

When evaluating LLMs for static analysis, I see four main dimensions: accuracy, coverage, context size, and cost.

On accuracy and coverage, today’s LLMs feel nowhere close to replacing dedicated SAST tools on real-world codebases. They do better on isolated snippets or smaller repos, but once you introduce deep dependency chains, results drop off quickly.

Context size is another bottleneck. Feeding an LLM a repo with millions of lines creates huge problems for reasoning across files, and the runtime gets impractical.

That leads to cost. Running an LLM across a massive codebase can be significantly more expensive than traditional scanners, without obvious ROI.

Where they do shine is at smaller scales — reviewing PRs, surfacing potential issues in context, or even suggesting precise fixes when the input is well-scoped. That seems like the most practical application right now. Whether providers will invest in solving the big scaling problems is still an open question.

Curious how others here think about the trade-offs between LLM-based approaches and existing SAST tools.

aafanah 254 days ago |

Interesting. LLMs are already shining at PR reviews even if they struggle with massive codebases right now. And they are evolving fast enough that those scaling limits might not stay limits much longer.

kogatlas 254 days ago | |

I'd love to see your evidence that "LLMs are already shining at PR reviews". We've used a handful of them here where I work for months now and they are rarely correct, and thus, rarely useful. Instead they tend to just summarize nonsense that wasn't even introduced in that PR, make shit up entirely, or recommend bad fixes to things that would be better solved by being removed entirely.

aafanah 254 days ago | | |

Fair point. I think the bottom line is that it depends a lot on the context and how the prompt is framed. For PRs with small enough scope, I have seen LLMs provide decent value, mostly in surfacing potential issues or offering quick summaries. That said, the Semgrep analysis highlights that accuracy and coverage still fall short even in these narrow cases, so clearly there is still a lot of work to be done before this becomes broadly reliable.