TL;DR: I wrote this because I believe the hype around AI agents in observability is getting ahead of reality. After building an MCP server for our observability backend, I'm convinced they are powerful hypothesis generators, but not yet reliable problem solvers.
After reading a few articles claiming MCP would be the "end of observability," I felt the need to write down my own, more skeptical take, based on my experience building one of these systems.
My core argument is that these tools are effective at identifying known failure patterns, but they struggle with novel issues. During a high-stakes incident, the risk of following a confident-sounding LLM hallucination down a rabbit hole is dangerously high. Verifying the AI's suggestions can often be just as much work as finding the root cause yourself.
Ultimately, I see these agents as a co-pilot that can brainstorm, but can't yet be trusted to fly the plane.
Curious to hear from other SREs and developers: how are you really using these tools? Are you finding them reliable for RCA, or are you also spending significant time manually verifying their "confident" suggestions?