To all Security Analysts/Engineers (and likeminded folks) out there, do you have any positive experiences about cybersecurity AI products such as Microsoft Copilot for Security? Have they actually increased your productivity, increased the quality of your work or decreased the amount of time it takes to analyze security incidents?

Background: I am a L2 analyst/security engineer at an MSSP and our management is tripling down on technology driven-AI by heavily focusing on technologies such as Microsoft XDR with Copilot for Security, among others. We do not have an "in-house AI" and we just use/configure third party tools, while simultaneously streamline internal detection engineering by re-allocating the resources to do consulting rather than work on improving the threat detection capabilities. We have hired a new "AI team" who so far have done absolutely nothing to improve the quality or the efficiency of the work we do at the ground-level. Nonetheless, the management sees their achievements as a great success as they have managed to "fully integrate AI capabilities in to the security analysis workflows".

My personal experience is, that using these AI-technologies slows down my work and decreases the analysis quality, as the tools cause me to chase red herrings more often than before, or fails to provide critical information that would have been obvious if I had just looked at the raw payloads myself. With prompting, I am having hard time to get the AI to produce any information that wouldn't be obvious by looking at the incident sheet or the raw payloads.

Only area where I see a big improvement on is the generation of text for instructions, incident reports and other type of work that is not happening in the moment (i.e., AI provides a starting point and I finish it myself).

Do I just have to git gud and try harder, or are the something fundamentally wrong with the cybersecurity AI products at their current state?