Ask HN: What 'AI feature' created negative ROI in production? Not demos, real usage. What broke first: data quality, evals, cost/latency, user trust, or support load?” |
Ask HN: What 'AI feature' created negative ROI in production? Not demos, real usage. What broke first: data quality, evals, cost/latency, user trust, or support load?” |
The AI would confidently misroute 15-20% of tickets, requiring human review of ALL AI decisions and the Customers lost trust after a few bad experiences and started explicitly requesting human agents also Support agents spent more time correcting AI mistakes than they saved
The breaking point was data quality - our training data was too clean compared to real customer queries. We ended up rolling back to rule-based routing with AI as an optional suggestion tool instead.
When you rolled back, did you keep AI as suggestions only + rules-based routing? And what metric exposed it fastest for you: recontact rate, handle time, or escalation to humans?