Ask HN: AWS CloudWatch Fraud?

7 points by uLogMicheal 1 year ago | 8 comments

Over a year ago I migrated one of my companies away from AWS and deleted all EC2 and Kubernetes instances. I recently noticed that regardless of these cancellations, AWS has consistently billed for "CloudWatch".

The service team sent back a bunch of general information about costs and instructions to remove a "Metric Filter" which from my follow-up I can confirm does not exist on my account.

The detailed billing shows charges like this across US regions:

$0.00 per request - first 1,000,000 requests 239,785 Requests

$0.01 per 1,000 metrics requested using GetMetricData API - US West (Northern California) 295,570 Metrics

Do others have experience with miscellaneous charges that are impossible to cancel? Any tips for dealing with support on this to get answers beyond copy/paste support articles?

cddotdotslash 1 year ago |

Based on "GetMetricData", you're not paying for services, but rather something with access to your account is making API requests to CloudWatch. Do you have any third-party monitoring tools (Splunk, Datadog, etc.) in use? Can you check your IAM portal to see if you have any users/roles with recent access?

Reubend 1 year ago |

AWS definitely doesn't do any "fraudulent" billing for CloudWatch, but there are sometimes very complicated pricing schemes for AWS products, and people often setup complicated systems that they don't fully understand the cost implications of.

In your case, I'd guess that some part of your system, or perhaps some integration that you added to your account, is making API calls without you being aware of it.

ashitvora 1 year ago |

Not sure about the fraud but recently we had a very heavy bill from AWS (6x of our usual AWS bill).

After much investigation I realised that one of my dev has setup complicated stuff using some Terraform config he found on Github.

I feel that AWS has very bad UX.

re-thc 1 year ago |

Just in case: the thing people often forget is to change regions when checking if something exists. Did you check us-west-1 when checking if this exists in CloudWatch?

Maybe try the aws cli to list and delete?

uLogMicheal 1 year ago | |

Checked all regions, no services on the AWS side. It seems maybe our old Datadog was still spamming the GetMetricData API by default after cancellation. No logs are collected but they still seem to query the API for every service enabled by default.

brodouevencode 1 year ago | | |

Yes, that's how the DD->CW integration works.

Side note: even if you shut down every service in the account, if something outside of AWS is connected to it, like Datadog, you will still incur charges. I'd recommend deleting the account if you can.