DATADOG got hacked?

4 points by ssalat 9 years ago | 4 comments

We've faced a nightmare weekend and ran into a locked account at AWS. After more than two days they told us that our key at DATADOG got compromised?

Hello XXX,

Thank you for reaching out for your patience. I’m reaching out to you in behalf of my colleague David as he is currently off shift. Regarding this issue the EC2 team have new information, and they mentioned that spot instance requests are block due to a compromised key in your account. An email regarding this issue was sent on Tuesday, July 12, 2016 at 4:26 PM PDT from no-reply-aws@amazon.com to the email X@X.X whit the following Subject “Informational Message Regarding Security Incident at Third Party ‘Datadog’ [AWS Account: X]”

Here is the information regarding the compromised key in your account XXX: Type: Access Key Pair Credential: XXX IAM User: datadog

To rotate access keys, you should follow these steps: 1. Create a second access key in addition to the one in use. 2. Update all your applications to use the new access key and validate that the applications are working. 3. Change the state of the previous access key to inactive. 4. Validate that your applications are still working as expected. 5. Delete the inactive access key.

Here are some resources that you might find useful: [1] https://blogs.aws.amazon.com/security/post/Tx15CIT22V4J8RP/How-to-rotate-access-keys-for-IAM-users [2] http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#rotate-credentials

Once these actions are taken please update the case so we can reach out to the EC2 team and they can remove the blocking for spot EC2 instances. If you have any questions or concerns regarding this issue please let us know we will happy to further assist you. Thank you!

XXX Amazon Web Services

Are there anybody else out there with the same issue? Maybe the DataDog team could provide their perspective? We're now sitting on the additional AWS costs.

irabinovitch1 9 years ago |

Ilan with Datadog here.

On Friday, September 30th a number of Datadog-AWS shared customers received an erroneous email notification from AWS about compromised AWS access keys associated with Datadog. AWS intended to reach out to shared customers that had been lax in deactivating and deleting access keys associated with the 7/8/2016 Datadog breach (https://www.datadoghq.com/blog/2016-07-08-security-notice/). AWS accidentally sent out a standard compromise notification to the original list of shared customers. False positives abound! You can validate this error by contacting your AWS support contact.

As recommended in our original communication, if you have not you should deactivate and delete any service integration credential shared with Datadog on or prior to 7/8/2016 immediately and for AWS transition to Role Delegation as outlined here: https://help.datadoghq.com/hc/en-us/articles/210376966-Revok...

The AWS communication was made in error. If you received the initial message you should have also received a retraction. Customers that have been lax in deactivating and deleting these access keys will receive additional communication directly from AWS.

subie 9 years ago |

Did you make that key public at any time? Its fairly easy to leak keys on github that get picked up by automated bots. AWS is very good about covering charges from account 'hacks'

I wouldn't jump to conclusions on Datadog unless you can verify they only had access to the key.

(Just went through a key leak with AWS)

ssalat 9 years ago | |

100% definitely not! It's a specific IAM key only for the use of DATADOG.

ssalat 9 years ago | |

We've never received this email either...