Ask HN: How do you manage AWS root MFA in a remote work environment?

13 points by gfisher 3 years ago | 12 comments

As offices are going more and more remote, how do you handle your AWS root 2 Factor credentials? We have typically used physical hardware keys that we store in a safe, but with employees leaving the city and the state, we are running low on in-city employees.

Is there a smart solution for multi factor authentication that can be shared between people who are remote?

GauntletWizard 3 years ago |

No, there's no good answer to this. Don't ever use the Root account itself. Use a corporate Yubikey, kept in a safe or better yet - Corporate Safety Deposit Box - that only the highest-level technical leads (3-5) have access to and keys to.

You should be using IAM user accounts for everything else. Unprivileged base users with access to two different roles. One for general use, one that has administrator access. Build a lambda that sends a daily e-mail summarizing logins to the second and actions those logins took to your security team. You only need to be quite this complex as your corporation grows, and you probably should only ever have a (under twenty) list of people with access to that second role.

The other key to this puzzle is separate AWS Root Accounts for different use cases; Run your Dev in one and give out the administrative role willy-nilly, "Prod" in another that's tightly locked down. AWS SSO is one good solution to managing all these accounts (there are others). Run your "Prod Data but not visible to customers" in another if you must have it (I suggest investing in automation instead - Set up a service team to handle understanding requests for "Prod" data, that will run queries in prod after vetting them and anonymizing, but don't ever load that data into staging; Have them also help engineers sufficiently fake that data).

gfisher 3 years ago | |

Yes, you are totally correct. We don't use our root account now, and we keep our our corporate Yubikeys in a safe with a 2-key access lock. The issue is that fewer and fewer people have access to the safe due to geography. I'm worried about our in-city bus factor.

GauntletWizard 3 years ago | | |

Tread carefully with permissions; when making changes to admin roles make sure you've got a backup role you can use to undo.

Remember that if all else fails, AWS will send your company a physical letter that support will verify and unlock your account.

imduffy15 3 years ago |

Not a solution for shared multi factor auth but maybe some ideas…

- the root account should not be used. Disable it from being able to do anything with an SCP

- new accounts created with aws organisations by default have a random password and no mfa. Access is granted by going through the password reset process. Switch to this process for existing accounts, randomise all the passwords, grant break glass access via password resetting (ensure your contact details are valid). The password reset typically requires access to the email account (make it accessible via SSO) and potentially a phone call, ensure a virtual phone number is used and root holders can point it at their phone.

- use the likes of azure ad, keycloak or okta to store your organisations identities. Require MFA on them via yubikey. Enable access to multiple aws accounts via aws sso.

- for ssh access switch to using aws ssm.

trelliscoded 3 years ago |

I try not to handle my root account credentials. Everything on the list that requires root account access is pretty scary:

https://docs.aws.amazon.com/general/latest/gr/root-vs-iam.ht...

At one place, they gave them to outside counsel, and at another place they're in a safe that only the trusted company ops person has access to, and she isn't remote.

Generally, you should be breaking your footprint into different AWS accounts under an org with SCPs if you really need root account access for something, not sharing the credentials for the main account.

gfisher 3 years ago | |

I hadn't considered outside council. That's a good idea. The issue with everybody being remote is that our corporate safe is difficult to get into (we are on a two-key system), but an outside entity is worth looking into.

abrookewood 3 years ago |

The root account should not be used to do anything except set up Admin accounts in IAM. That said, to ensure that the credentials aren't lost if someone leaves, what we do is save the QR code to an offline secure database (like KeePass) that is backed up regularly (e.g. on Dropbox). That way all the admins can use the same MFA account.

gfisher 3 years ago | |

That's an interesting idea, thanks. And I agree- we do not use the root accounts for anything. I'm much more concerned about the bus factor with physical key access.

acranox 3 years ago |

You can use the TOTP 6 digit code method, but instead of just scanning the QR code with your phone, you save the “serial” in your password manager and then all your employees can add that serial to the app on their device. Did that make sense? That’s what I do.

marcc 3 years ago | |

Are you advocating that everyone can log into the root account? Or maybe I’m not completely understanding here.

acranox 3 years ago | | |

Just the people that need to, usually the security or operations team that handles creating or deleting accounts, or setting up billing stuff, or creating the initial roles or users that will be needed. All the other responses just say, “don’t use the root account”. That’s true, but there are some things that need the root account, and in a midsized company that shouldn’t be controlled by one person, but usually a small team. I don’t use the root account for anything outside of what it’s explicitly needed for, but I do need access to it for those times, and the rest of my team need that access too. And we have audit trails that let us know whenever someone uses the root account. Tada.

mannyv 3 years ago |

Our CEO has the root creds in his 1password. Everyone who needs it had admin roles with billing.

If he gets hit by a bus we know enough to recover everything.