HashiCorp and Google: easing secret and infrastructure management(cloudplatform.googleblog.com) |
HashiCorp and Google: easing secret and infrastructure management(cloudplatform.googleblog.com) |
I've heard a lot of good things of Hashicorp Vault (https://www.vaultproject.io) but been hesitant to go with it.
https://docs.google.com/presentation/d/1ipP2eB9pW5j3WDvzCGz9...
At the end of the day, Vault gives you secret keys to manage which makes automating the unsealing process not fun if you want full automation.
Talking with Armon from Hashicorp they planned to work on some much improved docs around vault which should help with a lot of the issues of making vault usable, because quite frankly they are very challenging to understand right now.
- name: unseal 1
expect:
command: '/usr/bin/vault unseal'
responses:
'Key \(will be hidden\): ': "{{vault_seal_key_1}}"
echo: yes
when: vault_sealed_result.rc == 2 and vault_seal_key_1 is defined
tags:
unsealhttps://docs.ansible.com/ansible/latest/playbooks_vault.html
You can also specify vault-password-file in ansible.cfg [1] It can be a shell script rather than plaintext, so you can use it to call the CLI password manager "pass"[2] for instance. This is handy for automation.
[1] http://docs.ansible.com/ansible/latest/intro_configuration.h... [2] https://www.passwordstore.org/
Oddly enough though, I recently tried AWX (the open sourced Ansible Tower), and it wouldn't decrypt our vaults when trying to get the inventory, even though I gave it vault credentials (there was nowhere to associate it with the inventory run though).
So, we are still using RunDeck for a web UI/scheduling/web triggers/Slack integration of our Ansible runs.
Secret management is as complex as the system which relies upon it. Vault is not as easy as many other tools designed for simple systems. GPG by itself is often enough to manage secrets.
There are probably 40 or more secret management solutions out there, many tailored for specific uses. Most CMS's have their own secret management baked in. Most orchestration and infrastructure tools do too. Four different solutions are called 'Vault'. There are at least 5 solutions just for Amazon. Depending on your platform, something other than HashiVault may be easier to adopt.
Here's some reviews of a dozen different solutions: https://gist.github.com/maxvt/bb49a6c7243163b8120625fc8ae3f3... https://medium.com/on-docker/secrets-and-lie-abilities-the-s... https://www.threatstack.com/blog/cloud-security-best-practic... https://www.youtube.com/watch?v=OUSvv2maMYI&feature=youtu.be
http://blog.fleetsmith.com/authenticating-to-hashicorp-vault...
The worst part about vault is managing access in a secure way and granting access to the right parts imo. We leverage the pki backend (or rather, three dozen pki backends) to map nodes to their respective policies and that required quite a bit of tooling to make work. But now it does, it's secure, and if need comes, it should be easy to revoke secret access for a cluster.
This is admittedly a bit low tech, but could someone more well versed than me tell me what's wrong with this setup.
What kind of secrets do others recommend relying on third parties like this to manage?
I work at NuCypher (YC S16), and we're doing some pretty interesting stuff on the Ethereum block chain to build a decentralized KMS using proxy re-encryption.
Key Management Service creates and maintains private keys for you and provides an API to easily encrypt or decrypt some data. Basically call method KMS.Decrypt("name_of_key_to_use", <bytes>) and get back decrypted content. Secrets are simple text files encrypted with KMS and stored in a private storage bucket. For example we have something like "database-secrets.dev.json.encrypted".
We have a small library that took a day to write, used in all of our projects that does the following on startup: open private storage bucket, download encrypted file, call the KMS API, decrypt the file, and parse the raw contents as json. Now the app has the secrets in-memory to be used anywhere. No infrastructure required, nothing on disk and this is universally accessible whether inside Google cloud or on local machine. Takes under 1 second when running in the cloud.
I dont think I can do better than the documentation but let me know if you have any questions.
Service accounts are necessary to run anything in GCP anyway but can be used externally (like the gcloud CLI on your desktop) or a similar setup specific to AWS or Azure if that's your primary provider.
BTW Mozilla's sops [1] is quite interesting. I've been testing this for a while now.
If I create a kubernetes secret will it be stored in vault if I set some magic switch? Or are we not there yet?
Actually I'm more worried about how Facebook and Amazon treat non compliance, but Google sure seems to be getting shadier every day.
This combined with the W3C evolving into a corrupt entity just makes me want to get out of tech completely. Maybe if I could get some awesome dev job at the EFF?
There's not much detail in that. But, you could certainly read it in a way that using Hashicorp products might be lower friction than using other products on GCP.
This is a genuine collaboration effort by two of the players whose services many people are already using together, and they are making that experience better for their users.
The kind of dismissiveness and hyperbole in your comment is why we can't have nice things.
The reason we can't have nice has much more to do with naivete than paranoia when the paranoics are right.
You concern is probably worthwhile, but I think this is not an example of it.
We, like HashiCorp, cooperate with Google on a lot of things.
They don't pick winners. What works best for Google is to get your workload into GCP. It matters little whether the bits you run are Pivotal bits, HashiCorp bits, Docker bits, Red Hat bits, IBM bits, Microsoft bits or your own bits.
What matters is that they're being processed on GCP atoms.
The analogy I have used before is that Shell, BP and ExxonMobil don't care whether you burn their fuel in a Ford or a Toyota. They mostly care that you burn their fuel.
I don't mean to paint a cynical picture here. As a partner Google is excellent, responsive and respectful, our engineering cultures have good compatibility and there are deep common interests. But Google's goal is to make GCP the most attractive place to run your workload. That means that they are going to be ecumenical. They want to help us to win, but they want to help everyone to win, because that helps them to win.
I 'm not sure where you're coming from or going with this. Do you have an example to illustrate? What are you considering collaborators and non-collaborators?
What exactly is your problem?
Take the database secret backend for example. Getting that same feature out of other simple systems, would be a lot of work. Audit trails are another low effort high reward feature. When you start to get into combining those features, it is an even greater pay off. When you start to take into consideration HA clusters...well if you want to put that together on your own have at it.
Much of "easiest" has to do with familiarity as well. I've seen new users get Vault up in minutes that still don't have GPG setup "cause it is hard". If you are already working with DynamoDB or Consul, you already know how to setup the storage. Those are common skills. I'm sure there are folks that fall on the other side of that use GPG often, but it doesn't make either side more objectively easy.
If you only have the need for a simple system, then Vault may be overkill. I would say Vault is comparable to Kubernetes. Does everyone need it: No. Simpler stuff can be done with config management or tools like Nomad. Does it have features that most people will eventually want to use: Absolutely.
Secret management is one of those things were simple solutions end up easily becoming more and more involved just like container orchestration. Especially when you get into chicken/egg scenarios regarding stuff like some of the tools you mentioned.
Side note: I've tried at least 12 other tools for this purpose , and I would recommend Vault over all of them for most every scenario that is more involved then "Use 1Password".
You can limit access to the keys based on IAM instance profiles. So that only certain instances can access specific credentials.
For some projects I've been on, that wouldn't be a big deal. For smaller projects it can easily be several times your opex though.
You can run an open source cluster in HA without it though. The big draw for Enterprise is the multi-cluster and HSM support. Anything HSM is typically quite expensive so their pricing really isn't out of line.
That said, there's been a lot progress recently in this area recently. Starting in Kubernetes 1.7, you can optionally encrypt etcd at rest: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
You also have a few good choices for the crypto. Two of the choices are Secret Box (XSalsa20 + Poly1305) and AES-GCM with random nonce.
Full list of providers, including info on strength + other considerations:
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
I think a large part of our problem is that we are using Ganeti for most of our VMs, rather than something supported native by AWX like OpenStack/EC2/Azure. I have an inventory script we have been using, but couldn't get it to run in AWX due to AWS credentials not being made available in a way that boto recognized (the inventory is pulled both from Ganeti and EC2).
I have very little experience with GCP and Azure, but it seems like Hashicorp is reinventing the wheel in AWS with Vault.
You say this is not tinfoilism, but the tinfoil in this one is strong.
I don't think they have anything to do with each other. My point is that we know some weird things are happening and it's a put-down and a conversation stopper to call someone a tinfoil hat wearer.
(The best you can say is that the person you're insulting might have a mental illness.)
This is a pet peeve of mine because I knew about some of the hijinks that have gone down (and are going down) since before Snowden flushed his life down the toilet to get people to pay attention and I've been dismissed with that exact term. It's naive in a post-Snowden world to not postulate conspiracies.