We ended up creating https://github.com/meltwater/secretary to allow storing encrypted secrets in config files checked into Git by devteams. The encrypted secrets are passed as env vars through the continuous delivery pipeline, Mesos/Marathon and into containers. They're then decrypted and injected into the app environment at runtime, safely inside the container.

At startup the container reaches out to the Secretary daemon that holds the master keys, using public key cryptos to authenticate itself. The Secretary deamon uses Marathon to authenticate containers (checking their public keys stored in env vars) and validate that they're authorized for the specific secret in question (checking that the encrypted secret is indeed part of the containers env vars).

Meaning that Marathon is the single source of trust of which container can access what secrets. The problem then becomes controlling who and how changes are made to the Git repo containing the CD config, which is something Github does well with roles, status/deployment API and pull requests.

We had a similar problem as some describe with the distribution of the initial secret (i.e. Vault token) and one time Vault tokens being cumbersome in dynamic scaling envs. We didn't want the cleartext token ending up in config files nor in the https://github.com/meltwater/lighter config we use to drive our continuous delivery pipelines that go into Mesos/Marathon. We also had some other aspects like

* Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.

* Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)

* Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.

Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?

user:	mikljohansson
created:	January 19, 2016
karma:	13
about:	Engineering manager at Meltwater https://www.linkedin.com/in/mikljohansson/ https://github.com/mikljohansson

1.	Optimal Shard Placement in a Petabyte Scale Elasticsearch Cluster(underthehood.meltwater.com) 11 points by mikljohansson 7 years ago \| 0 comments
2.	Using Machine Learning to Load Balance Elasticsearch Queries(underthehood.meltwater.com) 10 points by mikljohansson 7 years ago \| 1 comment