Under the Hood of Amazon EC2 Container Service(allthingsdistributed.com) |
Under the Hood of Amazon EC2 Container Service(allthingsdistributed.com) |
In the task definition - it would be MUCH better if you could select the EC2 instance type you want, collect them in an ASG and have the task slicing scale the ASG accordingly.
Right now - you have to manually determine the slice size for each container-to-EC2 and manually scale the ASG.
Further, it was noted that ECS is actually NOT AZ aware and it will spread load over EC2 instances in the pool -- but it wont also balance the tasks across AZs....
So, its a fantastic version one... but these are some resiliency and scaling features that should have already been included.
Also, would you mind answering my other question about whether or not I can run Marathon or Chronos against ECS since it runs Mesos under the hood?
You have to manually scale the AutoScalingGroup, you say? :-)
More seriously, what's the barrier to configuring the ASG to grow and shrink automatically?
There is a workaround, apparently, which is to create a custom metric but aws says this has not been tested to their knowledge... So any asg will be "static" implicitly...
Other than that, Kubernetes works on AWS out of the box, with a one line setup.
Full disclosure: I work at Google, on Kubernetes.
However this really wasn't the point of my comment, which is that security for application secrets(and AWS API access) is currently a sore spot. It would be nice if kubernetes would adopt some of hashicorps stuff like consul, templates, and vault. Maybe that's too far up the container stack though and a popular bundling of technologies will appear.
The way Werner describes the ECS data store sounds very similar to Google's Megastore:
To achieve concurrency control, we implemented Amazon ECS using one of Amazon’s core distributed systems primitives: a Paxos-based transactional journal based data store that keeps a record of every change made to a data entry. Any write to the data store is committed as a transaction in the journal with a specific order-based ID. The current value in a data store is the sum of all transactions made as recorded by the journal. Any read from the data store is only a snapshot in time of the journal. For a write to succeed, the write proposed must be the latest transaction since the last read.
BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...
In my mind, there's definitely a trend towards consistency here. I'd love to see an AWS blog post about the reasons behind this!
In general consistent vs available (or neither, that's possible too of course) is a trade-off and you'd want to pick one vs other depending on your business case.
Some of those APIs are sort of odd, admittedly, and could be covering up eventual consistency under the covers (Route53 in particular springs to mind there!)
Edit: And S3 and SimpleDB now expose more consistency than they did at launch.
[1] http://aws.amazon.com/s3/faqs/#What_data_consistency_model_d...
With ecs I can imagine a metric that keeps track of the number of tasks and hosts, or ports used or something. I think asg is the perfect tool to use to auto scale ecs clusters.
Also - all my information is only from the one day container session I attended at the pop-up last week... so I am certainly not an SME on ECS.
Thanks for clarifying!