Modern cloud architecture on AWS: server fleets and databases(start.jcolemorrison.com) |
Modern cloud architecture on AWS: server fleets and databases(start.jcolemorrison.com) |
EC2 - You need a server.
RDS - You need a database.
S3 - You need to store files.
Lambda - You are building an API with short lived requests.
For a few quick examples, you should be using Datadog over CloudWatch, Snowflake over Redshift or Athena, and Terraform over CloudFormation.
Before someone comments on how TF is “cross platform”, all of the provisioners are vendor specific.
As far as what other services to use, if you are hosting your own services on AWS instead of using AWS manager services, you’re kind of missing the point of AWS.
But a few other services we use all of the time are CodeBuild, ElasticCache (hosted Redis), ElasticSearch, Route 53, load balancers, autoscaling groups, SSM (managing the few “pets” until we can kill them), ECS, ECR, Fargate, SNS, SQS, DynamoDB, SFTP, CloudTrail, Microsoft AD, we are experimenting with the recently announced Device Farm/Selenium service, step functions, Athena, Secrets Manager, and a few more I’m probably forgetting.
1. You're using Terraform already for resources outside of AWS (cdn, monitoring, dns, anything else) and want to stay with a common tech.
2. You're running into cases that CF doesn't support and have to generate your descriptions externally, or use sparkleformation hacks.
3. You want to manage a new AWS service. (CloudFormation support lags behind Terraform, new services don't get CF resources for months)
DataDog is great, but the way it polls data means you can't rely on it being available for a long time: https://docs.datadoghq.com/integrations/faq/cloud-metric-del...
> If you receive 1-minute metrics with CloudWatch, then their availability delay is about 2 minutes—so total latency to view your metrics may be ~10-12 minutes.
If an alert delayed by 10min matters to you, DD is not viable for alerting (could be still used for dashboards).
CloudWatch Metrics is a totally separate beast, which happens to share a similar name. You can set up basic alerts in CWM, and you can trigger certain types of events from those alerts, but it is still very limited. If you want real monitoring and alerting, then CWM isn’t even the easy 80%.
Assuming that the operator has the skills to manage Postgres.
It's not like RDS does something complex like Geodistribution, right?
Also what is the scaling like? Is it automatic? How quickly can you handle more connections? Because my understanding was that it was slow.
I did have a play with their RDS Postgres nonths before, and I managed somehow to crash it requiring a restore from snapshot. Also their smallest instance was quite expensive for the performance.
Source: Microsoft SQL Server performance tuner who runs an app that centralizes SQL Server performance data from thousands of servers. You'd think I would be running MSSQL on the back end, but the licensing costs just didn't make sense compared to Aurora Postgres.
Just don't use the AWS Database Migration Service if you can help it, that thing has a couple of badly documented pitfalls. (Fe. tables can't have ENUM fields)
It had so many gotchas and broken features. I'd be amazed if anyone got it really working on large applications without weeks of time invested.
It would be so cool though if it worked seamlessly. It addresses some of the hardest tasks in DB management, namely zero downtime server migration.
EC2 is your only choice if you want a database that AWS doesn’t support, such as Rethink or Cassandra (they just recently launched a managed Cassandra service though). EC2 is also your only choice if you need full control of the DB, such as using many Postgres extensions and foreign data wrappers. Even some triggers and UDFs are limited.
A self-managed, auto-scaling, cross-AZ replicated DB setup is no small matter with EC2. Not to mention logging, metrics, patching of the DB and underlying OS. It’s 100% doable, but one should only proceed with that course with understanding of the human costs.
Personally, I’ve been choosing FaunaDB these days when possible. It’s a no-ops managed service and has on-prem/VPC options. I just write graphQL clients and move on with my life, the rest just works.
When it scales up, queries start to fail and there are long delays. I had to add a try/retry loop around all my inserts to avoid losing data.
I definitely wouldn't recommend it for bursty applications where it needs to rapidly scale up.
The thing that bugs me is that I know a similar workload works fine on a relatively cheap c3.xlarge so I'm not actually saving much money with Aurora.
Provisioned IOPS is one area that can get expensive very quick, but people often don't realize that you get 3 PIOPS included with every 1GB of allocated storage, so you really don't need to pay for provisioned IO if you have a decent amount of storage.
If you want auto-scaling you need to look at Aurora or Redshift, which are quite different and significantly more expensive. I've not used those.
In general, I've found it makes sense to pay the premium for RDS and spend my and my team's time on more valuable work than db admin tasks.
RDS takes care of tons of administrative tasks such as backups, replication, failover, and database upgrades. Yes you can setup backups yourself, but the on going maintenance is going to be a pain. You need to deal with what happens when a backup fails, have a playbook for restoring from backup, cleanup your old backups, etc. These are tasks that are extremely dangerous if you get something wrong and they are completely taking care of for you by RDS.
If you're a tiny startup or hobby with literally no money, it might make sense for you to manage it yourself because you have no choice.
Once you have some money and a viable business, then your value is no longer your ability to spend your time running Postgres, ensuring backups and restores work, creating replicas, upgrading software, and setting up all of the monitoring tools. You provide MUCH more value spending your time and abilities building things that are core to the business that let you make money and grow.
No doubt you can do it all and save some cash. But you have to do it regularly if you want confidence that everything you have built still works. With RDS, you pay them some extra for a near guarantee that it will all just work 100% of the time.
Once you become a large company with tons of engineers and you start to bump into limits of RDS, then it might make sense to run it yourself again. It is a significant burden to do it correctly 100% of the time. Your entire business can fail if you don't do your job right.
In fact, the only time that I would consider migrating to a DB on EC2 is if the database isn't supported by RDS (DB2, for example, though I think there are solutions for this now), the licensing for RDS makes it cost-prohibitive or if the client needed to be able to really tweak the database engine or daemons.
RDS MySQL != RDS Postgres && RDS (MySQL || Postgres) >> RDS SQL Server
The performance dashboard is especially nice.
The reusable sets of configuration is convenient and the UI makes it easy to compare original vs. changed values.
The instance upgrade is not seamless, though, it is normally scheduled for the next maintenance window, unless you decide to apply and reboot immediately.
I agree the service is expensive, but setting up a db server for production takes a lot of time and expertise.
I definitely agree that a DB on EC2 is tech debt. But sometimes that is something you have to accept for immediate financial reasons while you figure out how to create a product with revenue to cover costs.
that seems like a colossally bad idea
If you don't need the striped storage model and guarantees that Aurora provides, it might not be the service for you
There was no back off because it was never needed before, even the shittiest instance could handle it without issue but Aurora failed spectacularly.
Examples for creating them in Java, Python and Node are here
https://github.com/stelligent/cloudformation-custom-resource...
Just add a few lines of code for create, update and delete for your resource.
For Node and Python, you can write them in the web console, test them, copy the code to your git repo and export the SAM CF template for your CI/CD process.
Unfortunately there's a lot more too it than that. You need to handle when the backup job fails or dies, have a process for deleting old backups, etc. Not just that, but if you have multiple Postgres instances, you need to do this work for each machine. I've seen first hand this kind of stuff become a huge distraction. It's often worth it to pay AWS a bit more in exchange to not worry about this stuff.
Is there though? Consider what I would argue to be the "average" case:
* Your database never exceeds > 40% resource usage
* You service fewer then 1m queries/day
* You never burst more then 1k queries/minute
* You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails
* You have a script tied to a cronjob which deletes old backups, with basic error handling that sends you a Slack DM if it fails
What percentage of companies need more then that?
How do I know it doesn't exceed 40% usage? Better yet, who's holding the pager when it does? If/when it does, who's product launch is dead in the water while the db is reconfigured onto a larger instance? What product isn't being delivered because we're faffing about with the database instead of product code?
> * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails > * You have a script tied to a cronjob which deletes old backups, with basic error handling that sends you a Slack DM if it fails
Who's responsible for restoring from backup every week/month/quarter, to assert they actually work, with whatever changes have been made recently? Untested backups are Shrodingers backups.
Just how well tested is this script? Does it properly error out if the script fails to be run? What if a firewall rule accidentally gets set that blocks egress from the backup box to the Internet (for security); who/how/what gets notified instead? Who's deliverables are slipping because the backups randomly stopped working?
> What percentage of companies need more then that?
That's a fair question, but Amazon's done far more research than I, possibly you on that topic. The real question is, of companies that don't need more than that, how many companies want to hire somebody to take on those responsibilities part-time? How many companies have the expertise to even hire somebody qualified to do that part-time? And since those people are managing the DB part time, how many of them are giving it the attention it needs, and aren't distracted by other responsibilities to the company?
None of those problems are insurmountable, but they're far from most business' core competency, and time I'm spending dealing with postgresql.conf (or my.cnf) is time I'm not dealing with other issues. Don't get me wrong, there's still a time and place for managing database instances, but IMO small business (small > tiny) aren't the appropriate place for that. I'd be interested in hearing if someone's run the numbers to justify it though! (Especially if it falls in favor of running it yourself.)
Source: I’ve built very high volume services that continue to run production workloads and use SQS as the buffer between components.
We're using different definitions of "consumer". By consumer, I'm talking about a group of workers that processes the data for one purpose. For example you may have one consumer read from the queue to generate various metrics and a second consumer read from the queue and write to a DB. With vanilla SQS, when you process a message, you need to perform all the tasks simultaneously. With Kinesis and Kafka you can have independent groups of workers (i.e. independent consumers), each performing one of these tasks. Each consumer is able to process the queue at it's own rate. The way Amazon recommends doing this in SQS is to have SNS fan out a single SQS queue to multiple SQS queues. Then you can consume each queue independently[0]. That will multiply your costs by the number of queues you have.
> Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage.
I'm talking about retaining a message even if it was successfully processed, on the order of days or weeks. I've used this feature of Kafka before to implement a recovery log. Under normal operation, Kafka writes data to a DB. If the DB goes down, you can quickly recover the last N days of data by going through the data retained in Kafka.
[0] https://forums.aws.amazon.com/message.jspa?messageID=865925
If your database goes down, you have point in time recovery and read replicas that can be promoted as needed.
There's also dead letter queue and retries for messages that aren't properly serviced.