> If "7 full production instance cost $150" then your application is tiny and you don't need 15 AWS services.

Did you miss that this is oriented around serverless? It doesn’t mean their application is tiny, it just means they can scale down a long way. Which, given that they are using serverless, is unsurprising.

Sure, if you are talking about dedicated EC2 instances or something, then a $150 “production” instance is tiny. But that’s not the situation here. $150 for developer load on serverless doesn’t correspond to $150 for a full production service.

Hey, author of the post here. Apologies if the wording wasn't clear, but these are our developer AWS accounts, i.e. not production workloads. I was just highlighting that everyone runs a copy of the production infrastructure and since serverless is "pay for usage" we mostly only pay for when developers are firing in the handful of requests they do during development.

None of our engineers have so far managed to generate gigabytes of data or millions of requests like our production account :)

I think he is saying that each developer have their own environment with full production like architecture. There are benefits to having each developer running in their own sandbox, with full freedom to deploy breaking changes at any time. Besides storage, costs are usually 100% variable based on use. Each developer may not have a full replica of production data, as with TBs of data, storage cost might get expensive. Many serverless designs are impervious to the size of dataset, with sharding and auto-scaling, so the performance and scalability risks of running a subset of production data during development are typically minimal.

I'm confused as well. How can you run an RDS instance alone for 7 people for $150 a month? Let alone the other infra? You probably need a VPC on each account, with a NAT Gateway, which easily costs $20/month or so.

They mentioned that most of the services are on-demand pricing, that’s why it’s cheap for a personal instance. Production presumably costs much more due to much higher usage.

This sounds like an interesting setup, but honestly my alarm bells are seriously going off in terms of cognitive load for every developer. I would love to see how much time is spent on ops at this company vs actual dev time. Also once they are hitting scale I am really interested how all of this holds up. You have 7 devs now, what happens with 20 devs on different teams.

Also since this is a beta product, I am doubting that you would hit too many problems with any deployment solution at this point. Why are you using Aurora + serverless + event bridge at all? Its magical because you are all using the same dependencies as prod, but why are you using these dependencies at all? All you need is a single postgres database (easily spun up on any developer machine) + crud app (easily able to use full stack typescript with nextjs or the like).

Word of caution about having this many moving parts, they will bite you in the ass. Now it seems all rainbows and magic but there is a reason people on this website preach simplicity -- because when shit hits the fan the simple solutions are easy to debug and fix without spending 1000s of dev hours. The complicated magic solutions are the ones that cause 24+hr downtimes and need to be re-architected. You are a customer service tool, not a devops company, the choices here don't make too much sense.

Looks very painful, or at least it looks to me like doing programming in the 60s: we need to connect to the "mainframe" in order to be able to do our jobs.

So much vendor lock-in, so much dependance on internet connectivity, so many buzzwords around... so much magic (as the post's title describes). I really dislike having to deal with a lot of third-party magic when doing programming.

This was an interesting read, I’m currently facing a similar set of decisions. Good on you guys for finding an approach you’re excited to write about!

Throwing my two cents into the conversation, perhaps one day we’ll be able to have our cake and eat it too: If http://localstack.cloud (replicate AWS services locally) ever makes it to a 1.0 release, I may strongly think about vendor lock-in friendly approach like this. Unfortunately localstack recently (v0.14.1) forced me to give up on each ambition in my ideal develop-n-test-it-locally wishlist: first I gave up on a CDK provisioning resources locally, then I abandoned using ECR images for localstack lambda, and then I abandoned localstack altogether. Their S3 mock still works for local dev though!

The other thing is... if you want to develop locally but still on top of lambda, you either skip the api gateway altogether, or you write an undeployed shim for like expressJS or node http to fake that restful pattern locally. As great as not using APIGW+lambda-integration is, Lambda doesn’t believe in return codes other than 200 (ignoring throws and server failures), and you should probably use the AWS-sdk/client-lambda package. This all scatters weird lambda anti patterns. Yuck, that’s gonna be a big “chore:” commit if you ever get sufficiently sick of it. And I’m actually not in favor of using the lambda response interface, but APIGW lambda integrations are a seeming sht show. VTL and selection pattern seem so bad I’d rather just have more Kubernetes experience. Knative is good these days, I hear.

For me, the magical serverless developer experience was Google Cloud Run.

I write a conventional 12 factor app, with no vendor-specific code which could be executed on my local machine for development, Heroku, or anywhere else, and hand it over to Google Cloud Run. It's really an amazing service.

I had a few wishes on their service (eg integration with Papertrail, an easy way to run background workers, etc) but overall the whole thing is the best of all worlds:

* No vendor lock in due to platform specific code

* Easy local development

* Serverless scalability and pricing

Are there particular reasons why people go through so many hoops to use Lambda when such superior experience exists?

We’re doing more and more full stack typescript because it’s incredibly powerful resource (as in money) wise to have the same environment across everything in non-tech enterprise.

We also do a lot of “serverless” but the way we do it seems far less vendor dependent than this. Basically what we do is layer out the “node” part of our “serverless” application and let our cloud provide act as what is essentially the role of what expressJS or similar might have done for you 5-10 years ago. We’re also handling a lot the federated security through a combination of ORMs, OPA, AD and direct DB access control for the very rare BI application that needs it.

This way we can leave our cloud vendor at any time. Not without figuring out a suitable replacement but far easier than this article, and maintaining almost all of its advantages.

Interesting read, and if you’re certain AWS is a good home for the next five years from your most recently deployed service, then I don’t see too much of an issue with vendor lock-in.

I've worked on a major project that went all in on AWS Serverless and all AWS latest and greatest, What a pain the amount of CDK code 1:1 to application code, Number of pipelines and speed of deploy abysmal. Dev experience sucked. In a rare one to one comparison we had to reimplement a major subset of the project as a mostly monolithic app running in a single container. (On prem version). Was done by 4 people in 2 month. Orders of magnitude productivity boost.

My company started from nothing on serverless about 18 months ago with a small team and now ramping up to several developers. Unlike the author, we're using more run-of-the-mill tooling (like AWS SAM) and more unit testing than integration testing, but the set up is otherwise very similar (TypeScript frontend and backend, a fully deployed version of the application per branch).

Our application has evolved from a simple API Gateway + frontend architecture to numerous asynchronous processes, dozens of integrated Lambdas, numerous S3 buckets and DynamoDB tables, third-party system integrations, cross-account integrations, etc.

It's been great for our velocity, costs next to nothing to run in development, and has been generally straightforward to refactor any "monolithic" parts into smaller, reusable, event-driven units. We operate in a very traditional industry and interface with much older style enterprise systems.

I wouldn't say any part of it is truly "magical", but the ability to give each developer their own fresh copy of the environment for each new piece of work at little to no cost is incredible. It's wonderful not having to concern ourselves with scaling each of our resources or worrying that they'll go down, as well as not needing the regular maintenance that comes with servers or containers.

It's great to see how others are doing serverless development. It's definitely not something you learn to do well overnight. I'm glad to see someone get past the learning curve and write about what works for them.

I've been experimenting with serverless for some time now and came to many of the same conclusions written about here. The biggest takeaway for me is that there are pitfalls to an overreliance on lambdas. You really need to offload as much as you can to the other serverless solutions AWS provides.

I've been using Appsync for my graphql API instead of API gateway + Lambda, and I have had a good experience with it. A lot of logic can be offloaded into mapping templates, making lambdas unnecessary in many cases. Debugging is still a bit of a pain for me, but the end result is fast, cheap, and reliable.

I hope people considering this architecture read this article and reconsider if they can build their applications using mostly portable services.

I'd be curious what kind of use-cases people have that makes this complexity worth it.

> In isolation without impacting other engineers due to everyone having their own AWS account.

This is super interesting. I'm almost certain they mean AWS keys (with appropriate IAM permissions etc.,). Otherwise billing, developer onboarding/de-boarding and such quickly go out of control.

Back in 2015 at my previous company I had setup two AWS accounts, one for test stack and one for production. That in itself was quite a bit of pain.

Terrible name choice:

https://github.com/serverless-stack/serverless-stack

Given their most obvious competitor is already ambiguously named and extremely popular:

https://github.com/serverless/serverless

Reading through the comments one thing is clear: almost no one here seems to understand _why_ you use serverless implementations.

If your architecture follows domain driven design principles, you can build very large, complex, scalable, and maintainable systems.

The tower of Babel that "serverless" requires to do the simplest tasks (like running tests) should scare anyone. And yet... "We deploy to cloud and then shim calls via websocket to dev machine which then shims something else while requiring individual accounts with full prod access" is magical

Wow, the stack in the article feels like a ton of innovation tokens[1].

I hate to be an armchair expert, but I'll do my best to give the _counter_ opinion to "this is a model of a good startup stack".

If you're looking to build a web app for a business on a small team, some guiding values I've found to be successful (that feel counter to the type of values that lead to the stack in the article)

1.) Write as much "Plain Ol'" $LANGUAGE as possible[2]. Where you do have to integrate with the web framework, Understand your seams well enough that it's hard for your app _not_ to work when it receives a well formed ${HTTP/GQL/Carry Pigeon/Whatever} request

2.) Learn the existing "boring" tools for $LANGUAGE, and idioms that made _small_ shops in similar tech stacks successful.

3.) Learn $LANGUAGE's unit test/integration test framework like the back of your hand. Learn how to write code that can be tested, focus on making the _easy_ way to write code in your codebase to be to write tests _then_ implement the functionality

4.) Have a strong aversion from adding new technologies to the stack. Read this [1], then read it again. Always be asking "how can I solve this with my existing tools". Try to have so few dependencies that it would be hard to "mess up" the difference between local and prod (you can go a LONG way with just Node and PostgreSQL).

Some heuristics to tell if you're doing a good job at the above points,

1.) You don't have to prescribe Dev tool choices (shells, editors, graphical apps, git flows, etc)

2.) You can recreate much of your app on any random dev machine, and feel confident it acts similar to production.

3.) Changing lines of code in your code base at random will generate compiler errors/unit test failures/etc

Most every real world software project I've worked on in the SaaS world ended up with "complexity" as the limiting factor towards meeting the business's goals. When cpu/network/disk etc was the culprit, usually the hard part of fixing it was trying to _understand_ what was going on.

Plain may be very successful in their flow, but I'd say most everything in this article runs counter from the ideas that I've seen be successful in the past.

[1] https://boringtechnology.club/

[2] At our shop we'd say "You're a ruby programming, not a rails programmer, your business logic is likely well factored/designed if it could be straight-forwardly reworked into a rails free command line app"

Vendor lock-in aside… azure has a very good experience for developing azure functions locally.

As hit and miss as my azure experience has been… azure function core tools, vscode, and azurite were excellent for purely local development.

Would help to know what exactly their product is. When you are building a crud type simple website / web-app this seems like a big overkill.

I am doing a lot of data-engineering and serverless is an absolute god-send. I have set up all our data-eng infra on lambdas and on aws-batch and I am super happy. The only issue is we have a lot of waste dressources on our always running Postgres instance.

I would love to hear more about your experience with Aurora Serverless PostgreSQL. The main turn-off for me is that it only works with postgres version 9.something.

Interesting read, but I think it exemplifies some of the problems people currently have with Serverless development. For the main population of people here on HN that have experience with deploying applications, this will seem like a lot of learning and effort to recreate an already positive experience they with whatever framework/setup they currently use to build applications. For new developers, it is now a mountain of tools and services they must learn, which takes away one of the selling points of Serverless as an easy paradigm to learn and start building apps on (hence the selling point of not needing to worry about how to setup servers).

With all that said, I am working on building a framework that I would describe to the crowd here as Django for Serverless. It is in early stages but you can check it out at https://staging.cdevframework.io/. One of the main focuses is to make it easier to write code that is not dependent on running on a Functions as a Service (FaaS) model, so that your code can eventually be bundled up and deployed on a traditional server when a FaaS platform becomes uneconomical.

Wow, the vendor lock-in with such a setup is incredible. Should AWS decide to ban you for whatever reason, your entire business is shut. It's crazy how people don't see that as a huge risk/threat.

Makes me wonder why AWS hasn't released a "next gen serverless", where things like endpoint urls (with CORS config), cron-like scheduling, and so on are built-in without having to cobble them together.

They did accidentally release documentation about "Lambda Function URLs" at the end of last year, and then pulled it back. So perhaps some of that is coming.

The issue I have with this article is that it sounds too good to be true. All pros, and seamingly no cons. Either the authors are embellishing it a little bit, or don’t want to think/write about cons a lot. Or both. I wish they balanced things a little bit more.

> Using so many services involves a lot of configuration, permissions, and infrastructure that needs to be developed, tested, and deployed.

Magical

This seems to be a lot of post-hoc rambling to justify why a suite of serverless tools were elected, rather than posing a problem seeking a solution. The conclusion does not follow from the premise.

For example, before a satisfying answer on why a serverless implementation is better, the article delves straight into tests, and how things are complexified because the serverless architecture means more hinges, glue code, moving parts which now need to be tested. Since this particular serverless interpretation involves relying on Cloud as a Service, a hermetic local development doesn’t exist, and so they build a solution around interactive serverless development.

This article is a solution seeking a problem. Many of the solutions here seems like caveats of pursuing this serverless stack, rather than a feature. This is enlightening and useful to know what’s needed to get a fully integrated serverless DevEx, but it doesn’t motivate why people should elect serverless versus a monolith.

> We need to make sure that the few engineers that we do hire can make the most impact by delivering product features quickly while maintaining a high quality.

> This meant being able to run our services at a scale at a low cost without needing to have a whole department just looking after a homegrown infrastructure.

> as well as how we’ll be able to build a successful business in the next 5 to 10 years without needing to do foundational replatforms

All these points can be true with a monolith. If anything, these points make a monolith seem viable.

There’s been article posting around of using Postgres as a hammer for everything. If anything, vendor lock-in to server less platforms will likely involve replatforms in 5-10 years.

> serverless applications. One of the main differences is that you end up using a lot of cloud services and aim to offload as much responsibility to serverless solutions as possible

This contradicts the desire to avoid replatforming.

> Having personal AWS accounts allows each developer to experiment and develop without impacting any other engineer or a shared environment like development or staging. Combined with our strong infrastructure as code use, this allows everyone to have their clone of the production environment.

Again, I’m not sure how this is a feature. This adds another incidental complexity to debug. Engineers may have different permissions and roles. Alternatively, if the engineers are going to have the exact same “environment” as the deployed environment, then the individual accounts don’t provide much value.

> full-stack TypeScript ... The ability to move between the frontend, backend, and infrastructure code without having to

This is an orthogonal concern to serverless.

> developer features... this In isolation without impacting other engineers due to everyone having their own AWS account.

This is orthogonal to serverless. In fact it barely seems like a concern of infra. If local environments exist, developers can just have their own local development branch in Git. They can also test and deploy in separate remote branches. Here, the problem was created because local development is against live services.

Typically these developer blog posts are intended to evangelize the company. Personally I find many red flags in the company culture. Seems like a fun and cool place if you want to try out the latest and greatest tech trends, and have a chance to scratch your own engineering itches. But if you care about job security, you’d want to understand what is the business impact of all this research and development?

I don't know their product well enough, but do they not have a website? Or the need for a single persistent server? How does one run an entire non-trivial application just on Lambdas?

For personal projects I have used Serverless Framework, and I have no qualms about the experience as a developer.. though they're not huge projects so things may vary with those.

"Magical" represents the extremes at both ends of the spectrum:

> "Good magic decomposes into sane primitives."

and

> "If it _isn't_ composed of sane primitives, steer clear."

This feels more like the latter.

Also abandoned localstack, most parts of it are hardly reliable.

Using SAM Local [1] to emulate API GW + Lambda and DynamoDB Local [2]. For the rest we have flags we can switch to either call the real deal on a personal AWS account, or mock responses locally.

Localstack is a total mess, cryptic errors, it either works flawless or with lots of bugs.

I definitely like the idea of using AWS services as much as possible, unfortunately each dev getting their own account was shut down by my current employer.

I've worked with API servers that moved from a dedicated service to lambdas. We were using Django in this case, and just added a tiny wrapper that converts lambda>asgi (using Magum).

For local development, we continued using Django normally inside its own docker container with matching dependency versions and like.

That is interesting, Cloud Run does look much closer to what I want out of a "serverless" type service. I had no idea it existed.

I remember once there was a whole discussion point when people decided it was no longer possible for a single human to learn "all of Windows". I feel like we are at that point with cloud services ... it is now beyond the realm of possibility for a mortal human to obtain a comprehensive knowledge of cloud computing.

GCR is certainly very good. Note that there are quite a few good alternatives too:

Fly.io - https://fly.io/ (gives you a Postgres DB too)

Digital Ocean App Platform - https://www.digitalocean.com/products/app-platform

Scaleway Compute Containers - https://www.scaleway.com/en/docs/compute/containers/quicksta...

AWS also has an “easy” mode for running containers: AWS App Runner (https://aws.amazon.com/apprunner/)

Cloud Run was also the sweet spot for me. Containers feel like the natural service unit.

interesting that i've never heard of it, isn't this basically AWS Fargate? write your custom stuff in docker/kubernetes and it becomes serverless?

Do you mean that you could write a regular Flask/PHP app and it will automatically make it serverless? What about long running tasks that gets triggered by http?

To answer your question, its just a lot easier to use a serverless framework to fully utilize separate AWS services (authentication, message streaming, database) while with the solution you described would benefit from custom platform depenedent binary applications (getting ffmpeg/specialized version of PIL to work on aws lambda was a nightmare).

Hopefully somebody else can chip in here, I've never used Fargate or Cloud run but neverthless open to when i can use it.

Not that it would be a good idea really, but I could put Cloudflare in front of a mostly static site living on cloud run in order to be protected by big cloud run network/transfer bills right?

Cloud Run does look promising - I'd love to move towards a model of "just push container" without all the other hassle.

Are there many other good contenders in this space?

> full stack typescript because it’s incredibly powerful resource (as in money) wise to have the same environment across everything in non-tech enterprise.

but with typescript are you not ending up writing more code? curious to know more about this and are you seeing tangible benefit from not having to switch between backend and frontend? im somehow skeptical of this and im always curious to hear from others who might prove otherwise.

we are using python on the backend (aws chalice) and just react javascript. don't really find much issue switching back and forth. and post 3.5 there is now runtime type hinting too so I didn't really see the appeal of using typescript.

now if there was only a way to write frontend applications in python that would be a dream :)

> We also do a lot of “serverless” but the way we do it seems far less vendor dependent than this. Basically what we do is layer out the “node” part of our “serverless” application and let our cloud provide act as what is essentially the role of what expressJS or similar might have done for you 5-10 years ago. We’re also handling a lot the federated security through a combination of ORMs, OPA, AD and direct DB access control for the very rare BI application that needs it.

hmmm I guess we are locked to AWS sort of (since we use AWS chalice) but I mean it could easily be ported to a python Flask server since the syntax is almost identical and we could absolutely run this on a VPS server if we choose to.

I haven’t been super impressed with Typescript, to be honest. It’s fairly verbose, and it’s a gigantic language with a complex type system that still isn’t really type safe.

I’d lean towards either something dynamic and quicker to develop with, like Elixir, or else use Rust for the parts where stronger compiler guarantees are absolutely vital.

I was once working at a place which was heavily invested in CoffeeScript (at that time, the syntax and the handy sintactic sugar were awesome compared to what JS had to offer). A few years later after I left the company, I had a chat with an excolleague of mine and asked they if they have migrated to JS + babel transpilation. They didn't. As of today they are still working with CoffeeScript. It's a huge codebase.

I see TS just like CoffeeScript.

At this point TypeScript is pretty boring tech, and using it on the back-end is an extra compilation step - with the benefits that brings. And it's easy to say use "boring tech", and in some cases it might call for it, but when you need to have applications/services communicating with one another, need high availability, don't want to manage infrastructure, there is an added value that cloud providers give you.

man this comment needs to be top, quite refreshing tbh from the constant cognitive overload we are used to as developers.

instead of innovation token I would call it attention token :) but seems people might mistake it for some crypto coin.

to me i think the reason this is rampant in the SaaS world is that many are burning through millions of VC money every month and they have to justify enormous valuations by inflating problems, creating solutions for problems that used to be solved by boring tech, and then marketing it to developers on social media which creates the illusion that if you are not on the gravy train you are somehow not an expert.

always fascinating to see just how much marketing and politics goes on consciously and subconsciously.

Just ("4: Have a strong aversion from adding new technologies to the stack.") is worth paying you a couple of grand for every start-up to be that reads this and follows the advice. Large numbers of companies absolutely drown in various stack components, sometimes multiple similar ones within the same tiny team.

I agree with all of this, but I've also have come to appreciate bells included packages like spring or django.

I'm currently working on a large Flask App. It really should have been Django from the outset. There's a different kind of hell in having the almost but not as good version of everything.

The argument on boringtechnology.club could not be more true. Simple tech stacks are always best

I know a company (don't ask) that only uses AWS, yet for unspecified "security" reasons (all theatre) and to avoid vendor lock-in, they've built their own terrible implementations of services provided by AWS. Don't get me wrong, if you've got some good infra/sec people who can deploy and properly manage Vault, an entire observability stack, KOPS Kubernetes etc, good luck to you. But these guys didn't really have the chops to do this properly, nor did they hire sufficient people support and maintain each ball of crap when they moved onto the hobby-build. And they lazily avoided documenting anything, pretending that this was intentional ("documentation is code", except their codebases are sprawling spaghetti with very few useful markdown downs or comments). There aren't even any strict naming conventions. So basically it's incredibly difficult, complex, manual and scary to deploy a new data center, or indeed, even modify the existing ones. There are maybe 3 people in the whole company who gatekeep security access and change. I genuinely wonder if they had a race with the guys at Plain, who'd get their stack off AWS the quickest? My guess: Plain would win.

Sorry, I needed to vent :( But my point is, sometimes it's easier to avoid building the things that your team doesn't know how to build. And maybe you can't hire the right people to build it for you. If they're all comfortable with their severless stack, they're far better off than the other company I'm talking about.

Oh god is it Monday morning again?

If you consider that a huge risk/threat I can only congratulate you for running a business without real issues.

> Should AWS decide to ban you for whatever reason

If this is one of your concerns maybe you should rethink what you're doing.

Working at a multi-billion dollar company we don't have those worries, so we're 100% in on AWS.

I mean it's a real threat, but I can count on one hand the number of accounts I've heard of aws locking out, but don't have enough appendages to count how many Google (gcp) accounts have been locked out.

Vendor lock-in is a real thing, but vendor lock out is usually a vendor issue.

An even bigger risk than being “banned” or losing your account would be mundane things like pricing changes that break your business model or the discontinuation of a service your product relies on at a deep level.