Every place I've been, the costs of microservices get overlooked in favor of the illusion of decoupling. Microservices will absolutely make simple features fully contained within a service easier, but as soon as a feature spans services or even impacts the contract of a service, you're in for more pain than in a monolithic architecture. Microservices sieze local simplicity at the cost of integration complexity.

Microservices, as a philosophy, is encoding your org design at the networking layer. I hope you get it the factoring right the first time, because it's going to be painful to change.

I'm all for small services where they make sense. But just like the advice that "all functions should be small" microservices has been myopically adopted as a design principle, rather than a debatable decision.

The world did not finally "crack" distributed systems. Most companies created multiple microservices, putting their small dev team underwater because "this is the way at the FAANG".

All of your DRY principles are out the window, and now you have to debug in production - the new word for that is OBSERVABILITY.

Not to mention that the actual reason for distributed systems is to scale multiple parts of the system independently, but you need to KNOW what has to scale individually before you do it. What I see is that the topology really reflects the company structure, of course. It's not "what has to scale separately", it's "team Y is working on X, and team Y does not want to talk to team Z, so they will create a service to make sure they don't have to talk to people".

Except that this is a giant self-own. We all still have to talk to each other, like, a LOT, because things just keep breaking all the time and no one knows why.

Dropbox, Instagram, StackOverflow - these companies are largely monoliths to this day. You thinking that your small outfit needs to be like Google is highly arrogant.

And don't get me started on the amount of money, people, CPU cycles, and CO2 emissions wasted on this solving of the problem most people don't have.

I often see appeals to Conway's Law when discussing microservices, but teams don't organize themselves this way. Instead, teams work on a macro services: the email delivery team, or the monitoring team, or whatever. In most cases these macroservices would be best implemented and deployed as a monolith, and then presented to the outside world over a reasonable API.

I did the full trip on this microservices rollercoaster. Monolith => uServices => Monolith.

I used to vehemently advocate for using microservices because of how easy it would be to segment all the concerns into happy little buckets individuals could own. We used to sell our product to our customers as having a "microservices oriented architecture" as if that was magically going to solve all of our problems or was otherwise some inherent feature that our customers would be expected to care about. All this stuff really did for us is cause all of our current customers to walk away and force a re-evaluation of our desire to do business in this market. All the fun/shiny technology conversations and ideas instantly evaporated into a cloud of reality.

We are back on the right track. Hardcore monorepo/monolith design zealotry has recovered our ship. We are focused on the business and customers again. The sense of relief as we deprecated our final service-to-service JSON API/controllers was immense. No more checking 10 different piles of logs or pulling out wireshark to figure out what the fuck is happening in-between 2 different code piles at arbitrary versions.

Why touch a critical web service relied upon by literally every product you have when you don’t have to? Your business still functions if account management is offline; but not when authentication is offline. Even short outages to auth were unacceptable (millions of customers) and any updates had to be performed in constrained windows due to the criticality of the service.

So we cleaved off the mission-critical parts, stuck them in their own repos to be versioned independently, which let us move faster on the account management work since we could confidently deploy code that wasn’t 100% working because we didn’t need to wait for a maintenance window.

In a monolith, you just call another function/class, but in microservices that function is a http call. I guess the benefit of microservices is the ability to independently scale different microservices, being able to choose different languages for different microservices and less conflicts in the repo as more people work on it, but you still have to deal with backwards compatibility and versioning of endpoints.

I think lambdas are interesting when you look at it this way. A microservice is essentially a set of functions which is constantly deployed as one unit. But with Lambdas, each function is a single unit that can scale independently.

He talks about the scaling issue; they were using Django at that time and scale up to 50 millions users with a small team of developers.

I do not think a majority of companies have more than 50 millions users and absolutely need to go full microservices.

https://www.youtube.com/watch?v=3pvpNKUPbIY

They're fine. But what's NOT fine are nanoservices. I did security on a project once where it seemed that every function was its own microservice. User registration, user login, and password resetting were each a separate microservice. It was an utter nightmare.

I have a couple layout comments:

First, I love that you have a high contract text-to-background. That is really helpful for me. There was/is a trend to have light gray on white backgrounds for blogs; this is absolutely a terrible pattern. I appreciate that you did not go this route.

Second: serif fonts are difficult to read when the font size is relatively small. Something like Jura or similar could maintain the "terminal" feel without getting bogged down in serifs.

Third: I have a really hard time reading content when it uses smaller fonts and uses a minor fraction of the screen. This is what I see: https://imgur.com/a/NPCBkHJ -- I am getting older, and reading smaller fonts is increasingly difficult for me. I tend to keep my zoom at 150%, something about this page forced it back to 100%. I am not well versed in responsive design, so I don't know the technical details for it, but having zoom maintain or using a larger font would save some cognitive cost to older users like myself needing to zoom in.

Thanks for your thoughts on microservices!

This only protects against some threats related to insecure code, but layers of protection is the key to threats and it is useful for the parts it does help.

Lots of devs don't know infrastructure that well, though this is changing with the adoption of Kubernetes. Additionally, most devs don't want to go on-call when their app crashes unexpectedly.

https://twitter.com/htmx_org/status/1462636513578135556

For me, other than the obvious size difference, the difference between microservices and "large" (?) services is that a single team breaks down their domain into sensible layers, abstractions etc.

I mean yes, I know that each of these problems can be solved, sometimes in a relatively straightforward manner. But who really has all these aspects covered and doesn't run some services that started to smell weirdly a couple of months ago?

But Unix isn’t all small tools. We have servers for the heavy work- like databases.

The challenge then becomes; how do you design that IPC mechanism? Maybe it exists! I don’t know the answer yet. But it’s something I think about a lot and I haven’t seen compelling evidence for “microservices are always bad, no exceptions”

@carnage4life on Twitter

https://mobile.twitter.com/carnage4life/status/1311702322024...

Now, I’ve not had the pleasure of working in an organisation using Microservices and so I’m only informed anecdotally. But I always assumed they where best used as an API boundary between teams, rather than adding more complexity to a teams work.

ie. if you work alone on a problem, you can only solve for the issues you know.

This is probably a good point, however isn't the entirety of the story.

Personally, i agree that most teams shouldn't start out with microservices, monoliths can be entirely sufficient and are easier to run and reason about. Otherwise you might end up with so much operational complexity that you don't have much capacity left to actually develop the software and make sure that it's actually good.

However, you also need to think about the coupling within your monolith, so that if the need arises, it can be broken up easily. I actually wrote more about this in my blog, in an article called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to...

Where this goes wrong, is that no one actually thinks about this because their code works at that point in time, so they make their PDF report generation logic be tightly coupled to the rest of the codebase, same as with their file upload and handling logic, same with serving the static assets etc., so when suddenly you need to separate the front end from the back end, or extract one of the components because it's blocking updating to newer tech (for example, Java 8 to Java 11, everything else works, that one component breaks, so it would be more logical to keep it on the old/stable version for a bit, instead for it to block everything else), you just can't.

Sooner or later, containers also have to be brought up, since they can be a way to do a multitude of applications in a manageable way, but at the same time it's easy to do them wrong, perhaps due to not understanding the tech or some of the potential concerns.

Many out there think that "doing containers" involves taking their legacy monolith, putting it inside of a container and calling it a day. It isn't so, and you'll still have plenty of operational challenges if you do that. To do containers "properly", you'd need to actually look into how the application is configured, how it handles logging, external services, and how it handles persistent data. And it's not the "No true Scotsman" fallacy either, there are attempts to collect some of the more useful suggestions in actionable steps, for example: https://12factor.net/

(though those suggestions aren't related directly to containers alone, they can work wonderfully on their own, outside of container deployments)

Lastly, i've also seen Kubernetes be used as almost something synonymous to containers - in some environments, you can't have a conversation about containers without it being mentioned. I've also seen projects essentially fail because people chose it due to its popularity and couldn't cope with the complexity it introduced ("Oh hey, now we also need Istio, Kiali, Helm, oh and a Nexus instance to store Helm charts in, and we'll need to write them all, and then also have a service mesh and some key value store for the services"), when something simpler, like Docker Swarm or Hashicorp Nomad would have sufficed. I actually have yet another blog topic on the subject, "Docker Swarm over Kubernetes": https://blog.kronis.dev/articles/docker-swarm-over-kubernete...

(honestly, this also applies outside of the context of containers, for example, picking something like Apache Kafka over RabbitMQ, and then being stuck dealing with its complexity)

In conclusion, lots of consideration should be given when choosing both the architecture for any piece of software, as well as the tech to use to get it right. In some ways, this is slower and more cumbersome than just pushing some files to an FTP server that has PHP running there, but it can also be safer and more productive in the long term (configuration drift and environment rot). Sadly, if the wrong choices are made early, the bad design decisions will compound with time.

I also agree with some of them.

Having to push back on that over and over was frustrating.

We didn’t even have a real devops guy, or a vpc with properly partitioned CIDR blocks to segregate our databases from the public web and we’re going to start adding the complexity of a microservice architecture?

For what! We didn’t even have _users_ yet.

But try to get folk to dogfood our application since we had no actual users besides the founder and it was like pulling teeth.

Totally backwards to me.

You can have pieces of functionality in a monolith that make sense to scale independently, and those should not be micro, they should be meaningful pieces of functionality that justify the overhead of spinning them out.

In a way your comment reflects this, a lot of the places that justified microservices were at a scale where their "microservice" was serving more requests than the average company's entire codebase.

It's "big data" with 10 GBs of logs all over again.

It seems like this is not coming up because the most common context of application is one or more teams working on a single large project probably forced to grow as fast as possible because of the funding structure involved. Increasingly, though, there are companies doing software development without the need for focus and scale that is so common to venture capital powered groups.

I'm within spitting distance of the end of a project of collapsing a service-oriented system back into a Majestic Monolith. Every step of the way has reduced the lines of code, fixed bugs, saved money, saved time. It's been such a joy that I'm considering doing only this as a side hustle. "Saving" companies who were sold an over-complicated dream.

> but in microservices that function is a http call

I think that maybe you're understating the complexity that distributed systems may involve.

For example, see all of the following:

  - https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
  - https://blog.erratasec.com/2012/06/falsehoods-programmers-believe-about.html
  - https://medium.com/@kenbantoft/falsehoods-programmers-believe-about-networks-30a328c25c50

The blast radius on deployments can be smaller.

You've already done the work to build up IPC/networked communication so you can make big decisions in a service (like using a better suited language for some feature) without worrying about integrating it with every other feature in your monolith.

You can tailor the instance type to the service. Say you need some kind of video ingestion (or something) that is low use but needs a high memory ceiling. Would you rather pay for that memory across all your monolith instances or just run a few instances of a high memory service?

There's a lot of differences you're not thinking about.

You can have a single function that handles all data base writes. A single function that monitors for failures. A single function that sends outbound notifications... etc

If these functions are call in a way that: allows for A/B testing, SUPER high latency (or no response), failure notifications sent to the code owner, automatic retries of failures, independent code deploys

Then, congratulations, you have a monolith made up of microservices!

> In a monolith, you just call another function/class, but in microservices that function is a http call.

That right there is the fundamental difference.

For one thing, calling a function in the same process is going to be orders of magnitude faster than a network connection call and all that it entails. Even if performance doesn't matter at all in some use case, it's also additional cost to be running all these additional instances.

And then, complexity went up since a network call can fail in all kinds of additional ways that a jump to a function address in the local process cannot. So the code has to deal with all those. The complexity of correlating your logs and diagnostics also just went up. Your deployment automation and versioning complexity also now went up.

All these are solvable problems, of course. It just takes more people, time and budget. If the company is large enough sometimes it's worth it for the dev team decoupling aspects. If the company is tiny, it's approximately never worth taking on all this extra work and cost.

In J2EE that difference is a configurable technical detail. You have 1 service that calls another, and the protocol they use can be a normal function call, RMI, SOAP (or I think REST nowadays) depending on dynamic configuration.

The difference is enormous.

This is just like those people who consider all DB calls to be "free". Not that this went away either, it's just way more egregious now.

I'm a big SOA fan, but my experience with any non-trivial lambda architectures has soured me on the FaaS concept writ large.

The biggest problem I've seen is that early applications are not built in a modular fashion. More as a maze of twisty little functions calling each other, where you quickly end up with circular dependancies and other "challenges". If your base monolithic architecture mimics the world of microservices, modular single purpose functions and event busses to pass around information.

Good design is good design.