Make Your Back End Layer as Thin as Possible(kartick.substack.com) |
Make Your Back End Layer as Thin as Possible(kartick.substack.com) |
The only mention of security hand waves away the idea of access control to a "thin back end". That's the whole point of this post! The details of this are critical!
> you can define an API that takes an SQL query from the client, runs it, and returns the results. This query can be run under a database user account that has only read access, that too only to the friends table. This can accommodate unforeseen uses, like getting the number of friends rather than the actual list. Or if there’s a search box where the user can filter his friends, you can do a LIKE query. Or you can limit the number of friends returned to how many will be displayed in the UI.
Without mentioning how this has been solved elsewhere, this is an incredibly reckless piece of advice. For one, it's nearly impossible to parse and check an arbitrary SQL query for malicious intent. Besides ensuring that queries that should read only read, you also need to make sure that queries aren't designed to intentionally DoS your DB.
And moreover, it now passes the responsibility of query performance to the FE engineer. Are appropriate indexes in place? Does the query make inappropriate JOINs?
And on top of all of that, schema changes now mean that you need to update your front end code. This means guaranteed hard downtime, because you can't control what JS folks are running in the browser.
There's a lot wrong with the ideas presented here.
> This article doesn't use the word "permission" or "validation"
The article talks about it multiple times: using a database user account with limited privledges, exposing some tables but not others, exposing read but not write access, giving users access only to data they own.
> it's nearly impossible to parse and check an arbitrary SQL query for malicious intent
Which is why the article doesn't propose trying to parse and check queries for malicious intent.
> it now passes the responsibility of query performance to the FE engineer. Are appropriate indexes in place? Does the query make inappropriate JOINs?
No, in the proposal, we have backend engineer(s) to advise and assist frontend engineers. Having appropriate indices and JOINs doesn't mean you have a layer that doesn't add business value. And when it does add value, write it!
> schema changes now mean that you need to update your front end code. This means guaranteed hard downtime, because you can't control what JS folks are running in the browser.
I don't know if I understood you, but you can just force a refresh in the browser.
> you also need to make sure that queries aren't designed to intentionally DoS your DB.
That's a valid point I didn't think of.
> There's a lot wrong with the ideas presented here.
It's hard to take your criticism seriously when many of your reactions are a result of your own misunderstanding of the proposal.
> ensuring that queries that should read only read
seems to be the purview of the DBMS, setting appropriate account permissions. The bit you quoted said "a database user account that has only read access".
Never mind that, how do you prevent denial of service attacks? I can run any allowed query on that database. Including 50 times cross join of the table I can read, sorted by random.
1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious. Validations must be duplicated in both the UI (to show people that they can't do the thing), and the back end (to stop them from the doing the thing).
2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
3. The entities that make the most sense for storing your data into a database are not the same entities that make the most sense for using in a UI. Forcing your UI to use the same entities as your database will mean more work for both, and often ends up influencing the database design badly.
4. There are ways of writing APIs that allow for rapid change. Uncoupling the storage entities from the display entities is a key step, so you can change things in the database without affecting the front end (and vice versa).
5. There are ways of writing database schemas that allow for rapid change. My favourite is adding a hstore "metadata" field that I can use to create "temporary" fields in the table for experimental features.
I've written a few back ends, and the advice in the article strikes me as dangerously sensible-sounding while being mostly wrong. I can see a newbie product manager buying into this completely and making a huge mess.
Your DB is likely to contain internal state that has no business being sent to the client. Over time applications change and fields can change meaning, become deprecated, or become dependent on other fields through business logic.
If you do this, any business logic not captured in your backend has to be duplicated in each front end. Imagine needing to write and maintain the same business logic in a JavaScript framework, Java for Android, and Swift for your iOS app.
How do you make breaking DB changes if you take on this strategy of API design?
In most cases in my professional career, making a choice to have a thin RESTful API that corresponds closely to the DB structure has been a mistake that only time made clear. RPC-like APIs fared much better.
This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten. I would not recommend it for anyone wishing to build a lasting architecture.
Don’t take my word for it. Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
Yet, this is what my backend does:
* make sure the user is allowed to move that particular card.
* update the database.
* add an entry to the board's activity log.
* notify all connected clients listening to events from this board that a change occurred, so they can refresh the UI instantly.
* if some users asked to be notified about changes, generate notifications.
* if some of these users are offline but still want to be alerted, generate and queue some emails to send.
* add an entry in a raw text log for easy debugging.
* register the event in some kind of analytics storage for future stats.
* if the board is integrated with e.g. Slack, call Slack's API.
* if some users registered webhooks through my API, trigger those.
I'm so glad my frontend engineers actually do not have to worry about how to do any of that.
Edit: format
The article says, "build a backend API when it has something to do, not as a pass through to your database."
It doesn't say "never write any backend code".
I get the allure of this kind of approach, but the costs in usability and defense against pathological queries (given nontrivial data volumes) are way too heavy to bear for the superficial convenience. And the justifications related to the sorts of problems avoided don't pass the smell test.
> every change to your database will break your client unless they are updated simultaneously every time (impossible for mobile apps)
You can just put up a UI forcing the user to update. Which is perfectly fine since you won't be making backward-incompatible changes to your database every month. Adding a new column doesn't break existing clients.
> or your client is aware of every backend change (very complex)
You can just put in a build date in your frontend and compare it against the date a breaking change was made. If it's <, refresh.
sounds very user-friendly
Even at MVP stage, many backends do quite a bit more than CRUD.
This is why I wouldn't pick e.g. https://github.com/PostgREST/postgrest even when it's a perfectly fine (and probably well thought-out) project.
As a DevSecOps Engineer, I field requests like this from frontend developers all the time. The answer is always no.
In this case it would be "I am sorry that you are not happy with inter-team planning and communication; please raise this with the appropriate project owner at the next scrum meeting. Sadly poor communication can not be allowed to drive well established systems planning and architecture best practices..."
Um, yeah, and now you have front end security issues, since all the expressive power that you have just given to your front end engineers is also available to any damned fool who can fire up a browser console.
I strongly agree that you should make the distance between the front end and the data as thin as possible, but this article comes at that from the front-end developer perspective.
There is another option: make the front end as thin as possible. Eliminate the huge front end framework and go with something closer to the HTML.
This keeps the expressiveness on the server side, which is a trusted computing environment, rather than exposing it on the client side, in an untrusted computing environment, where security concerns become much more complicated.
Wat...
This article reads like a jumbled mess of ideas. But, the above is a total wat...
And it also assumes that your backend doesn't have to consume and tie together several resources or that you need a caching layer and so many other naive assumptions - and I'm not even addressing the security implications.
Take this article with a heavy cup of salt.
To me it just reads like something someone would write because they want full control and does't like working with and/or being dependant on other people/teams.
Fast forward almost a decade, and this approach has generated a huge headache. The database is an incomprehensible mess, there's a whole class of features we cannot implement because of this (lack of) architecture, business logic is all over the place (with a million ways to do the exact same thing), mostly in the clients, performance is bad because every client is doing all sorts of queries all the time, and training a new developer either takes months or doesn't really work out at all - because who wants to read a decade's worth of spaghetti code. I've been working on a way to fix this, but it takes a lot of time and effort. Also management still doesn't understand the concept of "overwhelming technical debt".
The only reason this has even been hanging by a thread for so long is that we only make software where the back-end and the clients are run on-premise in a separate network. Otherwise it would be a security nightmare as well.
This article is basically a set of instructions to get to where we are. It's like saying that traffic lights are unnecessary because if you just floor it through an intersection, you'll get to your destination much faster. And you know what, it might just work the first few times. Until you find out why these things exist in the first place.
And as someone else pointed out, you can't enforce security on the frontend.
The line is getting blurrier between front and back these days with new frameworks, e.g. React. Personally, I'm not ready to move away from mature languages generating HTML on the backend, even though it means giving up some interactivity on the frontend.
You have discussed multiple ideas that fit for different use cases.
SQL is for transactional type of workload that's where it's best so data returned need to be less. In first part you are talking about not making any changes to the program when a new column is added. Now a days we are talking about column masking. So querying all columns is a no. Otherwise there needn't be a columnar DBs.
Seeing a generalized rest APIs(selecting many Columns than required) are also no but architects are recommending it nowadays. Here unnecessarily you are retreiving lots of data than required and filtering in frontend after retrieval (you have already done the heavy duty work).
Firebase is a special case, UX designer basically dictates the database schema rather than architects or dba. When you hit the node in firebase you retreive all the data in that node. Here there has to be duplication of data as it's designed based on UX no foreign keys or checks. To be efficient if UX changes you will have to change schema. If you are not changing UX but just adding more info or taking out things you can easily add more fields, no disruption.
What you do with firebase cant be doing with RDBMS, adding a column needs to be planned and also theres boyce codd normal form to avoid duplication and repeated groups. Again there is a reason why they had to build firestore when they have firebase.
Also one api call or three api calls again that depends on use case and performance testing. Three separate calls to get data from three separate tables or join table query again it depends in use case.
Don't do this, "Do a SELECT *, iterate over all the columns, and copy them to the JSON that will be sent back to the frontend."
As the data grows you will end up thrashing memory and unnecessary network data transfers.
Basically do work where there is lots of memory.
If you have dedicated backend developers and front-end developers — or worse, separate teams — then each and every change becomes painful. Writing bog standard UI and backend code shouldn't require in-depth specialist knowledge, and you can usually collapse those down into a single role.
I also agree that the frontend requirements should drive the shape of the API. If you have a backend team they tend to picture an idealized consumer of their API. They stress over whether the API follows RESTful principles and how to model things out correctly, and version breaking changes so their consumers can have their own update cycle. But when you step back and realise that the API has one, or maybe two, consumers. And it turns out writing some weird, esoteric, heavily coupled endpoint for a specific use case is not so horrible.
FWIW, Reviewable (https://reviewable.io) follows roughly this design and, after 6 years, it's still a fine system to develop in. It runs on top of Firebase RTDB which takes care of the security rules; if anything, I feel better having all the security rules articulated in one central location (the schema) rather than spread throughout the backend. Communication between client and backend also flows through the database (with very few exceptions), which lets the backend get away with an easy to scale queue-based design.
There are downsides, of course. As many posters alluded to, changing the schema gets tricky. It basically must remain backwards compatible for at least a few releases so as not to inconvenience the users, and some kinds of changes are just not feasible. And yeah, how the client wants to fetch data does affect the schema as well, though in practice I haven't found the impact to be too bad. Also, Reviewable has only one (browser) client, so I don't need to worry about duplicating logic -- I probably wouldn't pick this architecture if multiple clients were a requirement. Putting much of the business logic in a client-side model layer integrated with the database structure actually works out really nicely, though I still struggle with how to efficiently share some of the logic with server-side code.
Overall, I'm satisfied with how things worked out and would definitely consider using a similar architecture again in the right circumstances. The article is right on the money that if you're just getting started on your MVP it's worth thinking about a thin- or no-backend approach.
It's good to see a considered response, rather than the typical knee-jerk reactions common on Internet forums including HN.
Alternatively, it speaks of a dysfunctional dynamic between front-end and back-end developers, but no amount of technical solution fixes that.
Man, this article is just handing out terrible advice left right and centre
> 1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious.
This is already taken care of in the proposal, by permitting clients to access only data they own.
> 2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
No, because a) not every startup starts on multiple platforms b) you can always put common behavior into a piece that's reused. This can be a library linked in to multiple frontends, or a backend API. The post says your backend should be as thin AS POSSIBLE. It doesn't say you shouldn't have a backend at all costs, even if that means duplicating code.
> Forcing your UI to use the same entities as your database.
Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
> The advice in the article strikes me as dangerously sensible-sounding while being mostly wrong.
I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
But you don't propose any method for that apart from hand-waving about "database user accounts with read access", which is largely irrelevant unless you're proposing to give every one of your customers a database account and do row-level database security.
Let's talk about a concrete example: A call arrives at the back end to change the user's name to "dickwad".
In a conventional backend, you'd check the session, work out which user that session was logged in as, check that it's the same account as the name that's going to change, and maybe then run a profanity filter, or not.
How would you build your minimal backend to do that? Remember, you don't actually know that this call has come from the front end, it could be sent from curl.
> Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
True, but that's not mentioned at all in your article, and defeats half the point of it - the front end engineer still has to worry about what the database needs and how to do that mapping when it changes.
> I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
What misconceptions? I understood you (almost) perfectly. It's a well-written article. Still wrong, though.
I'm curious how this works. In a database that stores something like an address, how do you prevent someone from slightly changing the query to give them someone else's address stored in the same table. I've heard of giving per-table access, but never per-row.
> Your DB is likely to contain internal state that has no business being sent to the client.
Which is why not all tables need to be sent to the client. The post gives an example of exposing the friends table to the client but not others.
> If you do this, any business logic not captured in your backend has to be duplicated in each front end.
No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
> How do you make breaking DB changes if you take on this strategy of API design?
You change the frontend!
> This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten.
Which is why the blog post begins by saying, "Say you've started a startup today"
> I would not recommend it for anyone wishing to build a lasting architecture.
In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
> Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
What if the "is_active" flag takes on more nuanced meaning over time, as you add "client_activation_status", and over time expand that meaning. All frontend clients of the API need to be aware of such low-level changes. An API acts as an insulation layer.
> No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
At one point the post says to get rid of it altogether and use SQL, making the architecture 2-layer. If you do have a backend layer, and it does have business logic, that business logic will stand between your DB and the client. This logic will vary in thickness, but in a mature application will be substantial.
> You change the frontend!
How do you force a synchronized iOS app update to all the people who have your app installed?
> Which is why the blog post begins by saying, "Say you've started a startup today"
Fair point. Still, it is not something I would do except in throwaway prototypes or proof-of-concepts.
> In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
Agreed on this advice. And you need to ask yourself how you will gradually adapt the software you've built so that it can change with the needs, risk profile of your clients, and growth of your company. An approach to API design you describe is fundamental, and it would be difficult to adapt it without scratching it altogether and starting afresh with v2.
> As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
Agree about not cargo-culting. Agree about adjusting approach to company maturity. My experience has showed me that the approach you suggest has too many disadvantages for building any production software.
I wish you best of luck and hope that at the very least I and others on this thread were able to offer another perspective.
In a traditional “MVC” style architecture this would be described as “thin controller”. But that kind of architecture often tightly couples other things that shouldn’t be, because they hide other transport/interface boundaries.
The way I approach this is pretty simple:
1. Interfaces between user input and “programs” of any kind are used for validation, sanitization, deserialization and interface translation. Only. No state changes can occur here. In the language of the article, this is the “thin backend”.
2. Any runtime business logic, using any technology for that runtime, clearly defines its input and output types. This layer is allowed to make local state change as appropriate, but any external side effects are forbidden.
3. If the layer described in #2 needs to communicate with an outside network, system, storage or service layer, recurse to #1. Contracts between these boundaries are only that and imply no implementation details.
4. If you implement #3 in the service providing #2, those responsibilities are explicitly separated.
5. Do business logic. Return data consistent with the business logic’s contract.
6. Rewrap the onion and eventually serialize the result for the outside client according to that layer’s expectation, at whatever interface boundary you’re at.
You're right, but the original post seems to believe there are only three layers: frontend, API, database. They haven't left any place for business logic at all, except maybe in the way the front-end would write the SQL queries. (Which I think is a terrible approach.)
The article says "build a backend API when it has something to do, not as a pass through to your database."
> Fast forward almost a decade
If your system lasted a decade, that's a success, and any startup that just started would love to be in your shoes.
> management still doesn't understand the concept of "overwhelming technical debt".
That's a problem with your company, not with the suggestion in my post of adding in layers of abstraction when needed rather than ahead of time.
But that's what I've been missing from your article - more on the distinction between the approaches, and the drawbacks of having a thin back-end layer as the product / company is growing. In the beginning I thought you were talking about how to make your startup more efficient and flexible - basically consciously building some technical debt, having weighed your options.
But if that's not the idea - personally I don't know how to scale with this approach, neither technologically nor in terms of team structure. The thin back end layer could work though if you have a lot of small products that are independent of each other - developed by small teams that can afford to act like early startups. Maybe that's what you meant, but I didn't really get that from the article.
They tried this system, and then realised they painted themselves into a corner. Businesses wants features they can sell to customers, and sometimes fixing technical debt is a luxury. Especially when technical debt hidden to customers.
Adding layers of abstraction when needed might sound like great advice to a startup that needs money ASAP. But down the line when requirements change, maybe those layers of abstraction cost too much time to implement, especially if you're firefighting a dozen other poor choices.
It's a pretty impressive house of cards.
Refactoring (or rather re-designing) this sort of thing while still churning out new features is tricky to say the least.
Not really. You can do row or column level permissions. Details very much depend on your DBMS.
> Never mind that, how do you prevent denial of service attacks?
I have no real answer, and I never suggested that I would. Denial of service attacks (deliberate or accidental) are one piece of the "broader concerns" that I very much share.
In theory, a sufficiently advanced DBMS might be able to assign users quotas, and if your users are stable that may be sufficient for some use cases, but I am skeptical.
However this article is insane because of the security nightmare of trusting all incoming queries