Taming the beast that is the Django ORM – An introduction(davidhang.com) |
Taming the beast that is the Django ORM – An introduction(davidhang.com) |
First, it let's you get started quickly and prototype. You can write unit tests, make sure everything is working as expected, then count queries and make sure you're being efficient with your SQL engine.
Second, and even more importantly, it's crucial in the definition of the app and the Schema. Thinking in high level "classes and objects" helps with the abstraction and the design of the apps. Even if you then default to raw SQU queries, thinking and building your model with class abstractions is huge.
Finally, there are some "tiny details" (but in my eyes, very important) that everybody oversees:
* Migrations: the way Django has designed migrations is just marvelous. We've migrated tons of data and changed the structure of our DB multiple times without any issues.
* Troubleshooting and "reporting": the ability to fire a quick shell, load a few models in Django's REPL, and do a few `.filters()` is for me key. On top of that, we add a Jupyter server connected to our read replica and we can do all sorts of reporting VERY QUICKLY. Not everybody needs a Data Lake :)
PS: We've ran Django sites accessed by millions of people per month. And we never had an issue with the ORM. Yes, sometimes I have to tell more junior devs that the gymnastics they're doing with `select_related` and `prefetch_related` can be more easily and effectively resolved with a query. But that's it. I'd say less than 1% of all the queries in our codebase have to be migrated to raw SQL.
It has happened that a dev in our team was populating an endpoint that used inheritance and when looking at the number of queries we were over 100.
But the solution in those cases is just OUTER JOINs and CASE. Especially since we use Postgres in the backend and it works so well.
So yes, there are some pitfalls, but they're greatly overshadowed by the advantages of the ORM.
It's like understanding memory allocation if you're writing c. It's just part of the job
I put that word in quotes because I don't want to imply it's a problem - it's just something you need to know how to handle.
Too often I come across Python projects that are hyped and when I dive into it I find it rather underwhelming to say it politely. Invariably it turns out that those people don´t know any other language than Python.
I see that as a general problem; Python is the language for beginners this day with an endless amount of tutorials, but it seems lots of those starters don´t get to have a look outside the Python world. I fear this is getting an even bigger problem because AI models are trained on a vast range of Python, not because it is better, but simply because it is the PHP of these days.
I don´t mean to imply that I know you have that "narrow profile".
I agree that the value in Python is for quick prototypes in case you know Python well. Outside of that (at the risk of a language war), I think one does better look into modern .net core or the JVM for a general purpose, high quality and highly efficient language/platform.
On the topic of ORM I think EF Core (.net core) and Doctrine (php) is strictly better.
> Too often I come across Python projects that are hyped and when I dive into it I find it rather underwhelming to say it politely. Invariably it turns out that those people don´t know any other language than Python.
To give a nice counterexample: Python's hypothesis library is really great, and compares favourably to its Haskell inspiration of QuickCheck.
Edit: That said, learning multiple programming languages (paradigms) is a very good idea. A reasonable dose of e.g. Java, .NET or C++ for example is great for understanding how programming languages and software should not be written. Learning modern JavaScript makes it obvious how Python really suffers from weak functional programming support, very bad performance and a horrible packaging system.
The widespread use of Python is a problem? People should learn java? Okay.
I'll note you don't talk about the usefulness of the projects themselves, only your opinion of what the code "looks like".
Meanwhile Django's ORM just does the job perfectly fine in the 99% common case, and for the 1%, again 99% of the time it has enough escape hatches to solve the hard problems without making things excessively complicated. But $DEITY save you if you're in that 0.01%...
I don't know how people build websites in any speed without such a tool. I guess they just waste time duplicating effort on an admin UI or pay for one of those tools that can generate one from an API (meaning they have to also build an API).
It's such a massive time-saver I switched to django after a week or so.
Also, I believe your code for creating an empty "data" migration is missing the "makemigration" command itself.
The last time I needed to do that, I ended up crying "uncle" and writing manual SQL. I wasn't happy about doing it, but I was happy that the framework left me an escape hatch so that I could.
Thanks, I will update the data migration code!
Routes, form validation, REST API, templating (if you don't need react), auth, etc.
You'll probably wind up recreating a lot of ORM and surrounding functionality at lower quality
Databases are the bottleneck your classic website. We choose to query these databases in a extremely high level language called SQL. This language is so high level that you need to come up with tricks and query analyzers in order to hack the high level query into something performant.
A better abstraction would be one that's a bit more similar to a standard programming language with an std that has query related operations/optimizers that can be composed so programmers can compose and choose query operations and avoid optimization issues that are hidden by high level languages like SQL.
We are unfortunately, sort of stuck with SQL (there are other options, but SQL remains popular because years of development has made it pretty good in spite of the fact that it's a poor initial design choice). This is already a historical mistake that we have to live with. Same with javascript (which has been paved over with typescript), same with CSS, etc. The web is full of this stuff. It's fine. My main problem is the ORM.
The ORM is just another layer of indirection. You already have a high level language you're dealing with, now you want to put Another High level language on top of it? ORMs are basically high level languages that compile into SQL.
What is the point? The ORM isn't actually making things easier because SQL is pretty much as high level of a language you can get outside of having an LLM translating direct english.
The point is OCD. Programmers find it jarring to work with SQL strings inside their beautiful language so they want to chop up a SQL string into web app language primitives that they can compose together. Query builders operate on the same concept but are better because they aren't as high level.
This is basically the main reason why Many programmers experience the strange counter intuitive phenomena about why ORMs actually makes things harder. You have to Hack SQL to optimize it. Now you have to hack another ORM on top of it in order to get it to compile it into the hacked query.
Then I tried the new generation of typescript SQL query builders that automatically infer types from your database and provide end-to-end type-safe query results, even for highly complex queries. And since then I became convinced the answer isn’t trying to shoehorn classes on top of relational tables, but having a sufficiently integrated and capable query builder that provides properly typed results.
And yet using it was such a shit experience I switched permanently to writing SQL.
I see zero advantage to using an ORM over SQL and in fact see many downsides.
Don’t use an ORM just learn SQL. It’s faster, more direct, more powerful, easier to understand, allows you to make full use of the power of the database and your knowledge isn’t suddenly valueless when you go to a project with a different ORM.
Good ORMs make sure your data model is always aligned with the DB, and there is no way in hell a runtime error can occur. What's more, it automates the mindless data class/table matching for 99% of your use-cases.
Once you got a more sophisticated query, please use SQL. But for simple stuff, use ORM.
I’ve read a lot of criticisms of ORMs, as I’m sure everyone else has. Some of them are certainly valid criticisms that are immovable and just inherent in what an ORM tries to do. Some of them just seem to be caused by not very many ORMs being good, and the writer not having used one of the better ones.
All you need is the django toolbar so you can check their efficiency, then keep telling it to make them more efficient.
It's been particularly helpful for aggregations, subqueries etc. You may not think LLMs are there yet but my project is proof. Try it yourself.
Imagine an electrical engineer saying, “I don’t really know what a bridge rectifier is, but I plugged these things into the breadboard where ChatGPT told me, and the output signal looks correct.”
I use https://pypi.org/project/django-nplusone/ for instance. Sentry also warns of these by the way.
Actually it seems they are: https://code.djangoproject.com/ticket/22492#comment:11
.prefetch_related (for whole models) and annotate/Subquery (for fields or aggregates) have existed for many years, alongside a pile of aggregate functions which have existed forever and have improved.
Whether or not you use the tools given to you is a sign of developer quality and experience. You can easily avoid n+1 99% of the time but you have to appreciate the problem exists.
I think the Django project demanding some competence is okay.
And it encourages CRUD thinking instead of thinking about business processes and user experience
It's great for tiny/personal projects but in an organisation it can be a trap IMO
- the main reason to use django today is the admin UI
- and the only reason to use django ORM is the admin UI
It feels like much of the value in that is achieved even better by using a proper database client, say, JetBrains‘ database integration. I certainly wouldn’t let any business people touch a Django admin panel, or at least those I’ve seen built by our data guys. Way too technical.
And don’t even get me started on extensibility. I wanted to have like, an additional page to include a Grafana iframe plus a bunch of buttons. Something that would take me about five lines in Laravel, for example. Good luck even searching the documentation for this…
It means you can focus on your business logic while buying time to get you through the initial development.
is it possible that the fact this article is written in 2006 simply makes it dated? it seems very catastrophizing but we've come a long way and are just more aware of how to work with or around the shortcomings and flaws of ORMs.
I've often written programs where I'm trying to encapsulate pure in memory state into business objects and run into the same type of issues people complain about with ORMs. programming is just hard, we don't have to be such dogmatists about it.
I don't think the article being from 2006 (or 2004) makes it dated (though it does have a date). I think it addresses a fundamental issue with ORMs that will always be there, which makes it a bit of a classic.
David Hang's Django ORM article has a bullet-point section on ORM cons, including "difficult debugging", "performance", "hides underlying SQL". Whereas Neward's article goes into depth on each of the following topics:
The Object-Relational Impedence Mismatch
The Object-to-Table Mapping Problem
The Schema-Ownership Conflict
The Dual-Schema Problem
Entity Identity Issues
The Data Retrieval Mechansim Concern
The Partial-Object Problem and the Load-Time Paradox
If you haven't read the original article, I highly recommend it. You'll learn a bit more about "Vietnam" (from a US perspective), and be better equipped to discuss and make decisions about ORMs afterward.
[My own opinion is that ORMs can be useful (I wouldn't say never use one), but that a programmer should be grounded in SQL and the relational model[2] first, so that they'll know when to, and when not to, use one.]
[1] https://web.archive.org/web/20220823105749/http://blogs.tedn...
> This eliminates the problem quite neatly, because if there are no objects, there is no impedance mismatch.
But:
> Integration of relational concepts into the languages. Developers simply accept that this is a problem that should be solved by the language, not by a library or framework.
> Over the last several years, however, interest in "scripting" languages with far stronger set and list support, like Ruby, has sparked the idea that perhaps another solution is appropriate: bring relational concepts (which, at heart, are set-based) into mainstream programming languages, making it easier to bridge the gap between "sets" and "objects".
I agree that bringing relations into your language is a good thing, but many languages are strong enough to do that as libraries, there's no need for baking support into your language. Just as hash-tables can be implemented as a library in most languages.
We have relations as a library in a Haskell dialect I was working with in a previous job. Worked like a charm, and no language support was needed. Python would also be flexible enough to use relations as a user-defined datatype. And so would many other languages.
So having an ORM in place that is tightly integrated in Python and Django lets even the junior developer fresh from the bootcamp make changes to an existing application.
It’s not a pretty story, but in my opinion the reason for staggering layers of complexity is the ability to move quickly even without experts on the team.
What? You have to understand a language to write performant code in it. That’s not a hack, that’s basic competence.
The poster is not referring to understanding the language, he/she is referring to having to guess at how to structure the query in a way that increases the chances that a good plan is within the constrained search space of plans (due to it being a combinatorial problem and the optimizer has limited time and information).
The garbage collector is something you should not think about when programming and most programmers don't even need to know it exists. You only think about it when you need to seriously optimize things.
For SQL it's a bad design choice because of what SQL is and what SQL is targeting.
SQL is a high level abstraction that automatically determines a query plan based off of high level input. The problem here is that it can choose a bad query plan. So you need to "hack" the query in order to trick the query planner into doing what you want.
It's also targeting the slowest part of the stack: Non volatile memory and IO. The slowest part of the stack should be targeted with a zero cost abstractions to maintain speed while the fastest part of the stack you can use a language like python for your web app it's fine because databases are magnitudes slower.
Although over the years my code trends more and more towards `.rawSql` or whatever equivalent exists. Even for the simple stuff. It’s just so much easier than first thinking up my query then bending over backwards three times to make it fit into the ORM’s pet syntax.
Plus raw sql is so much easier to copypasta between different tools when you’re debugging.
And before you say “but sql injection!” – that’s what prepared statements/parametrized queries are for.
Once you get sufficiently familiar with some paradigm the training wheels can come off.
“Raw” SQL is already an abstraction. Over time all the implicit magic will get on your nerves. Trying to shoehorn two completely different worlds into one abstraction is not worth it: you get to learn today’s untransferable funky ORM syntax and idiosyncrasies while losing sight of the actual skill that matters long term which is SQL itself.
I concede however that handling of SQL, the field names, the relations, is annoying. But it’s core to the problem you are probably solving (some form of CRUD). Plumbing is annoying but as a plumber I’d say get used to it instead of wishing to be dancer.
I notice this in other aspects of my work as well. When I switched away trom desktop environment to terminal I had the same feeling. It’s easier, less hassle, less wonky abstractions, more direct. Completely counter to what popular culture is telling me.
But the team chose EF due to it supposedly being easier to integrate with whatever database the customer might be using and that seems like valid reasoning.
I started using Django on I think version 1.2 or 1.3 in 2011, back when it didn't have database migrations and you had to use a library like South for it. Even then, as an ORM/query language it was apparently better than what other languages have now.
That branch merged on May 1st 2006: https://www.djangoproject.com/weblog/2006/may/01/magicremova...
I've long found Django's commitment to not breaking too much at once inspiring. The release notes and upgrade guides are really solid.
If you started from Rails and ActiveRecord, you're probably not be very appreciative of Django's ORM.
Define the object, hook it to a view, use our custom permission class, done.
GET, POST, PATCH, DELETE all urls, filtering, pagination, search, ordering, complex multi-org multi-user object permissions handled automatically. Need custom logic? Just overwrite method.
It’s a productivity superpower, sure there’s a lot to learn but that’s the price you pay for the tools that make you hyper productive
SQLAlchemy is leagues above and beyond Django ORM. I can’t say I have nightmares from dealing with it, but it certainly was not pleasure. A bit too much magic here and there, not flexible enough and provokes too much bad practices. Re-defining save() methods to do a lot of work, anyone?
The best “orm” that I’ve ever used was not an ORM but a query builder — HoneySQL from Clojure. That one is fantastic. No ORM bullshit, no extra data loaded from DB, no accidental N+1 queries. Not reading docs on how to do a GROUP BY, everything’s just easy to write.
Frankly, we often use SQLAlchemy as just a query builder too. Makes life so much easier.
The lack of support until very recently (and it's still lacking) for views is the main knock.
I'm rewriting a large Django project in Java (quarkus + jooq), because it's at the point where I need a type system now, but it still has a place in my heart.
I had the "pleasure" to use Doctrine once. Never again !
For query plans, in nearly every plan flip I’ve encountered, the root cause came down to either poor table / column statistics (because they had let the table grown too large without appropriate tuning), having an enormous amount of joins beyond the deterministic level the planner could provide, or sub-optimal queries. The latter is growing less common as planners get better and better at rewriting on the fly, but it’s always good to know how to do things optimally.
A good example is semijoins / antijoins. With modern versions of both MySQL and Postgres, you’ll probably have “WHERE foo IN (…)” turned into “WHERE EXISTS (SELECT 1…)”, but it’s better to write it the optimal way in the first place.
I will grant you that indexing can be bewildering, with its many rules, caveats, and gotchas.
Currently working in typescript and writing SQL queries out by hand since I just don’t trust anything to do the right thing wrt units of work.
The other person is the engineer here.
If you are not a fan of it however, you can use queries directly with '.FromSql(...)' retaining the convenience of object mapping. Interpolated queries use string interpolation API to convert them to parametrized queries that are injection-safe under the hood.
https://learn.microsoft.com/en-us/ef/core/querying/sql-queri...
Depending on the complexity of the product I think writing standard SQL92 will get you far as well.
I don’t get how an ORM will handle Postgres’ CTEs and windowing functions. Those are pretty basic and extremely useful features to me, but those require dropping to raw SQL. Each vendor has those useful particularities that you immediately lose by going ORM.
So in practice you are already required to make your queries as bland as possible where GROUP BY and JOIN is about as complex as it will get. I’d say just do SQL92 work around the limitations - which is something you’ll have to do anyway, but now more directly - and all major vendors will work OOB.
Django has supported windowing functions since 2.0, almost 7 years ago: https://docs.djangoproject.com/en/2.0/ref/models/expressions...
And people are experimenting with CTEs - this looks pretty natural to use, at least with the basic example, and it supports recursive CTEs even if the example looks clunky (though I think that's their choice of example rather than the actual recursive part making it look bad): https://dimagi.github.io/django-cte/ (repo: https://github.com/dimagi/django-cte)
The nice part is, in prod, that code _allows_ lazy loading since application stability is more important. Hopefully at that point you have a good performance monitoring tool that will alert you to that problem. Laravel also has hooks to more granularly fire events about lazy loading if you want to roll your own notification solution.
So I disagree with your assertion. Lazy loading is ONLY for production. While it is possible that such a feature could potentially bring down your DB server and therefor your app in general, if the feature is turned off locally, then hopefully you’re catching it well ahead of time. And if you’re running small app with no users, meh. For large apps, hopefully your team’s standards are enough to keep the monsters at bay.
Lazy loading is a sharp knife that requires skill to use appropriately.
It feels more like outsourcing said plumbing to someone that has done a lot of it in the past and will in most cases save you time, even if they won’t do everything the way you’d prefer yourself.
Throw in a bit of codegen and reading your current schema (if using schema first approach) and you’re in a pretty nice spot to be, except for the times when ORMs will get confused with non trivial joins and relationships, but typically then you have an escape hatch to have the query be in raw SQL while still mapping the outputs to whatever objects you need.
To be clear, I still think that mapping read only entities/DTOs against specialized database views for non-trivial data selection makes sense a lot of time regardless of the stack (or even some in-database processing with stored procedures), but basic ORM mappings are useful a lot of time too.
This is precisely why we introduced the "TypedSQL" feature in the Prisma ORM. For those who are interested in reading more on that: https://prisma.io/typedsql
You got to back up this claim. The technologies you mentioned are entirely orthogonal to the architecture you are using.
The benefits far outweigh the disadvantages.
Until it fucks up and you have no clue how to fix it, and either does the LLM.
This isn't something new.
And yes, I guess OP has now a large system that needs types enforced by the system to reduce the friction in evolving the stack.
> Note: The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc.
Then yes, Django model definitions are more like Java types in that they error if you try to use an incorrect type. You can't just ignore them like with type hints.
They also try to be 1:1 with database types, so for the most part any additional validation added on top of Django would be something you had to do anyway.
So have 2 components that ensure the types are correct? Why?
If you didn't learn it, you don't know it.
It doesn't matter if you can reason or 'understand programming' if you haven't learned what is necessary to understand this particular piece of programming.
Or is it that you think you can 'reason or understand' any piece of programming? And you possibly assume that is easier, to read what an LLM generates rather learning what is necessary to write it yourself?
I'm saying that your server side representation could also be done as a relation. No need for object orientation. I don't have a problem with the M part of ORM, but with the O part.
_If_ you are having different representations, than having an automated mapper between representations is good. Agreed, yes.
One big why is SQL. It's a horrible language/API but sadly practically all relational databases are SQL.
I’m really not a fan of ORMs, I don’t think they save any time because to use them in a safe way, you NEED to fully understand the SQL that will be run. There’s no shortcut from this and it’s plainly just an extra step.
I’ve been bitten too many times but one ironic problem is that they’re so damned reliable. 99.999% of the time your use of an ORM will be just fine. It can be hard to argue against sometimes even though that 0.001 time can be an extinction level event for the company.
So if you need to understand the SQL that will be run anyway and ORMs occasionally introduce disastrous behaviour, why persist with them?
Well to start with the obvious, SQL is not panacea, you can still be bitten with non-deterministic behaviour when you hand craft every query in sql (e.g. when was the last time you updated stats on that relation - will the optimizer select an appropriate strategy?).
But a stronger reason is that some harder queries just suck to write correctly in SQL. Maybe you’re working with some kind of tree or graph and need a recursive common table expression: Enjoy! It might be better in most cases to write such a query at a higher level of abstraction (ORM) and take advantage of the more productive testing facilities.
You can model relations in your language without using SQL. (You could have a mapper from your internal modelling of joins etc to how you talk to your database.)
In eg Haskell that works really well, but other languages are also flexible enough.
This is the trap of the local maximum.
As opposed to what mentality ? I mean any tool I worked with has the same problem.
> > Everything is great, you just
Part of the python mentality is to pretend difficulties aren't there in order to keep presenting itself as beginner-friendly.
https://docs.djangoproject.com/en/5.1/topics/migrations/#ver...
You just rename one of the two files, and add the dependency. If they touch the same fields, you obviously have to resolve that manually.
I've had the misfortune to use both Spring and Hibernate professionally.
SQL has other warts, apart from not being completely relational.