Structuring Clojure applications(yogthos.net) |
Structuring Clojure applications(yogthos.net) |
My theory is that this is result of no guardrails on how to structure your application. Clojure to be productive must be used by people who absolutely know what they are doing when it comes to structuring your app.
When it comes to Java you get hordes of devs still producing passable results. The structure is largely imposed by the frameworks (mostly by Spring/Spring Boot) and available help, literature. Even some antipatterns at the very least achieve some level of convention/predictability that is needed to be able to find your bearing around the codebase.
I would say, if you have a normal-ish project, care about productivity but don't have really stellar and mature developers -- skip Clojure.
Choose Clojure if you know how to use all that additional power, have need for it and understand what your added responsibilities are.
If you don't know how to wield Clojure's power there is very little you can gain by choosing it but a lot to loose.
Why did this happen?
- We had small common framework that everybody used, at the very highest level (think application lifecycle management). That imposed some amount of consistency at the most basic run-the-program stage.
- The devs communicated openly, a lot, so there was some general consensus on what to do, and what not to do.
- The team at large was very suspicious of introducing new macros. You could do it, but you'd better have a really good reason.
- When I went to make the changes, I didn't have to worry about spooky-action-at-a-distance kinds of consequences anywhere NEAR as much as I do in other languages. Being strict with your state management, as Clojure strongly encourages, REALLY pays off here.
The actual problems I had were entirely related to the overall build system, the fractured nature of the source control, and figuring out who was responsible for what code once we were 3 reorgs deep. The code itself was remarkably resilient to all this nonsense.
Python applications seem to benefit from this as well, and I've encountered a surprising amount of resistance to it from other developers. I think everyone has been burned at least once by "over-designing", building too much of the wrong abstraction. But the result is that they are never willing to commit to a common internal framework even long after the need for one has become painfully obvious.
Usually this happens among developers who have been solo developing a project for a while, and see themselves as YAGNI zealots fighting the good fight against excessive abstraction and overengineering.
I understand and sympathize with the sentiment. But when every design decision is ad-hoc and as-needed, it makes it really really hard for external contributors to make changes to the existing codebase. It discourages contributors from "big-picture thinking" and eventually leads to the dreaded Ball of Mud design, with some combination of:
• meandering flow control
• poorly-defined or nonexistent interface boundaries
• inconsistent naming
• lack of documentation, or incorrect documentation and/or comments
• redundant safety checks, or absent safety checks
• poor runtime performance
• difficult-to-test code that freely mixes I/O and business logic, requiring complicated test fixtures, tests that are difficult or impossible to change, and poor test coverage (antipattern and code smell: "idk how to test that, don't waste your time. did you run it on the QA environment and check that it worked?")
I think it arises as a misunderstanding of the forces that lead to overengineering and building incorrect abstractions in the first place. The problem is usually one of expanding the scope of an abstraction or framework too early, not of building the abstraction or framework in the first place.
I find the mistake people often make is to get clever with Clojure. Using macros when regular functions would do is a good example of that. It is absolutely possible to write impenetrable Clojure if you start doing weird things just because you can.
However, I find the beauty of Clojure is precisely in the fact that you can write simple and direct code that solves the problem in a clean way without the need to get clever.
My biggest advice for structuring maintainable Clojure projects is to keep things simple, and to break things up into small components that can be reasoned about independently.
Another is -- lack of good IDE support (which is connected to lack of strong typing). In something like Java you can always understand what is the thing under your cursor.
But an experienced developer knows how to deal with these problems. Lack of strong types means you have to be building clean, well specified interfaces. Lack of strong typing in language does not mean you don't have types in your program -- it just means that it is now on you to make sure it is easy to figure out what is the exact specification of piece of data at any point in the program.
I also found that spec rarely helps. One of the major points of something like Clojure is to be able to create general purpose functions that can do useful operations on very wide range of data and then use those functions to compose your program. Spec really hinders your possibilities here.
> It's a shame, really, since the language is vastly more flexible and powerful than most, but it goes to show that there are some things that have, in the wild, outsized influence on productivity for the 80%.
It is the curse of Lisp. It is the most powerful language that can exist and yet it will never be mainstream because it requires a whole other level of development mastery to really get the productivity benefits.
I personally use Clojure for my rapid prototyping which is to say -- "as long as I am sure nobody else will ever need to work on it with me".
I did a very efficient algorithmic trading framework proof of concept in Common Lisp and it was a joy to work with. But then when it came to actually productionising it I was forced to rewrite it in Java at the cost to performance and heaps of boilerplate code.
It because apparent over time that the lack of types and the other features provided by Clojure resulted in much smaller and simpler codebases compared to those previous languages and frameworks that I toiled in for so many years.
It has really highlighted to me the value of simplicity for better productivity and maintainability. I wouldn't even want a framework to build web apps using something like HTMX for example. Clojure handles HTMX in almost magical ways with a simple library or two such as hiccup.
Recently, I was doing some work on a Java / Spring project and was dismayed with the proliferation of classes and packages; really the complexity of it all. And remember, I am solid with Java experience, so it is a result of those types of languages and architectures IMO.
Totally agree. My very initial motivation that caused me to get interested in Clojure was my study of various API clients generated I think by Swagger Editor? All clients were littered with generated boilerplate -- except for the Clojure one which looked clean and exactly as you would imagine some intelligent programmer writing it. As I already had experience with Common Lisp I immediately understood what is happening here.
> Recently, I was doing some work on a Java / Spring project and was dismayed with the proliferation of classes and packages; really the complexity of it all. And remember, I am solid with Java experience, so it is a result of those types of languages and architectures IMO.
I also have over 20 years of experience with Java and I also share your experience.
Recently I have started mixing OOP with functional and FRP. For example, most of my code is now FRP (ReactiveX/Reactor) and for some strange reason Java is superbly suited to it. It is not a panaceum but I found that I can frequently write what would normally be large features even in minutes.
As you mention, if you're building websites/services, I think the biggest "problem" with Clojure is the community hasn't rallied behind any particular framework.
I'm reminded a bit of companies. The point of a large, successful company is to slow its employees down enough so they don't kill the goose that laid the golden egg. The point of a startup is to find the goose.
When it comes to programming, Java is the former. There's nothing particularly special about Java that keeps you from coloring outside the lines, so to speak. It just makes both coloring outside and inside the lines harder.
Of course, in steps the standard frameworks everyone uses.
I contrast Clojure a bit with Elixir. I find both to be similarly "powerful", in similar and different ways. But the community has rallied behind a web framework (Phoenix) and it's pretty easy for the average web dev to just read a book and be told how they should do everything.
If you agree web development isn't a solved problem, do you want to couple your entire language to a massive framework that is committed to solving an un-solved problem in the general sense using technique X that presumably will need to be replaced Y years down the line? Or not work in situation Z
Personally I don't want that for Clojure, instead I see lots of different approaches in Clojure (and some outside) that could be the next big thing:
- https://www.hyperfiddle.net/
- https://github.com/whamtet/ctmx
- https://github.com/leonoel/missionary
- Unison
Ultimately I want to go long on my programming language and short on "the one true framework" until we have a one size fits all approach for solving the general problem of web development, until then give me low coupling libraries that I can mix and match
And that's exactly what Clojure is doing right now, maybe in the end it will be something like chatgpt just mix and matching the approaches for you but for now I'm personally not looking for a framework to be the only way to use Clojure
Once you get past the syntax, a lot of the day to day thinking about how to model your solution in the context of the system is similar. to the point that I'd say elixir is Clojure with a ruby like syntax. elixir has let bindings but they are implicit and blend into the language. you don't really think about them. At the same time, you have all teh standard fare of a functional enough language without all the type ceremony.
Id say its more convenient to work with elixir day to day. Less tracking of parenthesis and the code naturally comes out looking more uniform. There is the price that macros are harder to write in elixir. That is not to say its difficult but lisps in general make writing a macro so damn easy that anything else feels difficult in comparison. I think its a good compromise. Its just difficult enough to make avoid creating macros until you REALLY need them.
Regardless of whether I agree, I'm struck by this thinking as a sad commentary on the state of our industry: devs can't be trusted to actually architect the code for maintenance, so they should be forced to color by number with frameworks.
If you had to consciously make every single decision about everything you do you would die. That's where habits and patterns come to reduce the number of decisions you have to make to a reasonable level.
Most software development projects really are the same as a lot of other projects. You get API services that basically shuttle data between database and web interface, you get frontends which basically shuttle data between APIs and UI, etc. There is no need to invent everything for every project you do, it is much more productive to just focus on things that are specific to your project and adopt the rest from either your experience or some ready made guidebook.
It is not necessarily a lack of trust. It is just that when I hear some development manager to start on a journey of reinventing everything for a pretty mundane backend project it immediately suggests to me a lack of common sense or prioritising personal goals over good of the project.
Honestly, I find myself structuring my own code to be dummy-proof and paint-by-numbers, even when I am the only user of it (:
I don't see it as a bad thing, necessarily; sometimes you want to "bake in" structure and conventions in various places so you can use your creative energy elsewhere.
Though I do take your point that this is not necessarily (or at all) communicated to junior devs, who are sometimes plopped into a little artificial coding cage and discouraged from reaching outside of it.
Depends on your perspective and intentions, I suppose.
It all comes down to execution, in the end.
This holds true as long as you're not buidling your startup on something truly outlandish like brainfuck or piet
But I understand where you are coming from. Understanding an existing clojure code base is exploratory in nature. You'll REPL into it and run the functions you have difficulties with.
This is a very different activity than chasing object references and hunting down method calls.
For background I'm a devops and software engineer and I'm comfortable with Javascript, Java, Python and C.
I am implementing my own multithreaded programming language and I have a switch based interpreter and a compiler that codegens for it.
I think other people's Clojure code is unreadable.
Clojure code resembles the author's mental model of the problem being solved. Which is often very different to how I model the problem to be solved.
It's not the complexity of Clojure codebases that I have trouble with, it's the syntax and the approaches to the problems being solved.
I would use LISP for AST generation and codegen but not for direct programming.
I think both eventually found their niche and now have plenty of developers that stuck around write appropriate code in it. But when they were shiny and new they indeed both created their fair share of failures.
- process / workflow (everybody goes on hacking a topic for weeks and then integrate ?)
- team size (I can foresee how clojure would make large team a problem, but small teams happier)Also, not every company can hire best developers (even though most claims so).
Imagine you are project manager for project X which is pretty mundane backend with pretty mundane API and you are given a deadline to develop it.
Will you:
a) try to find absolute best developers on the market and then use absolutely most powerful language (Lisp) to get it done?
b) try to find some developers that are available and use technology that is adequate enough that they can work with not shooting their feet off?
Take into account that you might not be able to hire best developers and even if you do, they might quickly leave not being satisfied with the mundane job you gave them.
Thanks for sharing!
The advantage over simply passing credentials is that the initializer for the client can validate itself when it loads. If you just pass the credentials and assemble the client when you try to send the message then if some variables weren't set correctly you only find out when you try to use the client at as opposed to when application starts.
Also, I saw numerous cases where people will copy/paste multimethod arguments without knowing what they are used for.
I still find case/cond more readable and way more performant, especially since the author uses the same type for a multimethod dispatch, but YMMV.
If your method only needs 1 argument then why should the other ones matter? I don't see a problem here. clj-kondo will guide people to name them as _ or _thing anyway so you don't even need to think about it.
From my perspective, I reach for multimethods when I want to provide my caller with the ability to declare their own "actions" as it were. The goal is to offer an open interface.
Without that need, yes, case or cond can be sufficient.
But Clojure does look amazing :)
Learning Clojure has made working in other languages much harder for me - pretty much all of my Clojure programming is functions plus transforming data structures (mostly maps). So few concepts to keep in your head!
The majority of my work is in TypeScript these days, and I always write the most Clojure-y TypeScript I can manage.
For testing purposes is easier to redef a function than implementing a full new test protocol.
I thought it was worth mentioning that you can use them to encapsulate any effectful code since they can be used as a tool to help enforce a bit of discipline. If you're just calling functions that can cause side effects then it can get tricky to figure out what all the functions that need to be redefined are. You basically have to read through all the code to know what you need to mock. If you stick all the side effects in a protocol, then you're being very explicit about what needs to be mocked out.
I'm wondering, though, doesn't the same apply to Ruby, Python, and Node projects?
I've over-hauled 80k line Python projects, and the "lack of typing" there seemed to apply as well.
Why don't Ruby, Python, and Node projects suffer from the same critique? Genuinely curious...
Model state where it belongs: in a database.
That's a lot of exits :)
I found ClojureScript worth it a few years ago. I’ve written two frontend apps (desktop-style interactive single-page applications meant for longer sessions) of reasonable size in ClojureScript. There’s a lot of non-glue code there (in fact mostly non-glue code, given the interactivity and statefulness of the applications).
Very little language and library churn has made maintaining them very straightforward really, and I won’t be rewriting them any time soon. Being able to use DataScript nearly made the choice worth it on it’s own!
The reason I’m mostly using TypeScript these days (for new things & backend code) is that it’s just too helpful for typing the data structures and reducing the "how many things do I have to keep in my head" burden. My TypeScript (like my ClojureScript) is mostly just functions and data structures (avoiding classes, inheritance, etc), and I avoid using any of the more complicated TypeScript features as much as possible.
It’s kind of heretical, but if Clojure had a well-adopted gradual structural type system (essentially what TypeScript has done for JavaScript) then I would find it hard to not pick Clojure for most things.