I love Erlang, and am one of the very few people who (in the past) managed to find work doing it full time. The actor paradigm it provides is surprisingly simple and elegant to write an app, even on a single node, and of course making distributed apps is fairly straightforward as a result of it.

I have found that I like Lisp-Flavoured-Erlang (LFE) a bit more, since I still find the prolog-esque syntax a bit frustrating. I've been doing this stuff for like 6 years, and I still occasionally get tripped up on when to do a comma, semicolon, or period occasionally. While I'm not a huge fan of LISP-2 semantics, overall I find the syntax for LFE to be a lot more consistent than raw Erlang.

This isn't to crap all over regular Erlang though; Erlang was the first language I had seen that fully embraced pattern-matching, and I find that kind of pattern to be incredibly elegant.

If someone reading this hasn't tried out Erlang, do yourself a favor, download and install it, and build a simple TCP chat server. I can virtually guarantee you that it will be easier than anything you've built in C/C++, and it's incredibly satisfying to have something you built in a few hours have the ability to scale correctly.

It's a gem of an observation that newcomers to an environment often only make it to the first few rungs of the system complexity / actualization ladder. Once you look, you see this everywhere, and not just in IT but with any sort of design exercise or institutional process, any complex system that's full of Chesterton's fences, and people will either evolve to debate their rationale from a position of experience, or be selected (or self-selected) away into fresh environments where the novelty and discovery-until-discouragement can begin again.

It also shows that the first few rungs matter a lot; this is the territory of easy answers, where following a few simple rules leads to rapid productivity, and there will always be people for whom that level is good enough, either through carefully weighed decisions from limited information, or ignorance and deferment of future problems and their solutions. You can't solve everyone's problems for them, but you can try to evangelize, and you can try to build your system in a way that best practices can be incrementally adopted from existing fumblings. Anticipate that most of it will stay mediocre.

History is littered with systems that, in hindsight, seem to have offered sensible solutions to complex problems, yet didn't survive in the end. Much knowledge and wisdom is lost, and others independently discover it when trying to ascend an unrelated stack. Erlang/OTP is truly the sort of environment that masquerades as a programming language yet asks questions at a much higher conceptual level: what do we want systems as a whole to look like if we have to maintain them indefinitely? Its architectural innovations have been copied elsewhere, where they rarely form part of a coherent stack, but at least expose people to the advantages of its model. This may also be the most viable source of adopters of Erlang at higher levels: people who've sought out similar model for its benefits, and could thrive with an offering that pays attention to these concerns throughout.

Re: those infrastructure components written in Erlang not attracting Erlang users in any permanent way—it’s not so much that they have more users than contributors, but rather just the bare fact of them being infrastructure components. A black-box infrastructure component can’t be a “killer app” for a language.

Rails is a killer-app for Ruby because it’s a framework; a developer who uses Rails in their project is inherently a Ruby developer. People learned Ruby to use Rails.

But you don’t need to write a single line of Erlang to use CouchDB or Riak or Ejabberd. They’re self-contained. (And, in fact, scriptable in not-Erlang languages; CouchDB for example spawns and manages a JavaScript engine to run its “design documents” on.)

Thus, these infra components becoming popular doesn’t cause there to be any more Erlang programmers than there were before.

In that sense, I would say that Erlang has never had a true “killer app” yet. (Outside of a few specialized domains, like telecom and fintech, where the OTP platform itself is Erlang’s killer app.)

And even in the places where Erlang itself is a “killer app”, it’s also not so much a viral thing that the whole industry gets excited about; rather, Erlang is more of a “secret sauce.” Nobody’s learning Erlang in their basement because their friend told them it’d be the next big thing in their industry.

How does anyone learn Erlang, then? Well, usually, engineers are forced to learn Erlang at one job (where the system architect there decided on it); they get to liking it; and then, when they later become a system architect themselves in another job, and choose a stack, they choose Erlang, thus forcing a new wave of engineers to learn it.

I find the key point to language adoption is that self-selection that occurs. What type of person and their personality does an ecosystem attract and what is that type of person good at?

The BEAM ecosystem is all about fault-tolerance, distribution, and concurrency. These BEAM concepts could be described as "upper ladder" ideas: they require more prerequisite understanding of systems engineering to appreciate. Despite being a fallacy of composition, to compare a given Erlang vs Ruby/Rails developer may be useful to consider.

A person who invests in learning Ruby/Rails might be motivated to do so because of that bootstrapping mindset of MVP's and failing fast. Our average Rails developer might be rolling the dice for that "killer app" opportunity more often than our Erlang developer prioritizing concurrency and reliability. This may be because those BEAM features are more useful at a later stage of product development. So while the BEAM features are uniquely powerful if they require an upper-ladder understanding our MVP-developer doesn't yet have - they're unlikely to pick Erlang as tool-set to invest in.

If we accept that the BEAM is good at infrastructure and that doesn't result in as many opportunities for these hype-cycle-causing-killer-apps, than what could be changed in the ecosystem to support this goal? Can we get both the later stage benefits of reliability and concurrency in addition to the early-stage productivity benefits? Maybe this early-stage development focus is where Elixir will break out of the infrastructure-niche Erlang seems to be in. Can Elixir's tooling reach a short enough new-product feedback loop more potential killer-apps get deployed into production? I think as Elixir brings in some different personalities to the ecosystem we could see more killer apps but that idea -> production feedback loop seems to be the key component.

Here's to another 10 years of Erlang/Elixir. The programming stack for the boring software developer that wants to clock out at 5 sharp. It's fantastic I tell ya. I have been blessed with opportunities to work with Elixir fulltime and it's hands down the best experience I've had. Compared to C#, Rails, Nodejs, it's miles ahead. (Although C# with dotnet is coming up fast and _crazy_ compared to what it was 9 years ago).

This ladder is a great blueprint, great article:

    functional programming
    isolated processes and concurrency
    reliable concurrency (links, monitors, timeouts)
    OTP behaviours and other system abstractions
    How to structure OTP systems
    How to build releases and handle their life cycle
    How to never take the system down, and how to operate it

This is how the page looked when Erlang posts caused HN to become slow.

http://www.waybackhn.com/?date=2009-03-11

Good observations.

'Getting' OTP, and especially how to structure an application the right way is definitely tough, because there's not a lot out there like it if you come from the world of C or scripting languages. It's not just code organization, but thinking about how to structure all the processes so that things fail in a 'good' way, and you have everything talking in the right way.

A big thanks to Fred for his book, and the many hours he has spent helping people on the mailing list, IRC and elsewhere.

I miss my time working with Erlang. I think it's still an indicator of an environment willing to try new things, but like he writes, you have to be wary of the places that are 'serial early adopters' too; I've encountered a few of those in my Erlang career.

Fantastic article; it paints a great picture of the journey taken & accomplished achieved.

My personal take is that the next major milestone for Erlang/Elixir will be a significantly more performance BEAM. Be it JIT or reworked Hipe.

Elixir (& Phoenix) brought a ton of interest from the Ruby community. The only remaining aspect that would keep a Ruby developer using Ruby is the perf vs Erlang. Today, for many raw perf task there isn’t a meaningful difference in speed (note I’m not taking scalability). As soon as the speed difference becomes meaningful, Erlang will get another wave of people.

As someone who loves servers and back-end work, Erlang is amazing to me. I too read that book by Joe Armstrong, but I never really got to use Erlang for much since I don't get paid to use it. Recently I was trying out Elixir and going through the documentation and was pleasantly surprised how much of it is Erlang-like.

I wish the industry was much more Erlang friendly.

Interestingly enough I started using Erlang around the same time and while I've continued to use it for the project I started then on and off, I've not had a lot of reason to use it on other projects. I don't ever think it'll be as wide spread or as popular for many of the reasons listed. I do think it will continue to be a tool used to create great projects with very few hands. In the right hands and for the right projects it is an incredibly productive tool.

"But the vast majority of them will be a continuous flow of serial early adopters who surf from tech to tech, sniffing the best opportunity to gain a competitive advantage by being first to use a type of framework, language, or toolkit." - Oh god, that hit me so hard.

I was there. Had a full time Erlang job even before the line numbers in stack traces. Later migrated to more enterprise-y languages... and for me, the big missing thing in Erlang is the IDE. And a company (or a big community) that is dedicated to catering to the developers. This was never a thing in Erlang.

Riak convinced me that any language pretending that it could magically create better products was full of shit.

Riak is not great. It performs well, but doesn't scale well. It has tons of operational overhead, a lot of bugs (and it didn't help that a slowly disintegrating company left less time to fix those bugs "until next release"), and any changes to the cluster while it's being used heavily leads to instability/unavailability. And many of its features were not implemented great; the SOLR functionality is a pig, for example, and "enterprise" replication is flaky.

On our team, we had a few engineers who could write Erlang, but they slowly moved to other teams, so troubleshooting production issues became a black box of pain. I'm so glad the product we had using it was sunset.

This is a fantastic article.

A question for rubyists out there: what does the "ladder of ideas" for ruby look like?

Was a bit more than 10 years for me. I had coworkers struggling with netsnmp to build out a mock control plane for a new and exciting data center management platform. I whipped up a mocked out version with OTP in a really short time and eventually the mocked version was good enough to plug into the real backend so we shipped it.

It was elegant and clear!

For what it's worth, I decided to pick up Elixir based on this post.

I have never written a single line of Erlang, but I run my own ejabberd. However, looking at the package build scripts and build instruction, I have the impression that compiling a project like ejabberd can be complicated.

Does someone know if this is an inherent Erlang problem or just due to some weird project setup in this case?

I started learning erlang and was stuck at 2nd rung, problem with erlang is(at-least for me) that there are not enough projects/use cases to learn/build for practice.

So, what's the quickest way to get into OTP?

To me it looks like Erlang itself isn't all that interesting, and OTP is where all the fun and magic is.

RIP Joe Armstrong

Functional programming has too long of a learning curve to the average programmer. A language should be judged by how long it takes average programmers to become proficient in it, not the "Sheldon Cooper" types. In typical medium and large organizations, it's difficult keep being selective about programmer hiring. (There are org structural/political reasons that would take several paragraphs to explain.)

This problem existed since Lisp was invented. The benefits would have to be large to overcome the downsides of this learning curve, and so far they are not, except in certain niches.

I hate the rain the functional parade, but it's been tried and retried for many decades, but will just not fly in the majority of the real world. I'm just the messenger. The jet-powered chainsaw works wonderful in the lab, but actual lumberjacks either can't figure out how to start it or blow their arms off.

If you disagree, please reply instead of negativate me. I would greatly appreciate that.

Agreed. This is actually why I think the killer apps for the BEAM are 2 Elixir based ones.

Nerves, which is already being heavily used for embedded systems and only getting stronger thanks to the likes of Scenic.

And Phoenix LiveView (not released yet), which I personally think pulls the strengths of Phoenix together in a way that can truly disrupt web development. It’s very difficult to do what LiveView is doing on other tech stacks.

The combination of those two will likely fuel things long term.

I don't think this counts, but I imagine a significant portion of people learning today learn it because they are Elixir programmers.

I think you should add to this comment "in my opinion."

Because in my opinion C# is miles ahead of Erlang. If I want to create an application, back-end, front-end, running on a wide variety of platforms, Mac, Linux, PC, IOS, etc, I would use C# 1000 times over versus Erlang.

The tools available, documentation, huge amount of libraries and support mean that developer efficiency in C# land is light-years ahead of Erlang. I could make a website + backend + IOS/Android app with all shared code while using tons of pre-existing OSS frameworks / packages and get it all up and running and prod/enterprise ready in no time at all and have it ready for enterprise load and scalability. Try that with Erlang!

Comparing Erlang to C# is a bit unfair. The latter is a general purpose language that can be used to build software on every layer of every platform. The former is a distributed programming DSL (an awesome one granted). There is 0 Erlang-native GUI libraries, there is no documented Erlang pipeline for iOS apps etc.

I'm hardly smart. The only thing I have going is I'm determined.

I have no CS degree, never went to university but have been doing freelance web dev work for ~20 years.

Been through a lot over the years but never took a shot with a functional language until I met Elixir recently.

It really didn't take that long to get to the point with Elixir where things somewhat started to click. I'm still very much a beginner and am learning multiple new things for every few hours I'm coding but I am able to make progress without feeling like I'm in way over my head. Of course I still have so much to learn about Elixir / Phoenix, Erlang and OTP but it'll come in due time. You don't need to take in all of the complexity at once.

I'm multiple thousands of lines of code into developing my first Elixir / Phoenix web app and there has been stumbling along the way but also great strides of progress.

The blog post is not actually about making FP accessible to the masses nor a how to guide about it, and if you read through the post, it’s even stated in there that it’s not important for the language to gain widespread use.

That’s why I downvoted you. It’s not because of your opinion, it’s because it’s probably not in the right thread.

Hi there,

I sometimes try to help normals relate to me by asking, "You know Sheldon Cooper from Big Bang Theory? I'm like a stupid, slighty-less-social-idiot Sheldon Cooper."

I identify with Dr. Cooper. (BTW it sucks being like that. Don't ever think we do it because we like it. There are a few perks but it mostly sucks. Also, it's like living in "Idiocracy". I can't watch that movie because it's too painful. That's my life. From my point of view y'all are running around pouring Gatorade on the plants talkin bout "It's got lectrolites!" It's getting seriously scary now IRL too: where the fuck are all the fucking insects!? We should all be fucking terrified right now.)

Anyhow, from my POV the "average" programmers should GTFO and stop peeing in the pool. I would fire 90% of working programmers. They're not needed and actively counter-productive.

Also, FP is coming on stronger today than ever before. You are in actual fact just wrong.

I think the current research on learnability doesn't come out quite this way. People are quite ok with declarative formulas, consider eg the popularity of spreadsheet programs among non-programmers. But people who learned imperative programming first need mental work to switch.

There are lots of things that were tried and retried for decades before they caught on, which are now standard. For example look at the 90s when Java and Python came around - GC and memory safety were radical ideas for the industry where C/C++ were standard, even though they had been around in various languages for decades.

> I'm saying that the non-elites are counter-productive, that they contribute negative productivity.

You may have concrete examples of that, but I couldn't agree based on my own perspective/experience. I've worked on a team with an elite programmer (in terms of actual 10x productivity) for nearly 3 years now. I know they're not all the same, but this guy isn't particularly sophisticated or cerebral in his approach; in fact, he wouldn't want to take on architectural concerns, refactoring, or really complex problems - I and another dev take care of that. He's just incredibly productive, by any measure you'd like to use -- LOC contributed, modules written, features implemented, issues closed. I've sometimes wondered if he's a front for an entire team behind him.

That doesn't mean the rest of us aren't worth having on the team. We are definitely contributing positively. In fact, he couldn't work anywhere close to the rate he does if he had to take care of the stuff the rest of us do.