BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers.

I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi.

It's not to say that some vendor couldn't reasonably build a BSD based supercomputer, it's just highly unlikely given how much stuff is missing.

This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specialty HPC hardware a no-brainer.

https://en.wikipedia.org/wiki/Donald_Becker

On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation. They know how to run their python analysis scripts there and the only thing they have to change when going to the cluster is the adoption of an environment managament system (such as http://modules.sourceforge.net/).

(However, I have to admit I never got in touch with BSD and don't know the differences in user space)

"Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc.

In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By November 2006, Linux made up more than 75% of the top 500. You get the idea. Over the years, there were a few attempts by microsoft to get into supercomputing, and there were BSD and Mac systems."

Since time is sold on these supercomputers they probably want to run all the same/similar OS so they can compete selling time on them. Also if one person has success everyone else will copy them.

https://linux.slashdot.org/story/17/11/14/2223227/all-500-of...

Slashdot has a ton of comments discussing bsd vs linux on this subject matter, but I didn't see anything to helpful.

My only thought is large companies like netflix use bsd more for CDN because from what I been told bsd has the best I/O handling. Why they don't use it for the rest of there infrastructure? Maybe linux is better at crunching numbers and bsd is better for network and security? No idea thats my best guess.

So, in no particular order:

- Linux came on the scene when BSD's were mired in legal uncertainty. After the legal issues were settled, Linux had already become the default choice for someone wanting a FOSS Unix-style kernel, and the BSD's never caught up.

- The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!).

- Due to Linux gaining the initial momentum, developers flocked (and keep flocking!) to it, leaving the BSD's ever further behind.

- Linux was more welcoming to new contributors, whereas the BSD's were controlled by a small circle of core developers sitting on the commit access. And of course, the BSD way of solving disagreements was forking the entire thing, further splitting up the already small developer base.

I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know".

As for FreeBSD, it might be a technically better platform, but it is living in Linux' shadow.

Personally i run FreeBSD for the excellent documentation, stability, features like ZFS, but nothing i run couldn't just as easily run on Linux.

Many of the vendors for HPC (I'm looking at you Mellanox) primarily develop and certify their products on Linux. While they might work on BSD ok, you're not going to get the full performance and all of the features on a BSD system. If you paid for Mellanox EDR 100G Infiniband switches and all of the fancy VPI network cards, you want to use them to the fullest performance capable. The vendor tells you to use Linux for that, you use Linux.

TL;DNR: Linux is what the hardware manufacturers overwhelmingly target and work with. HPC users use what vendors support best.

Dragonfly's design shows promise, but it's not anywhere near ready for supercomputers yet.

This can substantially reduce the time to deploy new software and cut down on overhead related to managing multiple modules.

Singularity, unlike Docker, is designed to require only minimal privilege escalation and as such it's an easy sell to HPC admins, who can (at least somewhat) get out of the business of helping users figure out what the heck is weird about their environment when trying to get something running on a cluster for the first time. You can also take these containers with you and be reasonable certain they'll work on another system.

http://singularity.lbl.gov/

Linux has two things that are extremely useful compared to BSD:

1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Sometimes it is billed to different departments, but in most universities or research groups the "billing" is really quota allocation.

HPC is generally a "softball" workload because the code is going to be more sympathetic to the hardware than many other computer usages. Processes will batch allocate a lot of RAM and peg runnable state for a long time.

SMP.. "it depends", again a parallel vector matrix multiply is just going to sit in the runnable state on all the cores and the kernel is pretty irrelevant. There is a lot of junior job stuff left in FreeBSD to move locks around. The VFS is quite bad. In an HPC type workload these things probably wont matter that much unless you see a lot of "system %". They will show up in profiles and are generally also easy to fix. But it's not hard to construct a microbenchmark showing Linux > $else in those areas.

SLUB.. no. What kind of HPC workload is going to care much about this? The Linux allocators are pretty awful at contig kernel memory allocation (see ZFS on Linux). I don't see why UMA would architecturally flop here.

NUMA is a sore point on FreeBSD. It should be usable in 12.0. Isilon and Netflix are paying Jeff Roberson to work on it. Some folks on my team are also doing minor NUMA and locking work, but for commercial CDN workloads.

Mellanox does a pretty stellar job on FreeBSD Ethernet and Infiniband support. Unfair dig at them. I generally prefer Chelsio, but Mellanox has lowest latency which is relevant for HPC.