Worst: CDC's NOS/VE on a Cyber mainframe, closely followed by their NOS OS. The naming was a bit like Javascript and Java, NOS/VE was vastly different than NOS, but both were very, very bad in very different ways.

After mainframe OSes, I'd have to say VAX/VMS is the worst I've used. This is partly because of the aggressive "need to know" culture that typically kept the official documentation locked away, but also because the OS just didn't make sense. It was very idiosyncratic, from file naming, to command line flag style, to its process model.