HN was designed by one person (pg) who is an inveterate minimalist, has good taste, and likes text. He wanted to make a site that seemed bookish (his word) so it would attract people who wanted to read and discourage people who didn't (but in the gentlest way, by seeming boring). He also wanted to make a news site that looked like top, the terminal app.
One thing that I don't think is widely appreciated about HN's design is the balance it finds between information density (i.e. fitting a lot on the page) and minimalism. If you try to move that needle in either direction it makes things worse: more cramped or less informative. Most attempts to redesign HN add tons more whitespace, but this lets you take in fewer stories at a glance. Of course, tastes vary widely and many people disagree.
The link to Reddit is genetic, since pg gave the original idea for Reddit to Alexis and Steve, and only made HN when his feature requests were no longer at the top of their list. Presumably that is also how the idea of keeping HN small got started :) There was also an early version of Delicious that looked a bit like what HN and Reddit later became.