Parler Hosting Requirements

Parler Hosting Requirements(twitter.com)

59 points by prepperdev 5 years ago | 29 comments

prepperdev 5 years ago |

Copying their requirements as text (original grammar preserved, including GB and Gb confusions).

It's a good glimpse of what unlimited cloud budget does to companies. I quote:

"""

* 40x i3.Metal instances (64 vCPU's, 512 Gb ram (more ram, of course, better for those)), 14 Tb nVME e/a (Scylla cluster)

* 70-100x (96 vCPUs, 768 GB RAM) - 4 Tb nVME e/a (PSQL cluster, could use more RAM and CPU, but not easy to get)

* 300-400 various other instances, less picky, generally 8-16 cores w/ 32-64 Gb of ram as available

* Internal traffic ~300-400 Gb of traffic/minute

* External traffic ~100-120 Gb of traffic/minute

"""

ev1 5 years ago | |

Wait, is this a joke? This sounds like the worst built infrastructure I've ever heard of where they had more money than engineering sense.

Also, Gbpm.

prepperdev 5 years ago | | |

I don't believe it's a joke. It still might be a deliberate lie (a random twit is not a real source of knowledge), but so far this is consistent with what we already know. See https://news.ycombinator.com/item?id=25769730

eejit_compiler 5 years ago | |

By my calculations, this is somewhere between 11,680 and 18,560 vCPUs and 83,840 to 122,880 GB of RAM they're asking for.

Seems rather excessive for a mid-tier social networking website.

h2odragon 5 years ago | |

What's that total up to in "expected AWS bill per week", say?

ev1 5 years ago | | |

It's at least 6 digits a month for the first item alone not including bandwidth, IP, NATGW, or load balancer

sschueller 5 years ago | |

What language is this thing written in that it requires this kind of computing power? Is it an Electron app? /jk

tinus_hn 5 years ago | | |

It’s the server side

zxcvbn4038 5 years ago |

Everything we’ve seen so far suggests they didn’t (don’t?) have the best tech people working for them, but they do have a lot of money at their disposal. This looks really over-provisioned compared to the reported size of their site, there is no caching layer that I can make out, I bet they are using those forty i3s to host user content as blobs in a database. If they added a couple varnish servers and moved the user content to S3 (or similar - you can get much better deals with the same api from other vendors) they could probably ditch > 90% of this.

When I was at Tumblr they also overbuilt a huge amount of infrastructure just because they had the investor money - they dropped something like 1.4 million on two Cisco routers, each one could probably handle all the traffic in South America - but you need two of them, right? They dropped another half million a year on four network engineers to manage the two routers, and they had nothing to do most of the time so they sat around staging gladiatorial contests between a couple turtles and whatever the pet shop had that day. I’m sure the users would have been outraged and canceled Tumblr if they knew that was going on.

jfengel 5 years ago | |

I'm surprised that they don't have better tech people. HN has a fair number of people expressing support. I'd have thought that they'd be able to attract some engineering volunteers.

zxcvbn4038 5 years ago | | |

Would be a fun technical challenge but current people might be defensive about what they built, could end up being more politics then progress.

Havoc 5 years ago |

Given that they forgot to strip EXIF data I’m going to go with there being something completely absurd behind this.

Actually that might be it. If they didn’t strip metadata then maybe they’re shuffling around all images and videos in whatever format and quality they got it? ie one big data lake of mystery content so to speak

floatinglotus 5 years ago |

Gigabit per minute? Do they have a single knowledgeable networking architect on staff?

Why not just measure things in random-sized-image-files-per-full-moon?

zxspectrum1982 5 years ago | |

I'm afraid you have misread.

They are talking about traffic volume (bytes per minute; usually you'd see megabytes or gigabytes per hour but they are massive so they said per minute).

You are talking about transfer speed (bits per second).

You may have a very fast link (transfer speed = 40 Gbps) but transfer very low traffic (1 byte per hour), thus network is mostly idle.

Or you may have a very slow link (100 Mbps FastEthernet) but transfer a ton of traffic (gigabytes per hour), thus network is very busy (which requires good network architecture, server architecture, etc).

As others said: these requirements seem insane for the size of Parler. Maybe they are looking into massive redundancy, different locations, or even doing something else?

throwawaysea 5 years ago |

According to Wikipedia, Parler has only 30 employees total. That presumably includes everything from management to engineering to legal. I wonder how big the engineering team is and how that affects these requirements. My casual guess is Facebook and Twitter are an order of magnitude more efficient with their hardware given their large staff and scale?

h2odragon 5 years ago |

I have said it before, I will say it again: Own your infrastructure.

rvz 5 years ago | |

> Own your infrastructure.

Absolutely. I have also said this countless times before in relation to self-hosted git. [0] Each time GitHub went down every week last year, I have made the case against the 'centralising everything' argument.

The social networks that didn't survive de-platforming are the ones that didn't self host. Parler is on that list.

[0] https://news.ycombinator.com/item?id=23849565

x86_64Ubuntu 5 years ago | |

It's hard to "own your infrastructure" when you needs are so extreme. Part of the job of tech workers is to make things efficient so that the company doesn't end up with an outsized OpEx or CapEx. With all this hardware required, they have made it impossible to own their own hardware.

cbozeman 5 years ago | | |

Uh, is it just me, or does that sound like a hell of a lot of hardware for a site with only 15,000,000 registered users and an average of 2,300,000 active ones?

jazzkingrt 5 years ago | |

That doesn't solve the the software/architecture problems that led to these huge hosting requirements.

chipsa 5 years ago | |

Just need to own your servers, network up to a backbone peer, CDN, DNS registrar, credit card processor, and bank.

touiexpress 5 years ago |

Not sure the original screenshot is true. Some people mentioned a Jarod but he's not listed on Parler's LinkedIn page.

More informations :

- line 1 : the Scylla DB Cluster would cost 59 000 USD according to AWS Calculator. Scylla DB is the C++ version of Apache Cassandra. It's a database.

- line 2 : this sounds like a bunch of postgreSQL server. Estimated price of 250 000 USD. Might be cheaper than Amazon RDS (?)

- line 3 : 400 instances 16 cores 64Gb -> 136 000 USD

Grand total 446 000 USD

We heard that it was around 300k USD on the news, so who can tell if it's true or not... anyway.

I don't see any wordpress nor specific technologies requirements.

elect_engineer 5 years ago |

How were they paying for this?

I don't see any sources of revenue that could fund the hosting described.