I figure you could "build a Steam" in a couple of years, with the right engineers hitting the main features. There's very little magic at the technology level, and you can make life simpler and forget about minor things like the hardware survey or the pretty graphs. I'm not saying this is trivial, but it's definitely doable.

This is a far different statement than "You can build something and compete with Steam in a couple of years". Most of the really hard problems are not technical. Success ain't gonna happen without a bunch of pain, sweat, and strategic stumbles on the part of the competition.

This is such a Hacker News comment.

It's just a crud app - why do they need more than 10 employees?

A lot of the secret sauce of such things are not that secret but just take a lot of work.

Building and maintaining infrastructure simply takes a lot of people, time, relationships and whatnot.

They get good at it over time which I guess could consider some secret sauce but there isn't like some secret code that makes the whole thing way better that now you'll see tons of competitors.

Everything is easy to build until a small nation state’s worth of people want to use it at once.

Just stream the video, it’s easy!

I mean doing Youtube is even easier; it's just a wrapper around HTML5 video.

Everything is just a crud app with a few extra steps.... yet you're not Zuckerberg or Dorsey

You don't need a genius. You need a few good people, and a lot of hands. I think the best way to look at things like Twitch is to compare them to cathedrals, bridges, things like that. You might be able to have the idea and sketch the plans by yourself, but it's physically impossible to build it yourself.

Like all things web, the problem is scaling the platform and moderation/security. It wouldn't be hard to build a toy Twitch clone no. But it takes tons of people and money to scale it / secure it. And even with all the security, they still got hacked...

This reminds me of the Albertsons guy on Blind who inadvertently created a meme when he said that Facebook could be rewritten with a small cluster of Oracle dbs. The meme is that Albertsons people are so elite, they work and think in a higher level of existence, way above the scalability bs us commoners are accustomed to.

Right, just like a plane is a car with wings.

Sadly, Reddit stopped updating the public repo for their main application in 2017:

https://github.com/reddit-archive/reddit

Oh I'm sorry, here's what I read:

"Building apps is easy as long as you don't have millions of users. For that you have to actually think about bottlenecks, the larger architecture etc."

(I agree with that)

What I wanted to express is that lots of engineers I personally know instead say

"Building apps involves thinking about every bottleneck in advance and optimizing for every possible user scenario and a global user base, regardless if the number of users is only ~100."

I’ll grant you immature, nothing else.

It’s just luck. I mean, if I was a storyteller, what story would I have to tell if there was no story.

They hit.

It’s sort of like we all hold Golden dice, so we marveled, by our own eyes, at the gold.

Dealer: You rolling those?

Us: no, it’s gold.

They fucking risked it. It’s not a engineering feat, we’re all a bunch of pussies.

Twitch is easiest site to build, you might as well show me a todo app (which will be sieged and dismantled), scale is solved, we will eat your applications, the barbarians.

Rome falls.

Yep, we saw it happen live.

Any bits you recall from the chat?

It is still fraught with problems, while you (knowingly) wouldn't do it with PII, is not all that reassuring, others could, or compromised system could be used to exfiltrate this data, if the only control is just trust on the users behaving well with their access

That fact in general industry the controls on how PII data is accessed internally is so lightly managed should worry everyone

I've heard (but don't have any actual evidence more than hearsay) that Twitch generally operates independently of Amazon/AWS. I'm sure that they share some things, but I wouldn't be surprised if their source was separate from the "main repo"

Remember that Amazon runs one of the biggest multi-tenant service platforms in the industry! A separate business unit like Twitch is likely to be set up a lot like any other random AWS customer, and you wouldn't expect that compromising servers used by one AWS customer to automatically compromise the underlying infrastructure.

(I would also expect that the Amazon retail systems are in most senses "just another tenant" on AWS, albeit with much more liberal quotas!)

I'm working on a project and just had to repull my workspace after some local corruption. I pulled 1.2TB out of the office and never got an email. I think it's pretty common for places not to monitor egress that closely.

on firefox, disabling referrer means you won't see the image

    network.http.sendRefererHeader = 0

> When you're owned by Amazon you don't need to optimize too much to achieve web scale... just leverage AWS services. It's not like you're going to get a bill.

Oh you're be surprised. Divisions get billed constantly for the AWS resources they consume, and this bill gets taken out of their annual budget. From what I hear, this is a common practice in most large organizations.

Also, the AWS services you can access from within Amazon are almost identical to the AWS services you can access as an external customer. It's equally easy/hard for a random company to achieve web scale, compared to Twitch.

That's a bold claim.

I'm not sure what point you are trying to make. If you look at most of the high profile hacks and leaks in the past 20 years, very few of them are from web 2.0 tech companies (e.g. Google, Facebook) rather than dinosaurs (ex. Target). Those that have (like Google) have only been successfully breached by nation actors (e.g. China, NSA).

As far as I can tell, there's no data to back up the assertion that these large tech companies are disregarding security if favor of profits, except for Twitch now, which is why this leak is interesting to me.

> In my experience with corporations there is little to no technical sharing between different parts of the company.

Amazon is all about sharing efforts with the company. That's the whole point of AWS - its a monetization of this efforts. Most older AWS services started out as internal services that someone realized was generally useful.

Parent company was no different.

The better question is, why did random engineers have access to the financials of the streamers on the platform, without having to go through a break-glass, audited, emergency access escalation.

> And it might be disguised as a video stream coming out of the video streaming servers.

Just log in to FB messenger or Discord and egress it as small data chunks that way. Lots of people have private chats on work computer for practical purposes.

Discord allows for bots, so you could easily write a script to chunk data and egress, and another to re-assemble.

I don't see how this precludes just-in-time access. Even if people can re-up on their own, you can still observe the data access patterns and manage the risk. Further, when you see someone is getting blocked a lot you can improve the experience for them so they are unblocked, or have more efficient access to the data. This is just mature data and security management.

Quality of life and developer experience are important topics in many ways, but should they really trump security consistently? It's always going to be dependent on people's risk assessment and comfort, but frequently it skews the wrong way because the people making the decisions know that they'll be gone.

I’ve worked on a lot of DLP projects in big enterprise, and I have a very dim view of the entire category of product. A lot of their functionality is just magic black boxes, that unsurprisingly achieve very little. The primary motive for deploying them is not that they’re particularly effective, it’s so that you can tell auditors and other scrutineers that you’ve got a “DLP solution”. The idea that you can grant people access to huge quantities information, but then very strictly control what they do with it is fundamentally flawed. Especially on networks that require large amounts of in and outflow for BAU. Even the most tightly controlled data in the world cannot be protected from an inside leaker (or adversary who has taken control of an insiders access), because it runs into the same “analog hole” issue that DRM products have.

The enterprise DLP tools were deployed fleetwide as agents and at network choke points; getting access to the raw data wasn't the problem.

The streaming part or the downloading/looking at code?

You can look at leaked source code for educational purposes in most places (not legal advice). As far as I understand leaks are commonly used in vulnerability research for example (if the bad guys can use it so can bug hunters).

Streaming copyrighted material is a separate issue - but using it for "criticism, comment, news reporting, teaching" should fall under fair use, no?

What's wrong with looking at public code? The code is public, regardless of how it became public - this isn't someone's personal life being exposed. If twitch is damaged by streaming this, it's only because their poor code quality is being examined publicly.

I can certainly understand why twitch banned this and don't blame them (although I think it's stupid), but I see nothing unethical about openly talking about this code in the public now that it's already there.

Its still more secure. Rubber hose cryptanalysis applies to both equally, but that doesn't mean there aren't other attacks that apply to totp which don't to yubikeys.

More secure != perfectly secure.

Yes.

I don't know which protocols they use (obviously), but if they use WebAuthn, everything is public-key signatures. Even if you leak everything from the server, public keys buy you nothing.

https://webauthn.guide/

Oops, didn't mean to be too too negative. I say embarrassing in the sense of, I've definitely shoved out awful code because something needed to get out(tm). And with large companies, deadlines that cause that situation are inevitable.

But I also say it like that because, well, I've seen code that causes (objectively easy-to-fix) crashes but still ships because of one reason or another: laziness, politics, inexperience. It's a part of software engineering I'm still trying to accept.

The whole point of devops is to automate everything according to best practices, so fuckups are a thing of the past! The only fuckups, of course, will be Terraform state issues.

Note that it was code that was leaked. Preventing developers from leaking the codebase they are working with is outright impossible. Now combine that with a "monorepo" and even the most junior developer has access to practically the entire company codebase and version control history.

And you can try to prevent them from accessing live/real customer data, but the cost is that they will never be able to debug issues in production. Most companies, even very large ones, are just not able to pay that cost. Not to mention that once you have access to the codebase there are a million ways to leak customer data anyway -- it is a lost battle.

Adding to your pet theory I think that WFH has led to a lot of people being casual about their workplace security. For example, leaving a laptop unattended at a Starbucks.

This is just a guess but I wouldn't be surprised if companies have to start taking stricter precautions with their security in a WFH world.

Really, you can't simply copy files from a code repo you're working on? You work on a isolated workstation, not connected to any external network, where you are not allowed to bring anything other than plain clothes (TSA-style)? With a sizable army of developers all working this way?

And if it's a remote FB/VNC connection, what is preventing you from just recording the screen? Not really hard...

Most companies I've seen could see all their code extracted with one malformed NFS packet. These are "air gapped" systems holding the type of industrial secrets that we don't want to leak to china. Practically the only real line of defense they have is employee screening, which does not really stop the lone man guy.

> You also get access to every streamer's dashboard and their analytics

I would classify that as access to production systems.

Why is getting access to prod, or prod data, considered a perk, exactly?

This statement makes no sense.

The leak includes source code of multiple active websites and applications that are operated under the umbrella of Twitch/Amazon.

Why would an intern have access to this data?

Why would you need to log into your computer with a yubikey? Wouldn't any computer (including the attacker's computer) work?

There are much better cases than this; in this case a monorepo makes it slightly more likely to be caught rather than less. (A monorepo can get to Google size and then you can't check it all out at once and it needs bespoke tooling, which can make it harder to pull this off.)

On the flip side while many smaller repos _can_ have independent ACLs, you are very unlikely to set those up until you reach a certain scale -- and then when you reach that scale it gets hard to implement ACLs across everything at once. So your engineers probably all have access to all your repos until you reach a very large size anyway. So the question becomes just "can someone write a for-loop over all of the repo names and check them all out," and it's like, yeah, that's not terribly hard, I as a programmer can do that pretty easily in bash.

Ideal repo size should not in my view be directed at "how do I prevent compromise to the external world," because VCS is not designed to give you the superpower of being resilient around being compromised. Rather VCS is trying to give you the superpower of time travel. So you should probably scope your repo to "what is the unit that makes sense to time travel with?" -- in other words if you are adamant that you have these independent services which operate decoupled and running this one backwards by a year should not affect that one, then those services should be in separate repos. If on the other hand they have some moderate coupling and rewinding this service by 1 year would break the APIs that that service uses to communicate... then those should ideally be in the same repo so that you can coordinate changes between them to their shared protocol.

You can still have a monorepo and restrict who has access to certain parts of it. You just have to build the tools to do it.

Google, for example, has a small number of subdirectories in the tree that only certain engineers can view (the really sensitive stuff, like the actual ranking algorithms for search and ads) but the build system is setup to allow you to still link against it.

Not particularly - unless different teams are highly focused on certain subsections of the repository. If everyone might have to look anywhere than you'll need to download all the repos - whether that's one or five hundred.

Clean OS install or new hardware should both be daily events at even mid sized companies. Because even if it’s once every 2-4 years per developer that still becomes extremely common in aggregate.

I suspect lots of junior devs will clone fresh, push changes, nuke repo and repeat. I did when i was young instead of syncing state and rebasing.

That's something I've wondered - do password hashes tend to be the same across platforms? Is everyone using the same hashing algorithm? Isn't this also what salting is for?

Never implemented auth myself.

Pretty much this. If they gain one email/username password combination - they can use it elsewhere.

The salt is usually stored next to the password. The point of a salt is just to make the hash unique to prevent the use of rainbow tables, it's not a separate secret.

I think it is pretty common to store the salts alongside the password hashes. They are used by the same pieces of code so it is generally unrealistic to think that your salts will be secure if your hashes are obtained.

Salting isn't really supposed to make a hashing algorithm secure by being secret but by being unique. Unique salts make hashing more secure because an attacker can't re-use a single rainbow table for multiple hashed passwords. That, combined with a sufficiently computationally difficult hashing algorithm, it makes it prohibitively expensive to reverse the hashes of all your users.

This may not be enough to protect high value users or those who use fairly common or easily guessable passwords. This is part of why it is so important that you don't reuse passwords. It's also why your application should reject all known passwords using something like https://haveibeenpwned.com/Passwords or any of the "common password" list you can find online.

Edit: If you do include a secret that is stored seperatly that is added to the password and salt when hashing, this is called "peppering" and these peppers are generally not unique per user.

I've heard this before, and queried how feasible an attack would be, as people always talk about just how bad this is but yet I've _never_ heard of someone having an account compromised through this vector, and I'd like to know how feasible it really is. Here's the sha1 of an unsalted password b85ffa7dae2cbed04e7d3335f6ebc43c8a5764dd

How long does it actually take in practice to break something like this? I would love it if someone could prove it to me.

Salts being exposed is not a massive risk in of itself, as the purpose of the salt is to prevent the use of pre-computed tables to reverse a hash into plaintext, forcing an attacker to bruteforce each individual hash+salt instead of being able to reuse work.

With regards to crypto mines being used for breaking hashes, if you have one based on GPUs, yes, you could reuse GPU mining hardware for cracking hashes, albeit with relatively low hashrates for current best practice hashing algorithms.

If you're looking at something like Bitcoin's hashrate and thinking that it could be used to break SHA2 hashes, as far as I understand ASIC miners, this is not possible, as ASIC miners are designed only for mining, and they don't really accept non-mining related inputs (ie, no arbitrary inputs to be hashed, unless it matches Bitcoin's specific steps for iterating over nonces).

> Ideally that would be useless because things are properly salted and you don't know the salt

I'm really curious where people get their ideas about salting. It's not just a word. It doesn't make one password any more difficult to crack. It makes cracking every password in a given database more difficult to do. A password's salt is public information.

I think many people are trying to say that in this world at many companies all that data is indeed stored and accessed together.

Maybe the super secure siloed world doesn't really exist outside of military/government organizations.

I have first hand info, and this is how it’s done. Don’t call somebody ignorant if you don’t have first hand info. Leave that for somebody who does.

Man oh man, big "No thanks" to a perk like that from me

"Hey, pick through everything I say with a fine-toothed comb and treat it as the official company stance!"

I don't really follow your argument. You've never heard of a hash being brute forced? I've done it myself multiple times, both for pen testing purposes and for password recovery on systems I control myself.

The LinkedIn password leak contained hashed (but not salted) passwords, and some of those where cracked and exploited in the wild.

My old gaming PC with a 1060 can apparently do ≈ 6300 * 10^6 hashes per second. Assuming your password above is az-AZ, 0-9 = 62 possibilities (with no salt) it would take me 10 seconds to test all combinations for 6 characters and 30 days for 9 characters. And it's a trivially parallel problem, making it easy to throw money on to make it wall-clock quicker.

It's just a simple brute force problem, I don't see what there is to question (beside the choice of SHA1 for password hashing...).

If they are the same everywhere, you can precompute a huge database of hashes (called a rainbow table) and simply lookup the hash in the table when breaches occur to find the password. By salting, every provider who stores credentials has different hashes for the same inputs which makes the approach far less attractive at a large scale.

It doesn't matter at all if one person's hashed password is identical across two of that person's accounts on two different websites. The identical hash will instantly let an attacker (with access to both hashes) know that this person shares the same password across two accounts. But that is of no value; the attacker is going to start by assuming that it's true anyway.

Salts are there to ensure that two accounts on the same website which have identical passwords nevertheless have different password hashes.

Couldn’t find it in 1.4 Trillion combinations. Used rockyou.txt with dive.rule.

Took me 6 minutes to try all 1.4 trillion passwords. So either you have a strong password or I messed something up. What is it?

In theory if your password was weak enough to be on this list it would take on average 3 minutes to break it on a GTX 1080.

The point of the salt isn't that it makes it take longer to break any one password. What it does is prevent you from re-using the rainbow table you generate breaking one password when you break the next one.

Sha1 is not a very secure/expensive hashing algorithm and thus does make it significantly cheaper to break even with a unique salt.

What if you did something like hash(plaintext_pw+"twitchsalt") <browser> ---> <server> hash(browser_hash + db_salt)

"Salting does not affect this attack in any way." Yes it does. If you habe unsalted passwords you can just use a rainbow table to look passwords up.

> It's not so much "how long does it take" as it is "how much does it cost"

So the answer is "It's too expensive to figure out in practice, unless you're being explicitly targetted by someone with nation state level credentials?", i.e. it's pretty much fine?

> Using a more appropriate hashing algorithm with a sufficient cost factor can massively increase the amount of compute needed.

But by the sounds of it, SHA1 is more than enough (given that nobody here is willing to brute force the hash I shared above?)

> Preventing the re-use of that computational effort on additional users is why unique salts are important.

The person who "cracked" my first hash found it in a list of passwords which was actually gotten from a plain text dump 15 years ago. That wasn't found by reversing a hash, so the compute wasn't reused. You are right that once it's cracked, it's cracked and that's that, but if your password _isn't_ cracked it's moot whether it's hashed with SHA1 or something more secure, as per above?

The reason I didn't give any more information on the password above is because you don't have any extra information on a dump of hashes from a twitch database either. If a password is only feasibly brute forceable for a specific algorithm by reducing the search space by many orders of magnitude, it kind of shows that there's not really any risk even if the passwords are unsalted for a person who hasn't reused a password.

That's for generating collisions, not preimage resistance. It's not particularly easy to reverse.

Salts do nothing for people with predictable passwords though. The salt is in the dump, so I can hash known plaintext with the algorithm and the dumped data.

Even if I can only hash a million a day, if your password is one of the top million most popular, and I have a good list, I'll have your password in a day. And if you re-used it...

Salts do make naïve brute-force, all-possible-strings approaches useless, yes.