Ask HN: Best way to get hands on large scale system design exp

15 points by jessehorne 2 years ago | 4 comments

Hi HN,

I’ve been in the industry for some time and unfortunately the reality is because I haven’t worked for companies with millions/billions of active users, I don’t have hands on experience designing those systems. I can pass system design interviews usually that are related to that but I believe that’s only because the lack of time takes away priority from an actual, practical, real, scaling plan.

I’d like to be more confident in my system design not just for interviews but because I’d like to actually be prepared to design a system to serve millions of active users. I’d like to have the experience to say with assurance what services I would use, what common challenges are, etc. I really want hands on experience versus purely theoretical.

My first thought goes to designing a system and using simulations/stress testers but I feel that could get really pricey.

Are there services or resources out there to help? What is your advice for me? Are there internships working on these sorts of systems for engineers who have been in the game for a while? Also would love any book/reading resources to expand my knowledge.

moomoo11 2 years ago |

You can look into systems design primers and just build stuff.

Honestly until you actually need to do it in production it just needs imagination and some core understanding. At least that’s my experience from working at a unicorn where we scaled from a few customers to millions, with tons of background work and queues and what not spread out globally.

You can try stuff at home and emulate how bigger stuff might be working. Or read up on their systems design and try it yourself.

Nobody knows until they have to address it. But they can only do so if they have some level of imagination/creativity and understand how computer/networks operate.

https://github.com/donnemartin/system-design-primer

palashkulsh 1 year ago |

Imo it's about learning base concepts which remain the same and get utilised in bigger systems more forcefully. It's similar to learning algo and ds we might not get to use them in our current work but people in places use them and need them.

Coming back to system design, when I say concepts i mean design patterns. Like two different rate and capacity systems are coupled using buffering systems etc.

Good resource for this is Microsoft system design blog series. Go through them , they have standard problems and scalable solutions that can be adopted.

Another part of system design is distributed systems, how do you attack distributed system design again learn the design patterns that are frequently utilised in distributed systems.

How you can learn that is via excellent blog series and book by unmesh Joshi of thought works.

If reading is not your medium then there are ample video series on YouTube.

Nutshell- learn design patterns in system design.

ipaddr 1 year ago |

You can simulate many users by scaling down your resources. Try the smallest box and max out one aspect and then try another.

phba 2 years ago |

If you want to get better at system design, there is a single question that you should ask yourself at every single step:

  Do you really understand what is going on?

You already noticed that system design interviews don't necessarily represent reality. Together with all the content from self-proclaimed system design "experts", you risk fooling yourself by thinking that your knowledge of high-level concepts actually makes you competent. Don't fool yourself, ask the question.

I suggest you start small and don't jump into the deep water (yet).

1. Pick a really simple project that has potential to scale, for example a webshop that sells a single item.

2. Limit the scope to the absolute minimum, since it's a learning project. Don't bother with layout or design or fancy JavaScript. Plain HTML will do, anything else is a distraction.

3. Set up a single webserver and make the project work (be able to create a user, log in and buy a quantity of the item; ignore payment systems, just make up a currency).

4. Your first thought is good: build some bots that make purchases, test parts of the system or otherwise simulate real usage.

5. Now start thinking: What can you do with this single server? How many users does it support and why? What can you do to support more users? How many requests can the server handle? What happens when the requests exceed this threshold? Where are the bottlenecks of the system? What happens if the server is unexpectedly shutdown? One question will lead to another. By adding features you will come up with even more questions.

Assuming that you want to build large web systems, you must first understand what you can and cannot do with a single webserver. There is no point going further, until you got that.

In order to get better at system design, you must gain understanding. In order to gain understanding, you must observe systems and ask questions (and find answers). There is no easy way, pick one question at a time and work your way forward. That is how you become an expert.

Good luck.

PS: This is a book I personally found useful because it covers many aspects you will inevitably encounter in a large system:

https://ocw.mit.edu/courses/res-6-004-principles-of-computer...