Ask HN: How to design a good technical interview test (take home or live) What are some principles in test design that make it revealing and easy to time-box, regardless of role/stack? Any specific test you designed or solved as an interviewee that stands out? |
On the other hand, I also don't like on-site interviews. A random question not-related at all with the position you're interviewing for is given, and have to solve it while you're beholded by a bunch of strangers... Being tired/nervous after a 3/4 hour in-site interview, scenic panic... To sum up, tThe recipe for disaster.
Whe I was the interviewer long ago, I gave them a questionary that could be solved in 1 hour. I made sure to tell the candidates that don't get stressed, look for whatever info you want in internet, and that there's no good/bad answers. In fact, I gave them the questionary and once the time has passed I would get them to check the answers.
There was several level of questions, to prune candidates at different levels. That also helped to determine the level of the candidate (I was working for a 200 employee non-tech company at the time).
I suppose that the questionary could have been asked to be completed in-place, in an empty room and without internet... But, what's the point of that? In the real world a developer is allowed to use internet, in fact, knowing to search is a skill I'd like to check.
Besides, if they had some open-source projects I would take a look at that, and at the end, there was a technical conversation between them and some senior developers about their career.
- Online shops.
- Hackerrank questions.
- Read from a big log file in an efficient manner with Python (did it but there was also a 3-4 hour on-site interview).Always present real time challenges, so that you are helping the teams pass obstacles and not providing tests that make no sense in teams therefore resulting in steep learning curves.
1. Design tests to be open-ended and demonstrate how a candidate thinks, rather than ones that drive toward a binary pass/fail outcome. Asking why a candidate worked on the pieces they did and where they would focus if given more time gets you high signal without requiring as much time from the candidate.
2. Provide clarity on the grading rubric. When candidates understand how they'll be evaluated, they'll spend less time on work that doesn't result in higher signal to the company. For example, having a candidate write some tests or talk through how they would approach testing is great! Expecting full test coverage isn't as useful.
3. Scoping is the toughest part about test design. Try to choose topics that closely mirror the day-to-day work in the role itself but that can be boiled down to a 1-2 hour task. You want to make sure that candidates who perform well are more likely to succeed if hired, but you also want to keep it short. From interviewing 50 hiring managers who designed the hiring processes at Basecamp/Medium/Mailchimp/etc, they revealed that the candidates who were ultimately hired consistently spent 2x the recommended time, which means that designing a 3-hour take-home is asking top candidates to commit 6 hours. It's a tough ask in the current hiring climate.
Here are 3 examples for inspiration:
- Full-stack role for Flow Club (YC S21) that runs virtual WeWork sessions: https://app.mightyacorn.io/flow-club/founding-front-end-engi...
- Front-end role for Playhouse (YC S21), building TikTok for real estate: https://app.mightyacorn.io/demo/playhouse-hiring-challenge
- Back-end role for zero5, a parking automation startup: https://app.mightyacorn.io/zero5/challenge-fullstack-softwar...
Happy to share more details if you have specific questions!
Source: my team spent the past 6 months designing personalized take-home tests for 20 VC-backed startups hiring founding engineers. We evaluated ~500 candidate submissions.
2. Needs to be explainable in 1 page with example data (MIN(completely))
3. Needs to not require any complex interaction. No web request, no complex libraries, just a unit testable piece of code. (MAX(impl speed))
4. Make it fun and something you and all your coworkers could easily do.
This has been my approach and it's been a good filter. If it's done before the interview have them come in and add a feature. Make sure they bring a machine with their prefered IDE / editor to maximize similarity to their real programming flow. Offer to do something else while they code so they don't feel "watched"
I prefer a 2 - 6 hour training day (depending on whats possible) for the best applicant, where he / she meets the team, sees how they are working and most of the people I met, were interested, because they benefit from this, too.
When you MUST provide a take home one, you could take a look at adventofcode problems as inspiration. They are fun to do, not too complicated and do not depend on a specific technology.
Learn about them, ask them to participate in a code review if you have to, pretend you work with them already.
HackerRank questions typically don't meet criteria 1 or 3. The best tests are often ones that the engineering team has designed themselves.
For example, this is an example of a test that meets all the criteria above: https://app.mightyacorn.io/flow-club/founding-front-end-engi...