We Prefer Coding Challenges(gravitational.com) |
We Prefer Coding Challenges(gravitational.com) |
* The challenges often don't actually matter, and are simply part of the recruiting hazing ritual.
* They're totally subjective and just a proxy for the same bullshit friends-and-resumes process people were using already.
* No consideration is given to the amount of time the entire recruiting process is taking.
* The challenges themselves are totally divorced from the day-to-day expectations of the job, so people are doing CS102 algorithms problems for jobs that will mostly be about moving bits from one place to another.
I understand why people don't like them. But that doesn't really matter, because code challenges are so much more effective at qualifying people that it would be crazy for any competent shop to stop using them.
For a couple years (and, of course, for many years before at Matasano) we've been hiring resume-blind and with almost no interviewing (we do a final on-site, but it's a formality with no tech-out, which is something we tell candidates). We calibrate our challenges so that they'd ideally, for a qualified candidate, take less time than interviews and phone screens would, and have the advantage of being scheduled at the candidate's convenience (all in one night, spread out over a couple weekends, whatever).
We have candidates coming out of our ears (volume has been the single biggest problem with our recruiting practice), and the quality of those candidates is just as good as anyone else's comparable funnel.
This is the right way to hire technical staff.
1. When you don't control all the HR process, you can't guarantee that the tech assignment is the main filter.
2. Because resume are generally so useless, if you look for a lot of people, you need to be able to review assignments very quickly. Also, to convince people to do the assignment, I make sure to have a phone call before the assignment, which is time consuming since there is no filter.
3. Writing good assignments is hard. You want something that does not take much time (half a day max), is quick to review, interesting enough for the candidate.
I have also not been able to make sure the assignment is the sole filter. I still filter at the (single, < 90 mins) tech IV. But that is at least partially because of my domain (ML engineering), where profiles are more diverse than pure software engineering.
My personal favorite is when interviewing for a standard full stack web development role I'm asked to more or less re-implement some feature of an existing standard library rather than, for example, build a minimal slice of functionality that requires touching each layer of the stack in question.
My finding on this is that companies in general tend to hire for traits and skills that will never be used in the role they're hiring for. In fact, if I'm a dev lead and a dev on my team reimplements a hand rolled version of java.util.concurrent.LinkedBlockingQueue rather than use the one that Java provides then we're going to have a problem.
I'm sympathetic to the hiring manager here and I do understand that some people cant code a lick despite being able to talk reasonably well about it in an interview. I want to test drive a car before I buy it. I like to try on pants before buying them. Believe me I get it.
Its been my experience that the coding challenge companies want to use is often a self aggrandizing, chest pounding kind of virtue signal about How Smart We Are Here. I suppose that if I had ever been given a reasonable assignment related to the actual job problem domain I might feel differently but this has never been the case.
I also feel that coding challenges are just another manifestation of Analysis Paralysis where the ole "if I just had A LITTLE MORE INFORMATION" kicks in. An experienced interviewer (note I didn't say experienced developer) can smoke out 90% of this stuff in the space of a 2 hour interview.
To be fair to the author here I found his write up to be an excellent effort to solve the ongoing problem that is hiring in general. An honest attempt to improve the situation.
But consider the following:
Hello Candidate, we need a [your problem domain here]. Verbally tell me how to build it while I ask you open ended questions about each step in the process. Once you get through telling me about each major component, responding to my questions along the way, we'll review and discuss the workability/pros/cons/compromises of the solution you describe. You may resolve any issues that arise in this process here.
They either know the answers from the experience of having done it or something like it before or they don't. Call me names if you must but personally I've never understood whats so hard about this. Every Good Job I've ever had has hired me by this method.
Everyone has secret sauce that works. Whether it's you or leeny or any of the other well-known industry blogger-veterans offering some fix for this problem. What a great way to cash in on name recognition!
Maybe the reality is that any process that has sufficient buy-in from both sides of the hiring pipeline works. That all that matters is aligned expectations and the means for all parties to deliver on them. Forgive my cynicism but maybe all we have to do is get everyone involved to be honest with themselves and each other.
Everyone is saying something different and has the data to back it up, but maybe it's all just fluff.
The cargo-culting actually seems like a problem for both employers and applicants, because it poisons the market. You have to know whether the company is treating coding challenges seriously. Because a company that will ghost you or decline you for unclear reasons after giving you a coding challenge has wasted a lot of time.
Or put another way, if it's a hoop to jump through for candidates, coding challenges are awful. If it really means a good chance at that $10k+ raise, coding challenges are a great investment.
This is an irrational statement and empty claim. You can't analyze something you can't see nor have data on.
- Do you have a view into "anyone else's comparable funnel?"
I doubt it. Therefore your claim is dubious and empty.
My skills give companies extreme multiples of ROI, and no other fields have nearly as ridiculous a hiring process. Have someone come onsite for a day and pay them, we can both decide if it’s a decent fit. But I don’t know you, your company, your team, the work process, the supporting teams, your funding, etc... honestly, one of those things are probably really awful. I’m not jumping through hoops for free. You’re not google, you won’t pay me $150,000+. Pass.
Most people seem to agree that white-boarding is a poor way to judge technical skills. Coding challenges seem a good compromise, assuming they don't require more than a few hours of time.
Yeah, I don't typically work for free either. Are you paying candidates for this time?
"Hey, I need you to come in and stock shelves for four hours, you know, just to see if you're a hidden gem for shelf stocking"
1) They asked for some publicly available samples of my code «I’m proud of». Okay, I prepared three samples from different areas with detailed descriptions and comments, explaining what that code did and why it was good.
2) After that, as a coding challenge, they asked me to write a simple app using the Unsplash API.
3) Finally we had an over an hour long interview during which I was again given three simple coding problems.
And then they rejected me. The question I have after all that is do you really need a multi-staged procedure like that to determine a candidate is not a good fit?! When exactly did they decide I wouldn’t fit? If that was after step 1 or 2, why didn’t they stop right there? If that was only after step 3, then what’s the purpose of steps 1 and 2? They don’t mean anything if step 3 can just cancel them out. Then why didn’t they start with step 3 not to waste my and their time?
Let a candidate pick a ticket from an open source project and come in to pair on it (or shoulder surf if uncomfortable with pairing). I don’t see the issue until the day we are pairing. Goal is to let them be the expert and as comfortable as possible.
They get some code on their GitHub, open source benefits, and I get a peak into their engineering skills.
If you’re thinking, “hey that’s not apples to apples,” well people aren’t fucking apples.
I had someone come in once that didn’t have an “issue.” She had been working with a library that had poor docs and wanted to work through the library and document it. THAT! That was an awesome experience. Best engineer? I had no idea. Willing to bang her head against a wall to make the experience better for others. Yes. Fuck yes, get on my team.
You may be gaining access to hidden gems, but you're definitely turning away proven and experienced developers from even entering your hiring funnel.
It all depends on what type of employee you're wanting to hire.
My personal experience has been that it's a lopsided process with the candidate putting in hours of unpaid work and the company spending one, maybe two engineering hours if we're being generous, reviewing the submission.
If the reviewer decides to reject the challenge, the loop closes, HR sends a nice e-mail, and the candidate has nothing to show for it.
The only way to find out what kind of salary is possible is to ask for numbers that get rejected a few times. Additionally, there are many companies out there, with a surprising variety of job parameters. Finally, many people are not good or comfortable with the application process, so you need to practice.
This means you have to apply to many companies and have to have many job interviews. Long, multi-round interview processes and take-home assignments - reasonable or not - make this very difficult and expensive for the employee.
These regular company blogposts seem to respond to the pushback of applicants and try and establish this kind of application process as a norm. Unfortunately this would decrease the market transparency for employees even more - many people go through their working lives underpaid and in subobtimal working conditions without knowing any better. This procedure also feels asymetric - in a time where software people are supposedly in demand, it's the employee who is supposed to jump through hoops to get a job.
Saying "We will pay you to do this exercise" is a simple (and cheap!) way to show you value candidate's time.
I was contacted by a recruiter a short while ago asking my availability for a phone screen- and I replied simply with my availability. The recruiter still hasn't gotten back to me, and I think that it may be because I didn't go out of my way to thank them for reaching out to me.
Are all of the niceties in your email inbox really that important to people? It seems pretty manufactured and boilerplate when people send me emails that they "hope find me well".
I may just have to get over it and start letting people know how "excited I am to speak with you and learn more about this opportunity!"
Likewise, while nobody really cares about thank you emails, not doing that suggests you didn't even bother to Google "Interview Tips" and implement the ones that took less than a few minutes. That's not a good signal to send to people that are going to have to rely on your professionalism and quality of work for the next few years.
Let me shared my experience.
Alexander is fantastic as an interviewer and co-worker. I was interview by ALEXANDER a few years ago. I ended up not getting an offer. It was my fault.
The challenge is the hardest thing I have to research on my own to implement because google found no result for me at the time. I learned a lot when working on his challenge and the idea of being able to solve it is more imporant than passing to interview to me.
The way they conduct interview is really great. You are given help by their team.
All I can say is Gravitational takes hiring to their heart and setup for success of candidate.
TO Alex, I'm Vinh if you remember me :-).
> Not communicating. Candidates who submitted all the code to master branch, which does not give us the ability to provide feedback on the various implementation phases. Because we are a distributed team, structured communication is critical to us.
how do you ask your candidate to split your answer using PR ? We would like this during interviewing but it ends up being a single PR. I dont want to impose create separate branches and a PR for each small incremental feature thing.
I'm assuming you are using Github
I'm just wondering if you found there to be any cognitive overhead here.
It was done one site as part of the interview process. It was related to the work I'd be doing, however it wasn't super complicated. They expected that you'd be able to finish it in about 30 minutes, but you had as much time as you needed. I finished it in about 20 minutes. Since it was over lunch time, they bought me lunch. I figure lunch for 20 minutes of work was a fair reimbursement.
I don’t know if we’ll ever fix it, but code challenges certainly aren’t the answer.
I’ve been developing professionally for 20 years.
I take about 4 interviews a year when _not_ looking for a job to stay practiced. Held DoE, Staff, and Principle roles.
Anecdotal evidence from this year:
- given a decent sized MVC app with bugs and missing features. Tasked to green the test suite. PR comment from a pretty established engineer was “this is perfect, I wouldnt change anything.” I used 1 of my 4 hours and thought my work was poor
- given a “simple”[1] sort and map/reduce a large log file and didn’t finish after 2 hours because the interviewer kept interrupting to ask me about the details of how internals of the language and GC worked - things I was familiar with, but I can’t focus on code and answering questions at the same time. I was ranked as “junior” and not a fit for the role, cute.
This last one rolls up into one of the problems I have (and assume others have) in interviews: “work through the problem as you would, but talk out loud about your process”
If I’m talking, I’m not in my zone. Let me shut the fuck up and solve the problem and we can talk about it afterwards.
[1] “simply” parse this 500k line CSV file and aggregate some results, no dependencies. CSV file was actually a STDOUT log file that multiple containers were writing to with different formats, JSON, CSV, TSV, SSV, and random ass unstructured logs from a prod system.
So I am currently only accepting interviews that give a 20% raise - and I am already quite high priced for my region.
I think you might not have seen many other professional hiring processes if you think spending a few hours doing an coding challenge is out of the question. Most professional occupation hiring practices are long and drawn out. That is part of what being a professional means.
That said, our hiring process leaves a lot to be desired and there aren't really great ways to determine if someone really had the skills they claim on their resumes.
Honestly sounds like Gravitational's hiring filter is doing its job...
But, it depends who you need to hire. If you need someone to just do the job. A coding challenge is fine. If you want someone who's above the rest in their CS knowledge, and problem solving skills, than a whiteboard interview is best.
Just my opinion. I have no data for this.
A whiteboard interview is pretty much a coding challenge where I can observe the candidate's methodology and their general approach to tackling problems. That's super valuable information.
From being a candidate myself, I've found that having someone watch me and being in a room with the stress and all motivates me to show my best. Whereas coding challenges I really lack the motivation and I end up rushing them, doing them last minute, and really not showing my best.
Ultimately, I think they both aren't ideal, but we don't have anything better short of an internship (and even those don't always pan out, since interns are treated differently and provided more hand holding).
In the end, it's just hard to reduce work that in practice spans multiple weeks per project, and involve all kinds of side process, to something you can test for in under a day.
OTOH, whiteboards or live-coding under interview conditions is a great way to watch me do worse work than I did when I was less than a year out from writing my first hello world. And yeah, I'm cool & collected under normal shit's-broken pressure, or client's-pissed-off pressure, on the actual job, which is totally different.
This statement is assuming that short coding challenges measure something relevant to job performance. If everyone agreed that is true, this debate would not exist. We'd all agree it is filtering for the correct skills, we'd all hire the right people.
But the fact that it is such a perpetual debate is because a substantial percentage of developers disagree. I have certainly never seen any coding challenge (whether whiteboard or not) that has any relevance to the job. So as an interviewer I don't subject people to it.
I kind of wish https://www.kalzumeus.com/2015/03/09/announcing-starfighter/ had taken off, as instead of doing a coding challenge for every company, you can do one really well designed challenge, and use that challenge with multiple companies.
Note: I'm an employee of gravitational.
That's especially worrying to me because it was a company that was still building out the team I'd be working for, so it would be hard for me to be really confident in the skills of my coworkers. Maybe the interviewers were just really good judges of character and skill (I think I probably was a good fit for the position), but it was hard to look past the fact that I didn't think I'd proven that I fit the role to them.
Not doing any technical/coding problems makes me worry that the company doesn't see technical skill as important at all. Maybe they could preface the process with an explanation of why they don't, and that could be persuasive, but outside that, even a basic FizzBuzz makes me feel like their hiring philosophy is considering those skills.
It's also possible that they are simply willing to fire people who are incompetent and prove to have lied about their accomplishments. If that's the case, it shouldn't be necessary for you to "prove" your skill, which is much, much harder to evaluate than most interviewers think.
I was asked to write an AWS integration tool once, and I declined to proceed with the interview process, because I didn’t feel like spending hours learning the Amazon API and re-learning a language with a useful library (I’ve since come to appreciate Python and Boto3, even if I’d rather be Erlanging).
It also wasn’t a sufficiently interesting problem to make the time worthwhile.
In this scenario, we are asking for similar amount of time, but with no travel, time that works for everyone and has low stress with an interesting task folks can review beforehand.
Besides, before even going through this challenge, any engineer can make up their own mind whether it's worth for them because we present compensation brackets upfront on the call.
No, they don't. Like most people that champion these types of nonsense "code challenge" interviews you present a false dichotomy of "you have to either torture someone in person for multiple days, or torture them with a bullshit made up test for hours!"
You can talk to them for a while over the phone (or video chat), split up over a few days, and get to know them just fine.
You've effectively filtered for people who are malleable and desperate for jobs.
But yeah...people in tech seem to believe all kinds of weird stuff about hiring...if you are asking someone to come in for 2 days for a non-executive position, you are lighting bags of cash on fire.
However, asking that candidates donate 1-2 days of their time (probably burning PTO at their current gig) just to have a chance at an offer? That's a very unbalanced risk/reward for the company vs the candidate.
Maybe you pay candidates for their 1-2 days of interview time + comp PTO for time spent if they get the gig?
Technical writing should be as concise and clear as possible
I can't speak to any other companies, but at least this one did pay me, like they said they would.
> You may be gaining access to hidden gems, but you're definitely turning away proven and experienced developers from even entering your hiring funnel.
You're describing hidden gems.
My point was that for someone who is internet famous, more people will be willing to jump through hiring hoops, whatever they may be, just as they are for FAANG, but that doesn't work for most hiring companies, because most companies are small and you've also never heard of them.
Also, no amount of "works for me" argues against your process turning qualified people away unless you have some out of band way of knowing that that isn't happening.
Since the company can't make time for a potential "experienced candidate" (their words) during the hiring process, I wonder what else they can't make time for? Red flag for me.
Companies won't change until this is enough of a negative signal for them.
I think it's perfectly reasonable to hire people based on resume, references, and conversation. If they can't handle the job once hired then they should be let go.
I do not think that the idea that all candidates are complete idiots unless they can be "proven" to be good engineers (by faked tests that rarely cover the actual job requirements but do have a way of discriminating against those who don't enjoy brainteasers, didn't go to college, or are older) is good or healthy.
At least that what my experience when I was a job candidate.
This does assume that the interviewer is an expert in the area and can take that friendly conversation to the necessary depths.
I have, however, on two occasions probably personally convinced someone they had dodged a bullet by passing on me when I bombed that sort of quizzy interview—they were, I am entirely confident given the job descriptions, my work history, and actual feedback I've received from managers and peers, quite wrong, but I bet they were very sure I was useless and nowhere near being fit for the job.
[EDIT] I guess I should add that I have a history of not realizing that something I'm good at is not actually easy or obvious to others, so possibly I'm just unusually awesome at assessing developer suitability through conversation—I really doubt it, but maybe that's what's going on. I've also not found a way to make this fit with any sort of "we ask everyone the same set of questions because we want spreadsheets at the end" process, as you've got to tailor the interview to both the position and to the candidate, as presented on their résumé and related material. So if you want spreadsheets and quantifiable-everything then I'm not sure how you match that up with my preferred style.