Interviews are Broken(erniemiller.org) |
Interviews are Broken(erniemiller.org) |
1. Finding decent people that are both technically competent and a good culture fit is bloody tough and can take a lot longer than you'd ever expect.
2. Interviewing/screening these people is simple. Don't make it more difficult than it needs to be.
You've done the hard part and you're confident the person/people you've found are worth at least a couple of hours of your time for an interview so why waste that time by applying a boilerplate interview process that every other company uses?
Give the candidate a realistic technical challenge to complete in a realistic timeframe, in an environment that will be indicative of the environment they would expect to work in should they succeed in getting the job.
If you're happy with the technical results, spend at least an hour with them having an actual conversation. Don't sit there throwing stock interview questions at them, don't try and nitpick on their CV. Just talk to them. Get a feel for their personality, the things that motivate them, the things that annoy them, the things that inspire them, etc.
1. I was repeatedly made to feel welcome, comfortable and calm as I possibly could be by everyone I met.
2. I was given technical questions that were appropriate and highly correlated with what I would be doing on a daily basis.
3. I spoke directly with two founders who, despite having my resume, opted for talking to me about my background, my goals, and offered answers to any questions I could possibly think of.
4. I was interviewed by four people total, and only one at a time.
It was honestly the most enjoyable interview I've ever had, despite the fact that I was initially nervous and didn't know what to expect, and it took the better part of a day. It was literally fun - if nothing else, this experience has given me a model for how I would interview candidates in the future if I am ever a hiring manager.
I wish every company operated this way - on some level, I think a lot of people who interview candidates don't know what they should look for, so they fall back on the "legendary battery" style of asking a suite of questions that only a recent 4.0 MIT CS graduate could answer confidently (and who could totally fizz out when asked to do something that isn't in Intro to Algorithms).
The problem is that a realistic challenge involves figuring out a realistic system. Realistic systems tend to be large, complicated, hairy. They tend to involve a lot of domain knowledge. They tend to have subtle considerations.
Working with a realistic system does not generally take a realistic amount of time. Not for an interview, at least.
I became a chef, where the interview could be moved to having them watch me make a dish or show knife skills. Then out of interest, I went through a biomedical engineering degree and am back into the tech field.
Still, due to my negative experiences with the filtering process, I have stayed in the realm of freelance and contracting. I would jump back into a tech job if interviews, like the above, were more common.
Thanks.
What exactly are you trying to determine from this? The problem with such open-ended types of questions is that all objectivity is lost. You end up picking people who mirror your own motivations, annoyances, etc. Unless you are sure your question will produce actual signal in an objective manner, you shouldn't ask it. You are just biasing the process in favor of people that will flatter your own ego.
[1] https://news.ycombinator.com/item?id=5227923 (this earlier Hacker News comment gives full references for the statements in this comment)
What I've done should be proof enough. I don't want to jump through a bunch of arbitrary hoops so that I can get an incrementally higher paying job (maybe) and devote my life to working on someone else's vision. Gee, can I?
I already have a job, and it might be different if I didn't. But maybe not.
That, and I wonder if there's a certain fraternity-like hazing element. One of our HR people recently brought up the possibility of removing one of our hiring steps (a take home programming assignment), and I found myself irrationally against the idea. I even joked that I wanted new candidates to suffer just like I did. On reflection, that might not have been a joke... Not particularly proud of it, but part of me resists changing the process because I did really well at it.
Anyway, good luck with that.
1) they were hired before the company started doing it
2) they new someone going in so they got hired without as much rigmarole.
3) they started as a contractor/freelancer and were converted to employee without interviewing
4) they came to the company through an acquisition
It seems like if you're not good at these annoying pre-offer activities that you're better off taking one of the above routes into a new job rather than trying to play these games.
[edit formatting]
And if we are talking about major corps, it would require lots of investment in terms of internal training, would take a while, in other words - it's expensive. And they still get a lot of top quality candidates, so why bother?
This is a great point. I've noticed that culture tends to be pervasive. The good experiences I've had with companies started from first contact. So have the bad experiences.
In fact, a candidate might have experience in something completely different but maybe just as impressive - publishing a paper on an optimization for an algorithm, creating a programming language, submitting Linux kernel patches, etc. How would one differentiate a candidate like that with someone with "a list of public repositories on GitHub as long as your arm"?
In fact, that's kinda the point. Each candidate should be evaluated on the unique experience they bring to the table. Trying to create the illusion of a level playing field by standardizing on generic code challenges in interview exercises is just that, an illusion.
As for the original article... if a company puts you through a long interview process, you should take advantage of it to learn about them as well, and see if it's where you want to work. Don't just be a replaceable part.
The game changes however if you need an expert consultant/contractor/freelancer who can solve a very specific problem with a very short period of time. For those, go hard on the specific tech. questions.
And, yet, our demand for developers with experience surpassing the average co-op or college grad doesn't change. It can be hard to make much progress with a team of mostly younger developers when every technical discussion someone is asking, "what is FTP", or "I read that GET is the only safe way to pass values", etc. Sometimes, it _really_ pays off to just hire someone who has already figured out a certain level of the fundamentals and can hit the ground running into more advanced walls like lack of business domain knowledge.
1. Write some piece of code that you think is cool. It doesn't have to originally be yours, and if it would take too long to recreate exactly, write it out in pseudo code and explain how it works and why you chose it.
2. (Slightly more common, and I've asked this before) We are having Problem A, and it would be within the scope of your responsibilities. How would you approach it?
the interviewer asks follow up questions to both and hopefully learns something in the process.
If your problem with the question is trying to think of a response that the interviewer would want to hear, then you aren't approaching the question correctly and/or neither is the interviewer.
He threw out 1:n interviews being bad as if it were the norm.
I also have no idea what he meant by this:
"All of this for a position that will involve working remotely (you are working remotely, aren't you?)."
Stopped reading after that because it didn't sound like he and I were in the same industry at all.
Something so simple became more difficult under the pressure of someone watching you the entire time.
It's also nothing I've ever even thought of building nor something I would ever be asked to build in a real work environment.
I have stellar reviews from my former employers. I know I don't suck, but hell if that didn't shake me to the core.
Maybe I'm just venting.
What filter? I didn't see him mention the filter he's using. Maybe he means he rejects a company unless it has an adaptive interview. But, how can you do that unless you take the interview?
Making decisions about managing technical debt, adding architecturally-significant changes, balancing good OOP with responsiveness, knowing the difference between future-proofing and conscientious coding -- all of those are both crucial (in many cases the most crucial) to day-to-day work, and also so highly context-specific that those decision-making traits are nearly impossible to identify during a technical exercise.
So for coding challenges, that leaves short-term tactical/analytic/algorithmic exercises, which in (anecdotally) 95% percent of cases cannot begin to approach a 'work-sample' environment. I've yet to encounter a technical challenge that would tell me much more about a candidate than basically how fluent they are with their tools, how well they know syntax and some general design principles, and what, for instance, their TDD (or lack of) workflow is like. Probably some insight into line-level analytic and algorithmic ability.
All of that is helpful, but -- Trust Me Here!! -- can also be very deceiving. The same coders that can knock those challenges out of the park can also be highly-proficient Debt Machines, all the more destructive because of their special genius for cranking out architecturally suspect code at a breathtaking rate.
To get into a real 'work-context' flow of a large app requires weeks, sometimes months, and only then can you get full perspective on how a given coder is going to contribute to your team on an ongoing basis. To get a feel for what that will look like in an interview, I've found I have to pretty much rely on the candidate's past projects, and informal conversations around larger architectural and OO principles.
[1]http://www.psychologie.uni-mannheim.de/cip/Tut/seminare_witt...
The validity coefficient provided by Roth and Bobko is likely more accurate. That isn't to diminish their value as they are still valuable, but the aren't the cure-all we'd like them to be. They do continue to show promise in reduced adverse impact though, which is great (note: the full article is behind a paywall - what is the HN-approved method of sharing the information?):
[2]http://onlinelibrary.wiley.com/doi/10.1111/j.1468-2389.2010....
That is for gender. With regards to ethnicity, the evidence isn't quite as optimistic yet. Like other predictors including cognitive ability tests, if they are showing notable adverse impact you may be in trouble Like other predictors including cognitive ability tests, if they are showing adverse impact you may be in trouble despite their validity.
[3]http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.2008....
It's a problem with lots of predictors, though scope of the problem varies. There is work being done all the time, even in the most reliably stalwart predictor, the cognitive ability test:
[4}http://psycnet.apa.org/journals/apl/92/3/794/
Anyone interested in the great "diversity-validity dilemma" can check out this link for more information, though there's always progress. It's a great article.
[5]http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.2008....
For my money I endorse integrity tests as a part of the solution. Decent validity, including incremental validity over cognitive ability due to a low correlation between the two, and small sub-group differences.
[6]http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.2007....
Having said all that, I imagine the efficacy of work samples is moderated by the type of work, and I'd have to believe they are more amenable to demonstrations of technical skill like coding (I don't know of any references for this now, but I'll look later). Coding-related jobs would be nice because it would be possible to blindly judge on the output as well, and in programming-type jobs it would be much easier and cost-efficient to test large numbers of applicants than it would for many other jobs. Cost and ease of large-scale administrations are their big problems, so overcoming those would be gravy. I don't know how subgroup differences are impacted though.
It really isn't. It may be indicative of your technical ability however your past experience doesn't give me any indication if you'd be a good culture fit.
Just like in social situations you can judge anyone pretty well in the first 5 minutes of a phone conversation, see if they will "click" with your team - hell you can almost do that just by looking at their open source (how they think/design). I don't need a high school coding exam and a 2 hour panel interview to see if someone is a "cultural fit".
The trouble is that, well, I hate to be like Kant but a lot of recruiters do just seem to have conversations that are very... spanish-inquisition-y... with virtually none of your personality there.
Like, when have I ever had a joke with them? Or an interesting discussion? When has a recruiter ever asked me about the people I work with/my friends? Or spent any real time talking with me my about my hobbies?
Hard to bring your personality out when people are asking you things like - "Would you consider yourself competitive?"
There's an underlying thing there where they're trying to guess at qualities they think contribute to behaviours they want in a very structured way rather than having a real conversation. And I'm sure we both know that people are more diverse than that, you can have lots of reasons for behaving in a particular way - not all of which the other person's going to be able to guess about.
It just feels really awkward, you know? You want to know how I'll fit in with your culture, invite me out for a glass of wine or something and we'll find something to laugh about. You want to get fake-happy-happy me then stick me in a horribly formal situation.
That said, I'm sure there are recruiters that treat you as human - and the more of them that are out there the better.
What does that mean, how is it determined via an interview and how is it weighted against all the other factors that make up a good hire?
Did I say it was the only thing? No. I said that the burden of proof shouldn't require three phone screens, a multi-hour (7 - 8+?) coding test, and then a 5 hour on-site interview. Companies are taking time out of people's lives to string them along on a hiring process that could last up to a month and a half, and still could result in absolutely no offer.
Second, I'm not merely referring to the list of things I've done. Like many people, I have public projects online that you can download and use, and in many cases, these projects have source trees that are publically available. Yet, most of the time, companies don't even care enough to do the background research of downloading the application and using it, or looking at the source of these applications to see if they are any good. I've submitted examples of work I've done and when I got to the on-site interview, the interviewer had no idea what I was talking about - hadn't used the application, hadn't seen the source, and in one case (where the application was actually quite popular) had never even heard of it, despite the fact that I'd mentioned it in at least two of my interviews.
Companies should be respectful of people's time. They shouldn't make the burden of proof so high that only the most desperate would ever agree to jump through these hoops knowing what they are in advance. I shouldn't have to complete a 10 hour programming test and then spend 5 hours proving to you on a white board that I can code. At what point do you say that my knowledge is general enough that I can pretty much approach and come to a meaningful solution for any problem that you can throw at me ? The combination of a short, in-person interview and a review of my work history and my references should be sufficient to get a grasp of my work skills and this vague concept of 'cultural fit' which seems to amount to, "Are you nice and respectful and a team player?"
Any thing beyond that is disrespectful to the candidate, especially if you are trying to recruit them away from another full-time position.
I don't think anybody reasonable is going to suggest that there's a single criteria on which a hiring decision should be made, but if you don't think that what you've accomplished should be pretty high on the list, finding candidates with an ability to create real business value is probably going to be much harder.
Who would you rather hire: a person with a track record of delivering working solutions or a person who can code on a whiteboard?
So when was the last time your preferred language's sort function magically disappeared and you had to come up with your own?
The interviewing process is criticized so frequently because companies focus on petty questions that are unrelated to a candidate's ability to do the job.
No offense intended to you. I doubt you meant it like this, but I got the same impression.
FYI, responses like this serve only to confirm the initial impression that you're insufferable.
The article was peevish, but I identify with the author's viewpoint. I suggest they interview with smaller companies, where for instance you can meet everyone and chat more informally. Like a startup.
I think you are on to something here. People want to do the safe, easy, and known thing. I cannot imagine most people, when finding out a poor employee was hired would blame the interviewer or the interview system. They assume that those are fine, because "Joel, Yegge, Google... everybody does them" and so they become a mental blind spot. The focus falls on the bad employee, "wow, he seemed so good, he must be a con artist or something".
>And they still get a lot of top quality candidates, so why bother?
This, I do not believe. By definition, most developers are not "top quality", rather just fit in just well enough to stay out of the "fire zone". From what I have seen, unless by "top quality" you mean "best development value for your buck", most developers at most shops are far below "top quality". Most are either overly conservative or overly aggressive, and have skills just barely good enough to do their job, which is good, because if they were truly the "top quality" they would tire quickly of your work and leave. The only way someone gets to be even close to "top quality" is by constantly doing new things, constantly pushing themselves, constantly getting better. Those traits _sound_ good till you consider what then will happen when they master the job you want them to do and get bored. That is what I mean by "best development value for your buck". Chances are, most development shops want most developers to be just barely good enough to get the work done in an arbitrary time with an arbitrary level of quality. Like the old joke goes, anyone coding slower/poorer/less than me is "a lazy idiot!", and anyone coding faster/better than me is "an architecture astronaut stuck up jerk!". Hiring becomes, "are you within an acceptably narrow band of skill that will all share?"
So, companies get what they hire for. Chances are, they get mediocre programmers, because that is what they actually _need_. They don't need language-writing, development world colossi striding about their office, designing new paradigms for their slightly altered CRUD app. Most companies wouldn't know what to do with such developers if they had them. Heck, even Google apparently didn't know what to do with Guido. Instead most companies need someone happy sitting down to add a new field to this page, or make this checkbox work on this popup, or fix this bug that only shows up in IE7. Someone for whom that is just challenging enough to be interesting, but not impossible. Apparently, programmer quizzo is the cheapest/safest way to ascertain that.
If a worker is going to work remotely, is it a better evaluation of her skill to have her do her coding test remotely or have her do it in the office? Likewise, is it fair to ask a potential remote worker to take on an endeavor that's much more cumbersome than it would be for an in-office worker.
The other part of the argument is that the hiring company may be affecting its ability to evaluate a candidate by subjecting him to a situation that's pretty different from the job's reality. Ostensibly, the point of having someone work out of the office is to see how well he'll do if he gets the job, but in the case of a remote worker, this may not be a good indicator.
I think all of this is highly debatable, and there's probably no right answer, but there is lots of food for thought!
When I look for cultural fit, I mean: will you have productive conversations with your colleagues; have you shown yourself to be inquisitive about new things beyond what you're assigned so that you have a good understanding of what a solved problem looks like; will you force yourself to find a way around hard problems or will you quietly give up and call the problem impossible or blame someone else; are you interested in solving customer problems and iterating rapidly, or in building the most "well-engineered" solution; do you think your role in the organization is to build products or direct them.
These aren't easy questions to answer, and they do open themselves up to bias in how you evaluate candidates. I like asking about folks' CV - even though there's already a written summary - to see what parts of a problem you considered most exciting and hard; I think that's revealing. But may not work for everyone.
I may have responded harshly, but certainly in kind.
But it is completely unrealistic to believe that in a single meeting you will be able to predict with any certainty that a candidate will work well with your team, enjoy the position and remain loyal. You simply can't establish these things until a person is on the job.
If you do a good job not hiring obvious poor fits, the odds that your relationship with your new employee will be a fruitful one obviously increase, but don't fool yourself into believing that there isn't a lot of luck involved.
While that may be, it doesn't mean cultural fit is only used in those cases.
There is a vast difference between someone who cares about doing their job well and someone who does their job. And 'cultural fit' will be used to describe that. Are you the type of person to put in your own time to advance your knowledge, or do you require the company to pay to keep you up to speed on the latest advances. I've seen both types of people.
And the former is more valuable than the latter. And the former generally won't want to work with the latter, either. Granted, the former will cost more than the latter, as he brings more value to the table.
You can argue the merits of 'cultural fit', but it's not just a word used to hide sexism, racism, and prejudice in the hiring process. And, personally, I think it's important because I want to enjoy the people I work with.
I like this as a definition of cultural fit. I have worked a place where I felt like the only guy who cared about his job, and it was suffocating, similarly, I have seen the one guy who is just doing his job in a team of those to love to do their job well, and he was like a ball and chain.
Perhaps we need to stop using "cultural fit", as a replacement for "professionalism". When I think about "cultural fit", I think, "what do I like, and does this person like it too?" When I ask myself and others, "is this person a professional", I think, "do they exhibit: passion, discipline, dedication, drive, and care in their work, skills, and interactions with everyone?" I would rather tell HR, "not as professional about his craft as we like to see", than a wishy-washy, "not a good cultural fit". The first sounds like we are professionals who treat ourselves, our craft, and others with respect, the second like a frat house blackballing a pledge because he doesn't like the same beer we all do.
There is so little of a difference between those sorts of person that they are frequently the exact same person except on different days or in different working environments.
Where is that line?
Looking at something dumb, like TDD. Is it okay to hire someone who doesn't "believe in TDD" when the rest of us practice strict TDD? Somehow, almost certainly based on nurture of past jobs/mentors/failures (we all like to think it is nature, pfft) all developers look at the same facts of TDD, and yet some are on this side, some are on that side. TDD is an arbitrary bar. May as well be favorite color. But, when the house painting team is 15 guys who like red and so we only paint red houses, but you like blue... Maybe they just are not a good "cultural fit". Shrug. Would that one blue person "poison" the team? On the one hand, that blue person might open us up to new opportunities, get us to start doing both kinds of houses. Maybe they just end up fighting with everyone all the time, "CAN'T YOU SEE RED IS THE BEST WHAT IS WRONG WITH YOU?!" And so cultural fit seems like a way of saying, "people similar enough to our average skill to not make anyone mad, and similar enough to an average of our shared backstory so as to have basically the same conclusions about the Big Issues as we do: TDD, FP/OO, Pairing, NoSQL/SQL, IDE/Editor, languages, what goes on the Big Issues list, etc.
But in all that, aren't we really just saying, "I want to hang out with people like me"?
This reminds me: I think I've yet to see a tech company where the default assumption is that you're a teetotaler. Why do y'all SV-ites insist on taking me out to consume (legal) mind-altering drugs?
I'm sure that's the case in rare circumstances. The phrase 'cultural fit' is fairly common here in the UK but I agree it may be taken a bit too literally elsewhere. The cultural fit I'm referring to is attitude, work ethic, maturity, etc.
I don't need a high school coding exam and a 2 hour panel interview to see if someone is a "cultural fit".
Completely agree. I can't see how that scenario could give anyone an appropriate insight into a persons character. As explained in other comments below, a relaxed, two way conversation in a comfortable environment is one of the few ways of establishing how well a candidate would adapt to life at your company.
(FWIW, those two guys have turned out to be truly awesome developers).
Further reading (The “Leaf blowers for stamp collectors” thread): https://news.ycombinator.com/item?id=6373983
I truly do not think it makes sense to demand every person on a job is passionate about the job they are doing. There have been many times I have had to (even at a job I normally loved) do long, boring, and fruitless work. I did not do them out of my passion for the "Mission to Save the World of Enterprise Banking", but because I am a professional who does what needs to be done and who does it to the best of my abilities.
When I said passion before, I was trying to express two things.
First, I was intending to express the attitudes that cause someone to stay abreast of their field, keeps their own saws sharp, (and ideally helps teach/inspire others to do the same), comes in and works hard, and pushes themselves and those around them to be better. Perhaps the best word for that is just professionalism.
The second concept is that I would rather be around someone who, while acting like a consummate professional, also enjoys at least some part of it. I love the fact that software development has more than I could probably ever learn, and it is a worthy challenge to seek to master its different forms. I have worked with several professionals who really do enjoy parts of the craft that I only barely tolerate, and I enjoy that about them immensely. They open to me a new worlds of appreciation for the craft. Just recently, I had to work on a few hundred legacy bugs for a few weeks in a row. They were unexpected, not caused by me, and deeply tedious. Some took hours to locate the cause. One of my coworkers really came alive doing that. He just loved the hunt and chase, like a detective working out a mystery. While I was mentally moaning and groaning after week two, he was just getting started. We talked about it for a few minutes one day, and I decided to try and see things his way. While it didn't cause me to love bugs like a fine wine, I was able to bring my attitude around from a silent scream up to mildly interested. Maybe the best word for that is having a good attitude? Enjoys some parts more than others, and keeps quiet when they don't enjoy it? I am not sure, but that whole concept I find to be important. The person who reads books about development and enjoys reading them is going to bring a wonderful attitude to the team, and can motivate others to pursue their own interests.
> Why is that not enough?
Because they can find someone with the same qualifications that will also bring that passion.