This is the only part I take issue with. I tried a take home interview once that took the better part of a weekend and decided I'd never do it again. There just isn't enough time. Basically, if it takes that long, there needs to be a very high chance that I'm getting hired at the end, or it's not worth it.

At my current workplace, I have the tools I need to work. Most of my softwares licences are under my employer's name. I have access to my snippets, my previous projects and many build processes (minifying my code, preprocessing my CSS, etc.). My workflow is great.

If you give me a project to do at home, I can't use any of that.

I would have to spent hours working either from my bed, my dining table or in a coffee, on a laptop. I would also be required to work on this project after the 40h/week that I already do.

I'll take the regular interview any time.

One of my worst interview process was one time when they required me to do a project of 20+h in a week. They knew I was employed elsewhere, so they let me give it Monday instead of Friday. This was ridiculous.

Otherwise, everything was doing very well. However, the experience really discouraged me from working there.

But I guess it's not all that bad, because I most likely dodged a bullet. If they can't even give correct deadlines to future employees, I can't imagine how they threat their own employees.

I recently did a clojure problem where in the gist itself I asked for feedback and showed my thought process, but crickets until a month later when I followed up and they said they had moved on. I think it was for the best.

A good preventative measure might be agreeing beforehand on a time to discuss the solution. I know I certainly will next time.

Also hiring managers need to ACTUALLY read code. There have even been times I've asked if they've read my code before doing a lengthy drive, they've said yes, only to realize upon getting there they haven't. If you won't take some time to assess the code I've written and put my name on, I don't think I should be expected to drive 3 hours to meet you.

My test involves a basic task (that is related and already implemented in our codebase). You have to check out a skeleton project, add the solution, build it, and deploy it. The test takes me about 30 minutes and most applicants spend 60-90 on it. Everyone on the team reviews the test and the criteria is essentially "do you want to work with this person". It's very rare for someone to pass the test (which occurs between a phone screen and an in-person interview) and not get an offer. The exceptions have all been around compensation issues.

Do you like them? Check. Do you feel like they are telling the truth? Check. If you use the 'try before you buy' method: You can easily see if they can handle the job/handle the pressure.

Also, you probably aren't the only company/avenue they are applying towards. All of the calls/tests add up.

The question is though, since companies are already using take-home tests/interview questions... can anyone confirm that these methods are more effective than a 'try before you by/contract to hire' position?

http://www.thoughtworks.com/careers/application-process

It's a fantastic way to vet candidates and there are multiple exits to the process. For obviously bad submissions, it's probably a no-hire, but I've even seen incomplete assignments get someone to the next level and then a hire.

Some people get sent to ThoughtWorks University which basically trains them to code, work in an agile environment and how to be a good consultant. Everyone that goes says it is a great experience.

    Rather than pick a new project, however, they'll take the same project further, incorporating feedback from the 1st interview.

It's been invaluable in determining candidate fit.

I hope that you expect the same amount of prerequisite work for all of your candidates (marketing, finance, sales).

Obviously you want to vet the person technically but you should be able to do that through talking with them. Most developers put off take home tests even if they are excited about the company and by the time they do it they already have finals/offers from other companies.

It doesn't replace face to face interviews but I can see it as a good example of "how would face this real life situation you would encounter if you got the job?"

I keep seeing more and more dis-associative non-communication being presented where the interviewer attempts to do less and less. Its lazy and a crap way to treat people.

3 hours isn't too bad. I had a friend interview with some hedge fund and he said he probably spent 30+ hours on a take-home project (he didn't get the job). I thought that was way too excessive for just 1 interview.

Some advice to those doing take home tests: write (good) comments and include tests.

It helps to know if the candidate you hire will handle that, or fall apart just when you need him/her to do the job he/she is paid for.

Interview with 2 people for an hour or so. Then you get a Rails project, fix 3 bugs, choose and implement 2 features from a list, then 2 features they choose. Come back in a few days and discuss why/how you did what you did.

I'm all for improving the process, but as someone who at times has struggled on a coding interview, I believe that the onus to improve should be on me the candidate.

So I was an iOS developer getting considered for Android position, and they said "hey it's great you can code fast & all, but can you learn fast?" and told me to create an Android app that searched images on Google. I sighed because I had already been through 4 coding interviews, but oh well, this will be it I thought. Since I wanted to do a good job, I spent two hard days working on it basically all day every day. Since I was new to Android world, I had to learn that while being productive. You can see the result of my efforts here: https://github.com/joslinm/android-image-search-example. I thought I did a nice job because rather than using a HTTP library, I read the streams myself to give nice progress indicators for each image (which persisted even thru phone rotates).

I give it to them and they say, "ok seems to be working." After that, I was invited to SF to interview with them in person -- actually, interview is the wrong word, I would be working with them for 2 full days. So we did that. When I wasn't coding, I was being grilled about my work history (perfectly acceptable, but why didn't we do this earlier?). Finally, after that, I was told they had come to their decision. Thinking I was a shoe-in, I went to go meet with Andrew. Andrew's decision: "You're too entrepreneurial."

Now let's examine this for a quick second. They had come to a pretty fair conclusion that I was too entrepreneurial after talking to me. They realized, hey this kid is a go-getter and probably isn't the greatest fit for our #7. Yet this was AFTER copious amounts of coding and wasting my time doing all these code interviews and even joining them to code in-house. If they had just went about a pretty normal interview, they would have discovered this fact way earlier and not wasted either of our time. So why waste my time forcing me to code before they do their diligence? Because he's Andrew Chen; you can google his name and discover who he is. It would be a "honor" to work for such a prestigious start-up pundit. In other words, he can get away with it.

For those of you considering a start-up and actually have skill.. don't get taken advantage of by these bogus work projects. The employer wins because he's not paying you and gets a lot of evaluation for free. And if they do insist on you working with them for a couple days, you insist on getting paid for those couple days.

http://www.gayle.com/blog/2013/09/18/companies-who-give-cand...

At the core of the problem is that this approach can be used to burn a lot of a candidate's time without an equivalent investment from the company.

I've mentioned this before - I applied (maybe 5 years ago) to a company that first asked me to take a Java test (about 1 hour of work). Then the recruiter called me and sent me a take home project that should take "5-7 hours". I did this, crickets chirped for a month, though I did check in with the recruiter for a while (I gave up eventually). Finally the recruiter called me with a one-line brush off "we've decided not to move forward at this time…"

Granted, this was a bad experience, and it won't always be like this. But I'm pretty close to saying "never" to these exams.

I thought Gayle's suggestions were good. She goes so far as to suggest a 90% passage rate - that you should not be giving these tests to people who are unlikely to pass then, using them to confirm, not screen.

Perhaps take-home testing works if you want a "code monkey" who can sit in a box churning out code according to specifications, but I cannot think of a company that doesn't want more. The ability to interact with others in sometimes stressful environments is, imho, an essential skill for all.

There is also the danger that companies will turn these "tests" into work product. I did an interview last year (legal) that requested "writing samples" on very particular and timely subjects. Once I realized the game I asked them to sign a copyright agreement. They got screaming mad and tried to get me to sign all sorts of NDA junk.

- https://en.wikipedia.org/wiki/Propranolol#Society_and_cultur...

Can ask your doctor about this

In light of this, I think most interview processes don’t make any sense. I think that a screening phone/Skype call makes sense before either side commits any time/resources etc. In this process, I think there should always be room for the candidate to ask questions of the employer. Any situation that is unbalanced, in which the employer is playing the role of the one with the final decision is a completely skewed setup.

I interviewed a while back with a YC company and although I didn’t get the job, I was highly appreciative of the setup. Firstly, there was a Skype/phone call, which felt really like an open conversation; one in which I was free to ask questions and understand suitability too.

Then I was invited to an in-house hack. Essentially, I spent the day (a Saturday) hacking with the founders on a project. The project was a lot of fun and something that I learned a ton from. We actually ended up working till around 4am the next day. What I really appreciated is that although it was “an exam” for me, the investment of time on my part was weighed equally by their time investment. Also, I was happy to know that they had learned things during the hack too. I was at ease during the whole day precisely because they treated me as though I was their colleague already.

Coming out of that experience, I’ve wondered for a while why employers don’t take the stance that to hire excellence you have to commit time and resources. Most interview processes that I’ve heard of seem to be a cop out on the part of the recuiter/interviewer, in the sense that it seems that the priority is to hire the best people but spend the least time doing it. That to me is the ingredient for a broken process.

So, in a nutshell, I think talk of take-home interviews misses the point. It might not be scalable, but my intuition tells me that it would be much more effective to have the interviewee and company folks hack on something together. That way, the interviewee gets the opportunity to essentially interview the company too. I’d definitely preceed this with a few conversations on the phone/in person, to ensure that the time investment of hacking together is geared towards the best interviewee/company relationships.

Perhaps this is missing the point too, and perhaps it’s not workable in the real world. My anecdoctal evidence is simply that have spent an entire Saturday working on a project as part of an interview process and ultimately not getting the job, I don’t feel I wasted any time, nor do I feel bitter about it.

I do agree that interviews add a level of pressure, but what you're trying to gauge thought process. How does the candidate think about solving the problem; Can they iterate; Can we lower the space and/or time complexity; Can this be solved recursively, Are they receptive to input? A take home assignment does't help you figure any of that out. Usually, the team who is interviewing already knows the answer to the specific problem and all of the intricacies (based on having to solve it on an internal project or just because it's a question they've asked before), but the goal is to see someone thinking in real-time.

In my career, I've never had a time where I go off on my own, work on something and come back to the team with a solution. Even when I was writing TI-83 programs in High School to solve all of my science homework and finish 50 minute tests in 10 minutes; I still had classmates asking me about what I was doing along the way. There is always discussion and I feel like by sending me off on my own, all of that goes out of the window. You can't verify if I copied the code from anywhere else, you don't know if I had my buddy who is an engineer at another company break down the answer for me to spit it back to you, and you don't know if I paired with someone else to get the assignment complete. I'm obviously biased based on my experiences doing over 40 interviews in Silicon Valley/SF. My goal for this type of interview question would be to understand how the candidate thinks; I don't care if you write your answer in Brainfk[4].

[1] - https://github.com/joeblau/sample-elevator-control-system

[2] - https://github.com/joeblau/sample-top-ten-tweets

[3] - https://github.com/joeblau/sample-url-shortner

[4] - https://en.wikipedia.org/wiki/Brainfuck

I think this should stop. In it's place, developers can have a pet open-source project, which they submit with their application. The point is that the same project can be submitted for a hundred applications if needed, saving the candidate dozens or hundreds of hours of wasted time.

They could put that time into the project instead and even if they don't get a single offer, the open source community benefits, and the candidate got to do something interesting rather than convert roman numerals or aggregate an array.

I seriously wonder if we should all boycott dev tests. If they give a test, just give them a github link and say "Please review this as a good example of my work.". If they can't review code that isn't tied to a silly question maybe they are not worth working for.

I say this having done some such tests recently myself. (To be fair one of them said a Github submission is OK, I just didn't feel my Github was good enough at the time.). I kind of feel bad contributing to the madness!

Hell, in my experience you're not even guaranteed anyone will read it.

This is just a bad idea in an industry full of irresponsible employers. I'm not saying that Triplebyte is in any way irresponsible, but the rest of the industry has already ruined this avenue.

I would cram for hours before my blackboard interview making sure I knew all sorts of algorithms, just in case. My friend and I would quiz each other. I don't see how memorizing tree transformations relate to a Django application.

It made it so much more difficult to move jobs.

> For those people who prefer FizzBuzz tests to implementing interesting stuff, I suggest companies provide the option to do a test for those who don't want to submit their project code.

In fact a great personal GH repo could be to implement FizzBuzz in a bunch of different languages. I think most companies would get an appreciate the joke.

> You shouldn't have to write a OS Kernel or a modern browser in your spare time either. I'd say it would be about 4-8 hours work that shows off your skills.

Well, this is a different suggestion. Now we're only talking about a few hours of work rather than a pet project. I don't think the licensing makes much of a difference since I probably couldn't monetize four hours of randomly dorking around. I'm still not a huge fan of the idea, though, because it sounds hard to me to show off (adequately) skills I've spent decades acquiring with less than a day's worth of aimless work. I don't feel like it would be very impressive.

> Is there a field that requires 4-8hr competency tests where each test is wildly different from the previous, with completely opaque criteria.

Nobody is advocating for eight-hour tests. The OP said they expected their test to take less than half that. And most fields that require competency tests (e.g. cooking, acting, music) will expect you to do it live, on the spot. Take-home tests are being suggested in the case of programming specifically because programmers hate being expected to perform on the spot.

> At least with the provide your own project idea, if the employer has fucked up criteria the candidate has only wasted a few seconds of their time in sending the zip file.

Not all criteria that someone might test for are "fucked up." If I'm looking for, say, somebody to do a math-heavy machine learning job, and your pet open-source project is a jQuery plugin that makes web page backgrounds sparkly, the mismatch between those two is not the fault of my criteria.

That's less than you're already spending on salary for the interview slate, so you're at most doubling your already small costs for a critically important function for your company.

I don't think I would give out the same work to multiple people, let them put it into production on their first day and get the pride going ;)

Agreed.

Annual billable hours for consultants, be they developers, management, lawyers, et al, are considered huge if near 1500. That roughly works out to just under 30 billable hours per work week when vacations/holidays and non-billable time are taken into account.

As you surmise, just because someone _can_ bill X/hour doesn't mean that each hour they breathe commands the same remuneration.

I'd much rather freelance for that type of stress & when having to compare the economics time-wise, which a take home test simply does not match - that is the whole point.

If a company does not want to pay that much, maybe they should respect the prospective employee's time, especially since it is a job seeker's market.

    I almost walked out, but the salary was stupid high. Didn't
    get the job. Thinking back on it, maybe it was just a test
    and they did want me to say it was ridiculous.

- They are being unintentionally stupid by asking you to do this activity.

- If they are intentionally asking you to do a stupid thing and they expect you to revolt in the interview and decry the madness. How does that make you feel about your future day-to-day life at that organization?

In both cases those are not good "A-player" types of people to work for.

No need to torture yourself over more favorable prospects "lost" :)

If that was the case ... then you don't want to work there. I've worked with a few business people that intentionally filtered interviewees by strange characterists. [Aka... graphic designers that were juinor, but didn't have web dev experience (I want to say it was something stranger than that)]

Now you could debate which is a better representation of my work, the polished dog and pony show repo or the fast and dirty stuff that I push in my ongoing projects.

At the same time, I prefer the idea of presenting previous open source work instead of having a group of potential employers agree on a project that they can all judge. One benefit is that the work is less arbitrary, potentially useful for others. Another is it's more likely to be unique than an assignment is. Finally, do I want to work for any of those other employers?

Applying to a group that way would seem odd to me; part of me thinks that choosing companies is important, but another thinks that there's some chance that that's just because I haven't tried it.

Which, of course, leads me to agree that the OP is great as an experiment - it might well be a better solution in many cases. The fact that, in the OP, it's opt-in is a big deal as well. Though I'm still very skeptical of take-home interviews in general.

I rather put in those extra hours contracting (worst scenario), doing open source work (lots to be done that advances the productivity of tens of thousands of developers, if not more), into my hobbies, or into my friends.

I had that experience with an Amazon interviewer. The guy who told me he expected perfect code also gave me zero feedback as I worked (despite me literally asking if I was headed in the direction he expected) and spent the entire time staring at his laptop. He'd apparently forgotten that he was scheduled for an interview, and also his manners.

At that point, the candidate should not be hired.

If the candidate can't communicate while writing code, and can't communicate about code already written, at some point you have to be afraid the candidate just can't communicate.

If this is a person who can communicate in any situation except an interview situation, then it's a Catch 22 where their ability to communicate in a work environment just can't be verified.

(Unless you secretly record them communicating in some other work context? Probably not a scalable approach.)

There's a gap between knowing concepts and knowing how to apply those concepts. There's actually a certain (limited) amount of artistry to writing code. Hiring a programmer without seeing code is like hiring an Architect without ever having seen a house he's designed.

The interviewee (the vowel count makes me dizzy) can choose what he/she prefers. So that argument is weird. Maybe (if that's your point) it might be beneficial to go for the face to face interview, for direct feedback? But .. some people just _cannot_ do that. You're not talking "Ah, they didn't respond. If you'd just had visited them on-site..." here. It's "opt for the home project or don't apply". Which is empowering the candidate.

1) I didn't score well on the test. It's within reason they read this post and release my scores with a 'it was because he scored low'. Future employers then come across my scores and deny me.

2) Don't bite the hand that's like the hand that MIGHT feed you. You don't yell and storm out of investor meetings. Because then they tell other investors. Similar logic here.

3) This place is very optimistic. Pessimistic people aren't particularly desirable. Whistle blowers are included in the latter group especially by corporate entities. Who knows what it could cause.

It really boils down to the fact that it's not a safe environment to do so. Or at least it doesn't feel so.

> To solve the problem of interview anxiety, we're adding a second track to our interview process at Triplebyte. Applicants, if they choose, will be able go through our process by completing programming projects on their own time.

> The project-based track will require a larger time commitment (and we expect lots of people to stick with the standard track for this reason).

Coming up with something that works, versus explaining why it works or why you did it that way after the fact.

I think it's the "thinking out loud" part before you have a solution, that can fluster a lot of introverts.

We have a magic white board that can print out a big piece of paper showing everything on it. We also take pictures of it (and lesser white boards) to record designs agreed. Those pictures generally get put into a formal design document that is reviewed, signed, approved and all the rest of it.

Explanations work better without lengthy pauses. Ideally, I can explain the thing without stopping too long.

> Why don't you use documentation?

I do if it's necessary. Sometimes it's not necessary.

> Oh, and how do you version control the white board?

I don't.

It isn't but it strikes me we are looking very hard to find a new way to do things, when the old way was pretty damn good.

Sit me down and talk about technology for ~thirty minutes. If I don't have the social skills to successfully do this (minority issue) I likely would not be able to communicate well with a team and thus should not be a candidate anyways. You will then know immediately whether you want to hire me (or progress me to another round) or not.

I then get a call two days later and can progress with my life.

People pretend like all this hiring strategy is for the good of the candidate. It's not. As with everything else its for the good of the company and its investors.

Here's a thought:

Invest in a competent hiring manager who can see through applicant bull shit and identify talent within a reasonable range. Includes basic negotiation skills. And assume the rightful risk that is employing another human being.

But it's important to remember that a take home exercise cannot fully replace an interview. It can replace the coding part, the whiteboard "and now we ask useless trivia questions" part. But there's no way for this work to replace the "would you fit the team" interview.

Others already discuss this in different subthreads here, but basically I'd expect the company to clearly show how their process works and - ideally - filter by ~social~ criterias first ("You might fit the team, if you can code"). Doing work for free with potentially no feedback or a 'fail' in a later discussion is crap, of course.

The fundamental reason why this is OK is that the company is hoping to pay hundreds of thousands of dollars for the candidate's time. The person who pays the piper, gets to choose how it happens.

But that said, it is still fair. The effort required to hire a new person into a competent company is significantly greater than the effort it takes a competent person to get a new job. As a candidate it doesn't look like this because you see that you personally gave the company more attention than vice versa. However you don't see all of the other people who the company also paid attention to and ultimately rejected.

If you're a competent developer it probably still doesn't look like this because the company usually develops procedures that concentrate the required effort in the hiring manager and/or HR. Therefore you have little idea how much effort is actually spent looking for candidates.

But spend time as a hiring manager and it will be obvious.

For a typical whiteboard coding problem, I also wouldn't want to see 60+ commits.

Coding is a skill largely separate from other critical dev skills, e.g. design. Someone can legitimately have deep knowledge of, e.g., how to build a 3-tier service architecture without actually being able to code. This isn't "faking it". It's a different skill. You can study database partitioning strategies and learn about the latest and greatest frameworks for writing web pages and APIs without ever writing a single line of code. And sure, you can try to probe deeply enough to expose where their knowledge stops, but you're still not probing for coding skill. There's actually a decent chance that you'll end up probing for trivia which is not a very useful filter. "Oh, you don't don't know the auto-generated C++ class members? You clearly haven't actually worked in C++!" Yeah, right. Anything known broadly-enough known that it's reasonable to assume all coders would know it is broadly-enough known that non-coders could learn it, too.

Even if you could probe deeply enough in conversation to expose that they can't code, you could do the same with a few minutes of actual coding. When someone can't code, it becomes clear pretty quickly when you ask them to code.

* People who read a lot of blogs so know the lingo, but never code

* People who get the theory but can't actually implement something that works without bugs

* People that over-engineer simple things

When I was writing signal processing code, we had four levels of engineering.

- The first was an Algorithm Description Document. This document laid out, in mathematical terms, the algorithms used for various signal processing functions in the system. It was purely conceptual.

- The second was an Algorithm Implentation Document. This document mapped the algorithms to specific parts of the hardware and software system, laying out the logical module structure and data flow. It also specified how the algorithms would be realized in code, since parts of a particular algorithm might need to run in different parts of the system. The AID was still primarily mathematical.

- The third was a series of software design documents. These documents specified the details of module interfaces, code and file layouts, and specifics about what algorithms would be implemented in what functions.

- The fourth was the actual translation of the mathematical algorithms in the AID into actual C code. Most of the engineering at this level dealt with function-level optimization and some platform-specific stuff.

Of those four levels, only the last is what I would consider "coding". It is also the least important. Anyone who can really understand the first three levels and is willing to learn can handle the fourth. As I mentioned previously, there will be differences in efficiency, but not in capability. It would take willful ignorance for this not to be the case. I think this is also what you are considering "coding", but I'm not sure; I also think you might be merging #3 and #4.

Now, most commercial projects do not run this way and, as far as I know, few have dedicated systems engineers to manage the system architecture and APIs. Thus, the software engineers building the system generally perform the work in all four of these levels simultaneously as the system builds out. Despite this, I still only consider the work that fits level 4 to be "coding", and still consider it to be the least important.

What I look for when I interview are engineers who operate very well in levels 1 - 3. If I find that, level 4 is pretty much a given absent the rare pathological case or someone with a very "academic" attitude. The questions I personally ask deal with levels 2 and 3. Those questions cover some of what might be considered coding, like API design and class layout. I do not have candidates actually generate code, though. Some of the other interviewers have candidates write small functions in pseudocode or explain existing, uncommented Java code, though.

I also need to note that we have a take-home test that candidates need to pass before they get an on-site.We have also recently added an open coding challenge which may eventually replace the take-home test. This probably covers your requirement for candidates to code, even though it isn't done in front of one of us.