Ask HN: How to tell if a job candidate is using a LLM to cheat on a coding test?

2 points by dynamite-ready 2 years ago | 6 comments

A bit of a long one...

My organization invited a candidate to complete a Live Coding whiteboard test for a second stage interview. It was a really simple problem (essentially 'FizzBuzz' with a strong UI component).

When we invite people to take the test, we encourage them to talk us through their solution while they work. That way, we get a sense of how they approach problems, and how they work with others.

A recent candidate surprised us though. After we introduced the test via Teams, the candidate switched off both their mic, and camera, and started working on the test.

The first thing they did, was start writing a component (React) containing all the logic for the test artifact, with no markup/JSX, straight into the comment section describing the test! We pointed this out to show the candidate that we were there to help, and the session should represent a kind of collaboration. They replied, curtly, and didn't talk to us for the rest of the session.

It could have just been nerves, but the comment section actually sat above the module import section, so it was suspicious.

The logic was fine though. But then, when it came to adding the markup, the candidate encountered an error where they were calling variables outside of function closures (this was JS). The type of thing an experienced dev would fix along the way. Again, we raised the issue, but the candidate acknowledged the suggestion, and didn't touch this issue until the very end...

After another 10 minutes of absolute silence (muted mic), the candidate finished in record time, and only then, did they switch on their mic (still no camera).

In isolation, the two glaring mistakes (writing code into the comment section, and misunderstanding variable scope) could well be nerves. But when coupled with the order in which the candidate approached the test (writing all the logic first, without testing the rendering logic until the end), and the lack of communication, I personally found the situation suspicious.

Either the candidate had another, more experienced programmer in the room with them, or they were using ChatGPT. I raised an objection, my colleague gave a benefit of the doubt. Management progressed their application...

So I have two questions. Am I right to be suspicious in this hypothetical case?, And, going forward, what can my organization do to head off such concerns?

mmcnickle 2 years ago |

If you want to avoid this kind of cheating, you need to approach the test as if it were proctored. Communicate the expectations up front: you'll be required to screenshare, have mic and camera on at all times, what resources they're allowed to access and which they aren't.

There are still dozens of ways to cheat even under the above conditions, but it should eliminate egregious copy/pasting from a LLM or in-person help.

Were it me, I'd have made the expectation clear that we were interested more in the line-of-thoughts and explanation than the code. The lack of communication would probably be enough to not take the application further.

Quixotica1 2 years ago |

This is someone you’re going to be working with every day, if it seems like they are very difficult to work with, don’t take direction, aren’t willing to cooperate with a team, and they do things that seem suspicious, I think it’s fine to pass on the offer and maybe provide feedback to them so that they understand how they can perform better in the future. Whether or not a person solves some hacker rank problem isn’t really the determining factor on whether or not they pass a coding interview IMO

fuzzfactor 2 years ago |

You could just ask them respectfully.

Everyone involved should already be fully aware that any deception in employment application would be grounds for instant termination forever.

If not make sure the awareness is there.

Bring them in for an interview and ask this direct question so they can provide a written answer while you watch. And while you're at it you can do a live fizz-buzz to maybe hone your own sense of bullshit detection to keep up with emerging challenges.

If they are actually doing this, learn from them when you have the chance.

mtmail 2 years ago |

You're right to be super suspicious.

jjgreen 2 years ago |

Do the tests on-site.

JSDevOps 2 years ago |

At this point it should be blatantly obvious. If it’s not your in the wrong job