Ask HN: Hiring in the age of AI-assisted coding: what works?

29 points by nitramm 40 days ago | 19 comments

I saw the HackerRank (YC S11) hiring post (https://news.ycombinator.com/item?id=47667011) and it made me realize I no longer understand how to evaluate candidates effectively.

Specifically, we are changing hiring across 3 dimensions: > Tasks: Real-world tasks on code repositories vs standard algorithmic-style puzzles > Evaluation: AI fluency, orchestration skills vs functional correctness > Candidate experience: Agentic IDE vs a simple code editor

In the “old world,” you could ask multiple questions and triangulate skill from answers. Now it seems like evaluation depends heavily on tools and models that keep changing month to month.

So I’m curious: > What signals actually correlate with strong engineers today? > How do you design interviews that don’t become obsolete with the next model release? > Are algorithmic interviews still useful at all?

Would love to hear from people who have recently changed their hiring process or have been interviewed using this new approach.

Aurornis 40 days ago |

How to interview in the age of AI is one of the top questions in a manager peer group that I'm in.

Several hiring managers in the group went all in on AI-assisted interviews because they wanted the interviews to match the tools that engineers can use on their work. Most of them have gone full circle and returned back to no-AI interviews.

The main problem with AI-assisted interviews is that they become a test of how familiar the candidate is with the specific AI tool you're letting them use. They started getting inverted signals because the hardcore vibecoders knew all the tricks to brute force the problem with high token spend. They'd do things like spend the interview trying to spin up parallel subagents to brute force a solution.

Then the careful coders who tried to understand the problem and do it right were penalized because every minute they spent trying to do the problem (instead of offloading all cognitive load to AI) was time lost to letting AI do the work.

There were also simpler problems like when someone was familiar with some visual LLM interface tool but didn't have a familiar workflow for the CLI tool used in the interview.

Most people went back to coding interviews that forbid AI and test coding skills, combined with a discussion about their AI experience.

My takeaway was that it's easy to teach new hires how to use AI tools on the job, but it's much harder to bring someone with weak coding skills up to the level of someone with strong coding skills. It's even harder when that person is leaning on AI so much that they're not learning how to code anything.

adavid17 40 days ago | |

> They started getting inverted signals because the hardcore vibecoders knew all the tricks to brute force the problem with high token spend. They'd do things like spend the interview trying to spin up parallel subagents to brute force a solution.

Can’t you just…tell them not do this? Or give them limited model access instead of full Claude Code / Codex?

PaulHoule 40 days ago |

Even if you said AI programming is based on "knowing what to prompt" this still comes down to:

(1) understanding software engineering (for one thing knowing if answers make sense)

(2) subject matter expertise and the ability to communicate with SMEs, fake being an SME by reading books, see the old "knowledge engineer" construct from the 1980s.

(3) knowing specifics about AI coding.

I think (1) and (2) are 80-90% of what leads to success in the long term. My guess is the models are going to get better so (3) skills have a short half life and will matter less, but (1) and (2) will stay the same.

Maybe I'm cynical but if I was designing screeners for this thing I would ask people things like

"How many accounts do you follow on X about AI?" where the right answer is "I don't have an X account" and the higher the count the worse it is.

"What percent of your programming time do you spend thinking about AI programming tools?" and anything over 20% is suspect (but maybe it is a tooling job or something in which case I'd drop it)

That is, I want to see that somebody used AI tools to deliver something 100% done end-to-end that worked and I'd like to see them spending 80% of their time doing.

I'd also be thinking about screeners designed to detect FOMO attitudes and reject people for it.

mewpmewp2 40 days ago | |

What is an example of that 100% end to end that worked?

PaulHoule 40 days ago | | |

Could be as simple as “completed some tickets with quality code that I understood and that passed rigorous review and didn’t add technical debt”

The trouble w/ AI slop is that the people who make it don’t know it is AI slop, not that it was AI generated.

eleventhborn 40 days ago |

I'm interviewing currently. I've come across the following.

* Non-technical recruiters using a list of technical trivia questions and a keywords list to check if you answer correctly (e.g. what is the default isolation level in postgres) - this is the worst.

* Code reviews but on non-generic code, like on a specific framework/product/SDK code in which you may or may not have prior experience - slightly less worse, but if you are going to do this, it is counter intuitive to hire someone who doesn't have production experience in that area.

* take home assignments where they say "we recommend you to not spend more than x hours on this", but then, they expect a very sophisticated work and will reject you if you turn-in a "simple" implementation. Maybe they expect speed and quality?

* leetcode is still a thing.

* non-leetcode, but livecoding, on a likely OOP many to many relationship problem

* give a spec in the first x minutes and leave the interview, you do your thing and then they join in the last y minutes - this is my favorite.

I haven't yet participated in a Agentic AI interview. It is not that common AFAIK.

a coding interview that is objective, repeatable, doesn't put the interviewee under pressure (and doesn't trigger unconscious bias based on accent/appearence)

* take home assigment with minimum requirements criteria and tell them "add a feature of your own" or "any extra work is appreciated".

* Pay them for their time and tokens.

* Use a custom agent to review the code and see how many "high", "medium", "low" issues the agent identifies.

sminchev 40 days ago |

I believe that the new interviews will just get more questions, related to AI. I read it in an article: the AI amplifies. It amplifies the success of the good professionals and it amplifies the failure of the bad.

Good developer will give good prompts, because knows what to ask, and what might be the problem. Good developer can read the code and point badly generated one, and learn the AI how to perform better, which style =to follow. Good developer can evaluate if the used algorithm is the proper for the task and give suggestions, if needed. Good developer can optimize token usage, by using scripts, for example.

Yes, skills in prompting, knowing about new tools and how to use them is also mandatory, but not the most important one, in my opinion.

I guess better soft/social skill are needed. Some people just can't express themselves in real world, and probably they will have difficulties expressing themselves in free texts as well

kokken 39 days ago |

You should interview for the skill that is required in the age of AI-assisted coding.

That is, looking at code that's been written by AI and see what's wrong, what's superfluous and what's missing.

For preparing the code for the interview, I would suggest prompting Claude Code using a requirements document that's purposefully a bit vague so that the AI will have to make choices when writing that code.

When you have the interviewee come over, show them the code and have them criticize those choices and edit the code manually (I know) so that they can demonstrate that they can intervene in the AI process at the correct inflection points.

zambelli 38 days ago |

I've recently been tasked with hiring new college grads (less than 2 years of experience) AI Engineers. Some things that we've been doing:

1) Do they understand the ecosystem of algorithms and models? How do those coexist? Statistical models are the right choice sometimes. Sometimes it's XGBoosted RFs, sometimes NNs, LLMs, etc. And they're not mutually exclusive. I don't think that has changed since AI to be honest - though I get bad candidates that say LLM-everything of course.

2) AI-assisted fluency. Not just in coding, but in concept build. I don't expect them to have the velocity of an AI-fluent principal engineer, but I want to see that they're not resistant to AI-assistance. This is obviously new.

3) Experience with production systems, more than before. Pre-AI, I'd accept that a recent grad might be tuned towards models and algorithms, and wouldn't know much about frontend or backend, or anything you run into in production environments. Given the ease with which you can now setup a small DB, your modeling pipeline, and a full react dashboard or fastAPI frontend...I'd at least like to see they've dabbled in all of that, have a rough sense of it. I don't need them to be full-stack, or even comfortable with it - but AI has raised the breadth bar for me.

cdnsteve 38 days ago |

Have them show you and talk about what they built. If they are not curious and building side projects, doing open source for excitement and making an app for a friend's business, their family member or some passion project, to me that's a red flag.

We are in a new age when the barrier to building is literally the lowest it's ever been. If your GitHub still is a dead graveyard fix that immediately.

I have over 2k commits in 2026 alone and I'm not even an IC anymore. Not including my work account.

I have like 12 side projects being built usually, working on 3-4 in parallel. A few are open source and growing.

All of this has helped me learn at a crazy rate. If your candidate is still just slogging 9-5, doing stand-ups then clocking out without anything extra.... That's all you need to know.

Curious, self learning, high potential people with a track record and good communication skills. That's it.

reggz 39 days ago |

The signal I’d care about is whether they can scope the task, prompt the model, review the output, explain tradeoffs and clean up what the model got wrong.

PocketBot 39 days ago |

A tough middleground. Someone adaptable who doesn't waste resources and knows precisely what they're trying to build.

ex-aws-dude 40 days ago |

I used to ask them to code a binary search in 1 hour

Now I ask them to code Google Search in 1 hour

nirvanist 36 days ago |

instead of doing remote interview how about irl , I think that will fix many issues

pants2 39 days ago |

When I find problems at that AI does a bad job at, I take note, and those are exactly the kind of problems I give in a take-home assignment + interview.

If the candidate turns in AI slop and doesn't understand the fundamentals of what they're working on, reject. If they took the time to learn the subject matter and feed it their own ideas to improve the output, awesome.