From this perspective, the people who use these tools are beneath the people who design them. From this perspective, the rightful state of the world is for the superior members of the in-group to control the experience of the inferior members of the out-group.
By using these tools, you willingly accept the position of the inferior/the sub-human/the addict.
we will tell ourselves that we should wait eagerly to see if we will be graced by "tibo's" largesse, like peasants clamouring to see a royal throwing bread among the masses.
or that it is right and good that claude should refuse to answer on account of safety, but hack other companies on behalf of anthropic. surely we are too simple, and sometimes naive, truly we know not what is best for ourselves.
no more.
Vive la révolution!!
Meanwhile, how many jobs have AI critics created? Crickets.
So I'm increasingly uncertain about what and how to test. My default for now is still to rely on ability to write basic code fluently, but I'm open to changing this perspective.
I really want to know how this existing repo AI-assisted live coding test works, with example problems.
It seems the standard data structure puzzle type thing won't be feasible if you are using an LLM.
Also the latency for these agentic coding/prompts seems like it would make the interview a bit awkward.
Anyone been conducting or taking interviews with this kind of thing with thoughts to share?
I ask them to implement xyz thing. What I’m looking for is how they interact with the AI agent. Do they ask the agent to plan first? Do they review the plan? Etc. It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.
There are a series of follow-up questions about how to productionize the toy system, which gives some additional signal about how well they understand what they’re making. I sprinkle these in when there’s dead air waiting for the bot to think.
I think we’ll need to evolve and refine this problem and process as the models continue to improve.
It's also the kind of stuff that somebody can learn in a week, so not hiring the right person who just didn't spend this week of time yet for whatever reason is a loss.
- You get a leetcode question and if you're lucky is an easy medium that you can solve, if you're really lucky you already solved it and can pretend you are approaching the problem the first time. Good luck if you get a hard question and you never saw it before.
- You get a home assignment, in a framework you might not know but you're expected to be fluent with it, then waste 1 hour setting up the project structure, and one more hour to find out how the framework expects you to define the CORS allow list. You are expected to deliver the project in 3 hours.
The good I see in AI is that it completely removes the need to study just for interviews, and you can also delegate all the project setup to the AI. Then you can focus on what you would test (e2e? integration? what are the boundaries? what do we mock?), how to keep the documentation, how to structure your code. You have an expensive endpoint, do I make it sync or add an async jobs framework?
Imagine you're an expert in C++ interviewing for a Django position and the interview consists of fixing a big in a repo. The bug is that a function without type hinting is modifying what is expected to be a list, but the caller is passing a tuple. Trivial after a week you work in python and you have your environment set up for type warnings, also trivial with AI and definitely not an interesting problem that shows expertise with software engineering in general.
We also did this in our last interview at work, and it was a really good indicator to see if someone just copy pasted code, or understood it after it was generated. Some candidates had a unit test fail and couldn't debug it for his life, even if he "wrote" all the code himself. Others simply did not understand the architecture they wrote, and assumed that a function defined with "async" and awaited would run in parallel from the code that called it (as if you spawned a thread)
- We have embraced everything about the AI Era at our company
- Arbitrary topic about how that changes something
They have to earn it, as the tokens are not free.
Given that deskilling and over-reliance in AI assistance will continue to happen, putting a hard token limit <100k tokens in the interview process serves as a great filter to prevent the vibe-coders and "tokenmaxxers" out and forces a higher bar for quality, with clean code and reasoning across well maintained software with less tokens rather than increasing the slop.
Do you want a candidate that knows when to use AI and carefully uses tokens with in their limits, or do you want a candidate generating incomprehensible AI slop to be tokenmaxxing out your company limits and then draining your company bank account?
We explicitly said -- feel free to use whatever framework and AI assistant, just show us how you do it. The practical part had no leetcode too. Just build something really basic, then explain a snippet of code (3 lines) and generalize it. A trick question (with a disclosure it's a trick question) if the candidate did it fast enough that we didn't have to go into the overtime. A bit of theory about protocols, all in all an hour and we leave another 30 minutes on top to answer questions.
At the end of the day we just filter out with confidently bad takes, people who can't do 2+2 and ones that can't understand the question without rephrasing it three times.
The most bizzarre candidate didn't know anything at all, but was so relaxed and confident, that he spent all of the 30 minutes asking about the company and how his day would like and all that, while he clearly bombed it.
The difficult part is how to not filter out a competent person who doesn't necessarily agree with all of your takes, uses all the same tools and had all of the same experiences as both of the interviewers.
We resorted to filtering candidates in-person with 5 basic technical questions on pen and paper. And I mean _really_ basic questions.
This was surprisingly effective because it filters for many non-technical skills like being able to read and write English, follow instructions, and show up to the office, on time, and appropriately dressed.
The number of candidates that failed these basic skills was astounding. We had candidates show up 20 minutes late, or email 2 minutes before the "interview" asking for a Teams link even though the invitation stated the meeting was in-person (highlighted in yellow). Others couldn't write their own name legibly on the paper.
A candidate that cannot answer basic technical questions has no hope of being able to prompt an AI effectively or review the code it produces.
I agree. A technology professional who views themselves as "one of the cool kids" (read: easily manipulated by social media) is a legitimate security threat, as are many of the popularly promoted approaches to "LLM-assisted software developement".