AI broke the take-home test. Here’s how we hire engineers now.
Written by 
Andres Oller Monutti
Listen to this article
3:49
Introduction
AI tools can now finish a take-home test for a candidate. So we dropped ours and started watching engineers work with AI in front of us.
In late October, Felix Rieseberg, an engineering manager at Anthropic, wrote (opens in a new tab) that he had stopped giving coding interviews because Claude now beats candidates at them. He hasn’t found the replacement yet, and asked anyone who has to share it.
We have one. We dropped take-home tests in May, built a new interview loop over the summer, and since September it’s how we hire Flutter engineers for Wishlabs (opens in a new tab) and Wellture (opens in a new tab). Candidates spend about three hours with us instead of a week, and they bring their AI tools. Anyone can get working code now. We hire the engineers who own it.
Anthropic’s performance team (opens in a new tab) hit the same wall in January. Their answer was to move the test away from everyday work, into puzzles a model hasn’t seen. We went the other way, closer to the real job, because that’s where our engineers now spend their day: directing AI and checking what it produces.
Closer to the real job, and faster
Our engineers build with AI every day, and more of their day now goes into directing it and checking what it produces. So we built the new loop to look at exactly that.
It also had to be faster for everyone.
Hiring takes less time on both sides
Candidates now spend about 3 hours with us instead of a week on a take-home.
- Each interviewer’s time per candidate77% less
- With a take-home5–6 hoursNow1–1.5 hours
- Time until we decide44% less
- With a take-home4–5 weeksNow2–3 weeks
Can you explain the code you accepted?
We spend less time on whether someone can produce code and more on whether they understand it. Candidates read code they didn’t write and tell us what they’d change. They also work on a real problem with the AI tools they use every day, and talk us through what they hand to the AI and what they check before trusting it.
What we care about most is ownership. If you can’t explain the code you accepted, a working feature doesn’t tell us much. We’d rather see part of a feature you fully understand than a complete one you can’t explain.
Every interviewer writes down what they saw before we discuss a candidate, so each decision rests on what happened in the interview.
If you’re interviewing with us, bring the setup you actually work with. Use it the way you would on a normal workday, and tell us what you’re checking as you go. Some of the most useful moments come when the AI gets something wrong and you have to notice.
Good until the next model release
This process is a few months old, and it already looks different from the first version we wrote down in May. It will keep changing as the models get better every few months, and an exercise that tells us a lot today can be one a model solves easily tomorrow. Anthropic’s take-home story shows how fast that happens.
Practices that worked for years can stop working with a single AI model release, and nobody hands you the replacement. We have to keep trying new formats and be honest about dropping the ones that stop working, even the ones we liked.
So if you’re reading this before an interview with us, expect the process to look a little different from what you read here. And if you lead engineers and are working through the same question, we’d like to compare notes.
