Cheating is possible in the abstract, but I found a tight correlation between interview and take-home performance. For whatever reason, candidates didn't seem to cheat much.
The AI point is worth diving into a little. This was a year ago, so SOTA was worse, but I didn't find it terribly hard to write questions AI couldn't solve, whose answers you couldn't search for, and which good candidates could solve. The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section.
I don't think that moat will exist indefinitely, but today's AI just isn't very good at a lot of incredibly basic tasks unless the operator has enough outside knowledge to guide it in the right direction (and if a candidate did that I mostly wouldn't care because, by definition, they had the knowledge I was looking for). I use AI a lot, it's great at a lot of things, some even quite complicated, but it was weaknesses, and those are pretty easy to exploit.
Your description of the test and your replies to questions indicate you've come up with a pretty great assessment for the role(s) you hire for. Especially where you mentioned:
> The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section
I also like how you allow/encourage self-assessment, where if a candidate can't do the test in ~20 minutes under zero pressure, they probably won't be a good fit in the role itself.
The AI point is worth diving into a little. This was a year ago, so SOTA was worse, but I didn't find it terribly hard to write questions AI couldn't solve, whose answers you couldn't search for, and which good candidates could solve. The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section.
I don't think that moat will exist indefinitely, but today's AI just isn't very good at a lot of incredibly basic tasks unless the operator has enough outside knowledge to guide it in the right direction (and if a candidate did that I mostly wouldn't care because, by definition, they had the knowledge I was looking for). I use AI a lot, it's great at a lot of things, some even quite complicated, but it was weaknesses, and those are pretty easy to exploit.