How to evaluate developers in the vibe coding era
AI lets candidates submit working code without understanding it. Here's how to update your hiring process to test what vibe coding can't fake.
Table of contents
AI can write code now. That means candidates can submit working solutions without understanding a single line.
Andrej Karpathy coined the term "vibe coding" in February to describe developers who prompt AI, accept the output, and move on without reviewing or understanding the code. Nine months later, Collins Dictionary named it their Word of the Year. Your candidates know what it is — the question is whether they're doing it.
For hiring managers, this creates a new problem: traditional coding assessments assume the candidate wrote the code. If a candidate can prompt their way to a working solution, how do you tell whether they'll be able to maintain it, debug it, or extend it when requirements change?
The problem vibe coding creates for hiring
A developer who prompts well but doesn't understand the output will pass your coding test. They might even impress you. The problems start after they're hired.
We're already seeing the consequences. An analysis of apps built with Lovable, an AI app builder, found that 170 out of 1,645 applications had publicly exposed sensitive user data. The developers lacked the expertise to identify the vulnerability. They couldn't see what they didn't understand.
SaaStr founder Jason Lemkin was vibe-coding an app when the AI agent ignored his commands and deleted the entire database. Then it lied about what happened.
These aren't junior developer mistakes. They're what happens when people build without understanding. Your coding assessment might not catch the difference.
What your coding assessment should test for
Four skills separate developers who ship from developers who get stuck:
Understanding how code works. They can read a solution and know whether it does what it should.
Debugging. They can describe expected vs. actual behavior and systematically find the gap.
Architectural thinking. They know which components to build and how they connect.
Problem decomposition. They break vague goals into concrete steps before writing any code.
AI doesn't replace these skills. It makes them more valuable. Someone has to verify the output, catch the mistakes, and make the judgment calls. That's who you're hiring.
The red and green flags below indicate whether a candidate has these underlying capabilities—or has learned to prompt without them.
Red flags when evaluating developers
Watch for these signals during interviews:
- Can't explain their own code. Ask them to walk through a solution they submitted. Vague answers or "it works" as the only justification is a warning sign.
- Struggles when requirements change. Mid-problem pivots reveal whether they understand the architecture or are starting from scratch each time.
- Can't debug without AI. Ask how they'd diagnose an issue. If every answer involves "I'd ask ChatGPT," they may not have the fundamentals.
- No reasoning about trade-offs. "Why this approach?" should get a real answer. "It seemed right" isn't one.
- Vague on architecture. Ask how components connect. Prompt-dependent developers often can't see past the function they're generating.
These aren't character flaws. They're signs someone learned to prompt before they learned to code.
Green flags that indicate real understanding
Look for candidates who demonstrate:
- Can critique AI-generated code. Give them code to review. Do they catch issues? Do they suggest improvements?
- Explain trade-offs unprompted. "I chose X because Y" signals someone thinking through decisions, not accepting defaults.
- Break down problems before coding. Watch how they approach a task. Do they plan, or do they prompt immediately?
- Debug systematically. Good developers form hypotheses, test them, and narrow down. They don't guess randomly.
- Ask clarifying questions. Candidates who want to understand requirements before building tend to build better.
- Can simplify or extend. Ask them to modify their solution. Developers who understand their code can adapt it.
A developer with these fundamentals uses AI to move faster. One without hits walls the moment something breaks.
How to update your developer evaluation process
Recognizing these signals is step one. Here's how to surface them systematically.
You don't need to ban AI from your assessments. You need to test what AI can't fake.
Add code review exercises. Give candidates code to critique instead of (or in addition to) code to write. Can they identify problems? Suggest improvements? This tests understanding directly.
Test debugging live. Present a broken feature. Watch how they diagnose it. Are they systematic or random? Do they form hypotheses?
Probe architectural decisions. "Why did you structure it this way?" Follow up on their answer. Keep asking "why" until you understand their reasoning.
Change requirements mid-task. A developer who understands their code can adapt. One who prompted their way there often has to start over.
Allow AI, but require explanation. "You can use Copilot, but walk me through what it gave you and why you accepted it." This surfaces understanding without creating an artificial constraint.
Our advanced challenges are designed to test architectural thinking and problem decomposition — the same skills that matter in your assessments.
Start here
Vibe coding changes what you need to test for, not whether testing matters.
Pick one update to your process this week: add a code review exercise, test debugging live, or ask candidates to explain AI-generated code. See what it reveals.
If you're onboarding junior developers who need to build these fundamentals, our guide to AI coding assistants for beginners covers how to use AI without undermining skill development.
Looking for candidates with verified skills? Frontend Mentor for Hiring connects you with developers who've demonstrated their abilities through real projects — not prompts.