In short: a vibe-coded app has usually been used by exactly one person, the one who prompted it. To find out whether it works for anyone else, give it to three to five people who have never seen it, one realistic task each, ask them to think aloud and watch without helping. Fix what they trip over, then repeat with new people. You can run the whole loop yourself or from your coding agent.
Why vibe-coded apps need real users
Andrej Karpathy coined the term in February 2025: “There’s a new kind of coding I call ‘vibe coding’, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.” By November, Collins had made vibe coding its Word of the Year, defined as “the use of artificial intelligence prompted by natural language to write computer code.”
The method changes who checks the work. When nobody reads the diffs, the only review a feature gets is its author clicking through it, and the author is the one person who cannot get confused by it. They know what every button does because they asked for it.
The gap shows up after launch. Nielsen Norman Group describes the pattern in its 2026 piece on UX debt from AI: “AI lets teams build faster than UX can evaluate,” and the result “may work well enough to demo or ship, but user confusion and degraded trust surface later.” Developers see the same thing in code. In the 2025 Stack Overflow survey, the most common frustration with AI tools, named by 66%, was “AI solutions that are almost right, but not quite.”
A usability test is the cheapest way to find the “not quite” before your customers do.
Five steps to your first test
| Step | What to do | What you end up with |
|---|---|---|
| 1. Pick one flow | Choose the flow that decides whether the app succeeds | The flow and its finish line |
| 2. Write the task | Describe a real situation, not the clicks | One scenario without clues |
| 3. Find testers | Three to five people who have never seen the app | A short list of sessions |
| 4. Watch them think aloud | Record screen and voice, do not help | The moments where people got stuck, in their words |
| 5. Hand it to your agent | Give evidence, not conclusions | Two or three fixes, then the next round |
1. Pick one flow and a finish line
Test the flow that decides whether the app succeeds, not the whole app. For most products that is one of three: signing up and reaching the first useful screen, doing the core action for the first time, or paying.
Write down what “done” looks like, for example “the tester has created a project and invited a teammate”. Without a finish line, you cannot tell a success from a tester who gave up politely.
2. Write the task as a situation, not an instruction
How you phrase the task decides what you learn. Nielsen Norman Group’s guidance on task scenarios comes down to three rules:
- Make it realistic. Give the tester a reason to be there that a real customer would have.
- Make it actionable. “It’s best to ask the users to do the action, rather than asking them how they would do it.”
- Avoid clues. Remove any words that appear in your interface, or the tester will simply hunt for the matching button.
Compare two versions of the same task:
- Instruction: “Click New Project, add three tasks and share it.”
- Scenario: “Your team is planning an offsite next month. Set things up so everyone can see what still needs to be booked.”
The first one tests whether people can read. The second tests whether your app makes sense.
3. Find people who have never seen it
You need fewer than you think. Jakob Nielsen’s research found that five users uncover about 85% of the usability problems in a design, and his advice is that “the best results come from testing no more than 5 users and running as many small tests as you can afford.” Steve Krug’s Rocket Surgery Made Easy scales it down further: three people, one morning a month.
For the basics, almost anyone who is not you will do. Krug’s advice is to “recruit loosely and grade on a curve”. You need people from your target audience only when the task depends on domain knowledge, such as an accounting feature or a clinical workflow.
Friends are available but polite, and they rarely give up or say “this makes no sense”. Strangers are more useful and usually expect to be paid. The User Interviews incentives report, based on nearly 20,000 projects, puts the typical rate at $80 an hour for a remote consumer session and recommends $100 an hour as a baseline for professionals.
4. Watch them think aloud, and do not help
Ask each tester to say what they are thinking as they go: what they are looking for, what they expect to happen, what surprises them. Jakob Nielsen calls thinking aloud “the single most valuable usability engineering method”, and it is the difference between knowing that someone clicked the wrong button and knowing why.
While they work:
- Record the screen and their voice, so you can go back to the exact moment later.
- Do not explain, hint or apologise. If they ask what to click, ask what they would do if you were not there.
- Write down where they hesitate, go back, re-read or give up, with their exact words.
- Stop when they reach the finish line or clearly give up. Both are results.
As NN/g puts it, “the point of usability testing is to see what users do, not to hear what they would do.” Opinions about colours and fonts are noise. The moments where behaviour breaks down are the findings.
5. Hand the findings to your coding agent
A vibe-coded app gets fixed the way it was built: by prompting. Give the agent evidence rather than conclusions, one friction point at a time:
Tester 2, 04:12: on the plan step, the tester scrolled past the
Continue button twice and said "I don't know how to get to the next page".
Tester 4 did the same at 03:50. Make the next step obvious on a 13-inch
laptop screen without changing the plan cards.
Fix the two or three problems that stopped people, not everything anyone mentioned. Then run the next round with new testers: a fix often uncovers the next problem, and people who have seen the old version are no longer first-time users.
Doing it from your coding agent
If recruiting and watching sessions is the part you will never get around to, Hallway Test turns the whole loop into a tool call. Your coding agent describes the flow in one sentence, a fresh tester from our in-house team runs it on camera while thinking aloud, and the recording, timestamped transcript and friction points come back into the agent’s context within 24 hours. It works in Claude Code, Cursor and Codex.
The first test is free in exchange for a 30-minute feedback call; the planned price after public launch is $29 per test. The guide to usability testing for AI coding agents explains where a real-user test fits next to automated checks, and what is hallway testing covers the method this one grew out of.
What a usability test will not catch
A usability test shows where people get stuck. It does not check what they cannot see:
- Security. In Veracode’s 2025 test of AI-generated code, 45% of samples failed security tests and introduced OWASP Top 10 vulnerabilities. A 2025 vulnerability in apps generated with Lovable came down to “missing or insufficient Row Level Security (RLS) policies”. Review authentication and data access separately.
- Load and data loss. What happens with a thousand users at once or a failed payment webhook needs tests and monitoring.
- Demand. A flow can be easy to use and still solve a problem nobody has. That is a job for customer interviews.
Sources
- Andrej Karpathy, post introducing vibe coding, X (February 2, 2025)
- Collins' Word of the Year 2025: AI meets authenticity as society shifts, Collins Dictionary (2025)
- Anna Kaley and Raluca Budiu, The Custodial Era of UX: Cleaning Up After AI, Nielsen Norman Group (2026)
- AI, 2025 Stack Overflow Developer Survey
- Marieke McCloskey, Turn User Goals into Task Scenarios for Usability Testing, Nielsen Norman Group (2014)
- Amy Schade, Write Better Qualitative Usability Tasks, Nielsen Norman Group (2017)
- Jakob Nielsen, Why You Only Need to Test with 5 Users, Nielsen Norman Group (2000)
- Steve Krug, Rocket Surgery Made Easy, table of contents (2010)
- The UX Research Incentives Report, User Interviews (2025)
- Jakob Nielsen, Thinking Aloud: The #1 Usability Tool, Nielsen Norman Group (2012)
- Jens Wessling, 2025 GenAI Code Security Report, Veracode (2025)
- Matt Palmer, CVE-2025-48757 (2025)