Testing an app with real users means giving it to people who have never seen it, asking them to do something a customer would do, and watching where they get stuck. It works the same way for a web app, a mobile app or an internal tool, and it needs fewer people than most teams expect.
When should you test an app with real users?
Test whenever real people are about to meet something new: before launch, after a big change to a key flow, and when a metric drops and you cannot see why in the data. Analytics tell you where people leave; a usability test shows you why.
Three moments are worth a test almost every time:
- Before launch. Only the people who built it have used it, and they cannot get confused by it.
- After a big flow change. A new onboarding or checkout changes what first-time users see, even if every screen passed review.
- When a number moves the wrong way. A drop in sign-ups or conversions usually points to one or two steps. Watching five people go through them beats guessing.
An AI agent clicking through your app catches broken links while you build, but it does not hesitate or misread a label the way a person does. The comparison of real users and synthetic users explains where each one fits. If your app was built mostly by prompting, the shorter guide to testing a vibe-coded app gets you to a first test in five steps.
Pick one flow and define success
Test one flow per round, the one that decides whether the app works for its users, and write down what success looks like before anyone starts. A test of “the whole app” produces a pile of opinions; a test of one flow produces a list of fixes.
Good candidates are the flows where people either get value or give up:
- signing up and reaching the first useful screen;
- doing the core action for the first time, such as booking a slot or sending an invoice;
- upgrading or paying.
Then define the finish line in observable terms: “the tester has scheduled a recurring payment and can see it in the list”, not “the tester understands payments”. Without it, a polite tester who gave up quietly looks like a success.
Moderated or unmoderated, in person or remote
For most small teams, remote sessions with a recording are the practical choice; moderated sessions are better when you need to ask follow-up questions. In a moderated session a facilitator watches live and can ask questions. In an unmoderated one, the participant works alone and you watch the recording later.
| Format | Who watches, and when | Best for | Trade-off |
|---|---|---|---|
| In person, moderated | You, in the same room | Early concepts, complex or physical products | Scheduling and travel; fewest sessions per day |
| Remote, moderated | You, on a video call | Follow-up questions without travel | Silence on a call is hard to read |
| Remote, unmoderated | You, from the recording | Specific flows, tight timelines | No follow-up questions during the session |
Nielsen Norman Group’s article on remote usability tests states the trade-off plainly: they generally recommend in-person testing whenever possible, but when it is not, “remote testing is preferable to the alternative: skipping the test altogether.” The same article recommends unmoderated tests “when the main focus of the study is a few specific elements, rather than an overall review”, which describes a one-flow test well. It also warns that no-show rates are higher for remote studies, so recruit one more person than you need.
How many people do you need?
Three to five per round, then fix, then test again with new people. Jakob Nielsen’s research found that five users uncover about 85% of the usability problems in a design, and that a single user already uncovers about 31% on average across the projects he studied.
The point is not to stop at five. Nielsen’s curve shows that finding nearly all the problems takes about 15 users, but he recommends spending them on three rounds of five, because “the ultimate user experience is improved much more by 3 studies with 5 users each than by a single monster study with 15 users.” Each round of fixes changes what the next tester sees, and a fix often uncovers the next problem.
The one exception is an app with very different groups of users, such as patients and clinicians, or buyers and sellers. Nielsen’s numbers hold for comparable users, so test each group separately: he suggests three to four users from each group when there are two, and three from each when there are three or more. For what each option costs in money and staff hours, see how much usability testing costs.
Where to find testers who have never seen the app
Anyone who has not used the app and roughly fits your audience will find most of the basic problems. What matters most is that they are seeing it for the first time.
- Colleagues from other teams. Sales, finance or operations have never seen your flows. This is the classic hallway test.
- Friends of friends. One step removed is better than your own friends, who know you and tend to be kind.
- Communities where your users already are. Forums, Slack and Discord groups, meetups. Ask the moderators first.
- Existing customers, for a new feature. They know the product but not the new flow.
- Recruiting panels. Paid services that screen participants to a profile, for a specific job title or industry.
Avoid anyone who helped build the app, watched a demo or tested the previous round: they already know where the buttons are. Strangers usually expect an incentive; colleagues often do not.
Write tasks as situations, not instructions
Describe a situation the tester might really be in and let them find the way; never name the buttons. Nielsen Norman Group’s guidance on task scenarios sums up good tasks in one line: they are “realistic, encourage an action, and don’t give away how the interface should be used.”
Compare two versions of a task for a personal finance app:
- Instruction (bad): “Go to Budgets, tap Add category, create a Groceries budget of $400 and turn on alerts.”
- Situation (good): “You keep overspending on food and want a warning before it happens again next month. Set that up.”
The instruction tests whether people can match words on the screen to words in the task. The situation tests whether the app’s structure and labels make sense to someone with a real goal. Use the customer’s words rather than your interface labels, give the tester any details they need (a test card number, a sample file), and try the task on a colleague first: if they ask what you mean, rewrite it.
Run the session: think aloud and do not help
Ask the tester to say what they are thinking as they work, then stay out of the way. The struggle you feel the urge to rescue them from is the finding.
Jakob Nielsen calls thinking aloud a “window on the soul”: you hear the tester’s misconceptions, and “when users misinterpret design elements, you need to change them.” He also lists its weak spots. Talking to yourself is unnatural, and people tend to filter what they say to sound smart, so remind them gently to keep talking and that there are no wrong answers.
At the start, tell the tester you are testing the app, not them. Then:
- Do not help. No hints, no explanations, no apologies. If they ask what to do, ask what they would do if you were not there.
- Do not ask leading questions. “What do you expect to happen?” is fine; “Did you see the button on the top right?” is not.
- Let them finish or give up. Both are results. Set a time limit in advance.
What to record, so you can go back to the exact moment later:
- the screen, with clicks or taps visible;
- the tester’s voice, and ideally their face;
- the time of every hesitation, wrong turn, backtrack, re-read and give-up, with their exact words;
- for a web app, the browser console and network requests, so an error the tester never saw is still on the record.
Testing a mobile app: what changes
The method stays the same; the setup changes. Test on the tester’s own phone with a real build, record the screen and voice from the device, and plan for interruptions that never happen on a laptop.
The build. On iOS, Apple’s TestFlight distributes pre-release builds: you can add up to 100 members of your team as internal testers and invite up to 10,000 external testers, and builds for external testers go through App Review first. On Android, the Play Console’s internal testing track or a direct build link does the same job. Send it a day early so installation problems do not eat into the session.
Recording. On-device screen recording captures taps but not the tester’s face; a video call adds the reactions. Ask testers to silence notifications so private messages stay off the recording.
What to watch for that web tests miss:
- touch targets too small or too close together;
- the keyboard covering the field or the button the tester needs;
- permission prompts (notifications, location, camera) appearing before the tester understands why;
- hidden gestures, such as swipe to delete;
- the app losing its state after a call, an app switch or a weak connection.
Turn observations into fixes
Rate each problem by severity, attach the evidence, and fix the cheapest version of the most severe problems first. Then test the fix with new people.
Jakob Nielsen’s severity scale is still the simplest one to use: 0 means not a usability problem, 1 cosmetic, 2 minor, 3 major (“important to fix”), and 4 a usability catastrophe, “imperative to fix this before product can be released.” He rates severity on three factors: how often the problem occurs, how much it hurts when it does, and whether it persists or users get past it once they know about it. A problem that stopped three of five testers at the same step is a 3 or 4; a tester who disliked a colour is a 1.
Write every finding with evidence, not conclusions:
Severity 3. Tester 2, 06:40: on the payment screen, scrolled up and down
twice, then said "where do I confirm this?". The Pay button sits below
the keyboard on a 6.1-inch phone. Tester 5 stopped at the same step, 05:15.
Timestamps let anyone, including a coding agent, jump to the moment and check the interpretation. Then pick the cheapest fix that removes the problem: a clearer label or a moved button usually beats a redesign. Fix the top two or three, then run the next round with new testers to check that the fix worked.
Doing it from your coding agent
If recruiting testers and watching recordings is the part that never gets done, Hallway Test runs it from inside your coding agent. It is a remote MCP server for Claude Code, Cursor, Codex and other MCP clients; see what a usability testing MCP is for its tools and an example result.
Your agent sends a URL and a task in plain language. A tester from our in-house team who has never seen your app runs it while thinking aloud; the session itself usually takes 15 to 30 minutes, and results come back within 24 hours. For a web app the recording includes the screen, the tester’s voice, a transcript, a click and navigation log, and the browser console and network logs captured with Jam. A React Native or Expo app can be tested on iOS and Android through a mobile SDK, with TestFlight builds on iOS.
Your agent then pulls the evidence through MCP tools and writes the timestamped friction points itself, on its own tokens. Hallway Test runs no AI model and never sees your source code. Setup is one line, covered in connecting the MCP server. The first test is a free pilot, one usability test plus a 30-minute feedback call, which you can book here; after public launch the price is $29 per test.
Sources
- Jakob Nielsen, Why You Only Need to Test with 5 Users, Nielsen Norman Group (2000)
- Jakob Nielsen, Thinking Aloud: The #1 Usability Tool, Nielsen Norman Group (2012)
- Marieke McCloskey, Turn User Goals into Task Scenarios for Usability Testing, Nielsen Norman Group (2014)
- Amy Schade, Remote Usability Tests: Moderated and Unmoderated, Nielsen Norman Group (2013)
- Jakob Nielsen, Severity Ratings for Usability Problems, Nielsen Norman Group (1994)
- TestFlight, Apple Developer