How to Run a 20-User Beta Test
Contents
A 20-user beta test is small enough to manage closely and large enough to expose patterns you will miss in your own testing.
The goal is not to collect twenty opinions. The goal is to learn whether the right users can complete the main workflow, understand the product, return to it, and recover when something goes wrong.
- Decide what you want to learn before inviting anyone. - Recruit users who match the intended audience, not just friends who will be polite. - Watch completed tasks and repeat usage alongside written feedback. - Separate bugs, usability problems, missing features, and requests for a different product. - End the beta with a launch decision, not an endless list of comments.
Define the beta question first
Write one question the beta should answer.
Examples:
- Can independent tutors manage a trial lesson from booking to follow-up?
- Can tenants submit a repair request without sending a separate message?
- Can a small team understand the dashboard without training?
- Do users return to the product when the problem happens again?
A beta with no question becomes a general feedback exercise. Every comment sounds equally important because you have not decided what matters most.
Choose the right 20 users
Your participants should resemble the people you want after launch.
A useful mix might include:
- 12 people who match the main target user
- 4 people with a nearby use case
- 2 people who are less comfortable with new software
- 2 people who will challenge your assumptions
The exact mix can change. What matters is that the group is not made entirely of friends, other founders, or people who already know how the app works.
Track why each person was invited. You want to know whether a problem came from the product or from a participant who never had the target need.
Split the beta into small cohorts
Do not invite all twenty users at the same time if you can avoid it.
A simple schedule is:
| Cohort | Users | Purpose |
|---|---|---|
| Cohort 1 | 4 | Find blocking setup and workflow problems |
| Cohort 2 | 6 | Test the first fixes with similar users |
| Cohort 3 | 10 | Check whether the improved flow works across a wider group |
This gives you room to fix obvious problems before they affect everyone. It also keeps feedback connected to a specific version of the product.
Give users a real task
"Try the app and tell me what you think" produces weak feedback.
Give participants a task that resembles a real situation:
"You have a leaking kitchen tap. Create a request, add a photo, check that the landlord received it, and find the current status."
A good task includes:
- a starting situation
- a clear outcome
- enough information to act
- no instructions that reveal every correct tap
If users need a long explanation before they can begin, that is already useful evidence.
Prepare the app before inviting people
Before the first invitation, test the basics yourself:
- account creation and sign-in
- password reset or sign-in recovery
- the main workflow
- empty states
- slow network behavior
- denied permissions
- image or file uploads
- notifications
- account deletion or support contact
- test data cleanup
A beta user should not become your substitute for basic quality assurance. Their time is for product learning, not finding an obviously broken sign-in button.
Write a short onboarding message
Your first message should tell users what to do and how to report a problem.
Include:
- what the app helps them do
- the task you want them to try
- how long the test should take
- where to report bugs
- what information to include
- when you will follow up
Example:
"Please use the app to create one repair request, add a photo, and check the status later today. The first test should take about ten minutes. If something fails, send the screen name, what you expected, what happened, and a screenshot if possible."
Do not send a long product tour before users touch the product. You want to learn what they understand without a lecture.
Track behavior, not only opinions
Ask users what they think, but record what they do.
Useful beta events include:
| Event | What it tells you |
|---|---|
| Account created | Setup completed |
| First core action | Activation started |
| Core workflow completed | Main value was reached |
| Error or retry | A step needs attention |
| Return session | The problem brought the user back |
| Support request | Confusion or missing functionality |
The exact event names depend on the product. Keep them tied to decisions. Tracking every tap will create more data than you can use.
Ask for feedback at the right moment
Do not wait until the end and ask, "What should we improve?"
Ask smaller questions after a real action:
- What did you expect to happen here?
- What made this step difficult?
- What information were you missing?
- What would you do next if this were your real task?
- Would you use this again for the same problem?
One short conversation after a task is often more useful than a long survey full of general opinions.
Separate four types of feedback
Every comment should enter one of four categories:
| Type | Example | Response |
|---|---|---|
| Blocking bug | The app cannot save the request | Fix before more testing |
| Usability problem | Users miss the status button | Improve wording or layout |
| Missing requirement | A business needs multiple locations | Confirm the target customer |
| Feature request | Add a dark mode | Defer unless it supports the core goal |
A feature request is not automatically a product requirement. It may be a workaround for a usability problem, or it may belong to a different customer group.
Use a simple severity system
You do not need a complicated issue tracker for twenty people.
- P0: data loss, security problem, or the main workflow cannot complete
- P1: a common workflow fails or creates serious confusion
- P2: a problem has a workaround but should be fixed before a wider launch
- P3: cosmetic issue or improvement for later
Include the participant, device, app version, steps to reproduce, expected result, actual result, and evidence.
"Login broken" is not enough to fix a problem. "iPhone, email sign-in, after tapping the confirmation link, the app returns to the login screen" gives the team something to investigate.
Protect beta users and their data
Tell participants what the beta involves.
Explain:
- what information the app stores
- whether the data is real or test data
- how to remove their account
- how to contact you
- whether the app is experimental
- what may change during the test
Do not ask beta users to upload sensitive information unless the product needs it and your storage, access, and deletion process are ready.
Run a two-week beta cycle
A practical cycle can look like this:
| Day | Action |
|---|---|
| 1 | Invite Cohort 1 and observe setup |
| 2 to 3 | Fix blockers and unclear steps |
| 4 | Invite Cohort 2 |
| 5 to 7 | Compare behavior and feedback |
| 8 | Ship a small update |
| 9 to 12 | Invite Cohort 3 |
| 13 | Review evidence and unresolved issues |
| 14 | Decide what happens before launch |
This is a sample schedule, not a rule. A complex product may need longer. The point is to create a defined learning cycle instead of keeping the beta open forever.
Decide whether the app is ready
At the end, answer five questions:
- Can the target user complete the core workflow?
- Do users understand the result without personal training?
- Are the blocking data, security, and payment problems resolved?
- Do users return when the problem happens again?
- Can you support the product if more people arrive?
You do not need every requested feature. You do need confidence in the promise you make.
What to do with the feedback
Create three lists:
- fix before launch
- test after launch
- reject or park
Write one sentence for every decision. This prevents the same request from returning in the next meeting without new evidence.
Share a short update with participants. Tell them what changed, what did not, and what the next test will cover. People are more likely to help again when they can see that their time mattered.
Build and test the beta with Huxly
Huxly helps you build and refine a mobile app with frontend screens, backend logic, authentication, databases, payments, AI features, testing, and preparation for TestFlight or Google Play. You can use the first beta cohort to test a real workflow, then update the product based on observed problems instead of guessing from a feature list.
FAQ
Is twenty users enough for a beta test?
Twenty users can reveal usability problems, workflow gaps, and repeated failure patterns. It cannot prove that every customer segment will behave the same way or guarantee business success.
Should beta users be paid?
That depends on the difficulty and length of the test. A small thank-you can help, but payment should not replace recruiting people who genuinely have the problem.
Should I invite friends and family?
They can help with early technical checks. They should not be the only participants because they may be unusually forgiving or unfamiliar with the target workflow.
How long should a beta test last?
Run it long enough for users to complete the main workflow more than once when the product depends on recurring use. A defined one or two-week cycle is easier to manage than an open-ended beta.
What if users request completely different features?
Ask what problem the request solves. If it belongs to another audience or changes the product promise, record it separately instead of expanding the beta scope.
Should I launch after the beta?
Launch when the core workflow works for the intended users, serious data and security issues are handled, and you have a plan for support. The beta does not need to produce universal approval.
