How to conduct a B2B software beta program with little to no budget
What does a scrappy product manager do when they’re launching a new product or complex feature, but they don’t have a big enough budget for UX testing consultants or A/B test applications? How do you beta test your new feature with end users and get the validation you need that you’re on the right track prior to launch? This is of course not ideal, but if you work for a small company or an early stage start-up, this is probably a situation you will run into. Whatever you do, DO NOT just tell people internally the feature is on beta and hope someone will eventually provide some feedback. That’s a good way to doom your new feature to launch failure.
In this post, I’ll give a real world, step-by-step example of how to conduct an organized and effective beta test program in a B2B product launch scenario using tools you probably already have or can get for free.
I recommend running through these steps twice if possible: once less formally with a set of internal employee test subjects (I like to use subject matter experts from the professional services or technical support teams) and then again with a set of customer test subjects. Testing with internal team members first will help you smooth out the experience for your customer testers and hopefully leave them with a good first impression of your new feature. Ideally, you will want your internal team members to vet your beta testing scenarios and documentation to make sure they are easy to understand. They can also help you find the first wave of bugs and user experience issues to address so that your client end users don’t run into as many during their test.
Step 1 - Set goals
First, you need to determine what you want to get out of your beta test process. This will help guide you in your decision-making process later on when you’re determining how you collect feedback and what kind of feedback to collect. Some examples of goals you may have for your beta period are: finding bugs, assessing usability of the application as certain tasks are performed, determining if you have the right feature set or all of the above.
I like to keep notes during the design and development process about questions we had or controversial decisions we made. Then I can revisit those as inspiration when I’m determining what I want to validate during my beta test period.
Step 2 - Design testing scenarios
I have found that if you just set users loose to do whatever they want with your new feature and then ask for feedback, the quality of the feedback is rather low, and it can be difficult to tease out any patterns across different test users. Instead, I recommend that you identify some common tasks that you would like your users to run through to validate whether or not you’ve accomplished your goals.
For example:
Goal: Assess the usability of my new widget that allows the user to design a marketing asset.
Tasks:
Navigate to the new widget
Create a new marketing asset
Add an image to the design
Add text to the design
Change text colors
Change background colors
Save the design
Export the design
Step 3 - Identify test users
Next, you will need to decide who should test your new feature.
I like to ask our customer success managers and other client-facing team members if they can make recommendations on who we should try to recruit. I request that they consider the following guidelines for who to recommend:
They should fit the persona you had in mind when you developed the feature
They should be an active and engaged user of the software
They should have an open and curious personality
Bonus points if they were one of the clients who asked for the feature to be built in the first place - these are often the most enthusiastic testers
I usually start an Excel spreadsheet or Google Sheet a few weeks or a month ahead of when I want to start the recruiting process and I share this around to the client-facing teams so that they can add good candidates along with their contact information.
If you have a ticketing system for collecting feature requests like Service Desk, UserVoice, or Aha! you can also do a search for users who requested similar functionality in the past and add them to your list.
You may even wish to target an entire user type such as “system admins.” In that case, skip right to the next step and send an email to that entire audience.
If you don’t have any existing users you can draw from, you can enlist the help of your marketing team. They may be able to help you find users in the right persona through LinkedIn advertising if they’re willing to share their budget with you. Alternately, you can look for professional groups or forums targeting that persona and post there looking for interested participants. For example, if you’re looking for marketing researcher users, you could try the Reddit r/MarketingResearch forum. Make sure you read the posting guidelines first to make sure that this type of activity is allowed.
Step 4 - Recruit test users
If you feel really confident that you have the perfect list of test subjects from your CSMs, you can skip this step and proceed to Step 5.
Once you have a list of target users, create a screening survey to collect some preliminary information about them so you can be sure they are a good testing candidate. You can ask them questions to help you assess if they are the correct target persona and if they have the right background knowledge to complete the tasks you will assign to them.
I like to use Microsoft Forms or Google Forms for this. This survey should be quite short. It should be no more than 5 questions.
Then enlist the help of your marketing team to draft a recruitment email. The email should contain the following information:
What you would like them to do
How long it will take them to do it
Why you would like them to do it
What’s in it for them
Sometimes all you can offer is that their feedback will shape the future development of a feature which will eventually provide value to them
You might be able to offer them a free subscription to that feature for some period of time
You might have some marketing swag laying around that marketing will let you give away. We used some really nice Yeti mugs with our company logo once and those were very popular
If you have a little bit of budget, a gift card or charitable donation in their name is always nice
A link to your screen survey with a call to action to fill it out if they are interested in becoming a beta tester
A deadline by which you would like them to fill out the survey - I like to make it about a week away
Step 5- Internal training and communication
Sometime just before your recruitment email is set to go out, assemble the customer-facing teams (client success, professional services, sales, technical support, etc.) and make sure they are aware of your beta test initiative and prepared to answer customer questions about it. There is nothing worse than being blind-sided by a customer who knows more than you do about what your company is doing.
Step 6 - Documentation (optional)
If the feature you’re testing is very complicated and robust, you might wish to create some preliminary documentation about that feature for users to refer to during the test. However, if your goal is user experience testing, use documentation sparingly. If you’re trying to create an intuitive application and you want to know if you’ve done that successfully, you may not want users to go into the test with a lot of information up front.
Step 7 - Collect feedback
I like to divide my test subjects into two groups:
A small group of interviewees for a live, observed test
A larger group of testers who can provide quantitative feedback via a survey
Group 1
For the smaller group, reach out to them via email (consider Ccing their client success manager here) to schedule a mutually agreeable time for the interview. Then develop an interview script based on the tasks you identified in Step 2. During the call, ask them to share their screen (using Teams or Zoom) so that you can observe them going through the test steps. When I conduct an interview, I usually invite a team member along to take notes for me about their observations and things the user says during the interview so that I can give the user my full attention. Along the way, look for areas of UX friction where they cannot perform the task or questions they ask about functionality which may not exist. Take note of any potential bugs you see them run into.
Group 2
For the larger group, design another survey in Microsoft or Google Forms. This survey will ask them to run through the testing tasks you identified in step 2 and will ask them to answer follow up questions about how it went.
For example:
Imagine you have been asked to design a new business card for your company using XYZ new feature.
Task 1: Navigate to the XYZ feature in the ABC application and create a new business card document.
How difficult was it to complete Task 1?
Did you encounter any issues completing Task 1? Please describe the issues you encountered.
Did the XYZ feature do everything you expected it to do?
Is there anything else you would like to tell us about Task 1?
…and so on
When the survey is ready, once again enlist the help of marketing to draft an email to your finalized list of participants. This email should once again contain the following information:
What you would like them to do
How long it will take them to do it
Why you would like them to do it
What’s in it for them
Sometimes all you can offer is that their feedback will shape the future development of a feature which will eventually provide value to them
You might be able to offer them a free subscription to that feature for some period of time
You might have some marketing swag laying around that marketing will let you give away. We used some really nice Yeti mugs with our company logo once and those were very popular
If you have a little bit of budget, a gift card or charitable donation in their name is always nice
A link to your final test survey with a call to action to fill it out
A deadline by which you would like them to fill out the survey - I like to give them a couple of weeks for this if the test will take them a little bit of time
Consider setting up a reminder email to go out just before the deadline because people get busy and forget to walk through the test.
Step 8 - Act on feedback
Review your interview notes and survey answers for common themes. Create work items/tasks/tickets/product backlog items for your development team to address the most common areas of feedback. Make sure to use your product skills to prioritize the feedback accordingly. If needed, create a new roadmap initiative to add missing functionality identified by your beta users.
Step 9 - Make sure your testers feel the love
If you promised an incentive for participation, make sure you follow through on providing it.
If you implemented anything based on the tester’s feedback, it’s a great idea to reach out to them and let them know. You can even ask them to try it out and let you know if it met their expectations.
