External Beta Testing - Why and How to do it

Every business wants to build high quality products that customers love and truly fulfill their needs, regardless of whether they are building a physical product such as a chair, a hardware product or software product. To build a successful product, you need to put the product in the hands of different users early in the development process, and iterate over it to make it great. When the product is a mobile app, we as developers are fortunate to have tools that help us do this iteration at a rapid pace, and one essential step in this process is external beta testing.

As an app developer, it is easy to believe that our own ideas are always the right ones, and that what we are building is exactly what our customers want and need. Let’s face it: even perfectly clear requirements rarely capture the full scope of the real-world problem, and nobody wants to invest time in something people will not use. Putting the app in the hands of real users as early as possible in the development process is extremely valuable and it is the only way we can find out whether our approach delivers on what users need, and how we can improve our app to fulfill those needs.

We know that in the mobile world, administrative tasks like sending a specific build to a set of beta testers can add overhead to our already busy day, so we’ve built HockeyApp to automate the mundane details and to take care of the distribution of versions to beta testers. We recommend beta testing in a multitude of scenarios - regardless of whether it is your first build that ever sees the light, the version that you’re planning to publish to the store, or a new feature you have just added to your already successful app. Beta testing all builds with significant changes ensures you are getting constant feedback, and that you can quickly address issues before they turn into debt.

HockeyApp provides team management features and can restrict access to various versions of your app to specific groups or individuals, enabling you to create ringed or canary like releases where you slowly make the latest version of the app available to more of your users. HockeyApp makes beta test management simple, allowing you to focus on what matters most: build a successful app.

You can distribute an app to beta testers via HockeyApp in a few easy steps, with no SDK integration needed. HockeyApp has a free plan which gives the owner two free apps with unlimited number of versions with no other limits on crashes, feedback or user metrics.

  • Upload your build to HockeyApp. We support all major platforms: Android, iOS, Mac OS X, and Windows.

  • Distribute your build to testers by inviting them to your app or by simply sharing the link to the app’s public download page (you can read here about how the process works on different platforms)

  • Restrict the build to a subset of testers or devices, if you want

With these easy steps and no SDK integration, you can quickly distribute a new build to your testers and notify them via email (through HockeyApp). From the email, the tester can directly

install the app on their mobile device. In addition, HockeyApp provides in-app updates through which users can update to the newest version from the app itself. This feature is enabled by adding the HockeySDK to your app. As part of the Mobile DevOps process, we also recommend to automate the distribution process by using a continuous integration server or service, which enables deploying a build with each code commit and successful build.

A common process for quick iterations is the build-measure-learn cycle: build a product (code), put it in the hands of users (beta testing), measure the results by monitoring crash reports, usage data, and user feedback, and learn what needs to be changed to truly fulfil the users’ needs. Then repeat. Beta testing connects the build and measure stages, which then raise question like this:

  • Is the current version shippable?

  • Are there critical bugs that I should fix as soon as possible?

  • How are users interacting with my app?

  • What is the user feedback?

Integrating our open-source HockeySDK helps you answering these additional questions and you can use the insights gathered to learn what to do next:

  • With the in-app feedback feature, beta testers can easily provide feedback to the developers straight from the app, include screenshots, and allow you, the developer, to engage in a conversation with these users, making sure that you truly understand their needs.

  • When your app crashes, HockeySDK will collect the crash report and send it to HockeyApp. The crash report will be symbolicated to provide class, method, and file names as well as line number. You will also see how often the app is crashing and how many users are affected by a crash for each version.

  • The User Metrics feature helps you measure how many new users the app is gaining and how many users and session the app has on a daily and monthly basis. With Custom Events, you can instrument your app further to see exactly how users interact with it; for instance, when they perform a particular action in your application, when they click a button, or what is their preferred login method.

These features are useful during beta testing, but they will continue to work in production apps and we highly recommend to keep them throughout the lifecycle of your app. Understanding real world user behavior, and how stable your application is on devices in different connectivity states, configurations, and usage patterns can be the difference between a good app that goes bad, and a good app that becomes great.

External beta testers are an important stakeholder and help you and your team build a successful product. Make sure that you involve them early and often in the process, as well as before shipping bug fixes or new features to your already successful app. They have a fresh perspective on your product and will tell you the truth, most of the time. Their honest feedback might hurt sometimes. To make sure that their feedback won’t break your heart, make sure to ship updates to them as soon as possible. They will not hesitate to tell you when what you built is not what they want or when it is, and the sooner you get to measure their reactions, the sooner you will learn what to do next.

We’re looking to grow HockeyApp to meet as many mobile developers’ needs as we can, so connect with us on our public Slack channel HockeyCamp or on Twitter to let us know how you do external beta testing. We’ll do our best to make your life easier.