Device Selection for Usability Testing

How often do you sit down at a computer with an unfamiliar operating system with an unfamiliar browser, and immediately attempt to use an unfamiliar web site or application?

You did that once? Or when you have a new operating system do you instead explore around a little bit?

Usability testing of desktop web sites generally use Windows computers and a major browser. If testing an application, the participants are screened to have the proper operating system experience.

So why is it that I've seen so often a complete avoidance of this issue in the mobile space? The first two issues are simple ... you don't really have to worry about the problem in the desktop space (in other domains you generally provide the hardware as well as the software), and mass-market phones aren't really seen as having an operating system.

I've seen four basic approaches to handling devices when testing mobile applications or mobile web sites:

1. Ignore it. Choose a common phone and have everybody use it during the test.
2. Make everybody a novice. Require that participants have NOT used the device being tested.
3. Use the user's own device. Have them download and run the application or web site.
4. Match devices. Select three or so devices that represent a large portion of your user population. See what other devices in your user population have a user interface largely identical to one of the selected three. Select participants who use any of the above devices. Test the application on a device that matches the participant's device.

Oh - with the increasing popularity of add-on browsers like Opera Mini, be sure your test devices also have the other browser for a mobile web usability test. Let the user choose.

The first option, "ignore it", is not only not realistic, it will wreak havoc with your data. Some of your users will have an experience that matches their comfort levels, the others won't. At a minimum, note which ones are which and adjust your statistics accordingly.

The second option also has its challenges. How many phones has the user used before this one? Did they have a matching OS? Are you testing the OS, or just the application? Will users perform the task you are testing with a brand new device?

The third option, using the user's device, runs into problems with data plans and device management. Does the user have a data plan? Which one? What about a theme? Perhaps a theme that affects the performance of the application? Maybe there is insufficient data available? Maybe the user already has your application or a previous version? How's the battery level?

For most situations, I recommend the last option, match devices. This provides a fair amount of control over the test environment, while allowing you to test your application and not the device. You still get to know where the device and your application don't get along together. You keep costs and recruiting complexities down. It improves the quality of your results.