Device Matching for Usability Testing

Usability testing of mobile web sites and applications has extra complexities over testing on full-sized computers. Chief among these is neglecting to test on a device with which the user is accustomed.

If the participant is accustomed to using a Motorola RAZR (Openwave browser), the iPhone or even Opera Mini on the RAZR will have a learning curve. If a Sprint UI user is asked to use a Series 60 device, all their learned behaviors will differ.

A person accustomed to the Opera Mini rendering engine will be less able to predict behavior of the iPhone browser. A person accustomed to an iPhone will feel crippled with most mass-market phone interfaces.

Is this any way to understand user behavior with your web site? Or worse, your application?

In the desktop world, browser controls are fairly consistent across operating systems and browsers. A Windows Internet Explorer user will be startled but have a shallow learning curve moving to Safari on a Mac. In contrast, a user with heavy use of a Back softkey will do poorly if testing with a device with a hardware back button.

The images above show a web site rendered on three different devices or emulators. The rendering and browser controls differ wildly, as do the native user interfaces.

Our practice is to recruit participants who use one of a target set of devices. For the usability test, the participant uses a matching device. In this way, the participant evaluates our client's site or application, not the device being used.

One counter-practice is to use the "best available" device out there. This isn't a great solution. First, you'll not understand the true context of your application. Second, just because the device is "best" (which is arguable) does not mean it is familiar. That Nokia Series 40 user may not understand the iPhone. Finally, there are so many rendering differences between devices that a great application on one device could conceivably be utterly unusable on another. Capturing an array of devices somewhat guards against that problem.

Whenever reviewing the results of a usability test, be sure to consider whether issues were introduced by the tested application, the device the application was on, or the interaction between the two. We do everything we can to reduce device-created issues.