Usability Testing for Mobile Devices

This is the first of a three-part series on mobile usability testing. Part one is for devices and software native to devices. Part two is for web sites and applications that might go on multiple devices. Part three will talk about adding testing with physical and social context.

First, a definition: usability testing is a method for evaluating the ease of use, ease of learning, satisfaction, frustration, and potentially a few other measures of a device, software, web site, or other system. A typical test has one participant at a time, each of whom is asked to perform a series of tasks with a given system.

The "few other measures" can get into desire, delight, mood, and more, even though these are not measured by task times.

Perceptive Sciences performed a usability test comparing iPhone, Nokia N95, and the HTC Touch; this sort of work can give us a set of places to work to improve any design.

Testing the ease of use of a mobile phone or other personal communications device is not terribly difficult. A former colleague of mine over at Virginia Tech put together a systemic methodology for measuring mobile phone interfaces; such a system means that you can compare the results of one device measured by one team to another device measured by another team halfway around the world. We can also establish benchmarks to compare devices.

We use a device sled, tricked out with cameras to get two device shots (to cover multiple surfaces) and one face shot. The device just sits on top; rubber prevents it from sliding off. This, combined with a wide angle shot, gets fed into a quad processor and saved to disk. You can rig this up yourself if you have a bit of videographic know-how. And you can use it in a lab, in a coffee shop, or in a mall.

And yes, you do need to test on actual devices. The input mechanism will be different (mouse clicks are different than button presses, stylus taps, and multi-touch interaction), the response speed will be different, and the visual and tactile experience will be different. Remote testing tools won't work. If, in some future, they do work, be careful that your tool does not slow down the device.

With the proper set-up, you can use all your standard usability testing techniques, from setup to analysis.

Next up: usability test of web sites or downloadable software, and handling multiple devices.

The sled you have there is pretty big. One of the things I’ve noticed when doing user research for mobile devices is that when the mounting bracket / sled is too large or heavy, users will start to hold the device in two hands or use it differently in some way.

If its the kind of device people normally hold and operate with one hand alone, then that’s what you want users to do. Some people use a cap or spectacle mounted camera instead of a sled to avoid this problem.

Another problem with benchmarking studies is that there are always shortcuts for doing things, designed to be labour saving for regular users. If you have a die hard nokia user come in and test a nokia handset, they will know the shortcuts and completely nail the task time. This means you have to record the user’s handset preferences and account for this in your data analysis.

We’ve tested it out a bit, and it is well balanced. People take a wide variety of postures with it, from sitting on the table to slouched in the chair.

Are there step-by-step instructions for making a sled, including rigging up the video?

Xander: the video part is the hardest, as it requires swapping out lenses to get a good close-up shot, and finding a quad-processor that companies are no longer interested in making. We don’t have a written set of instructions.