usability testing of multiple-device software and sites

Tags: Design,  Usability testing
July 21, 2008 by Barbara

This is the second of a three-part series on mobile usability testing. Part one is for devices and software native to devices. Part two is for web sites and applications that might go on multiple devices. Part three will talk about adding testing with physical and social context.

In most of our work, we are designing applications or web sites that need to run on multiple devices. When we have control or even just the ability to predict what screen size, rendering engine, input mechanism, user interface paradigm, device interaction, and more is present, we consider that relatively easy. Complexity in the design process quickly arises from managing device diversity.

Developers deal with device diversity problems as well, with coding, porting, and testing. Best practice I’ve seen is to test in a simulator, check actual behavior on a small set of live devices, and then do thorough testing and debugging with a service like Device Anywhere. I’m hopeful that Mob4Hire will also do well.

None of this is relevant for usability testing.

If you are testing a prototype, perhaps focusing on information architecture, then you can do usability testing on simulators and get pretty good results. Most usability tests I’ve seen, however, are done on complete or near-complete products. You simply will not get reliable results from an emulator. User behavior changes, understanding of information changes, perception changes, delay times change, input mechanisms are mismatched, and rendering engines usually are slightly different.

Yes, we talk about this a lot. You can check out some slides, or of course my book, Designing the Mobile User Experience.

We have a few main recommendations for handling usability testing of cross-device applications:

  1. Test on actual devices for final product testing. If you are just testing information architecture or some other iterative question, you can use the emulators.

  2. Acquire ten or so different models, based on popularity in your market and rendering and UI differences

  3. Recruit users using a device the same as or similar to your devices

  4. Perform usability tests with these devices.

  5. Match the device’s user interface to the participant’s personal device’s user interface.

  6. Use the practices for all mobile usability testing, as explained in part one.

Avoid the tempting shortcuts.

  1. Use actual devices, not emulators, for final product testing. You’ll find different results, results not relevant to your actual product. You won’t find that a target size is too small on a computer, you won’t find that the page is simply too heavy to be sufficiently responsive. Go ahead and use emulators for iterative testing, as long as you have similar input mechanisms and device+network responsiveness.

  2. Use your devices, not the participants’. You have no idea what customizations, downloads, extra bugs, full memory, start state, or more might be on the device to affect your data. Besides, you don’t want to do anything to inadvertently cost them more money.

  3. Use an array of devices, not just high end. Unless your entire target market uses the iPhone or N95 or Blackberry, you’ll not get good data. I’ve seen too many “best case” usability studies. If you want a marketing study, be honest about it.

  4. Have participants show you their device. They may tell you they have an iPhone when they actually have an Instinct; they may say they have a Sprint device when they actually have a Sanyo device. It can be difficult to know what device you have.


«« free advice to keyboard inventors  |  greatness and danger in police car computers »»

No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

*
To prove you're a person (not a spam script), type the security word shown in the picture.
Anti-Spam Image