Humans Are Tool Users

Since I wanted to show off some tools, I did an experimental and sorta terrible (i.e. you can't really see any of the stuff I am holding up) video version of this. The video is essentially exactly what I say below (though not word for word in many places), so take your pick which one you use.

Click the image to play the clip (4m 36s, .MOV)

I have been thinking a lot about touch lately, especially for mobile phone use – some of it paid and secret – and the more I think about it the more I think there are two standard beliefs that I sort of disagree with.

First, I don't think we have touch screens yet. I think we just have "point screens." That means these are not too far from the mouse, and are almost exactly the same as the digitizing tablet I use every day. I have a "touchscreen" laptop as well, and there's an N800 on the desk in front of me.

Pointing, like with a mouse or digitizing pen, is just pointing. There's no functional difference between poking at the air and sliding your finger across a smooth glass screen. Touch on the other hand gives you some sort of feedback. You can feel the presence of a button, and when you press it get a response that you've actually moved it.

Which brings me to the second point. Fingers are pretty good at pressing buttons, but I am not sure they are much good at anything else. But people are tool users; we don't just point, or point more precisely with stylii and so on. We use tools to accomplish tasks. And not tools in the sense of buttons, or mobile phones. I mean we can grasp things to manipulate other things.

Your basic toolbox is a good example, but I went and grabbed by old art supply boxes, because yes I am partly talking about the fascination with the ability to draw on your iPhone. I have a degree in printmaking, so have an unusually deep set of specialized tools hanging around.

The pencil is less obvious than it seems (much less, if you are a Henry Petroski fan. It's not just a pointer, but a mark-maker. Each one has different characteristics, and interfaces with the paper in a way that makes it possible to draw, or draw straight lines.

Similar things happen with technical pens, charcoal and paint brushes, and more so the engravers I used in a certain kind of printing plate making. The interface between the tool and the surface guides it. For pencils and gravers, this helps you make more ordered lines.

I had forgotten about some of the tools I used. Burnishers, to polish or flatten items – several of them for different surfaces and end effects. Even a feather on a stick, to spread acid on plates for a certain kind of etching (called "spit bite").

In all cases, There is feedback, not just from the direct interface of you grasping the tool, but of the tool changing or running over the surface. Even digitizing tablets do not do this well, though they tend to have texture to get halfway there. The pen tips are replaceable like pencil leads for this exact reason.

Now I am not saying we have to stop being exciting with, say, the ability to draw on your iPhone. I just want to be very, very clear about terminology or even the assumption (all too strongly stated on fanboi forums) that we've reached the nirvana of interfaces. Presuming that any one solution is not just the best today, but the best that could ever be will tend to stifle creativity and development of wholly new ideas.

Things like calling pointing on glass "touch" when there is no feedback I think lead to confusion over expectations and future terminology. Now we'll have to explain what "haptics" means, and since that will probably launch in some half-assed manner, we'll have to come up with yet another term for "really good haptics" later on still.

That's the future I hope for in the relatively near term, haptic feedback will let you simulate the real world environment of things to manipulate on your flat screen. After that... I have no specific idea. Lots of things are possible. And that's a key point. We have to constantly remember to keep our minds open to what could be The Next Big Thing, and try to understand what really would be a natural UI.

For any design problem, never assume the first answer that meets a bunch of your goals is the one final solution. Look at how users actually employ their devices, and in this case consider that tool using is about grasping items, precision and specific solutions for specific tasks.

I'm excited for the future, and look forward to the next great new idea.

Great points. My suggestion for simpler and clearer terms would be “touch input” and “touch output” (rather than touch and haptics). Right now we mostly just have touch input.

The output part is really important, but don’t forget about other modes of feedback. Even though visual feedback without touch feedback is still not exactly natural, I think people adapt to it well and it goes a long way to having the same benefit.

I agree with you also that fingertip input is only a tiny subset of what people can do with their hands, so calling these interfaces “natural” as if they’re the ultimate is a bit silly.

A side note that isn’t directly relevant but I think helps in clarifying differences between input devices: I found your use of “pointing” here a little confusing because you’re combining indirect pointing (or indirect touch) like with a mouse, tablet, or touchpad, and with direct pointing or direct touch on a touchscreen with a finger or pen. I’m not sure if this makes a difference for your argument, but I think it’s a useful distinction. It would be that people are better at doing indirect input without feedback than direct input without feedback.

Oops, typo in that last line. I meant “it could be”, not “it would be”.

Steven,

I completely agree with you that the one-direction communication we have currently in touch interfaces is lacking, errr, the other direction.

“Which brings me to the second point. Fingers are pretty good at pressing buttons, but I am not sure they are much good at anything else.”

'Buttons’ are a recent invention, which I have had trouble tracking down the history. But it can’t be more than some hundreds of years. Before then, hands were mostly used for much more tactile-complex tasks. If you do some reading on our tactile skills, you’ll find them very impressive. Human skin has much more resolution than any current technology can meet the requirements of.

There are a few haptic commercially available interfaces around, like the Phantom, which apparently do a decent enough job — but the challenge there is that a lot of force (ie electrical power) required to provide reasonable haptic feedback.

I suspect we’ll need the haptic version of 'the icon’ — an abstraction of haptics/tactile information which takes a lot less power + resolution.

Because touch screens offer visual and sometimes vibration-based feedback (as is the case with my current mobile phone), isn’t the quality of feedback already much richer than a button simple button press? I’ll use the act of typing on a keyboard to illustrate my point. When I touch type a word on the keyboard, the only way that I can verify for certain that I’ve entered the correct sequence of letters is by looking at my computer monitor. Unless I type very slowly, looking down at the keys themselves as I press them offers little insight into how accurately I can type.

But, with a touch screen, the basis of the interaction occurs when “touching” onscreen objects, such as icons. The underlying metaphor is that I am touching an icon or object in order to select it—and, in response the object lights up, or (in some cases) the phone vibrates. The basis of the interaction itself comes when I “touch” an object that appears at a set location, and the feedback that I receive oftentimes originates from that same location (e.g., in the form of the icon changing color). Why is this type of interaction unnatural? And, this whole system of interaction doesn’t work if I simply “point” at the screen—I have to actually touch it. With gesture recognition systems, input methods such as pointing can work—not so with my mobile’s touch screen, however.

As an interaction designer who got my start in cognitive psychology, I am weary of the talk “natural interaction”. There isn’t much about the act of talking on a mobile phone to begin with that makes it natural, is there? What is natural, anyway? As interaction designers, shouldn’t we be more worried about other qualities of interaction that might enhance the user’s experience? Simply arguing that some interaction is natural or not has little bearing on the quality of an experience.

That said, I liked this post—clearly, I found it thought provoking.

Quick comments, as that’s a lot and I get too verbose. Just bug me again if I skipped the point:

- Now that you mention it, I admit I glossed over remote inputs vs co-located input devices for my argument. I have several touchscreen devices (didn’t have the laptop there when I did the video, so didn’t use it) and don’t find myself using them in different ways. Parallax bugs me, so seems to negate some of the advantages now.

- Visual feedback will always be inherently slow and require too much attention; my future world has you using touch devices in the car, while walking, etc. If only for navigation and simple supporting tasks, touch feedback (good term) seems a lot more useful than visual-only feedback.

- I certainly am slightly mocking the NUI trend. I hope someday it becomes like the way we say “intuitive” in quotes. No GUI is intuitive to anyone.

- He’s stopped publishing it, but I think Bill DeRouchey’s History of the Button blog is still out there. Read it for a history of the, well, button. Indeed, only so old, but I think most of the very fine sensing fingers can do is one way; you cannot gouge, write, etc. with fingers directly…

- ...though I didn’t explicitly say it, the finger sensing is a good thing and means that haptic systems should let you feel interfaces. I also didn’t quite say it, but I want stylii (and maybe several of them; love how there are a variety of them for the Wacom tablet, like an airbrush shaped one) but robust finger support is also important.

- I haven’t gotten to use a haptic responding device for more than trying-it-out time, but it is clearly the way to go. Exciting versions of haptics should someday be coming, where you can feel stuff before you touch it even, and that will be great.

- I have personally never been able to use gesturing (in the sense of make a delete swoop with the mouse to delete things)... actually, my innate pinch-sense for Apple is the opposite of their programming). This makes me worry others are not getting it, and adoption rates before iPhone seem to indicate this. So, I tend to worry about some of those, without enough data to back it up specifically.

- I am not sure (though clearly it’s open to discussion) that there is that much different now between touch and point. I am not sure mouse pointing is that different from any other type. Or, my pen tablet detects pointing above the surface; you touch the surface to select a button or lay down a line (a nice system, really, for the current technology). Perhaps future haptics will use the acoustical system that lays the feelable-interface slightly above the surface, so you feel something that isn’t quite real. Not sure what that will mean just yet.

- Not sure I totally buy the difference between typing on a touchscreen or a remote keyboard. In both cases the finger occludes the input, so something else has to say what you pressed (or for word entry, very often it is more useful to focus on the result, whether the word is getting spelled right, instead of the individual keypress). So maybe again I am overly generalizing, but my impression is that typing on a computer, on a keyboard/keypad mobile phone and on a touch device are very similar: input below, watch above for results, glance as needed at the keypad (more for hapticless touch, but still).
I think I personally spend more time looking at input when I am using stylii and can see the direct results of the action, but I am not sure if this means anything in the long run.