Gesture
People
John Lee |
How are ideas expressed for self and for others in gesture? How do those gestures map onto diagrammatic and verbal elements and strings? What aspects of ideas are commonly conveyed in gesture? What do listeners gain from gestures? These are some of the issues we are exploring. We have found that explaining how to put something together using gesture and prohibiting speech benefits both explainers and listeners more than speech and gesture or just speech (Lozano and Tversky, 2004). What makes gestures especially effective for this task is that they convey action. When people are allowed to speak as well as gesture, they use gesture to convey structure and language to convey action. Interestingly, portraying action is what makes for good assembly diagrams (Heiser, Phan, Agrawala, Tversky, and Hanrahan, 2004).
Although it is typically thought that gestures accompany speech, gestures often accompany listening (Heiser, Tversky, MacLeod, Carletta, and Lee, in preparation) and non-communicative thinking (Kessell, 2004). In both cases, they seem to serve to augment spatial working memory, much as sketching a diagram would. People hearing a route trace it on a map as they listen. In explaining problem solutions, gestures serve both to convey the representation of the problem and the solution to the problem. The hand often shifts role and meaning in a seamless stream of gestures, the changes paralleled verbally. For example, in the two-rope problem, the hand may first represent an object, a hammer (that gets tied to one rope), then demonstrate an action, setting the rope swinging, and then the path of the rope.
In collaboration with diagrams, dyads save speech by pointing and tracing on the diagram. Partners look at the diagrams and their hands, not at each other (Heiser, Tversky, and Silverman, 2004). Having a shared diagram to gesture on facilitates establishing common ground and finding a solution. It also augments solution accuracy.