Do Gestural Interfaces Promote Thinking? Embodied Interaction

de Ayelet Segal

Can action support cognition? Can direct touch support performance? Embodied interaction involving digital devices is based on the theory of grounded cognition. Embodied interaction with gestural interfaces involves more of our senses than traditional (mouse-based) interfaces, and in particular includes direct touch and physical movement, which are believed to help retain the knowledge that is being acquired. There is growing evidence that spontaneous gestures affect thought and possibly learning. The author was interested to explore whether designed gestures (for gestural interfaces) affect thought. It was hypothesized that the use of congruent gestures helps construct better mental representations and mental operations to solve problems (Gestural Conceptual Mapping). There is also evidence that physical manipulation of objects can benefit cognition and learning; it was therefore also hypothesized that manipulating objects through direct touch on the screen supports performance. These hypotheses were addressed by observing children's performance in arithmetic and numerical estimation. Arithmetic is a discrete task, and should be supported by discrete rather than continuous actions. Estimation is a continuous task, and should be supported by continuous rather than discrete actions. Children used either a gestural interface (multi-touch, e. g., iPad) or a traditional mouse interface. The actions either mapped congruently to the cognition (continuous action for estimation and discrete action for arithmetic), or not. If action supports cognition, children who use continuous actions for estimation or discrete actions for addition should perform better than children for whom the action-cognition mapping is less congruent. In addition, if manipulating the objects by touching them directly on the screen could yield a better performance, children who use a touch interface should perform better than children who use a mouse interface. The results confirmed the predictions.