Skip to Main content Skip to Navigation
Journal articles

Seeing, listening, drawing: interferences between sensorimotor modalities in the use of a tablet musical interface

Abstract : Audio, visual, and proprioceptive actions are involved when manipulating a graphic tablet musical interface. Previous works suggested a possible dominance of the visual over the auditory modality in this situation. The main goal of the present study is to examine the interferences between these modalities in visual, audio, and audiovisual target acquisition tasks. Experiments are based on a movement replication paradigm, where a subject controls a cursor on a screen or the pitch of a synthesized sound by changing the stylus position on a covered graphic tablet. The experiments consisted of the following tasks: (1) a target acquisition task that was aimed at a visual target (reaching a cue with the cursor displayed on a screen), an audio target (reaching a reference note by changing the pitch of the sound played in headsets), or an audiovisual target, and (2) the replication of the target acquisition movement in the opposite direction. In the return phase, visual and audio feedback were suppressed. Different gain factors perturbed the relationships among the stylus movements, visual cursor movements, and audio pitch movements. The deviations between acquisition and return movements were analyzed. The results showed that hand amplitudes varied in accordance with visual, audio, and audiovisual perturbed gains, showing a larger effect for the visual modality. This indicates that visual, audio, and audiovisual actions interfered with the motor modality and confirms the spatial representation of pitch reported in previous studies. In the audiovisual situation, vision dominated over audition, as the latter had no significant influence on motor movement. Consequently, visual feedback is helpful for musical targeting of pitch on a graphic tablet, at least during the learning phase of the instrument. This result is linked to the underlying spatial organization of pitch perception. Finally, this work brings a complementary approach to previous studies showing that audition may dominate over vision for other aspects of musical sound (e.g., timing, rhythm, and timbre).
Complete list of metadata

Cited literature [38 references]  Display  Hide  Download
Contributor : Christophe d'Alessandro Connect in order to contact the contributor
Submitted on : Tuesday, January 23, 2018 - 11:54:15 AM
Last modification on : Monday, December 13, 2021 - 9:16:26 AM
Long-term archiving on: : Tuesday, April 24, 2018 - 12:31:32 PM


Publication funded by an institution



Olivier Perrotin, Christophe d'Alessandro. Seeing, listening, drawing: interferences between sensorimotor modalities in the use of a tablet musical interface. ACM Transactions on Applied Perception, Association for Computing Machinery, 2016, 14 (2), pp.1 - 19. ⟨10.1145/2990501⟩. ⟨hal-01672241⟩



Les métriques sont temporairement indisponibles