Speaking hands

A new score notation for experimental musical theatre by video-controlled gesture recognition and an innovative sign language system to facilitate new mouvement compositions.



Creation of an interdisciplinary plateforme of research about gesture and some specifique aspects of the visualization of the hand gestures to open the way to new research possibilies of the gesture notation.

Course of action

A team of composers of the HKB delivered elementary gestures to the informaticien team of the TI Burgdorf. They digitalized the gestures by Hidden Markov Model and worked on gesture recognition. A graphic team worked on visualization by a new typographie of sign language and motion history extraction. In the final work etape the composer-team integrated specific visual data in a new interdisciplinary score.


The team of TI-Burgdorf reached 100% gesture recognition by HMM. The composer’s team created a score, where complex parameters are visualized by new notation and combined with pictograms of more simple mouvements.

Looking ahead

A new interface for the transformation of HMM-data of video-based gesture-recognition to a Midi-Database would open new research ways of artistic and psycholinguistic aspects of gesture in applicated projects (i.e.sign language learning).