Speaking hands
A new score notation for experimental musical theatre by video-controlled gesture recognition and an innovative sign language system to facilitate new mouvement compositions.
Factsheet
- Lead school Bern Academy of the Arts
- Additional schools School of Engineering and Computer Science
- Institute(s) Institute Interpretation
- Research unit(s) Intersection of Contemporary Music
- Funding organisation BFH
- Duration 01.01.2009 - 30.06.2010
- Project management Angela Elisabeth Koerfer
- Head of project Angela Elisabeth Koerfer
-
Project staff
Franziska Baumann
Christian Baumberger
Claudia Brieske
David Burri
Jérôme Heimann
Werner Jenni
Fabienne Meyer
Dr. Andreas Leopold Dick - Keywords gesture recognition, musical performance, sign language notation system, visualiszation
Situation
Creation of an interdisciplinary plateforme of research about gesture and some specifique aspects of the visualization of the hand gestures to open the way to new research possibilies of the gesture notation.
Course of action
A team of composers of the HKB delivered elementary gestures to the informaticien team of the TI Burgdorf. They digitalized the gestures by Hidden Markov Model and worked on gesture recognition. A graphic team worked on visualization by a new typographie of sign language and motion history extraction. In the final work etape the composer-team integrated specific visual data in a new interdisciplinary score.
Result
The team of TI-Burgdorf reached 100% gesture recognition by HMM. The composer’s team created a score, where complex parameters are visualized by new notation and combined with pictograms of more simple mouvements.
Looking ahead
A new interface for the transformation of HMM-data of video-based gesture-recognition to a Midi-Database would open new research ways of artistic and psycholinguistic aspects of gesture in applicated projects (i.e.sign language learning).
