Making music with images: interactive audiovisual performance systems for the deaf
-
Michael S. Grierson
Abstract
This paper describes the technical and aesthetic approach utilised for the development of an interactive audiovisual performance system designed specifically for use by children with multiple learning difficulties, including deafness and autism. Sound is transformed in real-time through the implementation of a Fast Fourier Transform and translated into a moving image. This image is adapted so that relevant information can be understood and manipulated visually in real-time. Finally, the image is turned back into sound with only minimal delay. The translation process is based on research in computer music, neuroscience, perception and abstract film studies. The system is achieving a significant user base amongst the deaf and disabled community through projects involving the British Broadcasting Corporation and the London Philharmonic Orchestra. Alongside continuing collaboration with the Sonic Arts Network, the system has now been made freely available to the public, and is being developed as a speech therapy tool.
©2011 by Walter de Gruyter Berlin New York
Articles in the same Issue
- Editorial
- Disability, virtual reality, ArtAbilitation and music
- Reviews
- Customising games for non-formal rehabilitation
- Aphasic theatre or theatre boosting self-esteem
- Warriors’ Journey: a path to healing through narrative exploration
- CaDaReMi. An educational interactive music game
- Extending body and imagination: moving to move
- Original Articles
- Making music with images: interactive audiovisual performance systems for the deaf
- An infrared sound and music controller for users with specific needs
- Sound=Space Opera: choreographing life within an interactive musical environment
- Cognitive effects of video games on old people
- Providing disabled persons in developing countries access to computer games through a novel gaming input device
- Voice articulatory training with a talking robot for the auditory impaired
- Using augmented reality to support the understanding of three-dimensional concepts by blind people
- Augmented reality application for the navigation of people who are blind
- Case Report
- Unintentional intrusive participation in multimedia interactive environments
- Listening to complexity: blind people’s learning about gas particles through a sonified model
Articles in the same Issue
- Editorial
- Disability, virtual reality, ArtAbilitation and music
- Reviews
- Customising games for non-formal rehabilitation
- Aphasic theatre or theatre boosting self-esteem
- Warriors’ Journey: a path to healing through narrative exploration
- CaDaReMi. An educational interactive music game
- Extending body and imagination: moving to move
- Original Articles
- Making music with images: interactive audiovisual performance systems for the deaf
- An infrared sound and music controller for users with specific needs
- Sound=Space Opera: choreographing life within an interactive musical environment
- Cognitive effects of video games on old people
- Providing disabled persons in developing countries access to computer games through a novel gaming input device
- Voice articulatory training with a talking robot for the auditory impaired
- Using augmented reality to support the understanding of three-dimensional concepts by blind people
- Augmented reality application for the navigation of people who are blind
- Case Report
- Unintentional intrusive participation in multimedia interactive environments
- Listening to complexity: blind people’s learning about gas particles through a sonified model
