chapter one

In recent years, we have seen the audio-visual field of computer science mushroom. Sensory I/O hardware has advanced exponentially, to the point where aspects of audio and video quality have surpassed the capabilities of the human eye and ear, with sound-cards capable of outputting frequencies higher than the Nyquist limit and graphics-cards, frame rates higher than the 72 Hz (the limit of the human eye).

   As processor manufacturers try to cram more functionality on a single chip, questions of audio-video integration and interaction are being raised. Visual sub-systems, like OpenGL [42], are being complemented with audio counterparts, such as OpenAL [25]. MIVI, which stands for Musical Instrument Visual Interface, represents a similar complement, this time from audio to visual – MIDI to OpenGL.

   This section will introduce the concept of MIVI, its intended uses and familiarise the reader with the structure of this report.

1.1 concept

For two decades, the MIDI (Musical Instrument Digital Interface) specification has set the industry standard for the electronic encapsulation of music. Instead of being a standard for audio waveforms or data streams, like MPEG, MIDI encodes music on a semantic level – essentially an electronic score format – and is used to send musical notes and performance instructions to devices such as synthesizers, sequencers, mixers, etc., for their storage, processing or auralisation, etc. In many senses, it forms a communication protocol, for musical devices.

fig.1.1 - typical
dataflow of MIDI in a
musical environment




   In this project, we envisage and implement a single application that will produce dynamically generated and visually accurate 3D musical instruments that respond to a MIDI input in real-time. That is; a program, which permits you to select a piece of pre-recorded music and simulates the visual and physical performance of one or more of the instruments involved, as the music is played back. In figure 1.1, we illustrate some of the typical applications of MIDI, extending the diagram to encompass this new visualisation functionality.

fig.1.2 - screenshot of
the MIVI application



   Figure 1.2 shows a screenshot of the MIVI application running a piano instrument model. In it, our application is running as the plugin to a host application – Steinberg Cubase VST. Cubase is the industry-standard music composition tool for the modern professional studio and its competences, as such, extend to recording, processing, arranging and even auralising both MIDI and waveform audio. The photo, in figure 1.3, shows the program running on the development system, and directly corresponds to the environment portrayed in figure 1.1. The user, through the attached MIDI keyboard, provides the MIDI Input. Cubase then sends copies of the input to the tone generators (below left monitor), for auralisation, and to MIVI (right monitor), for visualisation. Additionally, the computer can stream, to both these outputs, the pre-recorded notes in the open MIDI file (stored on the computer), which it simultaneously displays in score format (left monitor) throughout playback.

fig.1.3 – the MIVI
system in action




  1.2 application

It is a readily conceded fact that education is of paramount importance to society. Music education, though, is rarely given any amount of priority in today’s classroom. In direct competition with language, science, mathematics and social skills teaching, the value of a musical education is not immediately obvious.

   In retort, many argue that culture – of which music can be crucial factor – is also important to society. Music, however, hones skills beneficial to all of these disciplines. For example, literacy in music is an exercise in mathematics and languages. Membership of any ensemble is an exercise in social skills, discipline and being able to express oneself. These and other reasons have led numerous musicians [44] and sociologists [35] to conclude that music is of prime significance.


"Obstacles usually cited are (1) the lack of funds, (2) lack of instructors, (3) lack of space and [instruments], (4) time conflicts, (5) tacit resistance from private teachers, (6) concentration needed on other instruments and voice, and (7) teachers inadequately prepared. The fundamental reason, however, is the lack of precedent, stemming from the fact that the band and orchestra scores seldom call for the [instrument]."


   In the above quote, House [19] identifies eight obstacles facing the aspirant musician. As an aid to tuition, MIVI sets out to help tackle these problems; (1) by being easily accessible, (2) by not requiring instructor presence, (3) by being virtual and easily distributable, (4) by not restricting tuition availability, (5) by factoring out the human element, (6) by servicing all instruments, and (7) by providing dependable standards of tuition. Furthermore, as we shall see, House’s final complaint is, in part, addressed in MIVI by the default provision of appropriate backing ensembles.

   We will exploit the user input / visual output nature of MIDI and MIVI in such a way that will afford us an interactive tutor system. Simply; presented with music, the user will play it and MIVI should be able to give both guidance and feedback during the performance.

   It should be noted however, that as far as we set out to address these problems, in no way do we present the concept as a replacement for the traditional music teacher. As with other computer-assisted instruction (CAI) software [47], the MIVI application is intended as a visual aid to learning.

  1.3 about the report

This report discusses, in detail, the provisions and concepts of, and surrounding, the MIVI concept, proceeding to document the implementation of a core system, with basic functionality.

   The structures of each chapter and section are presented at the outset of each respective segment throughout the report. However, the overall layout, as implicit in the contents page, is as follows:

   We will begin by reviewing relevant literature and research available on the subject, then documenting both the design and implementation of a software prototype. In these sections, we highlight particular hurdles and problems that were overcome and, if so, how, while occasionally discussing implementational considerations of further programming and extension to our prototype.

   Finally, we will evaluate the project with respect to its objectives, using feedback from musicians, teachers and other experts, and, in conclusion, identify future work prompted by our research.

   The reader should find the source code of our implementation, given in Appendix C, useful. The code has been rigorously commented and indexed with references corresponding to explanatory paragraphs in the report. Such references are given at appropriate points throughout the text.


All content, including code and media, (c) Copyright 2002 Chris Nash.