background
  chapter two
 

In the last half-century, Music Technology has established itself as a highly respected line of research and study. Much work has already been done across this multi-disciplined area, ranging from electronically-inclined studies into digital signal processing (DSP), through physical studies into acoustics, to more mathematical and melodic studies such as composition through algorithms.

   It is important to ascertain the role computers are taking in today's music and from what springboard, technologically, MIVI's realisation must begin. This will entail a review of previous and current research and technologies in the field of computer music, with special emphasis on human-computer interaction (HCI), musically-driven graphics and computer music performance systems (CMPS).

   The fundamental basis for MIVI will be an interface for the interaction of man, machine and music, and therefore, in addition to an exploration of the technological, it will be worth quickly delving into the psychological, with reference to applications of technology in education.

   Finally, following reviews of all the relevant fields and technologies, we'll take a closer look at a similar project to MIVI, called DIVA [26], and see if the findings of the Finnish team who conducted it, are of use in our venture.

  2.1 MIDI

 

 

Since its conception in 1982, the MIDI protocol has played an important role in many music-related projects, sometimes even to the point of inspiring or prompting them. Indeed, the connection between it and this project goes beyond just the spelling. However, whereas MIVI may be founded on a MIDI base, the goal will require us to develop upon the specification and protocol. It will therefore be wise, in addition to reviewing the MIDI specification and General MIDI (GM) protocol, to consider some aspects that lie beyond their scopes, and the previous attempts to extend them.

     

2.1.1 introducing MIDI

 

 

 

This sub-section aims to furnish the reader with a basic understanding of MIDI. Contained herein is an explanation of all the concepts and terminology of the protocol, pre-requisite in the reading this report. However, for those desiring a deeper or less specialised overview of the subject, the author recommends the books, mentioned as items [30], [53] and [54] of the bibliography.

   MIDI stands for Musical Instrument Digital Interface and is the technical specification [31] for a language of music - an encoding of music at the semantic level. It contains no audio information or guidelines for auralisation, such as sounds and waveforms, but instead, like a score to a piano, specifies properties like pitch and duration for each note and characteristics like tempo for a whole piece.

practical uses

 

   In the common instance, it allows music to be recorded by a computer, or dedicated hardware, from one instrument, then output for performance by another. The process can be synchronous, by attaching the MIDI-out (output port) of the first instrument to the MIDI-in (input port) of the other, or asynchronous, by having the computer record the signals from the first device to a file, which it can store and play back through the second at any time. The reader is referred to figure 1.1, in the previous chapter.

SMF

 

   A performance can be encoded in this way and stored in an SMF (Standard Midi File), which can, in turn, be transmitted to a MIDI device. A piano, if MIDI-enabled, can then pose as its own pianist and play itself. Indeed, most modern electronic musical instruments now support the MIDI standard – for example, the Yamaha CS1x [55] keyboard or Korg Trinity-Rack1 tone generator.

multi-timbral

 

   MIDI performances can be augmented over time. You could record the right-hand of a piece of piano music, then rewind and record the left. The computer can then carry out the simultaneous playback of the two phrases. The number of total notes (pitches) a device can play simultaneously is denoted by its polyphony. If it is only one, the device is monophonic.

   The number of different sounds (timbres) a device can play at any on time is called its timbrality, where one timbre might be a violin and another a piano, etc. If it is more than one, the device is multi-timbral.

   Note that it is possible, and common, to have a multi-timbral, polyphonic MIDI instrument, composed of both monophonic and polyphonic voices – for example, a 16-part multi-timbral sound module with 128 voices might include monophonic flute and violin voices, as well as polyphonic piano and guitar voices.

sequencing

 

   An SMF can contain the encoding of multiple instruments, even a whole orchestra, regardless of what MIDI input device you use; you can play in a violin – or even drum – solo using your MIDI piano. It is the job of the computer, through a program called a sequencer, such as Steinberg Cubase VST2 or Cakewalk Pro Audio2, to keep time and co-ordinate the performance of every instrument – fulfilling the typical role of the conductor. The sequencer takes the performance (sequence), encoded in the SMF, as input and, when commanded, streams, as output, notes and performance instructions, as MIDI messages, to a MIDI device for immediate execution (often auralisation). An apt analogy to a sequencer is a cassette recorder – allowing both the playback and editing of music.

communication

 

   MIDI messages are very small packets of only a few bytes, which carry information on the note or performance instruction – MIDI event – that is to be played or executed. Each MIDI event is a packet of varying length comprising a single status byte, and zero or more data bytes. An example is the 3-byte Note On message, where the status byte (0x90 in hexadecimal) tells the device to start a note for the active instrument at a certain pitch and velocity (volume), as denoted by the accompanying two data bytes. By contrast, the Note Off message can then be used to terminate it. Alternatively, using a Note On message with null velocity will also stop the note.

General MIDI

 

   Most MIDI devices are capable of posing and performing as several instruments – ordinarily, up to 16 different instruments can be played at once, each receiving messages through their own channel. In 1991, a specification, called General MIDI (GM) [32], specified 128 different MIDI voices (the full list of these is given in Appendix A), which can be assigned to any of the channels. In addition, since percussion sounds do not vary in pitch, it is wasteful to have whole instruments dedicated to a single bass drum or snare drum, etc. The specification therefore describes a standard percussion kit, where note C in the lowest octave is a bass drum, D a snare and so on. For example, the first channel might be set to an Acoustic Grand Piano (voice #1) and the second, a Nylon-string Guitar (voice #25), but another (often the tenth, for percussion) to a Standard Drum Kit.

   An SMF is broken up into tracks, each having its own channel. The subtle difference between a track and a channel is that you can have two tracks with the same channel. Going back to our piano example, the left-hand could be on Track 1 and the right on Track 2, both being sent to voice #1 (piano) on Channel 1.

  This structure is illustrated in figure 2.1. The hierarchy has, as its root, the sequence, encoded in the SMF, which is broken up into tracks that have their own channel and contain all information about MIDI devices, etc. Each track contains all the information about the events from the opening to closing note of the piece, for its respective part.

system messages

 

   A sequencer is normally connected to more than one input or output device. Thus, in music, it is imperative that they cooperate and coordinate their activities with each other, like the sections of an orchestra. Three different types of messages exist to help the sequencer in this pursuit – System Common, System Real-Time and System Exclusive (Sysex).

fig 2.1 -
MIDI hierarchy

 


 

 

 

 

   The first two concern the timing of the piece. Whereas System Common messages give the absolute position of playback in a piece, System Real-Time issue the start and stop commands to control it, as well as transmit MIDI clock (tick) messages, which devices can use to synchronise with each other.

   The MIDI specification’s endurance, however, can largely be attributed to the third. Sysex messages allow for unrestricted byte-stream communication through a MIDI connection, between MIDI devices. Originally designed for the ‘bulk dumping’ of settings, or even waveform audio between devices, it has been brought to bear in the real-time environment and has been used by manufacturers to implement controls and functions that are necessary to fully exploit the functionality of new MIDI instruments, but not natively supported by the base MIDI command set. It is, therefore, not necessary to ever replace the specification, but extend it instead.

   As the name suggests, sysex formats are defined exclusively at the system level and, thus, the sysex messages for one device are not necessarily compatible with another. However, the emergence and monopolisation of commercial standards has brought some of these extensions into wider, sometimes universal, use. In the next section, we will briefly look at two such standards, with special attention to their extensions to the GM specification.

    

2.1.2 MIDI and GM extensions

 

 

 

Two of the manufacturing giants of the music industry, Roland and Yamaha, upon the widespread adoption of the MIDI protocol, recognised both the potential and inadequacy of the GM specification, and seized the opportunity to release their own extensions – General Sound (GS) [36] and Extended (or Expressive) General Midi (XG) [56], respectively.

   The extensions address exactly the same deficiencies of their predecessor and do so using almost exactly the same principles and methods. However, due to the competitive nature of the market, the implementations differ and, thus, most3 XG devices will not respond to GS commands and vice-versa. Therefore, for our purposes, reviewing one will yield as much insight into the other and, under the widely-held conviction4 that Yamaha’s offering is superior, we choose to cover the XG format. Before we review each of the improvements borne by the standard, though, it is important to establish the failing of General MIDI that both companies set out to address.

deficiencies in the
GM specification

 

   Music, as an expressive art, is an imperfect science – it involves and assigns value to nuances, quirks and irregularities – and demands the ability to step outside the norm. As a mature technology, MIDI was introduced at a time of relative simplicity in the computer – it had to be simple and efficient, too. MIDI is thus highly abstract and technical, and, as a medium for expression, crude and inflexible.

   A symphony orchestra conductor will be the first to notice that confining the number of different instruments to 128, as General MIDI does, is extremely crude. When you consider that Voice #41 is not only a violin, but also the MIDI 'ambassador' to all violins in the world, the deficiency is magnified. Properties such as the violin's size, maker and origin - all of which can have a profound effect on the timbre (character) of a note - are instantiated to one generic set of parameters. Imagine the reproduction of a string quartet, where the two violinists on occasion play the same tune. In the concert hall, though we can't distinguish which is which, we are still aware of two violins. On the average MIDI instrument, the waveforms are identical and, once superposed, would sound like just one – albeit either twice the volume or partially phased.

ensembles and
orchestral sections

 

   The proposed solution to this problem, in part, only tends to exacerbate the inflexibility of General MIDI. We notice that voice #51 and voice #52 are not single instruments, but string sections. Seemingly, this is to compensate for the loss in polyphony that would derive from emulating each violin individually. It is also provided as a quick and dirty solution to the problem of the superposed violins that such an endeavour might present.

   Interestingly, however, strings and brass are the only sections to benefit from any adaptation that would permit their usage in an ensemble context. Though, this is more a comment on the quality of the MIDI sounds available when the protocol was introduced. Although considerations of sound quality might deter classical musicians, who will afford themselves a real orchestra for any number of the reasons listed in this section, contemporary artists, particularly of the 80’s, are more tolerant of, and even praise, the synthetic sound of MIDI instruments.

   Nevertheless, although it would be fairly painless to design and implement MIVI as a multi-instrument application, there exists negligible practical advantage to displaying more than one instrument (ie. ensembles) at a time – as discussed in section 1.2, MIVI is principally for educational applications, and teaching multiple instruments simultaneously is merely a recipe for disaster. Neither will this report cover the implementation of either a string or a brass instrument. The reader, however, can assume that ensemble GM voices (#49-#52 and #62) would be reduced to their respective solo visual incarnations. On the other hand, it is conceivable that, in its maturity, and combined with other research projects [22][29], MIVI could one day be used as a conductor training tool.

   Thus, in general, one of the biggest criticisms of the specification was the lack of freedom of expression in terms of both instrument varieties and individual instrument usage.

improving upon
the GM soundset

 

   Instead of just increasing the number of available instruments to more than 128, Yamaha’s research department opted to make the voice list for XG multi-dimensional. So, for each instrument, there can be up to 128 sub-categories, drastically increasing the total number of available timbres to 16,384. An example structure of XG's Violin voices, taken from the Yamaha CS1x synthesizer [55], is illustrated in figure 2.2. It should be noted that only a small number of the 128 possible sub-categories actually differ in character from that of the original instrument and, furthermore, that any variation in timbre is the result of the original sound being put though an effects processor, as opposed to coming from a different source.

 

fig 2.2 - Yamaha’s XG
extension to GM

   
Voice 41 Bank 0 Basic Violin
  Bank 8 Slow Violin
  Bank 16 Bright Violin
  Bank 35 Octave higher
  Bank 36 Two Octaves higher

 

   This is clearly an improvement on the original specification, and manages to maintain legacy compatibility with General MIDI, but is still not ideal. Ideally, we'd want categories like 'Stradivarius Violin' and so forth. Sadly, their inclusion would be of limited use, since today's sound synthesis engines are not able to reproduce tones of sufficient realism, especially when applied to solo string instruments. Our exploration of MIDI extensions, thus requires a review of current synthesis techniques.

 

2.2 sound synthesis

 

 

 

From the principle that graphics and sound are intrinsically linked, often as products of the same device – hence, follows the visibility of audible objects and vice-versa – our exploration of the visual side of music will benefit from an analysis of the audible.

In our endeavour to recreate instruments, we must know how they work to produce sound. More to the point, to meaningfully teach these workings, it is imperative to understand how the instruments are manipulated to produce not only sound, but also music.

   Initially, this will involve a brief discussion of instruments in the real world. However, both instruments and sound generation techniques have already been extensively studied from an aural standpoint – a core component of the Music Technology field is the pursuit of more realistic and expressive technologically generated sounds and music – and thus, we shall cover, in more detail, previous translations of the art into science.

   As we shall see, much of these aural enterprises can give us guidance in our own endeavour.

    

2.2.1 instruments in the real world

 

 

 

Today, a large proportion of music listening is done with the absence of visual stimuli. This is made possible by the nature of sound – although involving the movement of objects, which are often visible, sound is the result of miniscule vibrations, undetectable to the human eye. The irrelevance of the visual, means that sound is definable from base physical principles. Music is reduced to simply the manipulation of these sound waves and we must thus consider the instrument from this aspect. It should be noted that this is a centuries-old area of study, and much more is known about it than is permissible or relevant in this report. The author recommends Rossing [39] as a truly remarkable book, yielding deep insight into the subject.

the physics of music

 

   A violin, for example, uses the friction of a bow to induce oscillations in a string. In this form alone, the induced sound waves haven’t the amplitude (in essence, volume) to be heard, but the resultant minor vibrations of the violin’s bridge permeate into the hollow body of the instrument, where the larger internal surface area gives rise to an amplified wave. This wave can then reflect off the interior of the body several times, before leaving through the f-holes of the violin. Their collision with the human eardrum causes physical displacements, which are translated into electrical impulses and sent to the brain. The brain then interprets the frequency and variations in amplitude that it receives, into the commonly recognisable violin sound.

   This process is similar to that in the other members of the string family. Furthermore, pianos and guitars, as ‘stringed’ instruments use much the same process – differing in only the initial induction of the source wave: a piano string is hit with a padded hammer, and a guitar plucked with a finger.

   Farther afield, even more diverse families employ degrees of the same process – both brass and woodwind instruments rely on the reflection of waves in a resonant corpus. This time, the body is an open-ended tube and the blowing of air induces the wave. Whereas for brass, this excites the resultant wave directly, for woodwind, it is used to excite vibrations in a reed that, in turn, produce the resultant wave.

   Each method contributes towards the unique timbre of sound produced by the instrument, but it should be obvious to the reader that the methods themselves are not so different within certain instrument groups.

    

2.2.2 sample-based synthesis

 

 

 

Most modern sound synthesis techniques are sample based – that is, for each MIDI voice, the notes have been digitally recorded from a performance in the real world. However, recording a quality sample for all notes is costly – a high quality sample for each piano key (88 in total) would fill the average instrument's allotted memory, leaving little or no room for the other 127 voices. Instead, one pitch is recorded and stored in memory, and by electronically varying the playing speed (or frequency) of the recorded pitch, the other notes can be simulated. Generally, however, as you get farther away from the original pitch, the electronic ‘transposition’ results in a noticeable loss of realism.

 

 

 

   Therefore, for a higher quality voice, a multi-sample is used. A piano voice, for example, might have the C note of each octave recorded in memory and use it to produce the rest of the octaves’ notes. The higher density of notes sampled, the better the quality the reproduction. In the extreme, some digital pianos do dedicate their entire memory to one or two piano voices, sacrificing variety for quality. Note, however, that they also sacrifice their adherence to the General MIDI protocol.

   The pinnacle example of sample-based synthesis came recently, in the form of Hans Zimmer5, one of the most popular composers in Hollywood. Speculating, to accumulate, he booked the London Philharmonic Orchestra for an extended private session, during which he proceeded to journey around the orchestra, instrument by instrument, section by section, and sample every note and phrase he (or they) could imagine. At the end of the session, he had stored gigabyte upon gigabyte of audio data, and has since managed to remove the need for all but the most trivial orchestral participation in his movie scores. It is also worth noting that the hardware used by Zimmer does not employ any accepted instrument / sample naming convention – such as GM, GS or XG – other than that set out by himself.

   In section 2.4.1, we compare MIVI to video tuition and note that video is not as flexible as the real instrument or a simulation thereof. Zimmer’s approach similarly suffers from this inflexibility – any musical phrase he does not have, he cannot synthesise without referring back to the orchestra.

   Sample-based synthesis, by itself, exploits no implicit similarities between instruments or their families. In addition to extending the voice list for General MIDI, GS and XG both tried to develop on the freedom of expression available to the instruments of the GM soundset, by using sound effects processing. Although not evident in the violin voice, in figure 2.2, these extensions do acknowledge the existence of such implicit relationships in other voices. In figure 2.3, which shows voice #25 – the Nylon-string Guitar – the readers should notice the Ukulele in bank 96.

 

fig 2.3 - Yamaha’s XG
extension to GM

   
Voice 25 Bank 0 Nylon-stringed Guitar
  Bank 16 Nylon-stringed Guitar 2
  Bank 25 Nylon-stringed Guitar 3
  Bank 43 Guitar Harmonics
  Bank 96 Ukulele


   The explanation is simple and inherent in the architecture of the XG system. Effects processes are simply manipulations of the sound at the waveform level. From the previous section, we noted the relative similarity of sound generation techniques. In this instance, the step from a Nylon-stringed Guitar to a Ukulele is trivial and their actual composition is very similar: they both have multiple strings, wooden bodies, etc. The only difference could be in the shape of the resonant chamber, the material of the string or type of wood. The generic XG effect process has modelled the consequence of these altered parameters and resulted in the abstraction of two instrument timbres to a common source waveform – an instrument has been added without adding a sample to memory.

   In the next section, we talk about the ultimate extension of this abstraction – the entirely synthetic production of sounds.

   

2.2.3 physical-modelling synthesis

 

 

 

Using the precise knowledge of the processes inherent to real instruments, as introduced in section 2.2.1 and detailed in Rossing [39], we can artificially fabricate the waveform by generating basic wave oscillations and simulating the appropriate reflections, refractions, amplifications and dampening, etc.

   Furthermore, once this is achieved for a violin, we can also adapt the algorithms for other string instruments relatively painlessly. It then also follows that the combination of more adaptation and further innovation would yield synthesis techniques for other instrument families and genres.

   This is not a new theory and has been the subject of considerable research and successful implementation already. One research project, in the late 90’s, was successful and mature enough to breach the academic boundary and enter the commercial market – Yamaha and Stanford University’s illustrious SONDIUS XG Virtual Acoustic Modelling system [57].

SONDIUS XG

 

   Contrary to what the name suggests, SONDIUS XG has no voice naming hierarchy as in its namesake, XG. Instead, there is a more implicit system of inheritance between voices, dictated by the modelling algorithms that they employ.

   Before this, the very acceptance of a convention, such as orchestral families, already recognised the similarity of a group of instruments. In both cases, the distinctions are based on both the method each family uses to produce sound, whether it be plucking, bowing, blowing or hitting, and other structural properties of the instrument – for example, its material, such as wood or brass.

   For physical-modelling techniques, like SONDIUS XG, these families translate conveniently to synthesis models. String instruments, like violins, violas and cellos, as well as most keyboard instruments, draw upon algorithms, which simulate the sound waves produced by the oscillations of a string upon plucking, bowing or hitting. Wind instruments, like oboes, clarinets and flutes, rely on the modelling of sound waves passing down a wood or metal tube. The SONDIUS XG system also suggests the possibility of partitions based on the driver and resonant body components, and further abstracts them from their physical models.


fig 2.4 - the SONDIUS
XG architecture
 

 

 

 

 

 

   Because most strings use a bow, the respective model is invariably generic: an algorithm coupling friction-based scraping, and string oscillation. The difference simply lies in the parameters – the coefficient of friction, the length of the string, etc. This implicit categorisation in the system’s naming of instruments as combinations of Drivers and Resonant Systems, is illustrated in figure 2.4. Note that one remarkable feature of this architecture is that any combination of driver and resonant system can be used, allowing for imaginative instruments such as breath-driven violins.

fig 2.5 - the string
family: (a) violin,
(b) viola (c) cello,
(d) double bass

 

 

 

 

 

 

   Furthermore, although it may be musical blasphemy to say that a cello is simply a big violin, visually, this is essentially the case (as can be gauged from figure 2.5). Thus, physical modelling can apply the same generalisation to the resonant body, simply altering the dimensions to enable the correct internal reflection and refraction of the sound waves.

abstracting instrument
composition

 

   This extrapolation is especially beneficial when we consider that the driver can change part-way through a piece of music. A violin, for example, can be bowed or plucked. Indeed, a cello can even be bowed, fingered and plucked simultaneously.

However, on the score, whereas all this requires is a comment above the stave saying arco or pizzicato (respectively), MIDI has no formal method of encoding this performance direction. To simulate it, you must assign the relevant music to a separate bowed violin instrument (voice #41) or plucked violin instrument (voice #46), invariably over two different MIDI tracks or channels. In MIVI, if we were to automate which instrument is displayed (or driver is used), based on the incoming MIDI data, this might create a problem. For this reason, amongst others (discussed later), control over MIVI’s instrument selection will rest with the user. Aside from a small burden to convenience, the impact is minimal, in the context of MIVI’s application6.

   As regards resonant bodies, the string family is perhaps too simple an example. By inspecting the brass family, we see that our categorisation of instruments should be more involved than simple segregation into families, and we are again encouraged towards the driver / body split. However, whereas forced air (or pneumatics) is the driving force, universal to almost all brass instruments, the resonant body and method for controlling the pitch can, this time, vary as well. For example, take the trombone with its slide as opposed to the trumpet with its valves.

 

2.3 performance

 

 

 

Returning to the problem of differentiating violins in our string quartet scenario, posed in section 2.1.2, the sound of an instrument in the real world is also varied by the manner in which it is played – the proficiency and technique of the performer, or lack thereof. Indeed, much of what tells us that something is real (or human) is, cynically, what is imperfect about it – the gasping breath sounds of a trumpeter, the pale scraping of a bow, sliding too lightly across the violin string, etc.

   Computer aided music is often condemned for its lack of expression, which, though one could blame on DJ's of today's music scene, is more likely attributable to the MIDI specification. Indeed, it has often been criticised for being too centred on keyboard instruments and interfaces [26], resulting in an expressive model which is little more advanced than variation in volume – velocity.

   Performance is central to our goal – MIVI is an instrument performance tutor. The lack of expression in MIDI, also identified in section 2.1.2, must be overcome if we are to instruct in performance of any instrument save the piano.

    

2.3.1 expression

 

 

 

Research by a number of institutions has produced CMPS's, or Computer Music Performance Systems, which are designed to record more expression in music, allowing the computer to more accurately emulate the human element in performance. These systems can also bring expression to instruments where it wasn't previously found, such as synthesizers.

computer music
performance systems (CMPS’s)

 

   FORMULA (Forth Music Language) [2] is such a system. It was an attempt, in the early 90’s, to build upon the MIDI protocol – to add emotion and expression, etc. to the audio output, through the creation of a music programming language. On p.23 of the paper, Anderson and Kuivila also briefly mention the potential of visual output, but concede that the hardware of the time, combined with their system architecture, would introduce ‘unacceptable delays’ and prevent a meaningful exploration of the medium.

 

 

 

   There are two immediate problems involved with promoting such data to the visual layer, in an application like MIVI. The first is technical; "How does one show emotion in graphics?" The pursuit of an answer to this could be defined by the history of ‘Art'. Realistically and, perhaps, crudely, the only way to do it would be to have a representation of the performer and his face as he (or she) plays the piece. Indeed, the face is often used in art as an interface to the emotion of a painting’s subject. It is, after all, not the instrument that humans perceive as having the emotion, but the player.

   The other problem is of practicality. In an educational program, the useful applications of carrying expressive performance directives through to the visual layer appear minimal, since expression is often an aggregate of one's own soul and experience – tacit knowledge, which is difficult, if not impossible to teach. However, there can often be more than one method, or technique, available for playing a note or phrase, which is important for the performer to be aware of.

   

2.3.2 technique

 

 

 

Technique, although related, is not synonymous with skill – it is a means to skilfulness. Although, on a piano, teachers will train students to use particular fingers on particular occasions, it need not always improve the quality of the performance. Instead, it makes the performance easier, so that the move to the next echelon of ability becomes more achievable.

fingering

 

   One aspect of performance that instantly presents itself, when stepping outside the piano, is fingering choice. A guitar, for example, has six strings, and the dynamic range of each overlaps with not only the adjacent, but beyond that as well. There thus exists a choice of methods to finger a single note, and some will be easier than others. If we are to display the instrument as it relates to the score, we must choose one of these methods. However, as an educational tool, we must make sure that our choice is suitable for the learner.

fig 2.6 - table of
guitar fingering

   


 

 

 

 

   In figure 2.6, we illustrate a diagram identifying the possible fingerings for various pitches on a guitar fret, where each number denotes the distance from the fundamental pitch (in semitones) – in this case E(00) – and each row is a different string. One can see the repeated occurrence of equivalent pitches across several different strings (equivalent C(20)'s have been emphasised). The problem facing the performer, and – by transitivity, our visual interface – is which to choose at any one given time.

   In the above scenario, let us assume that we want to play the C and that the succeeding note is an A(17), so, if possible, the algorithm should, for simplicity's sake, avoid changes of string or hand position. Intuitively, it would be best to pre-emptively place the first finger on an A, and the second (or third) on an adjacent C, so the transition can be made painlessly and involve only a simple removal of the extraneous finger. This rules out the C on the B-string, since no such A exists, leaving two options, which – under our current constraints – are equally attractive.

   So, we introduce another constraint - that of sound quality. Plucking an open (un-fingered) string produces a much 'cleaner' sound than if the player were to finger the note by depressing a lower string nearer the bridge (towards the right of our diagram). Simplistically, this gives us the heuristic: the further left the number on our diagram, the better the quality the tone. Readers interested in implementing a more detailed set of constraints that account for quality are referred to Taylor's [50] paper.

generic fingering
algorithms

 

   Incidentally, in the case of other fingered-string instruments, like the violin, violinists are encouraged to choose fingerings in preference to open strings, in order to maintain a uniform sound quality across all notes. Thus, for applications servicing multiple instruments, such as MIVI, it will be important to give careful consideration to the domain of their fingering algorithms. For example, how much of a guitar fingering algorithm might translate to a violin application? Even on a lower level – how much modification is required to adapt a violin-fingering algorithm to a cello? Does the introduction of the thumb, in this case, present a large problem, or can we simply treat it as a fifth finger? We leave these exercises to the reader and referenced literature.

recursive
algorithms

 

   Returning, and restricting ourselves, to our guitar scenario; we have seen, with just two constraints, a decision can be made for each note (the previous decision has been shaded in the diagram). Notice, however, that we have also made an assumption (the position of the succeeding A), which allowed us to further constrain the decision. A note's reliance on its successor suggests a ‘lookahead’ is called for – the succeeding note’s position must be calculated before the current note’s and, of course, the same is true for the succeeding note. This recursive relationship will propagate the decision to the final note of the music, where a decision, based on no successor, must be arbitrarily made. The results then cascade back to the initial note. Thus, we see that, in this implementation, although implementable using a constraint-satisfaction algorithm [18], fingering calculations cannot be a real-time operation, but must be pre-processed.

   This is how the DIVA system (discussed in the section 2.5) works. The music is analysed and all the fingering positions are worked out and stored before the performance. It is a simple approach and will result in a polished performance, depending on the algorithm’s constraints. It requires, however, a pre-processing step before each new piece of music – the duration of which will vary depending on the complexity of the piece and the number of musicians in the ensemble, and could, potentially be quite costly. In comparison to the typical human approach, though, the duration of this step will be drastically smaller than a standard ensemble’s equivalent during rehearsal sessions.

   However, let us consider the human performer more closely. The player’s approach depends on their level of skill and experience, most notably in the sphere of sight-reading. A lesser-experienced musician might play the piece through once, annotating the score with fingering observations as they occur, restarting, each time, to verify them – the human equivalent of the DIVA system.

finite lookahead
algorithms

 

   An experienced musician, on the other hand, might be able to perform a small lookahead in his mind and anticipate optimal fingerings, similar to a Grandmaster of chess deducing the next seven or eight moves before they happen. This latter tactic gives us insight into how to implement a real-time fingering system. By restricting the lookahead to a limited amount and forcing the arbitrary decision before reaching the final note, we could conceivably avoid the heavy load on the processor, enabling the decision to be taken as the note is played.

   Indeed, it is self-evident that the fingering for the final note in a symphony movement has little or no influence on that of the first. We can apply this analogy on much smaller intervals and, by selecting a suitable lookahead size, in relation to these intervals, not only reduce the processing overhead sufficiently to allow for the decisions to be made in real-time, but forego any significant hit in performance quality. There are several points in a score where influence does not propagate, such as rests (silences), and less critical points where even more involved fingering shifts are less costly, such as following long notes, or stretches of repeated notes, where thinking time is more abundant – both are examples of good lookahead limits, that occur frequently throughout most pieces.

   However, in the absence of such appropriate junctures, an algorithm might have to force a lookahead limit in order to prevent hogging of the processor, thus avoiding glitches in song playback. In this case, a decision is forced, for the current note, based on less than all the facts.

expert systems

 

   Such decisions can often be simplified by the use of expert systems or neural networks, such as Bayesian Networks [23] or Hebbian Learning [40] (respectively), where instead of simple deterministic choices, statistical reasoning is employed. Hence, after a modest lookahead, though we might not be able to say for certain what the best choice is, we have statistical weights denoting which choice is most likely to be the best choice. Bayesian Network systems that allow for learning, such as BUGS [45], can even be combined with cached experience, that can assist us further, by basing the fingering, for a phrase of finite length, on what action was taken last time the question, or similar question, was posed.

a hybrid
fingering algorithm

 

   Using a similar principle of learned experience, Sayegh [41] successfully employed the Optimum Path Paradigm (OPP), an approach based around constraint satisfaction, to tackle the fingering problem in stringed instruments. Taking the number of strings to be S, the number of playable notes on each string, N, with NT (= S × N) representing the total number of finger positions possible, the algorithm first populates an NT × NT matrix, W, with the cost of every possible transition between fingerings, penalising string and hand position changes – as in our original example. The matrix can then be queried one or more times, given a sequence of notes, to produce a set of solutions. All existing solutions are then combined to form a weighted, directed graph (see figure 2.7), which can be solved in polynomial time using the Viterbi algorithm [52], a refinement of the algorithm for finding paths of least cost by Dijkstra [15].

fig.2.7 – Sayegh’s use
of weighted, directed
graphs in the sequence
G, A, C, E

 


 

 

 

 

   For a classical guitar (S=6, N=36), the number of calculations to populate W is a phenomenal 216 × 216 = 46,656. Unfortunately, aside from a trivial identity mapping, no ‘optimising’ observations can be employed to streamline the process – fingering transitions are not normally commutative. However, the algorithm need only be executed once per instrument – the matrix can be stored for use in the future.

   Sayegh refines the approach by introducing a second stage, based on Hebbian learning [40], a form of expert system. He uses a second matrix of equal dimensions, W', the entries of which are initialised to zero. The results of the first stage then go through a learning phase, using W to generate responses to various training material in the form of sequences of musical notes. Each time the algorithm suggests a transition between two fingerings, the corresponding entry in W' is incremented by 1. After a sufficiently large sample input, the new matrix represents experience, which can be used to base fingering decisions on, either solely, or in addition to the original matrix. The accuracy of this sort of Hebbian learning (as with Bayesian Networks) is often difficult to accept from base principles, and those new to the idea often benefit from practical and quantative examples, which the reader should be able to find in most of the books on the subject [23][40].

progressive
algorithms

 

   In MIVI, we are not designing a playing aid, but a teaching aid. The performers – in this case, guitarists – need to learn how to apply the constraints and work out fingering themselves, and should not become reliant on the presence of a computer. The constraints should be considered equivalent to 'playing tips and tricks', and can be presented as such in the program. Therefore, given a piece of music, the user should be able to specify which techniques the computer employs, and thus which to tackle learning themselves. The true beginner starts with none, possibly using only his fore finger to adjust the pitch of his instrument, and can then introduce the 'tricks' one-by-one, adding them and advancing at his leisure. Furthermore, when the choice of employing a specific technique conflicts with that of another, or represents a different – as opposed to 'better' – practice, the decision that must be made is one of playing 'style'. Sayegh provides a convenient method of implementing such a system, with each set of constraints (or playing style), represented by a pre-compiled matrix.

playing style

 

   Whereas the 'tips and tricks' indicative of a playing style might be defined by the constraints of the algorithm's first stage, an expert guitarist's playing style might be encoded by running just the learning stage on them, instead of the algorithm – getting them to manually increment the contents of matrix M', as they make their own decisions, given the learning material.

   Hypothetically, a 'complete' implementation would not only allow you to define sets of constraints corresponding to the playing styles of guitarists (John Williams, Eric Clapton, BB King, etc.), but also more general sets, optimised for their genres (respectively: classical, rock, blues, etc.). Simply restricting the learning phase's training set to pieces in the required genre could develop such genre-orientated sets.

   The initial implementation of MIVI, documented later in the report, will not include such performance memories and fingering algorithms. However, when it becomes time to codify our flute, we shall see that a simple fingering decision is required. Furthermore, we shall attempt to ensure that if – or when – the MIVI system is extended for other instruments, such as the guitar, the architecture is able to support the inclusion of fingering algorithms.

  2.4 education

 

 

 


The awkward and visible sign is the syllabus, a table of contents which lays down what the student is required to do and what is examined… The syllabus narrows the students vision of the edge of knowledge and cuts him off from precisely those fuzzy areas at the edges of subjects that are the most interesting and rewarding.

Christopher Small, p.186-7 in Music-Society-Education [44]

 

 

The moral of this fable is that, if you’re not sure where you’re going, you’re liable to end up somewhere else.

Robert Mager, Preface to Preparing Instructional Objectives [28]

 

Today, when it comes to learning an instrument, there is a multitude of teaching methods available to the music student and, as a player of several instruments, the author has much experience in this role. In this section, we give a concise and critical history of the academic field, with focus on the practises of musical instrument tuition and use of computers as an adjunct to education

   

2.4.1 music education

 

 

 

Tradition demands [24][35] the tried and tested technique of one-to-one – teacher to pupil – lessons on a regular and frequent basis. During daily, weekly or fortnightly lessons, the pupil’s performance is assessed by the teacher, whose competence then determines the quality and availability of positive critical feedback. Furthermore, the instructor also sets the syllabus, perhaps via an examining body, and suggests homework and beneficial extra-curricular activities. It is in this way that the author has learnt to play the violin.

   The strict scheduling of lessons and proliferation of deadlines and exams can, however, contradict the rationale of music itself – to be an enjoyable and entertaining occupation. After all, for most people, playing music will be considered a leisure activity. The two passages, given at the beginning of section 2.4, are quoted from Swanwick [48], who is but one of many scholars [17][35] who have arrived at this conclusion.

   Additionally, and often more importantly, the expense of expert tuition can sometimes be prohibitive [19]. In the case of many instruments, especially during the sometimes painfully slow early stages, it can be difficult to identify what your money is paying for.

the self-taught
approach

 

   For some, this might persuade the student that a self-taught approach is more financially viable – if only to get them to a stage where they can evaluate the merit of expert, third party tuition. However, even at the individual level, there are a number of different approaches, many complemented by commercially available teaching aids.

Naturally, the first is the bloody-minded, do-or-die approach. In the case of learning to swim, this involves jumping in the deep end and, more often than not, results in either drowning or the somewhat crude and expedited erudition of the fundamentals of swimming – namely, the doggy paddle. In the case of the aspirant maestro and, at one time, the author, this might involve a complex piano sonata, Bach for example, and practising until they can play it.

   This approach will only work for some, notably pre-seasoned musicians, and, although coarse and cursory, will elevate the subject’s proficiency across a much larger repertoire than simply the chosen piece. Indeed, once one has deciphered Bach, little stands between them and most other Baroque piano music, and this is the scenario, respectively, for many other genres.

   However, music is an age-old pursuit, and it is naïve to think that just the score and the instrument will endow one with enough experience to make you a maestro. Indeed, the score has been as heavily criticised [35] as credited for its provision of information and encapsulation of the performance. As in the extreme of our aquatic example, one may be able to swim, but not necessarily to swim well.

the explicit, the
implicit and the tacit

 

   As an art, most notably a performing art, musical knowledge extends beyond the explicit – what is written on the page. In this way, traditional tuition will always have an advantage over the entirely self-supported approach, for it allows the elicitation of implicit knowledge – that which can be articulated but isn’t – and tacit knowledge – that which can’t – from the expert to the learner. So, whereas the self-taught student might be able to play the music, they will be ignorant of tips, tricks and shortcuts, to allow for easier playing and more freedom to express themselves – another defining characteristic of art. Indeed, for many instruments, such gaps in knowledge will inhibit the student from moving to higher echelons of play. Moreover, for instruments less intuitive than the piano, the lack of knowledge and guidance about how to equate the notes on the page to the instrument can severely hinder, often prevent, even the slightest progress.

   Music, incidentally, is the only art form requiring literacy [35] – whereas a painter need only apply the brush to a canvas, a musician is reduced to working through a layer of abstraction – the score. As we saw in section 2.1, such obstacles can be compounded with further layers, such as MIDI.

teaching materials

 

   Therefore, when teaching oneself, a natural step is to find literature on the subject. In searching for such secondary materials, one is seeking an expert who has tried to put everything they know on paper – both stating the explicit and articulating the implicit, so that it is, in its new form, explicit. To what extent the expert has achieved this goal, determines the relative quality of a material, and there are several different approaches to the problem available.

When one is looking for such literature, as this author endeavoured to do when learning the guitar, the sheer variety available is daunting. In its infancy, this market was dominated by books [34] where the expert and publisher employed text and nothing else. Soon after, these were superseded by ones [33] containing simple diagrams to help tackle explanations of fingering and posture, etc. Then, with the advent, and relative cheapness, of black & white [51] and, later, colour photography, we have books advertising the "all-visual approach to learning to play the [instrument]" [10].

visual learning

 

  In music, the importance of exposure to the visual aspect of playing an instrument can sometimes be as important as the audible aspect – the classic 'Monkey See, Monkey Do' philosophy is actually of benefit in this case and has been formalised, in research, more than once.

   Chappell [11] states that the ability to internalise music (to hear, or picture, it in one's mind) is of considerable advantage in the process of developing musical skill. Nevertheless, other studies [1][16] show this skill to be present in only a handful of the world's greatest musical geniuses.

   Ben-Or [4], whose views on the subject are widely held, states "If one can really perceive a passage of music with all clarity and represent it to oneself mentally as it relates to the instrument, then there is no obstacle left for the body to freely express it in sound". Thus, to present music in a visual 'instrumental' form, pre-converted from its abstract score format, should remove a number of such obstacles.

pictures and
illustrations

 

   Static visualisations, though a step in the right direction, often do not convey the subtleties of playing the instrument. On a purely technical level, pictures can rarely capture motion. For a simple example, the book may illustrate 3 key stages of strumming on a guitar; the up, the strum and the return to the default hand position. In all, the strum should last 1 second. With the average human eye, working at a frequency of 72Hz (72 pictures a second), it is up to the individual to anticipate 69 of them, or 96% of the motion.

   Strumming is a relative simple procedure and most, if not all, beginners are able to master it after but few attempts, allowing books to economise in the early stages. However, more involved motions, that should be accompanied by even more enhanced illustrations, are, instead, usually accompanied by none. Though the cynic might put this down to the price it costs a publisher to print pictures, the true reason is likely to be the limited level of information expressed in only one image. It simply isn’t plausible to express, in a page, complicated techniques where 10 or more key stages, each shot from multiple angles, are required to deliver a true understanding – a deficiency compounded in some books by a tendency to illustrate the instrument from the listener's view, rather than the players, forcing a translation, this time of geometry, on the learner. 

   Another, more general failing with 2D (diagrams) and static pseudo-3D (photos / illustrations) material, is the lack of the fourth dimension of time – a reader, following examples, note by note, cannot gauge his progress, since although they are playing the right notes in the right manner, they are not aware of temporal aspects of the piece. In the case of tempo itself, the speed (or lack thereof) of their performance is likely to be lower, allowing for deceptively easy playing.

audio examples

 

   Therefore, some books are accompanied by an Audio-CD [10] that demonstrates what the reader should be reciting. This helps to a great extent, but still lacks cohesion, in that, although the music on the page may be recognisable in the playback, the process that connects one to the other still relies on the quality of text and pictures, and requires some quick-thinking on the reader’s part. Slack [43] also recommends the use of audio aids in music education, but stresses the importance of accompanying explanation7. As far back as 1964, he advocated the augmentation of audio with visual aids [43] in the learning process – extolling the virtues of being able to zoom in and focus on parts of the instruments. Thus, as the technology became available, the trend took us from the written word through the spoken and illustrated, to the motion picture.

the motion picture
and MIVI

 

   This philosophy is not new to education, and has even been brought to bear in the form of multimedia music tutorials - CD-ROMs of text, pictures and video demonstrations. Videos are digitised film, which is to MIVI what the Audio-CD is to MIDI. Instead of a data stream, like videos and CD’s, MIVI will be built on an object-oriented structure for performances. Where MIDI can tell you the pitches, and volumes of each note plus allow you to alter tempo, etc., MIVI will draw upon this to provide total-immersion graphical playing environments, with editable angles, magnification and speeds plus allow the isolation of a single instrument and its components, for closer scrutiny.

fig.2.8 - domains of education methods

 


 

 

 

 

   In figure 2.8, the common stages and routes to musical expertise, as discussed in the previous paragraphs, are summarised with relation to their competences. Some features of the table are worth mentioning. Firstly, it is clear that when one becomes an expert, tuition is no longer required, and development now relies on a self-supported exploration of music and performance, although possibly with external influences from other performers. Secondly, we conclude that a degree of tacit knowledge, not serviceable by oneself or media, is required to achieve expertise.

 

 

 

   It should also be noted that the table holds no information relating to speed, ease or expense of any particular method, or combination of methods. Given the problems with current self-tuition materials, and yet, also those of expert third-party tuition (introduced in section 1.2), most learners demand the latter. It is hoped that MIVI, by plucking some of the advantages of the third-party, will close the gap between the two – in a sense simulating a virtual music teacher, for the earlier stages.

   As Roland – a world-renowned musical instrument manufacturer – states [38], arguing a similar case for its own interactive tutoring products, "[M]usic learning with private lessons begins with 30 minutes of excellent guidance and coaching. But, now a crucial difference takes place. Instead of the regular, routine play-through of the school ensemble music, the student goes home to a week-long period of unguided, occasional practice."

   They continue by identifying the distinctions between teacher and tutor – the first, is to provide direction, the second, to guide them on their journey. In their case and ours, the respective products’ competences fall into the latter.

   The promotion to the former might involve the inclusion of implicit and explicit technique theory, including the migration of expertise from the real to virtual incarnations, which has resulted in the division of MIVI into two domains in figure 2.8.    The first represents the intended capabilities of the implemented MIVI system as documented in this report; introducing the student to the instrument8 and teaching the student how to relate the score to said instrument. The second builds on the first with the ominous requisite of ‘performance knowledge’, and was discussed in the section 2.3.2.

   

2.4.2 computers in education

 

 

 

The modern buzzword in multimedia education is ‘interactivity’. Having argued the ‘learning by seeing’ case, many of the same principles can be, and are, similarly employed in advocating ‘learning by doing’ [9][47][48]. Using interactive materials, not only can the user get instruction, but feedback too. The interactive device can have algorithms to analyse the user’s performance and give qualitative and quantative feedback on their errors, as would a genuine tutor.

games, simulations
and drills

 

   E.R. Steinberg [47] mentions several methods of interactive learning through the use of the computer. ‘Drills’ are the purest form of Computer-Assisted Instruction (CAI) and simply aid the memorisation of symbols, such as the periodic table, or connections, such as our score note to key mapping, through the standard stimulus / response format. Even in this simple context, detailed feedback is possible, since the computer can tell the learner what they got wrong, where they tend to fail (statistically) and to what extent they are in error. Furthermore, the execution of the drill can be tailored from learner response – getting harder as they do better or vice-versa. Drills, however, only test knowledge already present in the user.

   Either alternatively or in combination, E.R. Steinberg advocates the use of ‘simulations’, where the learner is presented with a virtual representation of the subject material. She states, "Insights about complex scientific principles often come from experience with such concepts in an interactive environment".

   A flight simulator, for example, allows trainee pilots to familiarise themselves with the cockpit of an aeroplane without the expense (in terms of money, space and time) of a real plane or flight.

   Closer to our context, the use of typing tutor software is also an embodiment of this concept. As a drill and a simulation, it is able to show the user what to do, regulate the act itself and give qualitative and quantative feedback on their performance.

eliciting tacit
knowledge

 

   We have seen how interactive media can replace the implicit teachings of a music teacher, but how can we extend this to tacit knowledge? As it becomes more widely acknowledged that we are living in an age where information is paramount, industry looks to retain such knowledge, without necessarily retaining workers, through Knowledge Management and Information Systems.

   One approach is to concede that tacit knowledge cannot be encoded and copied, but ‘cloned’ instead. In our case, instead of telling an expert guitarist, to write a book about how the reader should play the guitar, which we have shown will not implant his expertise in the reader, invariably loosing something in the translation, we simply tell him to write a book about how the expert plays the guitar himself. Although the knowledge cannot be copied, the renditioning of a particular piece of music (the performance) can be. So, we instruct the learner to recreate the performance(s), trusting that, in their efforts and attempts to execute the task, the learner will derive the tacit knowledge for himself and will, in future, be able to sub-consciously apply it to the generic music score.

   Thus, returning to our discussion on the relative merits of MIVI over existing soft and hard literature we see that, whereas existing teaching material is often restricted to a specific genre, such as "Jazz Piano" or "Blues Guitar", MIVI should have the capacity to cross these boundaries, simply by selecting a MIDI file and performance memory from the desired genre.

  2.5 the DIVA project

fig.2.9 - the DIVA
system in action

 
 

 

 

 

In this section, we will introduce the reader to DIVA, a project closely linked with MIVI, and review some of the core principles and ideas surrounding it. We will also discuss the relative merits and shortcomings of DIVA, as well as the isolate the differences between it and our own endeavour, MIVI. Other more specific aspects of the DIVA project are covered elsewhere in the report, as they become relevant to our own endeavour.

   

2.5.1 DIVA history

   

 

 

 

 

Beginning in 1992, a team of scholars, from the Helsinki University of Technology, embarked on a number of research projects [21][22][26][49] that, in 1997, culminated in the design and development of a Digital Interactive Virtual Acoustic (DIVA) environment [26]. This involved the creation of a totally immersive 3D environment, embodying the performance of a small Virtual Orchestra of Virtual Musicians, led by a human conductor, with physically-modelled instruments and acoustics, both audibly and visually accurate down to the spatial dispersion of sound and fingering of instruments, as illustrated in figure 2.9.

   Drawing from expertise in multiple fields, such as digital signal processing (DSP), graphics, acoustic modelling and sound synthesis, the team has studied and successfully combined technologies in all these areas, climaxing with an interactive performance at the SIGGRAPH'97 conference.

   

2.5.2 the DIVA system

 

 

 

The impressively endowed system architecture (figure 2.10 (a)) comprises of no less than 3 multi-processor SGI workstations, each acquiring a select number of competences, networked to external synthesizers and an Ascension Motion Star for conductor input.

   Figure 2.10 (b) is adapted from Huopaniemi et al [21] and illustrates the DIVA system information flow, with the corresponding scope of the MIVI system appended. It should be noted that our project will, in fact, address an even more confined scope than that indicated by the diagram, with our MIDI-to-movement mapper and Animation control avoiding the overhead involved with the processing of virtual biped musicians and multiple instruments. It is intended that the MIVI system should be suitable for execution on a traditional, self-contained, uni-processor system - a home computer or other such workstation.

   However, while sharing some common attributes, MIVI and DIVA differ in objective and target audiences. Thus, we find that, in a couple of our interest areas, such as fingering and visual detail, the DIVA research is less extensive.

fig 2.10 (a) – DIVA
system architecture

 


 

fig 2.10 (b) – the DIVA
process flow

 


 

   This stems from the fact that the DIVA system is principally orientated towards the use of sound engineers and experts – be that for movie sound or general recording studio work. MIVI, on the other hand, will focus on establishing itself in an educational role, aimed, initially, at the unskilled user, possibly in a studio environment, but more likely in the school or the home. The group's research, as such, illustrates an emphasis on acoustics and audio realism, with principal focus on sound spatialisation, room response modelling and HRTF (head-related transfer functions), rather than graphical and performance accuracy that, as discussed, will be crucial in our own venture.

   Many articles and papers, by different members of the group, provide overviews of the various aspects of the project, and it is, therefore, necessary to direct the reader towards Lokki et al [26] for one with more relevance to our field – the visualisation of musical instruments in real-time.

   Recalling our discussion of fingering algorithms in section 2.2.3, DIVA employs a simple and computationally efficient, yet slightly musically naïve, least-distancing algorithm to solve problems arising when there exists more than one way to finger a note. When presented with a choice of fingerings, this approach comprises of simply choosing the solution requiring the least hand or finger movement – favouring ease over quality. Such decisions are made using Critical Path Analysis (see [40]) – an extension to Sayegh’s [41] use of weighted, directed graphs (discussed in section 2.2.3).

   Although, as claimed, the fingerings are "realistic" [26] – in that they are possible – they are far from optimal and, although adequate in an entertainment environment, should be further scrutinised in an educational or reference role. This compromise is also evident in the conducting styles recognised by the system when it processes data from the input baton – only the most generic are identified. The implementation of both features is, nonetheless, more than adequate at demonstrating the practicality of their inclusion. The team’s fingering efforts do, however, introduce an interesting aspect of visual simulation – dynamic (or timed) fingering. Actions like guitar strums or percussion hits require preparation, such as the ‘up’ movement (mentioned in section 2.4.1) or the initial movement of the drumstick towards the drum skin.

   Lokki et al [26] also concede that more weight has been given to the realism of the ‘scene’, rather than that of the individual instruments and musicians themselves. This is most markedly illustrated by the ‘3D cartoon puppet’ format of the musicians. Sighting psychological reasons [26], the DIVA team conclude that maintaining a high animation frame rate and focusing on realistic character motions allow for a scene that the human brain can more easily accept as, or equate to, reality. The appearance of instruments and musicians has thus been optimised for efficiency rather than quality. Indeed, from the stills, it appears that the flute’s visualisation does not extend to the rendering of the flute’s keys.

   To further minimise the computational load at run-time, instead of computing the fingering and musician animation in real-time during playback, the motions are compiled for the entire performance pre-execution. It is not clear from the literature whether the DIVA system utilises an automatic inline system to compile the motions – such as that suggested by Lytle [27] – or a separate standalone application, to the same effect, or whether the animation files must be compiled by hand. If either of the latter two are the case, then MIVI has the great advantage of an immediately accessible repertoire of MIDI files, unrestrained by the pre-requisite of an accompanying animation file – in theory, every Standard Midi File. Furthermore, the real-time nature of MIVI allows for spontaneity in the music, like the introduction of musical improvisation.

   Like the animation files, the DIVA team store the data for instruments in external files [26]. In both cases, the system parses ASCII descriptions and must be generic enough to be able to process varied instruments (flutes, violins, drums, etc.). Such a ‘plug-in’ architecture of instruments and players to the DIVA Virtual Orchestra, allows for flexible system-extendibility. How the team achieved this, and the extent to which they did, is not clear from the available literature, but its description alone encourages consideration of similar functionality in our MIVI application (see Chapter 4).

   MIVI's focus on just the instrument should also permit us to dedicate considerably more of the available real-time computational power to this problem, as well as allow us to research more elegant and involved solutions in this area. Thus, for MIVI, the lines from the MIDI file and Instrument definition boxes to the MIDI-to-movement mapper, in figure 2.10 (b), should be emphasised in bold, indicating the real-time nature of this flow. 

   Indeed, it would be interesting to see the fruits of the DIVA project updated for today’s hardware. The progress, particularly in 3D graphics hardware, during the last few years, has been pronounced. Excepting the distributed and multi-processor advantages of the DIVA hardware, the performance of high-end consumer systems, such as those used in the development of MIVI, could represent a comparable level of computational power. A combination of DIVA’s scene dynamics and MIVI’s attention to detail could lead to one of the most realistic real-time computer simulations yet.

 
 


All content, including code and media, (c) Copyright 2002 Chris Nash.