the development
of music software

   When technology first entered the musical sphere, its influence was limited to embedded systems. Composing, recording and mastering were all handled by dedicated music hardware and, indeed, many stages still lie under the purview of such devices today. However, as personal computers (PC’s) become more capable and user-friendly, the industry is gradually adopting them. A decade ago, this latter constraint gave Apple Macintosh the upper hand, and hardware and software manufacturers began to design for the MacOS platform.

   In recent years, the IBM-compatible PC’s success in the consumer market, due to its improved user-friendliness, flexibility and processing power, has convinced many of these manufacturers to include support for Microsoft’s Windows operating system as well. Even more recently, as the PC OS market becomes more competitive, versions of the industry-standard UNIX are demanding attention, featuring heightened accessibility and multimedia-provision, whilst still exploiting the same dependability and superior performance, as is seemingly the case with LINUX and BeOS.

   Established software developers are porting their audio editors, sequencers and other music utilities to these systems. One of these, Steinberg Cubase VST, merits our attention.

Starting out as a simple MIDI sequencer, Steinberg’s Cubase gradually acquired competences in audio recording, audio processing and music scoring. In its present form, it has the capability to host its own software synthesisers and software effects processors as plugins to the parent – or host – program.

   This is the point at which Cubase acquired its VST suffix. Standing for Virtual Studio Technology [46], this initially involved the host application placing waveform data in a buffer and passing it to a user-specified child process – the effect plugin – that would modify (process) the data in the buffer and return execution to the host. As the host’s song plays, such information is streamed to the plugin and the buffer is outputted to hardware, post-modification.

   The widespread popularity and adoption of this architecture prompted improvements in the VST specification to allow these audio plugins to respond to, and process, MIDI input. Upon request, the host now places MIDI data in a buffer, which is passed to the plugin as well. The combination of these buffers allows for many different applications, as illustrated in figure 3.1.

   As is visible in the diagram, MIVI represents a departure from conventional plugin architecture and data flow. Indeed, the author can find no evidence of the VST system being previously used in this way – with no direct and immediate influence on the music being played back. Indeed, when we get on to coding, in the section 3.1.3, the reader will notice the proliferation of the word ‘effect’ in programming keywords, such as AudioEffectX and AEffect. Such keywords stem from the early days of VST plugins when the only possible applications were audio effects. Unfortunately, their continued, and sometimes counter-intuitive, use is required to maintain backwards compatibility.

fig.3.1 - VST data flow
and applications

platform
independence

musical
representation

   Although commercial concerns are largely irrelevant at this early stage, one consequence of this decision is worth mentioning. Music software, such as Cubase, is often targeted, by way of features and pricing, at the professional market. This may beg questions about the logic of designing an educational utility for a post-education audience and the morality of restricting use to only the affluent minority of society – a contradiction to the intended audience that we identified in section 1.2.

   In response, it should be noted that, while VST hosts, designed and priced for the home market – including a ‘lite’ version of Cubase (called Cubasis) – do exist, MIVI is also envisioned in a classroom environment. Already, many music departments have invested in music workstations, including a copy of such software – often the professional versions. These are viewed as sufficient and necessary justifications for the choice but, if for some reason the availability were not enough, a dedicated, yet simple, VST host could be constructed to allow the plugin to be universally executable. However, any such host would lack the advantages of the score format, unless explicitly provided.

For many software projects, the natural choice is to start from scratch, coding the program as an independent (standalone) application. If a hosted environment, such as VST, is available, programmers, often with reason, cite problems regarding the opaqueness of operations – the inability to see and control what is happening behind the scenes – and thus choose not to use these platforms, despite the advantages and shortcuts they might offer, so as to engender complete faith in their implementation.

   For two reasons, we conclude that VST, while omitting our need to code for low-level hardware communication and providing us with added functionality (such as the score and interface), also satisfies these more stringent requirements.

   Firstly, both VST and Cubase are benchmarks of professional standards in music technology. In the studio environment, for which both VST and its plugins are designed, it is necessary to have meticulous timing – the host functions, and is written, in much the same way as a Real-Time or Safety-Critical System [6], requiring the same compliance from hosted plugins.

   Secondly, without discouraging or hindering development of such software, Steinberg regulates the access and existence of developer resources. All developers must register with Steinberg, before receiving the Software Development Kit (SDK), which contains the official documentation to the VST system [46]. This, combined with the Steinberg-run and -moderated Mailing List, form the only practicable resources for VST programmers. In so doing, Steinberg minimises the potential for misinformation, inherent in distance learning, while providing an unbounded source of information.

   Another system in development, here at the University of York, called MIDAS¹⁰, shares a similar rationale to MIVI – the visualisation of audio. MIDAS is a programming language for the visualisation of music and, in this capacity, represents an implicit specification for applications to that end.

   Although literature is scarce on the subject (due to its infancy), several themes of the research compliment our own. Firstly, MIDAS harnesses the power of OpenGL 3D graphics – although only a select amount of OpenGL operations are currently promoted as keywords in the MIDAS language. Secondly, MIDAS is designed to be platform independent and, similar to many of the technologies employed in MIVI, should compile on any platform with a C compiler.

   Unfortunately, several aspects of the MIDAS system make it unsuitable as a host for our project. As yet, no MIDI support has been stated or documented – the language is designed to react to waveform audio only. Furthermore, in this capacity, the graphics facilities available are designed to enable programmers to produce highly abstract, algorithmic graphical patterns, as opposed to virtual objects. Lastly, as with standalone applications (discussed earlier), MIDAS has no default support for the score or other musical notations.

By selecting the VST plugin architecture, we have pre-determined the programming language for our application – C¹. As universal as the other standards employed in our project, C has been an industry-standard high-level programming language for many years now. Hence, many argue that it has begun to show signs of age and, for that reason, we take the liberty, in our Windows implementation, of programming in C++ – an enhancement introducing, among other improvements, an object-oriented programming (OOP) environment. In addition to adding efficiency to our code, OOP will also present the reader with a much clearer picture of what is going on in MIVI.

   When such an event occurs, the code at the DLL’s entry point (its *main function – code ref. 15) checks that the parent is capable of hosting our plugin and that sufficient memory is available, then creates a new plugin object, effect. If all is successful the function returns, to the host, a pointer to the new instance of the plugin. Otherwise, an exception is raised, in the form of a null return, whereupon the host reports the problem to the user.

   In creating the plugin object, the object’s constructor (code ref. 18) is called. A constructor, as the name suggests, consists of code to initialise the object, instantiate class variables and handle any calling arguments. In the VST environment, we must tell the host about the competences of the plugin, such as its audio IO capabilities. From the information we provide, the host can work out which resources the plugin requires, and thus save time by omitting unneeded callbacks and data flow.

issues with the
MIVI output device

   It is worth noting that, by simply switching the track’s output to MIVI, the track will not be auralised through a normal audio device, and will be effectively muted instead. As yet, it is not possible for the host to send notes to MIVI, then for MIVI to pass them directly to a particular MIDI device. The SendVstEventsToHost function allows the plugin to send MIDI messages back to the host, but they will be treated as standard input and sent to the output of the presently selected track, which only serves to complicate matters.

   So, to maintain the audio output, it is necessary to split the track’s output in two, from within the host, and divert one to MIVI and the other to the desired hardware device. Of course, another, less tidy, approach is to simply copy the original track and select MIVI for output on the copied track.

   Alternatively, we can use this deficiency to our advantage. By muting the track and sending it to be displayed through MIVI, we are not muting other parts of the music and are allowing the user to superimpose their own input. A range of learning tools called Music Minus One¹² already employs this concept. Consisting of the score for a concerto, or other such piece, and an Audio-CD of the music without the solo instrument, the idea of the Music Minus One pack is to provide the consumer with a backing to their solo performance. This has the great advantage that the aspirant soloist can practise with a full orchestra in the privacy of their own room. A similar process also underpins Karaoke.

   Other lower priority capabilities are requisitioned when the plugin’s initialisation is complete through the plugins obligatory canDo function (code ref. 21). The host calls this function for each capability and it is up to the plugin to return true (1) for those it supports. In the case of MIVI, we state support for receiving events – in particular, MIDI events – and timing information (including playback status). The host, therefore, now knows to provide such data.

   Non-essential properties, such as vendor information and copyrights, are also solicited in a similar away, but in MIVI, have been largely ignored.

   One chore, of note, is the unique identification of MIVI to the host. To ensure that two plugins do not conflict when used together, a plugin supplies the host a four digit alpha-numeric key to identify them (code ref. 22). To prevent plugins using the same key, the developer must register their key with Steinberg before use in the public sector. Since MIVI will not be used in a commercial environment and since all the keys of plugins, available to run in tandem with MIVI, are known to be distinct, the "MIVI" ID, that we shall use, need not be registered with Steinberg.

   At runtime, VST will not activate the plugin by default – the user must activate it manually. This involves opening the VST Instruments window (because we have defined MIVI as a synth, rather than an effect) and selecting ‘MIVI’ from the list of available plugins. Then, by toggling the ‘power’ button on the VST Instruments console, we make MIVI available, as an output device, to the open MIDI sequence. Now, the user only has to go to the track they wish to have displayed and select ‘MIVI’ as the output device, thus diverting all the MIDI messages to the MIVI plugin.

In the introduction, we mentioned our selection of graphics platform – OpenGL. OpenGL has been chosen, not only for its graphical capabilities and suitability to our application, but for its cross-platform portability and programming language independence. As detailed in Appendix B, OpenGL is essentially permissible in any C-based programming environment and, as such, represents an ideal choice in trying to maintain platform independence.

   The reader is referred to the introduction to OpenGL in Appendix B for explanations of concepts and terminology, used extensively in this section.

   To streamline the design process, we shall employ the functions of the GLUT utility toolkit library, as discussed in Appendix B. However, the most common use for OpenGL and GLUT are in standalone dedicated graphics applications, and it will be necessary to modify the GLUT library to make it compatible with the VST host environment.

   Primarily, we want to give priority to the VST host system, so that music playback is not impaired due to costly graphic operations. In a normal GLUT graphics application, the environment is initialised and pointers to procedures that will handle the creation of 3D objects and the effects of keyboard input are passed to GLUT. Following this step, control of the application’s execution thread is transferred to GLUT, using glutMainLoop(). In the GLUT library, this procedure consists of an infinite loop, which, at each iteration, checks to see if the execution of the code to update the on-screen graphics is required (requested when the program calls glutPostRedisplay()) and checks the operating system’s message queues for pending input, executing the appropriate user-defined input-handling function. This repeats until the application, and hence loop, is aggressively terminated either by the user, through the OS, or by the program itself, through the calling of glDestroyWindow(), during one of the user-defined callback functions.

   To solve the problem mentioned earlier, MIVI must retain principal control of the execution thread in the VST environment. This involves digging into the GLUT source code and finding some way to avoid the infinite loop, in the file glut_event.c, so that calling glutMainLoop() will draw the screen, check the message queues, calling any appropriate callback functions, and then return. It would then be up to our plugin to call this function regularly enough to produce efficient and continuously updated OpenGL graphics.

   For us, commenting out the for loop statement would be sufficient, but to maintain backward compatibility, we move the body of the loop to a new function, glutMainLoopUpdate(), and also provide a call to this new function in the original loop, as illustrated in figure 3.2 (a) and (b). Now, other applications can continue to call glutMainLoop(), and receive the same functionality. We, on the other hand, can place a direct call to glutMainLoopUpdate() in the MIVIEditor::idle() function (code ref. 23). This idle function is called by the VST host, whenever both MIVI’s Edit Window is open (and, thus, as will our OpenGL window), and the host is unengaged in other, more critical, activities. In practice, this proves to be quite regular and frequent, and is more than adequate for our purposes.

   GLUT’s conventional standalone nature means that it is also not designed to create an OpenGL graphics window as a sub-window of another non-OpenGL application. On the other hand, GLUT is perfectly au fait with creating OpenGL windows subordinate to a parent OpenGL window. In this instance, the program need only call glutCreateSubWindow() and supply an integer, identifying the parent window, as returned by the original call to glutCreateWindow() to create the OpenGL parent itself. Deeper into the library, these two functions simply call a generic window creation function, __glutCreateWindow(), which optionally takes the integer, identifying the parent, as a parameter. If non-zero, the function extracts this parent’s window context handle (HWND) and assigns the new child window to it.

   Thus, it is a fairly simple process to modify the existing code to allow us to simply provide the HWND handle of the VST host application’s window (given as the *ptr pointer in the MIVIEditor::open() procedure – code ref. 24) for use as the parent. The code is this time located in the GLUT source code file glut_win.c. Once again, we should be wary about directly editing the code. We would need to change the calling arguments of procedures glutCreateSubWindow() and __glutCreateWindow(), so that they can take an HWND instead of an int, but doing so will interfere with the inner workings of the library.

   Instead, we create a new function, glutCreateMiviWindow(), which will mimic the actions of the original two GLUT procedures, using the HWND directly in the Microsoft Foundation Classes (MFC) call to create the window, as illustrated in figure 3.3 (a) and (b). Note, however, that we yet retain the functionality of the glWindow integer window identifier, so that we can pass it to the original glDestroyWindow() function at closing.

preventing harmful
OpenGL destruction

   Another modification to the original procedure is warranted, as we want the window to omit the close button, which allows the independent destruction of the OpenGL window. Were the window to be arbitrarily destroyed, havoc would follow when MIVI, ignorant of its destruction¹³, next tried to execute an OpenGL command.

performance
considerations

   The performance bottleneck of our system will be the frequency of idle() calls (code ref. 23) from the VST host. This can be observed by placing MIVI in SPIN_MODE, (where the model is simply rotated on all axes by a small amount each call) then clicking and holding down a button on the plugin’s interface toolbar. In doing so, focus is temporarily usurped by the plugin and rendering performance is dramatically increased. The only net effect, however, is on the frame rate – the frequency of scene refresh. The actual time available for scene rendering, per idle() call, is still enough, in normal operation, for complex scenes and graphics operations.

   Furthermore, when a plugin requests MIDI data, the VST host is obliged to provide it with calls frequent enough to enable its real-time processing, and thus, in combination with double buffering (described in section 4.1.3), we are assured of acceptable scene fluidity.

   The relative scarcity of idle calls is mainly due to the number of operations the VST host must itself perform, before idle time is available. Regardless of the absence of noticeable performance increases, cheaper and more efficient approaches have been employed, where available, in order to prevent extra loads on the host and improve performance if, at some stage, it can bestow more processing time on the plugin. Sometimes, though, when a lot of the host’s work is involved with digital signal processing (DSP), as is often the case with modern audio sequencers, this frequency will become too sporadic. However, this problem concerns all VST plugins, and is only effectively addressable in the host.

   Some plugins are known to compound the problem. Although there is a large movement, in the music industry, from hardware to software, processor-intensive programs, such as DSP, nearly always execute slower, resulting in longer latencies between input and output. Because the two methods are often used together, the performance advantages of hardware are usually compromised when the VST host is obliged to delay its response in order to synchronise outputs with software components. The net result is that there is a delay of up to several milliseconds before audio events are heard.

   One further deficiency in the usage of OpenGL with VST is worth mentioning. When the user closes the plugin, the host promptly ceases execution of any thread in the plugin. In most circumstances, this takes place without complaint. However, on the rare occasion when the plugin is caught making an external call to a GLUT library procedure, notably glutMainLoopUpdate(), such abrupt execution produces errors. Usually, the result is merely an error message from the host but, from time to time, it becomes necessary to restart the host following this error, before the plugin can itself be restarted. This could present another advantage of bringing the GLUT functions in house, where their calls could be more closely guarded. However, these functions proceed to call the core GL library’s procedures, and it remains to be seen if the problem wouldn’t simply propagate to the next layer of abstraction.

   Double-buffering simply allows the next frame’s graphics to be drawn in a second buffer, while the contents of the first buffer are displayed, thus minimising (often halving) the time taken to render each frame. To take advantage of this, all we have to do is tell GLUT that we support double-buffering (GL_DOUBLE) and call glutSwapBuffers() after each frame is ready for display.

   Colours for computer screens use an RGB palette, with each colour defined by respective amounts of red, green and blue. These components are then mixed in a similar way to light rays of corresponding wavelengths. OpenGL actually supports an extra dimension of ‘colour’ – the alpha channel (GL_RGBA) – corresponding to degrees of transparency. When combined with alpha-blending (GL_ALPHA), it simply blends a foreground object’s colour with that of the object(s) behind it, to an extent defined in the program. As we will soon see, such operations are, unfortunately, not available to us in our current implementation.

   Z-buffering refers to the third axis in Cartesian co-ordinates – that of depth. Previously, 3D scenes containing multiple objects had to be rendered from the back to the front. Otherwise, distant objects would appear on top of closer ones, contrary to reality. Were the object to be rotated 180° around the vertical, what was previously must be rendered in reverse order. The intervening angles additionally present even more of an ordering challenge.

   By employing a z-buffer, the depth of each object is stored (ie. buffered) in memory, so that we can request the frequent calling of GL_DEPTH_TEST, which will automatically work out the correct rendering order to maintain consistency. Unfortunately, this OpenGL function prohibits us from using elaborate quality improvements such as anti-aliasing.

   Were we to have more time, it would be possible to modify our code to implement our own dynamic depth-sorting function, which would nullify the need for GL_DEPTH_TEST and enable the use of GL_BLEND. This can be achieved by storing 3D objects in our own buffer, which we manually re-sort, using an efficient sort algorithm such as QuickSort when the scene changes, before sending them to OpenGL – a process known as alpha-sorting. As one might expect, the lack of blending capabilities also prevents other uses of partial transparencies.

   Instead, more global functions, such as the ‘camera’ rotation and zoom, are handled in the glDisplay() procedure. The parameters for such operations are stored in global variables, which can be set or modified anywhere in the MIVI scope.

   Because of OpenGL’s relative vector-based architecture (see Appendix B), implementing zoom is almost as simple as a single translation backwards. However, to maintain an efficient scene, OpenGL requires programs to specify a viewing range. Setting up the viewing matrix through gluPerspective(), requires you to specify the field of view (FoV), the pixel aspect ratio and both a near and far clipping distance. Objects (1) outside the FoV, (2) before the near clipping-distance or (3) beyond the far clipping-distance are, thus, simply not rendered. We must make sure that the translation to account for zoom does not place the ensuing instrument beyond the far clip distance. Thus, we use the zoom variable in setting this limit.