Everything You Ever Wanted To Know About System Exclusive (Part 1)
(But Were Too Afraid To Ask!)
Martin Russ kicks off our new series by recapping on some MIDI basics before starting to demystify the secrets of MIDI's System Exclusive commands.
Soap opera script writers are notorious for deliberately writing loose ends into their plots - it gives them a way out if something unexpected happens, like one of the cast leaving! Writers aren't the only ones who do this. You might be surprised to learn that the designers of the MIDI standard also left some loose ends on purpose. The idea is that by leaving in a few loopholes you allow the evolution of the standard into unexpected areas - and with the high rate of development in musical technology, they were right to expect the unexpected!
In order to find the loose ends in MIDI, we need to know something about MIDI itself.
MIDI has two basic types of message: Channel messages are used to convey information in the 16 permitted channels, and consists of messages describing Note Ons and Offs, Pressure, Pitch Bend, Program Changes and Controllers - all related to actual live performance data. In contrast, System messages are not associated with a channel, and so can be received by any piece of MIDI equipment connected to the network. This global information deals with the overall timing and control aspects of MIDI: the Timing Clock and MIDI Time Code messages; Start, Stop and Continue commands; as well as Active Sensing and Tune Request messages.
MIDI information is transmitted in the form of messages - each message normally consists of a status byte (1 byte = 8 bits), usually followed by one or more data bytes. The status bytes are indicated by setting the most significant bit (MSB) of the byte to 1 (the most significant bit of the byte is set to 0 for all data bytes) - this means that the basic interchange level of MIDI is really only 7 bits. The status bytes for Channel messages are split into two nibbles (4 bits): the high nibble indicates the type of status message, whilst the low nibble indicates the MIDI channel for which the data that follows is intended.
There are seven fundamental sorts of Channel message and these are assigned high nibble values from $8 to $E in hexadecimal notation. (Remember: the $ dollar symbol is just there to remind you that the hexadecimal numbering system is being used, as opposed to decimal). The $F high nibble value is reserved for the System messages.
System messages use the same two nibble split, but in this case the high nibble is always $F and the low nibble is used to indicate the type of message - there are thus 16 possibilities and they are split into three parts. $F0 indicates a special purpose message called 'System Exclusive', and the details of this will form the bulk of the rest of this series.
$F1 to $F7 are the System Common messages, used for controlling and enhancing other functions. The $F2 Song Position and $F3 Song Select messages are used to control drum machines and sequencers, for example. MIDI Time Code (MTC) sends its timing information as $F1 messages every quarter frame and additional information by using $F0 System Exclusive messages - all for film and video synchronisation purposes.
Analogue synthesizer users might employ the $F6 Tune Request message to activate built-in oscillator tuning routines in their instrument before playing. $F7 is used to indicate the end of a System Exclusive message, of which we shall see more later!
The remaining System status bytes, from $F8 to $FF, are called the Real-Time messages and they deal with the synchronisation of the MIDI system. $F8 is the MIDI Timing Clock, a complete message in one byte - sent at the rate of 24 clocks per quarter note. $FA, $FB and $FC are the drum machine controls - providing remote Start, Stop and Continue commands. Active Sensing uses the single byte $FE message to indicate if a remote keyboard or MIDI guitar is still active (ie. still transmitting MIDI data), and can be used to mute expander modules if the connection fails.
Finally, $FF is the System Reset message — a panic button designed for use only in cases of dire emergency, since it restores the equipment to its power-up state. In reality, most manufacturers seem to tactfully ignore this 'kill or cure' message, with the exception of some European companies obviously eager to prepare for 1992.
When I was a kid I used to read stories of the days when maps of Africa had large areas marked 'unexplored'. The 'undefined' parts of the MIDI map have the same fascination - but where are they? All the Channel messages seem complete, so perhaps a deeper look is needed - the status bytes may all be used, but within the data there are more possibilities. Simple data messages which convey just values will not offer any room for expansion, so the Note Ons and Offs, Pressure, Program Change and Pitch Wheel messages are not worth exploring. This leaves just the Control Change message, and it is here that we find our first 'undefined' region!
MIDI Controllers are defined by the first data byte following the $B nibble of the status byte. The 128 possible Controllers were originally organised into three basic types: Continuous Controllers (modulation wheels, pedals etc), Switches, and Mode messages to control note reception conditions. However, there has been some reshuffling and MIDI Specification 1.0 has complicated this situation. Continuous Controllers are now divided into 14-bit (Controllers 0 to 63) and 7-bit types (64 to 95). Switches are now assigned as a special case of a 7-bit Continuous Controller. Not all Controller numbers are assigned to specific devices like pedals, wheels, etc. In fact, most aren't. Those that are seem to be driven by historical precedent, although the MIDI ruling bodies are gradually imposing some order on things. I don't really count the Unassigned Controllers as areas ripe for exploitation, since consistency is what is needed here - you want a volume pedal to control volume, not LFO rate.
Mode messages, which you might have expected to find in a message all their own, are instead hidden in a quiet backwater - Controllers 122 to 127. The Mode messages control the operation of the MIDI peripheral which is receiving the data: Mono On/Off, Poly On/Off, Omni On/Off determine the way it responds to incoming channelised information such as note or pitch wheel data. Local Control On/Off (122) sets the link between the keyboard of a synthesizer-type device and its internal sound generators - if it is Off then you can play the keyboard and use the MIDI Out, but the only way to produce sound is by feeding note information into the MIDI In.
The final Mode message is All Notes Off (123) - a sort of catch-all designed for when you halt playback on a sequencer whilst some notes are still playing, although this message often seems to be used to indicate when all the keys on a keyboard have been released. This has the unfortunate effect of annoying people who use expanders for their source of sounds, since every time they release the keys on their master keyboard, the expander stops playing, even with the sustain pedal held down! All Notes Off used in this way is not illegal as far as the MIDI Specification goes, nor does it seem to be completely legal either, but the problems it can cause serve a very useful purpose in keeping MIDI consultants like me in work!
The astute amongst you will have noticed that I omitted Controllers 96 to 121. This group is split into two sections - the first covering 96 to 101 is the Registered and Non-Registered Parameters section, where you can assign up to 256 parameters to MIDI Controller messages without using up valuable 14- or 7-bit Controller numbers. In fact, you can use just Controllers 06 and 38 to control the MSB (most significant bit) and LSB (least significant bit) of all these extra parameters. The remaining Controllers, from 102 to 121, are officially undefined. (Except for Controller 121, which is used for the Reset All Controllers message - a sort of All Notes Off command but for Controllers instead of notes. This is not documented in my copy of MIDI Specification 1.0).
If you think the story so far is complicated, then you would be right. MIDI is still evolving, and the Controllers are definitely one area where developments are having a direct effect on the assignment of the commands. Because of this, treat what I have said so far as an outline - my information could be out of date by the time you read this. Even so, this brief look through the Controllers should have shown that the MIDI standard is capable of being improved and extended within its own framework.
The System messages are more restrained in their response to change. Outwardly, only one addition has been made: MIDI Time Code Quarter Frame messages are now sent as $F1 messages.
$F4 and $F5 remain as undefined System Common messages, and $F9 and $FD are undefined System Real-Time messages. All the big changes and additions are hidden deeper inside just one message - the System Exclusive $F0 message.
System Exclusive is rather like the blank pages at the end of a book marked 'Notes' - it is a place where the user makes his own mark on the book instead of just reading it. System Exclusive is a means of removing some of the fixed standardised nature of MIDI and letting individual manufacturers and designers use it for their own purposes. The bare minimum of restraint is imposed by the MIDI Specification - everything else is left to the designer of the MIDI device. This is the loophole which really counts in MIDI. It is the key to all sorts of expansion and improvement - turning MIDI from just a means of sending notes from one instrument to another into a powerful tool for hi-tech music. The remainder of this series will concentrate on a detailed look at some of the possible applications of the System Exclusive message.
It takes a brave person to start a series on System Exclusive and then talk about everything except that! As limited recompense, I offer an extension to the mix of theory, practical examples and printouts from software used in the 'Practically FM' series [SOS May 1988 to October 1988]. This time the software is an integral part of the articles.
Over the course of this new series, each episode will present a piece of software on 3.5" disk (for the Atari ST initially: users of other machines will have to persuade me to convert the programs!) which illustrates and expands upon the ideas in the text. These are available from SOS for £7 per disk. At the end of the series, you will have a valuable and varied collection of software tools for use with MIDI, especially in the context of System Exclusive.
For anyone who prefers not to use computers, I will also try to show ways in which hardware devices can be used to achieve similar results. Last month's article about the Lexicon MIDI Remote Controller (MRC) is an excellent example of the way that a specialised piece of equipment can be used to control MIDI without needing to look or behave like a computer.
A good entrance is important in show business, and so this first article has two pieces of software to accompany it - one old, the other brand new. MIDI Channel messages can be viewed with the MIDI Channel Scope program (SOS Software Disk 06) whilst the System Real-Time and some useful testing messages can be sent with the MIDI Utility program.
Alternatively, many master keyboards and some sequencers allow you to decode Channel and System messages, while the Lexicon MRC allows many similar functions to the MIDI Utility program, as well as much more control over MIDI Controllers. (See 'Alternative Ways' box for further details.)
If you currently use MIDI purely for sending Note On/Offs and Program Changes to your synths, sequencers, and signal processors, then the rest of this series will come as a shock. You will be interacting with MIDI instead of just using it, so now is as good a time as any to see and learn what is happening...
Use the MIDI Channel Scope or another monitoring device to observe what is happening on your MIDI network. If you aren't actually playing something then you should see very little action. However, some drum machines and sequencers send out MIDI Timing Clocks all the time, so you might see these, and some early synthesizers send out Controller data even when the controller isn't doing anything. The Korg Poly 800 is notorious for continuously sending out messages telling you that its Pitch joystick and Modulation joystick are both in the centre position! Active Sensing can also appear in a 'quiet' system if your master keyboard transmits it. MIDI Time Code signals tend to be restricted to MIDI systems which aren't running ordinary MIDI Clock based synchronisation.
Once you play an instrument, you should see activity on the appropriate MIDI channels - Note Ons and Offs, Program Changes, etc. Mixed in with these you will still see the Clocks and Active Sensing messages as before. Aftertouch and other Continuous Controllers are interesting to try, because they send large amounts of data - usually a message for each change in value. (You really do get 128 messages one after another if you move a 7-bit Controller from one end of its travel to the other extreme.) Some synthesizers can also be confusing: early models of the Yamaha DX7 Mk1 (serial numbers 1001-24880 and 25125-26005) and KX1 (serial numbers 1001-1105) implemented Aftertouch as a Controller instead of as the now standard $Dn message (where 'n' is the MIDI channel).
Program Change messages may also be accompanied by lots of additional data: some synthesizers send a dump of patch information when you select a different program - the MIDI Channel Scope will not show this (but the MIDI System Scope program, available later on in the series, will!). Note messages may also appear confusing if you are expecting to see both Note Ons and Note Offs - the convention has quickly become one of using only Note On messages, with Note Offs instead being indicated by a Note On with a velocity of zero. This format allows you to send long sets of note number and velocity data bytes without needing to send any status bytes, and is known as 'Running Status'. Some 'home grown' and public domain software can be confused by this short cut. There is a test in the MIDI Utility for checking this - if you hear sustaining notes then your synthesizer or expander does not like Running Status, and if your drum machine does not start playing then it doesn't either!
Notice that the Controllers are channelised, ie. the data is sent over a specified MIDI channel. Remember this when you find that a Pitch Wheel is only bending one instrument. The all-important Channel number is part of the status byte. It is the low nibble, eg. the 'n' in $En (a Pitch Bend status byte).
One of the most demanding Controllers is Poly Pressure ($An), as seen on the affordable Ensoniq SQ80 and EPS keyboards. This generates messages for each note which is being pressed down - so it is probably a good thing that the human hand is restricted in the speed at which it can play notes and then apply controlled pressure to them!
This first episode has been an introduction to the aspects of MIDI which form the background to the activity using System Exclusive that will follow in subsequent articles. A knowledge of what is happening, and the ability to interpret the often obscure response of real instruments, will be of great value over the coming few months. I will supply some theory, practical examples, hints and some useful software tools, but you will need to supply some effort too in order to use them effectively!
The MIDI Channel Scope and MIDI Utility programs referred to in this article are available on 3.5" disks for the Atari ST only at present.
£7 per disk.
SOS Software, P0 Box 30, St. Ives, Cambridgeshire PE17 4XQ. (0480) 61244.
Feature by Martin Russ
Previous article in this issue:
Next article in this issue: