Talking MIDI (Part 7)
Part 7. In this penultimate episode Jay Chapman explains the usefulness of MIDI's all-important System Exclusive messages.
In this, the penultimate episode of his illustrious introduction to the MIDI protocol, consultant Jay Chapman discusses the remaining pair of System Common Messages known as the System Exclusive Message and the End Of System Exclusive Message.
If you read last month's magnum opus you will know that the most significant nybble of the status bytes of both the System Common and System Real Time Messages is &F ('&' prefix indicates hexadecimal, '%' will indicate binary) with the next bit selecting between Common (%1111Oxxx) and Real Time (%11111xxx). This easy distinction is a useful one as we shall see next month because the Real Time Messages can appear anywhere - even in the middle of other messages - without MIDI getting lost!
Of all the wonders that MIDI offers, System Exclusive combines the best of power and pain, joy and despair or even love and hate! Almost everything that we have looked at so far in our pursuit of the secrets of MIDI has been to do with performance control rather than the actual synthesis of sound. The System Exclusive feature of MIDI allows us to bare the soul of all MIDI synthesizers provided that the manufacturers have taken advantage of the possibilities that become apparent - but this means that a whole new complex world of fine detail has to be understood and dealt with. Hopefully all will be explained below!
The difference between the System Exclusive Messages and the System Common Messages is explicit in their names. The Common Messages are for all devices to respond to if they can eg. if a Song Select Message is sent out then all sequencers, drum machines and computers listening on that MIDI daisy chain should select the specified song. If a drum machine doesn't have a multi-song capability it will hear the message but will simply ignore it. All devices on a given daisy chain will also hear System Exclusive Messages but will ignore them unless they decide that the message is exclusively for them. At the MIDI level, exclusive means 'for devices made by the manufacturer specified in the message'.
So, if a synthesizer, a computer or whatever, sees a System Exclusive Message with a manufacturer's identity it recognises, it can grab the message and process it. The message will carry on down the daisy chain, of course, so it may well be that other devices also take a copy if they so wish. The point here is that the message is not MIDI Channel-based, so every device can be addressed simultaneously like Common Messages. Only those devices interested in the specified manufacturer will probably know how to interpret the message as we shall see, and so the message is aimed exclusively at such devices. Naughty computer programmers (like me, heh heh!) can choose to disregard this exclusivity and grab the messages to play with anyway! If you're wondering why anybody should want to play with such messages, see the 'Dumping' and 'Editing' sections below.
The MIDI Specification tells us the format of these messages but it doesn't tell us anything at all about the content ie. we are not told what can appear in (all but one of) the data bytes, nor are we told how many data bytes there will be. The format is: &F0 &ii &dd &dd .. .. .. &F7. The &F0 is a status byte (%11110000, ie. the left-most bit is on) and it is followed by any number of data bytes, the first of which (shown as &ii above) is defined in the MIDI Specification and identifies the manufacturer of the device sending the message - see Table 1. Usually there will then be a number of data bytes (shown above as &dd).
Finally, the System Exclusive Message is terminated by the EOX flag (End Of system exclusive) &F7. If the EOX is not present (and manufacturers often omit it because it is effectively optional) then the next status byte terminates the message. It may appear that this message termination business is a bit ragged: for example, what happens if there is no EOX and no status byte is transmitted for a while, does the receiver wait, totally unable to action the message? Well, the answer is no, at least not if the manufacturer's software department is any good! Since the EOX is optional, the software will process the message as soon as the right number of bytes have arrived.
A good question - have I got a good answer? Yes! Essentially these messages are to be used as the individual manufacturer sees fit. The main uses are for voice program data, device specific control parameter changes (rather than MIDI controller changes - though the two get mixed up sometimes), configuration data, sequence data, waveform table data, sample data and requests for the sending of such data. Let's discuss an example or three and try to explain some of these possibilities.
The shortest System Exclusive Message I have ever seen was only two bytes long ie. just the status byte and the manufacturer's identification number. This message was transmitted constantly at short intervals (cf. 'Active Sensing' next month) when the synthesizer had nothing else to do. As an analogy, consider walking into your local pub and announcing, loudly enough so that everyone will hear, "Hello, I'm a Chapman" (substitute the name of your manufacturer) every few seconds. Apart from giving moral support to any other synthesizers of the same make slaved on the relevant MIDI daisy chain, I can't really see what this message was supposed to do - maybe I should have read the manual properly (hangs head in shame)!
Hang on a moment, don't go away... I decided to do what I'm always telling everybody else to do - so I just went and read the manual properly! The early DX7s had the mini-message just mentioned! The message was Yamaha's own version of Active Sensing; presumably they thought up their version before the MIDI standard was totally finished and therefore deserve kinder treatment and less of my sarcastic comments! Active sensing is what's known in the trade as a very good idea - but an absolute pain when you're writing MIDI software. The tome on Real Time next month will explain why.
In most cases manufacturers want to communicate between various devices of their own which understand each other to a greater or lesser degree. All the performance and common data, which different manufacturers' devices are likely to understand whatever the source, has already been catered for by the other MIDI message types. It is therefore unlikely that other manufacturers' products will understand any further details left to be sent out from this manufacturer's devices, since the various synthesizer internals are quite different! This is why the messages are exclusive to a single manufacturer where families of synthesizers have some hope of having something in common internally.
Unless a manufacturer only produces one type of MIDI device, and never intends to introduce another, there must be some way to exclusively aim the message at a specific type of device. For example, if we send out a DX7 voice program then the message will certainly be marked 'Yamaha' in data byte one and will therefore start off with the two bytes &F0 &43. Any Yamaha devices will perk up when they hear these bytes but not all should attempt to deal with the message. Another DX7, a TX7, or a TX816 module, or even a CX5 may all be able to make some sort of sense of a DX7 program but a DX100, DX27 or DX5 cannot.
So, we find that there is likely to be at least a second level of exclusivity within the data bytes of the message so that a device deals with a System Exclusive Message only if the manufacturer identification and any further 'device type' and/or 'message type' information all combines to specify a message that is going to be understood. Table 2 shows the first few bytes of several System Exclusive Messages from different devices both from the same and different manufacturers.
I'll admit that it's not readily apparent what byte, or part of a byte, identifies a message as being for or from a particular device. In fact, it often looks as though manufacturers were caught a little disorganised by the blossoming of MIDI and seem to have allocated messages in a slightly random manner! Provided that distinct message types can be recognised when required however, this need not worry us. Actually, messages for different devices can sometimes be identical eg. the Program Dump Request messages for the Prophet 600 and Six-Trak, both made by Sequential Circuits. If you had both of these synthesizers on the same MIDI daisy chain they would both respond to one request! The two dumps would be distinct of course since data byte two (nybble two of the relevant program dump message) is 2 for the Prophet and 5 for the Six-Trak.
Fortunately the dumps do not come out onto the daisy chain (where MIDI data arrives via MIDI IN and leaves via MIDI THRU) thereby causing something like the MIDI equivalent of my cooking - a slightly explosive mess in which you can't distinguish any of the original components! Figure 1 shows what data is actually flowing where: the dumps are actually transmitted via the two synthesizers' MIDI OUTs. Whatever devices you intend to capture the dumps must be connected to these OUTs. You can begin to see why MIDI routing matrix boxes are useful!
In certain cases one device will use what was originally another device's message, on purpose (this may well have been the case with Sequential above, of course). This is evident in the case of a TX7 working with a DX7 where the ' 1 Voice Bulk Data' message of the TX7 is exactly the same as the 'Single Voice Data' message of the DX7. This means that voice programs can be sent from either device to the other as required. Furthermore, since the TX7 stores voice function data both for its own and the DX7's voices, the TX7 sends out a 'function data' System Exclusive Message whenever a new program is selected on the DX7. In other words the TX7 responds to a Channel Voice Message with a System Exclusive Message. Interestingly the TX7 does not interpret all function data in exactly the same way as a DX7 (some 0 to 99 parameter ranges are 0 to 15 on the TX7 ie. a coarser, but still ample, degree of control).
We don't have to stick with sending just one voice program at a time by the way. On the TX7 we can send out all 32 program stores in one bulk dump. This sends 4096 bytes flying over MIDI which should take over 1 second to transmit; a TX7 actually takes about 5 seconds, I wonder why? A few seconds aren't the end of the world in any case... unless you use up 32 voices during one song! Generally speaking, this size of update would only take place at very infrequent intervals whether during a live performance or studio work. Even if you had four or five synths to update you would only need 20 or 30 seconds - hardly time for a quick fag behind the guitarist's Marshall stack!
Having described whole voice programs, we come down in size a little to consider individual parameter changes. Literally this means the updating of parameters as knobs, buttons or sliders are moved on the front of a master controller (or simulated in the bowels of a master computer!). One of the main reasons for dealing with individual parameters is the same as that for whole voices: if an expander module has no keyboard (by definition!) and few front panel controls, then how is it to be controlled from the master keyboard or computer?
Many expanders have just sufficient front panel controls to select their MIDI receiving channel and which voice program is to sound. With luck, voice programs can be dumped to and from the expander as described above but no editing could be done without dumping a voice into a computer (or the normal 'keyboard' version of the synthesizer that the expander is based upon), editing it there, and then sending the whole (edited) voice program back. If we can send individual parameter changes then this process becomes much more efficient as we shall see.
It is not always voice parameters that need to be communicated, however. If a synthesizer has some mode of operation that was not foreseen by the designers of MIDI, such as a 'doubled mode' or a temporary key transposition, then no explicit control facilities will exist in MIDI. We can therefore invent messages of our own for this purpose within the System Exclusive Message format. An example of this is demonstrated on the Casio CZ101 and CZ1000 where it is possible to set the range (0 to 11 semitones) over which pitch-bends are effective.
As I mentioned earlier it is possible to use the MIDI Control Change Message (status byte &Bn) for some parameter changes. On the Casios, for example, Controller 1 is used as a switch to specify Vibrato On or Off (&Bn &01 &7F and &Bn &01 &00 respectively). When assessing the relative merits of the two methods I think manufacturers consider three points: how long the message needs to be (one Control Change Message per varying parameter byte is required remember), whether it is conceptually a controller, and the fact that often it's of little consequence which method you choose!
Control Change Messages incorporate a MIDI channel number and so no confusion will be caused if you have two identical expanders - just assign them to different MIDI channels. Sometimes problems will arise because a master synthesizer uses a Control Change Message differently than a slave. For example, quite a few synthesizers recognise Controller 1 as being the Modulation Wheel (it's in the MIDI 1.0 Specification) so the Casio's Vibrato messages will be interpreted as modulation fully off to fully on in one step!
On the other hand System Exclusive Messages cannot be misinterpreted by any synthesizer not made by the given manufacturer! Also, if some sort of complex multi-parameter message is required, the MIDI overhead is likely to be less with a System Exclusive Message. Such multi-parameter messages include what I have referred to as 'configuration data' which specifies logical connections within a synthesizer and the state of internal bit-maps (such as the 'Switch Map' on the Juno-106) and so on.
In any case, it is possible these days to pick up a message in one format and convert it to a different type of message for consumption by devices further down the daisy chain, thus resolving all 'identity' problems. See the Yamaha MEP4 review elsewhere in this issue for an example of a dedicated MIDI processor - the same sort of thing is quite easy to do with a microprocessor system inserted into the daisy chain in a similar manner.
Having put these two in the same section heading, I'm not at all sure I should have done so! Both Waveform and Sample Data are in some sense part of the voice program parameters that we have already discussed, since they contribute directly to the character of the sound the synthesizer will produce. The reason that I have grouped them together is because of the potential applications of graphics that spring to mind. Since most of your average sampling synthesizers don't have a Visual Display Unit attached (although the forthcoming Roland S-50 keyboard sampler can drive an RGB display and respond to an ASCII keyboard or even to a mouse - sensible Mr. Roland!) it can be quite frustrating trying to work out what the hell the sample you are playing with actually looks like!!
Viewing and manipulating the sample/waveform via masses of computer power and scintillating colour graphics is a wonderful idea and makes life far more satisfying than three buttons and two sliders on the front panel could ever do... Before we can get to grips with this marvel of hi-technology music, we need to get the sample into the computer and the result back again. For many samplers this will require a special, non-MIDI, interface which usually operates in parallel (rather than serial like MIDI) at much higher data rates than MIDI can manage.
It is possible to use MIDI for sending samples though. In fact, this has the quite distinct advantage that you don't need to buy an expensive, and even worse, non-standard interface. If you want to send two seconds of sampled sound consisting of 40,000 12-bit samples over MIDI it will take approximately:
2 (secs) x 40000 (samples per sec) x 12 (bits per sample) / 3000 (bytes per second) x 8 (bits per byte)
ie. 40 seconds with a tail wind to transmit the 120,000 bytes involved. Since MIDI has no error-correction capability itself, there is a chance that not all of the bytes will either arrive or have the same value as when they set out even if they do arrive!
This introduces the need for some computer-based network protocol techniques and you will often find some sort of error-detection bytes tagged on to the end of the data in the relevant System Exclusive Messages so that if the data doesn't arrive correctly, the receiver can find out and ask for a retransmission. The other thing to do is not try to send all 120,000 bytes in one message - which is known technically as hara-kiri! Instead, the sample data is sent in medium size blocks which are each checked (typically via a checksum calculation — see a later article!) individually. Thus if one or two blocks get clobbered on the way over we don't need a retransmission of 120,000 bytes but only perhaps a couple of thousand. If the receiving end has some decent software and the transmitting end has a decent System Exclusive-based message protocol, then the receiver can organise any necessary retries for you and you won't even know that there were problems!
The Ensoniq Mirage sampling keyboard has a comprehensive protocol and uses MIDI to transmit samples as described. System Exclusive Messages are available for Dump Requests which include start and end addresses (within the Ensoniq's sample memory) so that the samples can be transmitted in checksummed blocks. There are also ACK and NACK (ACKnowledge and Negative ACKnowledge) messages so that the Ensoniq can inform any device transmitting samples whether (ACK) or not (NACK) data arrived safely.
Having mentioned the subject a couple of times above, let's just say that editing - whether it be of 'normal' voice program parameters, waveform data or sample data - via a decent computer system (colour graphics, keyboard, mouse, windows, icons, etc) is in a different universe from trying to do the same from the front panel controls and a two-character LCD!
Typically one would handle such an edit by pulling the full set of parameters/data into the computer via MIDI so that all the necessary information to display the voice program/sample to be edited is to hand. As the user interacts with the display by altering parameters, or even actually altering the displays themselves, he gets instant visual feedback confirming his efforts. As each parameter is altered, the computer's internal model of the program/sample is amended; at the same time the relevant System Exclusive Message is used to instruct the synthesizer to update its internal values as well so that it not only matches the state of the editing in the computer but can also be heard by the user so that he has aural feedback as well!
Not, as you may think, the province of your local council's Domestic Refuse Operative but another major use for all of the data we are throwing around the place via our System Exclusive Messages!
Remember the bad old days when synthesizers couldn't remember anything at all? When Rick "mine's a pint" Wakeman needed 20 synthesizers, and reprogrammed 10 MiniMoogs at a time with his nose whilst playing... Ah, those were the days of true keyboard heroes — where are they now? Oh, sorry, I had a quick attack of nostalgia - back to the point.
And the point is that we now have synthesizers with 32, 64, even 128 voice program memories and more; we have sample stores of a megabyte and sequencers of similar gargantuan proportions. We have voices like: 'Piano 1', 'Piano 2', 'Piano 3 with parameter 42 tweaked a bit as it sounded nice'. We even have samples like: 'Orchestra 5', 'African cow pat squelch', 'Beans Meanz' (heh heh), 'The sound that Dad made as he tripped over the cat after 14 pints of homebrew' etc, etc. But what does it all mean?
Well, practically, it means that we still manage to run out of however many storage slots we do have. So what we need is a comprehensive voice library facility. Now a few hundred voice programs at 200 bytes-worth of parameters a piece requires something in the order of a million bytes - which is only a couple of floppy disks at most! Applying the full power of our friend the microcomputer to the problem allows us full indexing and cross-referencing of voices for a whole set of different synthesizers.
That's enough on System Exclusive. Well, enough to give you a flavour of why such messages can be so powerful and so useful. Thanks are due to Graham Hinton of Hinton Instruments for supplying the extensive listings of MIDI details for various synthesizers. These form an Appendix in the manual for his MIDIC interface which I read very carefully before writing this article! (By the way, the MIDIC interface will be reviewed in detail next month.)
Feature by Jay Chapman
Previous article in this issue:
Next article in this issue:
mu:zines is the result of thousands of hours of effort, and will require many thousands more going forward to reach our goals of getting all this content online.
If you value this resource, you can support this project - it really helps!