Having looked at an overview of MIDI last month, the System Exclusive section of the System messages seemed to be the major place left open for real future expansion, apart from the other undefined messages. The best part about System Exclusive is that the MIDI Specification designers cleverly defined just enough to make sure that the few simple rules would be used by everyone, whilst leaving enough leeway for all sorts of unanticipated future uses. There is rather more to this than just a way of ensuring that the MIDI standard does not go out of date - System Exclusive is more like a complete standard in its own right; hence the need for a series like this one!

THEORY: SYSTEM EXCLUSIVE

But wait - all this from only one message status byte: $F0? Yes. Unlike all other MIDI messages, the System Exclusive message is not a fixed length of one, two or three bytes, instead it is 'packeted'. A data packet is similar in some ways to the packet that you eagerly wait for the postman to deliver - the valuable and delicate stuff is inside, but the outside is there to protect the contents and tell the postman where to deliver it. A System Exclusive message is sent in the form of a block of information surrounded by head and tail blocks, which indicate where the data starts and stops, as well as giving additional information about the block. This is similar to the arrangement used by trains: the engine is at the front, the guard's van is at the end, and the number of carriages in between depends on how many passengers are expected.

The basic format of such a packet is very simple and is given in the MIDI Specification. The Head information consists of two bytes - the System Exclusive or SYSEX status byte, followed by a data byte which is used to indicate the user's identification. The user in this case refers to the designer of the particular data block - usually the manufacturer of the MIDI device, eg. Yamaha, Roland, Akai etc. The data byte is often called the Manufacturer's Identification, but this is only partly true, as we shall see.

The rest of the main block of information is not specified! The contents are left entirely to the manufacturer; although in practice most use a few variations of a standard format. The Tail information is much simpler - it consists of a single status byte, called the End Of Exclusive or EOX. The resulting packeted message thus looks like this:

$F0 SYSEX
$ii Identification
[block of data]
$F7 EOX

Notice that the two status bytes surround the data bytes within - in fact, the EOX ($F7) status byte can be replaced with any other status byte except for a Real-Time message, although I have never seen this used. Real-Time messages can occur at any time - even inside the block of data (this is one reason why the Real-Time messages are restricted to a length of one byte; it makes the task of decoding MIDI much easier). So Timing Clock messages ($F8) or Active Sensing messages ($FE) can occur inside the block without terminating it - any other status message will have the same effect as an EOX byte. The setting of the most significant bit (msb) in the byte becomes very significant here: in status bytes it is set to 1, whilst in data bytes it is always 0. Thus, the packet is shown by the presence of 1's in the most significant bit of the bytes. The end of the packet is shown by a 1 in the most significant bit - the msb of the high nibble of the byte (one byte = eight bits; one nibble = four bits) - and a 0 in the 3rd bit from the right: the msb of the lower nibble of the byte. Real-Time messages can be distinguished because they have 1's in the msb's of both nibbles.

The msb of the status byte is thus used as a flag to indicate the edges of the data, whilst the data itself is limited to only the lower seven bits of the byte. This inconvenience is the price paid for making the packets easy to detect and decode. With data limited to values between 0 and 127, some manufacturers split data into nibbles, but most choose to arrange their data into the lower seven bits. If data is not restricted to seven bits then it will be interpreted as a status byte and action taken accordingly, depending upon whether it looks like a Real-Time message or not. Some manufacturers (Emu Systems, for example) used this approach in the early days of MIDI before the standard was really fixed, but more recent implementations have stayed within the specification.

IDENTIFICATION

Only one data byte is defined in the MIDI Specification - the first one, the Identification byte. Originally designed to indicate which manufacturer's instrument the System Exclusive message was coming from, it has been extended to cover other uses as well. The basic idea behind this byte is to make sure that a System Exclusive packet is only accepted by instruments which are capable of understanding it - usually other instruments made by the same manufacturer. Thus a Korg M1 can transmit to another M1, but a Yamaha DX7 will ignore the information because the identification byte is the one for Korg, not Yamaha. Lists of the manufacturers' Identification numbers assigned by the International MIDI Association are regularly published: a separate panel details some of the current ones.

An identification byte of $43 (hexadecimal; 67 decimal) would indicate a packet from a Yamaha instrument, but it also indicates something else: it tells the machine that the data bytes that follow conform to the Yamaha format. As I said earlier, the MIDI Specification itself does not lay down any format for those data bytes which follow the identification byte - the arrangement is left entirely up to the manufacturer. In practice, each manufacturer has produced their own format specification to which their own instruments adhere. Some of these are similar, but some use novel ways to format the data.

The most often used basic format consists of some 'sub-status' bytes, which are used to indicate what sort of dump follows, what machine it is destined for, how long the data is, etc. The block of data then follows, usually with a checksum at the end. The format of the information in the data block is entirely machine dependent - the only restriction being the seven bit limit. The format takes the following form:

Sub-Status
Number of Data bytes [block of data]
Checksum

Combining the Header and Tail of the System Exclusive message format, we have a rough guide to what a non-specific packet might look like:

$F0 SYSEX
$ii Identification
ss Sub-Status
nn Number of Data bytes
[block of data]
cc Checksum
$F7 EOX

The checksum is a means of determining if the data transmission has been achieved correctly. By summing the data bytes within the block of data, a number is obtained which will vary if any of the bytes are corrupted. By checking the number transmitted against the sum of the data bytes which arrive, a receiving MIDI device can discover if the transmission contained errors.

So, by treating some of the other uses for System Exclusive packets as if they were done by manufacturers, the MIDI standardisers have neatly integrated expansion possibilities into the system of assigning numbers. The first 31 numbers after zero (which is not assigned), from $01 to $1F (1 to 31) are assigned to American manufacturers, with Sequential at $01 because they were one of the founders of the MIDI standard. $20 to $3F (32 to 63) are assigned to European manufacturers with Bontempi first in at $20! Numbers $40 to $4F (64 to 79) are all assigned to Japanese manufacturers, with Kawai in the $40 top slot. The definition of a 'manufacturer' for these purposes is a company which produces a MIDI device that implements System Exclusive. Interested parties can apply to the MMA for a temporary number, and it will be confirmed if they produce a suitable product within 12 months. I do not know what will happen when the number of manufacturers exceeds the available numbering scheme!

EXPANSION POSSIBILITIES

The expansion is catered for by using the highest numbers available. So far three of these additional Identification codes have been defined: $7D is the Universal Non-Commercial identification byte, designed for use by Universities, researchers, etc. No formal formats have yet been defined for this number; $7E (126) is designated for Universal Non-Real-Time messages - and this is where things like the Sample Dump Standard and the initialisation part of MIDI Time Code are defined; finally, $7E specifies a format for the data block which is inside the packet, and this takes the following form:

$F0 SYSEX
$7E (126) Universal Non-Real-Time
<DeviceChannel>
<Sub ID 1 >
<Sub ID 2>
[block of data bytes]
$F7 EOX

This format is similar to many of the manufacturer's formats - additional devices, channels and sub-identification bytes precede a block of data in a specified sub-format. The Device Channel in this case is used to specify which unit within a MIDI network the data is designed for; remember that System Exclusive messages are not channelised, and so this provides a channel function with 127 separately addressable items available. Device Channel 127 ($7F) is the equivalent of an Omni message, ie. it will be received by all devices on the MIDI network. The Sub-IDs define the actual function of the message. Nine Sub-IDs are currently defined:

$01 (01) Sample Dump Header
$02 (02) Sample Dump Packet
$03 (03) Sample Dump Request
$04 (04) MIDI Time Code Setup
$06 (06) Inquiry
$7C (124) WAIT
$7D (125) CANCEL
$7E (126) NAK (No Acknowledge)
$7F (127) ACK (Acknowledge)

The first three are specific to the MIDI Sample Dump Standard, sometimes referred to as the MMA Sample Dump Standard (from MIDI Manufacturer's Association) in order to differentiate it from previous alternative proposals. The Sample Dump Standard was adopted in January 1986. The MIDI Time Code Setup message occupies the fourth slot, and is used to initialise the particular operation of a device which uses MIDI Time Code. The Inquiry message is used to get information about the instrument.

The final four IDs are used for handshaking - specifically for use with the Sample Dump Standard. Unfortunately, most manufacturers had already implemented their own handshaking messages within their own formats before these were formalised, and so there are many different alternative schemes used for handshaking purposes outside of the Sample Dump Standard. $7F is used for Universal Real-Time System Exclusive messages and the format is identical to the $7E format. This is used for the remaining Time Code messages in MIDI Time Code.

To summarise, there are two types of Identification code: Manufacturer and Universal. Manufacturer codes specify that the data which follows adheres to a particular manufacturer's format, whilst the Universal codes are used to indicate certain extended functions which can be used by anyone. The function of these extended codes is closely specified and forms an extension to the MIDI Specification 1.0. Included within the extended codes are the Sample Dump Standard and MIDI Time Code. This gives four broad areas of application:

System Exclusive
- Manufacturers
- Universals
- Sample Dumps
- Real-Times

SYSEX FORMATS

The MIDI Specification 1.0 uses a standard way of showing System Exclusive formats. The example shown above adhered to this - the sequence of bytes is shown vertically, with time running down the page. Indentation can be used to clarify the sections within the message. The official MIDI documentation encloses the labels for the bytes: eg. . This format is very useful for outlining the basic form of a System Exclusive message and is easy to annotate with comments on the right-hand side. Unfortunately, the vertical format has the disadvantage of occupying large sections of the page, and it can be hard to quickly assimilate the similarities between formats.

Because it is difficult to appreciate the correspondence between different formats and System Exclusive messages in a vertical form, I use a chart, which can be called a System Exclusive Chart, to indicate the format of messages. Instead of being vertical, it is horizontal, and uses hexadecimal or pseudohexadecimal numbers separated by commas to show the order of the bytes. The right-hand side is used for additional notes, much the same as in the official MIDI Implementation Chart.

For the overview of System Exclusive messages so far described, the chart would be as follows:

F0, ii, dd, F7 Manufacturer's Packet
F0, 7D, dc, s1, s2, dd, F7 Non-Commercial Packet
F0, 7E, dc, s1, s2, dd, F7 Non-Real-Time Packet
F0,7F, dc, s1, s2, dd, F7 Real-Time Packet

NOTES
ii = $00 to $7C: Manufacturer's ID
dc = device channel
s1 = sub-status 1
s2 = sub-status 2
dd = data

You will notice that the horizontal format suits the normal reading style, and that it is easy to see that all four messages start and stop with $F0 and $F7. This is particularly useful when describing the more complex formats used by a manufacturer, as the example System Exclusive Chart for the Lexicon MIDI Remote Controller (reviewed SOS March 89) illustrates (see separate panel).

There are several things to notice here: all the data is expressed in hexadecimal form; the chart is split into sections covering the reception, request and transmission aspects, followed by the notes. The Filters show how the incoming or outgoing data is modified; for example, 'M' indicates that the receive MIDI channel needs to be set correctly, whilst 'O' shows that an Omni mode can be selected to override this. Further detailed information is given in the Notes area at the bottom of the chart.

Notice that the transmit and receive messages are substantially identical - showing that the Lexicon MRC will both send and receive valid MIDI System Exclusive messages. In everyday language this means that you can connect two MRCs together and you will be able to transfer data between them. The Bulk Dump Request message results in a reply message supplying the appropriate data from the four available choices.

This is the format in which I will put all the System Exclusive Charts in this series. Personally, I intend to standardise all my documentation to this type of format in future. Remember that a standard MIDI Implementation Chart would show just two O's in the Receive and Transmit System Exclusive boxes, with perhaps a note giving further details. The System Exclusive Chart provides the missing information in a more usable form.

PRACTICE: LOOKING AT SYSTEM EXCLUSIVE

All this theory has probably made your head spin. The best cure for confusion is a few down to earth examples. In order to undertake some practical work, you will need some means of looking at the MIDI data - either the two Atari ST programs which I am about to describe or one of the alternative MIDI monitoring devices described last month and this month. The other necessity is a MIDI instrument which uses System Exclusive dumps. Virtually all current instruments which have MIDI In and Out sockets will implement some sort of System Exclusive function.

Before doing anything else, read the manual(s) for your equipment. Most people ignore the pages at the back of the manual describing the MIDI System Exclusive formats, etc. You will need to find out how to send and receive System Exclusive bulk dumps of sounds, patches or patterns, by pressing buttons on the front panel controls. We will look in detail at the uses of these dumps next month - for now, we are more concerned with becoming familiar with this aspect of the MIDI device, and the analysis of the MIDI data which results from using it. These skills will be needed again and again in the rest of the series.

I have written two programs for the Atari ST which are designed to make looking at System Exclusive messages easy: System Scope and SANE.

SYSTEM SCOPE

The Channel Scope program from Part 1 displayed any MIDI information from the Channel messages, and ignored the System messages. The System Scope program does exactly the opposite - it only looks at the System messages, taking no notice of anything happening on the Channel side of things. Splitting the functions in this way helps to reduce the amount of information displayed, and focuses our attention on the specific area of interest.

System Scope works with both high and medium resolution monitors. MIDI messages trapped in the current version include:

- Bulk Dumps
- Manufacturers, Sample Dumps, Universals, Real-Times
- MIDI Time Code
- Start, Stop, Continue
- Active Sense
- Tune Request
- System Reset
- MIDI Clock
- Song Pointers
- Song Select
- Certain Non-System messages: All Notes Off, Reset Controllers, Program Changes, Controllers (detects them on any channel)

The support for MIDI Time Code is limited by the time available for processing - there is not enough time in the main loop to decode each of the nibbles into a time display. Instead you get a time indicator, like the Clock and Active Sense indicators, to show the presence of MTC. Because of the requirement to trap as much of the incoming bulk dump information as possible, a software MIDI 'Thru' has not been implemented in this program. As a result, it is a MIDI Sink and so nothing useful comes out of the MIDI Out, ever!

SANE

SANE is a rough acronym for System Exclusive ANalyser, with some rearrangement of the letters (SEAN does not have the same effect!). Why another program? It turns out that the use of System Exclusive in practice can be more complicated than I have so far outlined.

Some manufacturers split and then send their System Exclusive data in small parcels rather than one long packet; other manufacturers send several sets of different packets for different types of information all within the context of a single System Exclusive dump. Both of these approaches tend to make the System Scope and some Generic Bulk Librarians like Hybrid Arts' GenPatch ST more difficult to use. SANE was developed in order to have advance warning of these multi-packet dumps.

By sending a System Exclusive dump to an Atari ST which is running SANE, you can see how many packets are sent, together with their length in bytes. For example, early MIDI devices like the Yamaha DX7 Mk1 synthesizer can send a bulk dump of all the voices in its internal memory, and this is sent as a single file of 4,104 bytes in this case. In contrast, the recently released Yamaha YS100/200 synthesizers send multi-packet dumps of up to eight separate (and different) packets for a bulk dump of the internal memory. Each packet has its own Header detailing the contents of the data which follows - they can be considered as being eight separate System Exclusive packets. Generally, as manufacturers have become used to the idea and applications of MIDI, the amount of provision for dumps, messages and parameter control has increased enormously.

When System Scope receives a System Exclusive multi-packet, it pauses after the first complete packet it detects. To view the next packet you just press the spacebar or the mouse button. Unfortunately, whilst the program is paused, the MIDI buffer fills very rapidly and some data may be lost. I am still working on this problem and later versions of the System Scope should provide ways around it.

SENDING PACKETS

Connect your equipment so that the MIDI Out is connected to the MIDI In of the Atari ST (or your alternative MIDI monitoring device) and run the System Scope program. In normal circumstances the System Scope will not display anything - if a MIDI clock is running or Active Sensing messages are being sent then you will see this indicated. If not, the program will wait for some System activity - playing notes will not generate a display. (Some messages associated with Channel activity are shown in some cases to extend the versatility of the System Scope, but they need not concern us here.) A drum machine or sequencer makes a good tester for the Real-Time section of the System Scope - pressing the drum machine/sequencer's Start/Play button should send a Start message ($FA). Depending upon the manufacturer's interpretation of the MIDI Specification, the MIDI Clock messages ($F8) will either start at this point or will have been sent all the time that the drum machine/sequencer has been connected. Some drum machines and sequencers also send Song Position Pointer and Song Select messages, and these will be displayed as they occur.

Once you have verified that all is well, you can send a dump to the ST. A new window will open on the screen, and this will fill with a listing of the System Exclusive information which has been received, in both Hexadecimal and ASCII text versions. The Header information will be decoded if the format is one that the System Scope program recognises, otherwise it will substitute a generic format.

Pressing the spacebar will show the next System Exclusive message (if there is one). Otherwise the program will just wait for more System messages. A screen dump to your printer (press ALT + Help) will provide a hard copy of the screen contents at any time. You can use this to verify and familiarise yourself with the sorts of messages that your equipment sends - some instruments include ASCII text in the Header to identify the contents, whilst others need to be decoded with the assistance of the MIDI pages in the owner's manual. Sometimes the information in the manual is not up to date and you may find that some of the dumps are different to those listed - the System Scope will help you find and document these additions. The accompanying 'Example Dumps' panel shows some typical examples of the sort of dumps you may come across.

CONCLUSION

I encourage you to try and look at dumps from all your equipment - the more you look at System Exclusive messages the easier it will be to understand them. Familiarity will help as we proceed with more utilities, and feeling at home with Manufacturer Identifications and formats will become essential when we use System Exclusive to perform editing! This is the groundwork which we will expand upon in the next few articles, so a little time spent learning now will repay itself several times over when we start doing the more exciting stuff (the skills are also very valuable for MIDI troubleshooting in general).

Next month I will explore uses for the messages described this month - the sort of things you read about but are never sure how to do! Another couple of very useful utility programs will extend the capabilities of those already described. Meanwhile, press those MIDI dump buttons, observe, compare, verify and learn - the practical work is all part of the fun!

DISK INFORMATION

The programs referred to in this article are available on 3.5" disk (for the Atari ST only at present). Channel Scope is separate and costs £7, whilst MIDI Utility, System Scope and SANE have been bundled together on one disk (along with extra programs for Part 3 of the System Exclusive series), and costs a mere £7.

SOS Software, (Contact Details).