Everything You Ever Wanted To Know About System Exclusive (Part 10)
(But Were Too Afraid To Ask!)
PART 10: Martin Russ continues his in-depth demystification of these powerful MIDI messages with a look at 'handshaking'.
If there is one subject which offers the maximum potential for confusion, it must be the problem of controlling communications between pieces of equipment. Most computer owners will be familiar with the problems of trying to connect a computer and printer together successfully using RS232. With only a few exceptions, everything sent using MIDI System Exclusive messages in this series so far has been one way - one unit transmits, the other receives. For the vast majority of situations this is perfectly fine, but there are certain occasions when something extra is required. This extra feature is handshaking, the subject of this month's episode.
The word 'handshaking' carries its own connotations: a shake of the hands signifies agreement or the acknowledgement of the completion of a deal. In communications terms the meaning is slightly more formal, and is designed to provide reassurance that all is well at both ends of the communications channel. Basically, each message that is sent is acknowledged by the receiving device, and this is called 'handshaking'. In addition, the transmission sequence is often dependent upon the acknowledgements received, making the result more like a telephone conversation than a simple 'it is here!' reply.
I have used train analogies several times already in this series, so here goes again. Suppose that you want to find out the departure times of trains to London. There are two likely scenarios which might happen. Firstly, without any interaction, an automatic announcement such as:
BR: Here are the departure and arrival times for the trains to London tomorrow. The first train is the 7:30, which arrives in London at 9:30. The 8:30 train arrives in London at 10:30. The 9:30 train arrives in London at 11:45. The 10 o'clock train arrives at 12:05. The 10:15 arrives at 12:45...
In contrast, suppose that you ring your local station's train enquiry number and speak to an assistant:
You: I would like to travel to London tomorrow.
BR: What time would you like to arrive?
You: At about midday.
BR: The 10 o'clock train gets in at 12:05.
You: Is there an earlier train?
BR: The 9:30 gets in at 11:45.
You: What about later?
BR: The 10:15 arrives at 12:45.
You: So the 10 o'clock is the fastest train?
BR: That is correct. Do you wish to book a seat?
In the first example, you just listen to the pre-recorded message and attempt to extract the information you require; it is not easy to work out which is the fastest train when you are trying to keep up with the recording. The second example lets you go more or less straight to the root of the problem: not only do you want to arrive sometime around midday, but you also want a fast train. The price that is paid for the rapid access to the relevant information is the necessary interaction between you and the assistant - you both need to wait for the other to stop talking before replying, for example. Although conversation like this seems natural enough, there is a lot of control being exercised: notice how the assistant asks you the first question about the time you would like to arrive, then you counter with a query about an earlier train. Even when you have decided on the train, the questioning flicks back as you are asked if you want to reserve a seat.
This to and fro process, with first one and then the other person being in control of the conversation, is typical of the process of handshaking, too. When two items of digital equipment need to communicate, a similar interchange of information may also be needed, with each side waiting for the other to supply the relevant reply. In MIDI terms, a typical use of handshaking might be a bulk dump request from a computer to a synthesizer:
|<Please send dump>|
|<OK - dump follows>|
|<Are you there?>|
|<Please send dump>|
|<OK - dump follows>|
|<Checksum wrong - repeat>|
|<OK - dump follows>|
As usual, I will illustrate the workings of handshaking with programs for the Atari ST, although the principles can be demonstrated with any units which implement handshaking.
Some hardware sequencers, like the Kawai Q80 for example, implement handshaking only with other Kawai equipment. In all cases, you will need an extra MIDI connection in addition to the more usual one between the MIDI Out of the sending instrument to the MIDI In of the listening device - the MIDI Out of the listening device needs to be connected to the In of the sending unit, so that messages can be transmitted in both directions.
The first program is called Request V0.1, and this is similar to the SYSEX program which has been gradually evolving through the course of this series. Request lets you send a MIDI message and then monitor the incoming MIDI response (if any). To do so you choose a Manufacturer ID, type in the hex numbers for the request message you wish to send, and then press the Return key to send it. The right-hand box on the screen display will then show you the first 15 bytes of any received message, decoded into standard System Exclusive meanings if appropriate.
Request V0.1 only implements a simple form of handshaking - it cannot cope with the complex conversation described above, but it does allow investigation of the principles of handshaking and can be a valuable testing tool, as we shall see later in the practical section.
The second program is completely new and takes the best features of SYSEX and combines them with Movie, resulting in Movie II. Movie II lets you send dump requests, so it can be used with equipment which does not have a front panel dump button. As with Request, the handshaking is still very simple and so complex interactions are not possible; this means that Roland and Casio instruments may not work with Movie II under all conditions. Movie II also allows the messages to be sent without saving the response - in other words, just like SYSEX. If you do not specify a request message, Movie II behaves just like the original Movie, therefore the combined program is compatible with the operation of both earlier programs.
For complex handshaking, there really is no substitute for either the manufacturer's own sequencer/librarian hardware or software, or a generic librarian program like Hybrid Arts' Genpatch ST, or Keynote's Chameleon. As I have said before, it is not my intention to try and replace this type of professional software - Request and Movie II are designed to be used for verification, troubleshooting and learning functions only. If you intend to pursue handshaking or any aspect of System Exclusive seriously, then you should consider buying the best software tools for the job.
To demonstrate some of the possible uses of handshaking, here is a pot-pourri of examples. These are intended as illustrations, to show the operation, rather than necessarily practical or usable functions.
Back in Part 3 of this series I said that it was possible to remotely interrogate a MIDI device and find out about the system software version etc. The example I gave was for the Korg M1 synthesizer, and it was possible to discover the ROM number and software version number by using a simple Information Request message. Rather than repeat the M1 information, here is a similar request and response from an Emax sampler:
|Misc Info Request|
|02||Misc Info Request|
|Misc Info Reply|
|32||Send Misc Info|
|0ppp pppp||current preset n umber (0-99)|
|000ttttt||master tune (0-31)|
|000c ccos||s = supermode on/off |
o = overflow on/off
c = arpeggiator clock source:
2: 24 ppq
3: 48 ppq
4: 96 ppq
|0rrr rrrr||sound RAM remaining bits 0-6|
|0rrr rrrr||sound RAM bits 7-13|
|000r rrrr||sound RAM bits 14-18|
|0ppp pppp||preset RAM remaining bits 0-6|
|0ppp pppp||preset RAM bits 7-13|
|0000 000p||preset RAM bit 14|
|0sss ssss||software revision ASCII letter 0|
|0sss ssss||software revision ASCII letter 15|
Part 3 of this series also showed a typical Yamaha Dump Request message using the $2n second sub-status byte. For Yamaha's new mega-synth, the SY77, the message is very similar:
Single Voice Bulk Dump Request
F0, 43, 2n, 7A, type-string, [$00x14], mt, mn, F7
where n is the device number, and type-string, mt and mn are shown in the tables below:
|1 Pan data||"LM-8101PN"|
|1 System data||"LM-8101SY"|
|mt is the memory type:|
|00 = Internal|
|02 = Preset 1|
|03 = Preset 2|
|7F = Edit Buffer|
Because it was produced by Yamaha's Computer Division, the FB01 tone module has several peculiarities which are not found on equipment from the Musical Instrument Divisions. The FB01 has a specialised set of messages just for handshaking, even though the MIDI dumps and other MIDI input/output does not use handshaking! Instead, the three messages are used to indicate the results or progress of dumps and front panel commands, and so can be used to monitor errors or as a way to verify that dumps have been correctly received. You can therefore use an external computer to provide the missing handshaking functions!
The three messages are a subset of part of the published Yamaha System Exclusive format. The format is an evolving one, with additional parts added when necessary. For example, the SY77 dumps described above add a new dump format ($7A) to the already existing ones ($0A, SOB, $7B, $7C and $7E). The three messages are: ACK, NAK and CANCEL, which stand for ACKnowledge, Not Acknowledge, and CANCEL respectively. The messages look like this:
F0, 43, 6n, 02, F7 ACK
F0, 43, 6n, 03, F7 NAK
F0, 43, 6n, 04, F7 CANCEL
ACK indicates that an action (like a dump) has been successful. NAK shows that a checksum error (when receiving a dump), framing error, data overrun or underrun has occurred. CANCEL is used to signify a memory protection error (trying to execute a save whilst the memory protect is on) or an invalid number (out of range). By monitoring this feedback from the FB01, it should be possible to write an FB01 editor program which ensures that all operations are carried out correctly; this is something which almost every other piece of Yamaha equipment does not allow!
The MIDI Sample Dump Standard (see separate panel) can use handshaking to enable the transfer of sample data between samplers. However, some manufacturers use conventional MIDI System Exclusive messages to set up alternative higher speed modes, which they then utilise to transfer the actual data at rates considerably higher than the 31.25 Kbits per second MIDI rate. These transfers are typified by the exchange of information between the two samplers, followed by bursts of data which is usually not readable by MIDI monitors and scope programs. The initial exchanges are made in order to confirm that both ends of the transmission link are capable of operating in high speed mode.
As an extension of the idea of having high speed modes used for sample dumps, at least one major sampler manufacturer has used MIDI type electronics running at the standard rates, but has 'hi-jacked' the MIDI messages and redefined them for their own purposes. The Yamaha handshaking used in the FB01 might be mistaken for this misuse of MIDI, because although it would be typified by a handshaking type activity, the usual handshaking messages would not be used, instead the special Yamaha messages would be found.
As we have learned several times in this series, the obvious answer is not always the correct one - so be careful to examine MIDI data fully. It might be a 'rogue' implementation, but it might also be a legal but unpublished version, or even one which has not yet reached your reference books! For instance, the Yamaha SY77 information given in this article currently falls into this grey area of System Exclusive, but this will be rectified once the full MIDI Implementation becomes available.
Handshaking extends the ideas of requests and bulk dumps from a simple one-way data transfer to a complex interchange of messages between two devices. The price paid in complexity and in extra cables is repaid in improved integrity of the transmitted data and increased functionality - longer dumps are easier to handle and arrive intact! As the size of MIDI dumps and messages continues to grow, handshaking will become more and more important and is likely to become an essential part of MIDI systems of the future. So far, we have muddled along without it, but handshaking is destined to be the way forward.
This month's coverage of handshaking more or less completes the theory element which I planned into this series from the start. We have now turned the last comer and are accelerating along the final straight; however, there may still be an unforeseen problem on the track ahead, and to avoid it you will have to read next month's exciting episode!
Feature by Martin Russ
mu:zines is the result of thousands of hours of effort, and will require many thousands more going forward to reach our goals of getting all this content online.
If you value this resource, you can support this project - it really helps!
Please note: Our yearly hosting fees are due every March, so monetary donations are especially appreciated to help meet this cost. Thank you for your support!