HANDSHAKING

If there is one subject which offers the maximum potential for confusion, it must be the problem of controlling communications between pieces of equipment. Most computer owners will be familiar with the problems of trying to connect a computer and printer together successfully using RS232. With only a few exceptions, everything sent using MIDI System Exclusive messages in this series so far has been one way - one unit transmits, the other receives. For the vast majority of situations this is perfectly fine, but there are certain occasions when something extra is required. This extra feature is handshaking, the subject of this month's episode.

The word 'handshaking' carries its own connotations: a shake of the hands signifies agreement or the acknowledgement of the completion of a deal. In communications terms the meaning is slightly more formal, and is designed to provide reassurance that all is well at both ends of the communications channel. Basically, each message that is sent is acknowledged by the receiving device, and this is called 'handshaking'. In addition, the transmission sequence is often dependent upon the acknowledgements received, making the result more like a telephone conversation than a simple 'it is here!' reply.

I have used train analogies several times already in this series, so here goes again. Suppose that you want to find out the departure times of trains to London. There are two likely scenarios which might happen. Firstly, without any interaction, an automatic announcement such as:

BR: Here are the departure and arrival times for the trains to London tomorrow. The first train is the 7:30, which arrives in London at 9:30. The 8:30 train arrives in London at 10:30. The 9:30 train arrives in London at 11:45. The 10 o'clock train arrives at 12:05. The 10:15 arrives at 12:45...

In contrast, suppose that you ring your local station's train enquiry number and speak to an assistant:

You: I would like to travel to London tomorrow.
BR: What time would you like to arrive?
You: At about midday.
BR: The 10 o'clock train gets in at 12:05.
You: Is there an earlier train?
BR: The 9:30 gets in at 11:45.
You: What about later?
BR: The 10:15 arrives at 12:45.
You: So the 10 o'clock is the fastest train?
BR: That is correct. Do you wish to book a seat?

In the first example, you just listen to the pre-recorded message and attempt to extract the information you require; it is not easy to work out which is the fastest train when you are trying to keep up with the recording. The second example lets you go more or less straight to the root of the problem: not only do you want to arrive sometime around midday, but you also want a fast train. The price that is paid for the rapid access to the relevant information is the necessary interaction between you and the assistant - you both need to wait for the other to stop talking before replying, for example. Although conversation like this seems natural enough, there is a lot of control being exercised: notice how the assistant asks you the first question about the time you would like to arrive, then you counter with a query about an earlier train. Even when you have decided on the train, the questioning flicks back as you are asked if you want to reserve a seat.

This to and fro process, with first one and then the other person being in control of the conversation, is typical of the process of handshaking, too. When two items of digital equipment need to communicate, a similar interchange of information may also be needed, with each side waiting for the other to supply the relevant reply. In MIDI terms, a typical use of handshaking might be a bulk dump request from a computer to a synthesizer:

Here we can begin to define some terms, which should help to make things slightly more understandable. The computer starts by sending a Request to the synthesizer. The synthesizer Acknowledges the Request, and then Transmits the bulk dump. The computer receives the Acknowledgement, and prepares to capture the data which follows. The computer then receives and stores the dump.

Does this seem OK? Can you see any potential problems? If this was our example telephone conversation, would you ask for information, listen to it and then hang up? Or would you say, 'OK, thanks for the help,' and then hang up?

This highlights another reason for the use of handshaking - it helps to improve the integrity of the data (information) transfers between the two units, since if there is a problem during the sending of the information, you can request a repeat ('Sorry, what was that again?'). If you regularly use programs like Movie or another Generic Librarian to store and move bulk dumps around, you may have found that the number of received bytes is not always correct and that you need to retransmit the data. This can be due to any number of reasons, but if you don't check on what is happening then you might store useless information.

Therefore, it might be a good idea to do some checking to establish if the device you are talking to is actually there, and if you have received the information correctly. The dump request might now look like this:

We are now sending four messages from the computer instead of one, but we have corrected the dump received with an error and requested a repeat dump, something which a non-handshaking system, or the simple request shown earlier, could not have done.

These two elements, the control of the flow of information between the two devices and the integrity of the information, are the two key reasons for using handshaking. As the length of MIDI messages grows, the possibility of transmission errors increases, and so handshaking should become increasingly essential. Manufacturers like Roland already take this into account in the design of their equipment - short messages can be sent without handshaking, but longer messages are split into packets and sent using handshaking. Similarly, the MIDI Sample Dump Standard can use handshaking to ensure that the sample information is correctly transferred, by requesting repeats of packets.

THIS MONTH'S PROGRAMS

As usual, I will illustrate the workings of handshaking with programs for the Atari ST, although the principles can be demonstrated with any units which implement handshaking.

Some hardware sequencers, like the Kawai Q80 for example, implement handshaking only with other Kawai equipment. In all cases, you will need an extra MIDI connection in addition to the more usual one between the MIDI Out of the sending instrument to the MIDI In of the listening device - the MIDI Out of the listening device needs to be connected to the In of the sending unit, so that messages can be transmitted in both directions.

The first program is called Request V0.1, and this is similar to the SYSEX program which has been gradually evolving through the course of this series. Request lets you send a MIDI message and then monitor the incoming MIDI response (if any). To do so you choose a Manufacturer ID, type in the hex numbers for the request message you wish to send, and then press the Return key to send it. The right-hand box on the screen display will then show you the first 15 bytes of any received message, decoded into standard System Exclusive meanings if appropriate.

Request V0.1 only implements a simple form of handshaking - it cannot cope with the complex conversation described above, but it does allow investigation of the principles of handshaking and can be a valuable testing tool, as we shall see later in the practical section.

The second program is completely new and takes the best features of SYSEX and combines them with Movie, resulting in Movie II. Movie II lets you send dump requests, so it can be used with equipment which does not have a front panel dump button. As with Request, the handshaking is still very simple and so complex interactions are not possible; this means that Roland and Casio instruments may not work with Movie II under all conditions. Movie II also allows the messages to be sent without saving the response - in other words, just like SYSEX. If you do not specify a request message, Movie II behaves just like the original Movie, therefore the combined program is compatible with the operation of both earlier programs.

For complex handshaking, there really is no substitute for either the manufacturer's own sequencer/librarian hardware or software, or a generic librarian program like Hybrid Arts' Genpatch ST, or Keynote's Chameleon. As I have said before, it is not my intention to try and replace this type of professional software - Request and Movie II are designed to be used for verification, troubleshooting and learning functions only. If you intend to pursue handshaking or any aspect of System Exclusive seriously, then you should consider buying the best software tools for the job.

PRACTICE TIME

To demonstrate some of the possible uses of handshaking, here is a pot-pourri of examples. These are intended as illustrations, to show the operation, rather than necessarily practical or usable functions.

Remote Questioning

Back in Part 3 of this series I said that it was possible to remotely interrogate a MIDI device and find out about the system software version etc. The example I gave was for the Korg M1 synthesizer, and it was possible to discover the ROM number and software version number by using a simple Information Request message. Rather than repeat the M1 information, here is a similar request and response from an Emax sampler:

The Emax sampler responds with not only the ROM version, but also a host of other miscellaneous information - some of which could be very useful to a programmer attempting to write an editor...

Yamaha implement a similar function in some of their instruments but they call it a 'Condition Acknowledge'. It uses a format number from their bulk dumps to dump a different sort of data than the more usual voice data, namely a complete snapshot of the front panel and internal conditions:

F0, 43, 0n, 7D, be, a, m, v, r, c, e, F7

where n is the channel number, be is the byte count for the condition data, a is the classification name (the Division within Yamaha - four ASCII characters), m is the Model ID (six ASCII characters), v is the software version number, r is the software revision number, c is the condition snapshot data (length = byte count — 12 bytes), and e is the checksum.

Not all Yamaha equipment implements this, but for a DX5 synth the a, m, v and r fields could look like this:

"LM--", "8933", "2", "1"

As a further example, here is a slightly different use of the same type of exchange of information but for a Kawai K5 synth:

Machine ID Request
F0, 40, 0n, 60, F7

where n is the MIDI channel (or device number in Roland-speak) of the K5. If this matches the receive channel set on the K5's MIDI page, then the K5 will respond as follows:

Machine ID Acknowledge
F0, 40, 0n, 61, 00, 02, F7

where the 61 indicates that this is an Acknowledge message, the 00, 02 shows that it is from a K5, and n is the transmit channel as set on the K5's MIDI page. Since the two channels can be different, it is easy to set them so that normal bulk librarians or editors will be unable to communicate with the K5, but by sending requests on different channels until the K5 responds, it is possible to automatically search for a K5, and if no response is found, to report back an error.

Information request messages are easily overlooked by software authors, since it is not immediately obvious what their purpose is. Most modern MIDI instruments implement some sort of response to information requests, especially in the context of the latest version (4.1) of the official MIDI Standard, where Universal Non-Real Time message number $06 is defined as an Information Request or Inquiry. By incorporating routines to check that the connections are correct and that the right instrument is present, the resulting software can offer enhanced reliability and user-friendliness for very little added programming effort.

Bulk Dump Request

Part 3 of this series also showed a typical Yamaha Dump Request message using the $2n second sub-status byte. For Yamaha's new mega-synth, the SY77, the message is very similar:

Yamaha SY77:
Single Voice Bulk Dump Request
F0, 43, 2n, 7A, type-string, [$00x14], mt, mn, F7

where n is the device number, and type-string, mt and mn are shown in the tables below:

mn is the memory number (0-63 for Voices, 0-15 for Multis)

The SY77 would seem to be Yamaha model #8101 (the DX7II was #8973, the DX5 was #8933 as shown above, and the TX7 was #8950). Also notice that only single voice dumps are catered for - there is no equivalent of the 32-voice dumps of the DX7. Of course, this standard format is only applicable to Yamaha instruments, other manufacturers implement their own request formats.

Handshaking on the FB01

Because it was produced by Yamaha's Computer Division, the FB01 tone module has several peculiarities which are not found on equipment from the Musical Instrument Divisions. The FB01 has a specialised set of messages just for handshaking, even though the MIDI dumps and other MIDI input/output does not use handshaking! Instead, the three messages are used to indicate the results or progress of dumps and front panel commands, and so can be used to monitor errors or as a way to verify that dumps have been correctly received. You can therefore use an external computer to provide the missing handshaking functions!

The three messages are a subset of part of the published Yamaha System Exclusive format. The format is an evolving one, with additional parts added when necessary. For example, the SY77 dumps described above add a new dump format ($7A) to the already existing ones ($0A, SOB, $7B, $7C and $7E). The three messages are: ACK, NAK and CANCEL, which stand for ACKnowledge, Not Acknowledge, and CANCEL respectively. The messages look like this:

F0, 43, 6n, 02, F7 ACK
F0, 43, 6n, 03, F7 NAK
F0, 43, 6n, 04, F7 CANCEL

ACK indicates that an action (like a dump) has been successful. NAK shows that a checksum error (when receiving a dump), framing error, data overrun or underrun has occurred. CANCEL is used to signify a memory protection error (trying to execute a save whilst the memory protect is on) or an invalid number (out of range). By monitoring this feedback from the FB01, it should be possible to write an FB01 editor program which ensures that all operations are carried out correctly; this is something which almost every other piece of Yamaha equipment does not allow!

MIDI Sample Dumps

The MIDI Sample Dump Standard (see separate panel) can use handshaking to enable the transfer of sample data between samplers. However, some manufacturers use conventional MIDI System Exclusive messages to set up alternative higher speed modes, which they then utilise to transfer the actual data at rates considerably higher than the 31.25 Kbits per second MIDI rate. These transfers are typified by the exchange of information between the two samplers, followed by bursts of data which is usually not readable by MIDI monitors and scope programs. The initial exchanges are made in order to confirm that both ends of the transmission link are capable of operating in high speed mode.

Non-MIDI System Exclusive

As an extension of the idea of having high speed modes used for sample dumps, at least one major sampler manufacturer has used MIDI type electronics running at the standard rates, but has 'hi-jacked' the MIDI messages and redefined them for their own purposes. The Yamaha handshaking used in the FB01 might be mistaken for this misuse of MIDI, because although it would be typified by a handshaking type activity, the usual handshaking messages would not be used, instead the special Yamaha messages would be found.

As we have learned several times in this series, the obvious answer is not always the correct one - so be careful to examine MIDI data fully. It might be a 'rogue' implementation, but it might also be a legal but unpublished version, or even one which has not yet reached your reference books! For instance, the Yamaha SY77 information given in this article currently falls into this grey area of System Exclusive, but this will be rectified once the full MIDI Implementation becomes available.

CONCLUSION

Handshaking extends the ideas of requests and bulk dumps from a simple one-way data transfer to a complex interchange of messages between two devices. The price paid in complexity and in extra cables is repaid in improved integrity of the transmitted data and increased functionality - longer dumps are easier to handle and arrive intact! As the size of MIDI dumps and messages continues to grow, handshaking will become more and more important and is likely to become an essential part of MIDI systems of the future. So far, we have muddled along without it, but handshaking is destined to be the way forward.

This month's coverage of handshaking more or less completes the theory element which I planned into this series from the start. We have now turned the last comer and are accelerating along the final straight; however, there may still be an unforeseen problem on the track ahead, and to avoid it you will have to read next month's exciting episode!