• Everything You Ever Wanted T...
  • Everything You Ever Wanted T...
  • Everything You Ever Wanted T...

Magazine Archive

Home -> Magazines -> Issues -> Articles in this issue -> View

Everything You Ever Wanted To Know About System Exclusive (Part 10)

(But Were Too Afraid To Ask!)

PART 10: Martin Russ continues his in-depth demystification of these powerful MIDI messages with a look at 'handshaking'.


HANDSHAKING



If there is one subject which offers the maximum potential for confusion, it must be the problem of controlling communications between pieces of equipment. Most computer owners will be familiar with the problems of trying to connect a computer and printer together successfully using RS232. With only a few exceptions, everything sent using MIDI System Exclusive messages in this series so far has been one way - one unit transmits, the other receives. For the vast majority of situations this is perfectly fine, but there are certain occasions when something extra is required. This extra feature is handshaking, the subject of this month's episode.

The word 'handshaking' carries its own connotations: a shake of the hands signifies agreement or the acknowledgement of the completion of a deal. In communications terms the meaning is slightly more formal, and is designed to provide reassurance that all is well at both ends of the communications channel. Basically, each message that is sent is acknowledged by the receiving device, and this is called 'handshaking'. In addition, the transmission sequence is often dependent upon the acknowledgements received, making the result more like a telephone conversation than a simple 'it is here!' reply.

I have used train analogies several times already in this series, so here goes again. Suppose that you want to find out the departure times of trains to London. There are two likely scenarios which might happen. Firstly, without any interaction, an automatic announcement such as:

BR: Here are the departure and arrival times for the trains to London tomorrow. The first train is the 7:30, which arrives in London at 9:30. The 8:30 train arrives in London at 10:30. The 9:30 train arrives in London at 11:45. The 10 o'clock train arrives at 12:05. The 10:15 arrives at 12:45...

In contrast, suppose that you ring your local station's train enquiry number and speak to an assistant:

You: I would like to travel to London tomorrow.
BR: What time would you like to arrive?
You: At about midday.
BR: The 10 o'clock train gets in at 12:05.
You: Is there an earlier train?
BR: The 9:30 gets in at 11:45.
You: What about later?
BR: The 10:15 arrives at 12:45.
You: So the 10 o'clock is the fastest train?
BR: That is correct. Do you wish to book a seat?


In the first example, you just listen to the pre-recorded message and attempt to extract the information you require; it is not easy to work out which is the fastest train when you are trying to keep up with the recording. The second example lets you go more or less straight to the root of the problem: not only do you want to arrive sometime around midday, but you also want a fast train. The price that is paid for the rapid access to the relevant information is the necessary interaction between you and the assistant - you both need to wait for the other to stop talking before replying, for example. Although conversation like this seems natural enough, there is a lot of control being exercised: notice how the assistant asks you the first question about the time you would like to arrive, then you counter with a query about an earlier train. Even when you have decided on the train, the questioning flicks back as you are asked if you want to reserve a seat.

This to and fro process, with first one and then the other person being in control of the conversation, is typical of the process of handshaking, too. When two items of digital equipment need to communicate, a similar interchange of information may also be needed, with each side waiting for the other to supply the relevant reply. In MIDI terms, a typical use of handshaking might be a bulk dump request from a computer to a synthesizer:

COMPUTER SYNTHESIZER
<Please send dump>
<OK - dump follows>
<Dump ....
....
....>

Here we can begin to define some terms, which should help to make things slightly more understandable. The computer starts by sending a Request to the synthesizer. The synthesizer Acknowledges the Request, and then Transmits the bulk dump. The computer receives the Acknowledgement, and prepares to capture the data which follows. The computer then receives and stores the dump.

Does this seem OK? Can you see any potential problems? If this was our example telephone conversation, would you ask for information, listen to it and then hang up? Or would you say, 'OK, thanks for the help,' and then hang up?

This highlights another reason for the use of handshaking - it helps to improve the integrity of the data (information) transfers between the two units, since if there is a problem during the sending of the information, you can request a repeat ('Sorry, what was that again?'). If you regularly use programs like Movie or another Generic Librarian to store and move bulk dumps around, you may have found that the number of received bytes is not always correct and that you need to retransmit the data. This can be due to any number of reasons, but if you don't check on what is happening then you might store useless information.

Therefore, it might be a good idea to do some checking to establish if the device you are talking to is actually there, and if you have received the information correctly. The dump request might now look like this:

COMPUTER SYNTHESIZER
<Are you there?>
<I'm here!>
<Please send dump>
<OK - dump follows>
<Dump....
... (Error)...
......>
<Checksum wrong - repeat>
<OK - dump follows>
<Dump....
......>
<Checksum OK>
<OK>

We are now sending four messages from the computer instead of one, but we have corrected the dump received with an error and requested a repeat dump, something which a non-handshaking system, or the simple request shown earlier, could not have done.

These two elements, the control of the flow of information between the two devices and the integrity of the information, are the two key reasons for using handshaking. As the length of MIDI messages grows, the possibility of transmission errors increases, and so handshaking should become increasingly essential. Manufacturers like Roland already take this into account in the design of their equipment - short messages can be sent without handshaking, but longer messages are split into packets and sent using handshaking. Similarly, the MIDI Sample Dump Standard can use handshaking to ensure that the sample information is correctly transferred, by requesting repeats of packets.

THIS MONTH'S PROGRAMS



As usual, I will illustrate the workings of handshaking with programs for the Atari ST, although the principles can be demonstrated with any units which implement handshaking.

Some hardware sequencers, like the Kawai Q80 for example, implement handshaking only with other Kawai equipment. In all cases, you will need an extra MIDI connection in addition to the more usual one between the MIDI Out of the sending instrument to the MIDI In of the listening device - the MIDI Out of the listening device needs to be connected to the In of the sending unit, so that messages can be transmitted in both directions.

Requests 0.1 - a MIDI System Exclusive request utility by Martin Russ.


The first program is called Request V0.1, and this is similar to the SYSEX program which has been gradually evolving through the course of this series. Request lets you send a MIDI message and then monitor the incoming MIDI response (if any). To do so you choose a Manufacturer ID, type in the hex numbers for the request message you wish to send, and then press the Return key to send it. The right-hand box on the screen display will then show you the first 15 bytes of any received message, decoded into standard System Exclusive meanings if appropriate.

Request V0.1 only implements a simple form of handshaking - it cannot cope with the complex conversation described above, but it does allow investigation of the principles of handshaking and can be a valuable testing tool, as we shall see later in the practical section.

Movie II - 'handshaking' has been added to this System Exclusive file mover utility from Martin Russ.


The second program is completely new and takes the best features of SYSEX and combines them with Movie, resulting in Movie II. Movie II lets you send dump requests, so it can be used with equipment which does not have a front panel dump button. As with Request, the handshaking is still very simple and so complex interactions are not possible; this means that Roland and Casio instruments may not work with Movie II under all conditions. Movie II also allows the messages to be sent without saving the response - in other words, just like SYSEX. If you do not specify a request message, Movie II behaves just like the original Movie, therefore the combined program is compatible with the operation of both earlier programs.

For complex handshaking, there really is no substitute for either the manufacturer's own sequencer/librarian hardware or software, or a generic librarian program like Hybrid Arts' Genpatch ST, or Keynote's Chameleon. As I have said before, it is not my intention to try and replace this type of professional software - Request and Movie II are designed to be used for verification, troubleshooting and learning functions only. If you intend to pursue handshaking or any aspect of System Exclusive seriously, then you should consider buying the best software tools for the job.

PRACTICE TIME



To demonstrate some of the possible uses of handshaking, here is a pot-pourri of examples. These are intended as illustrations, to show the operation, rather than necessarily practical or usable functions.

Remote Questioning

Back in Part 3 of this series I said that it was possible to remotely interrogate a MIDI device and find out about the system software version etc. The example I gave was for the Korg M1 synthesizer, and it was possible to discover the ROM number and software version number by using a simple Information Request message. Rather than repeat the M1 information, here is a similar request and response from an Emax sampler:

Misc Info Request
F0 Sysex
18 Emu ID
02 Emax ID
02 Misc Info Request
F7 EOX

Misc Info Reply
F0 Sysex
18 Emu ID
02 Emax ID
32 Send Misc Info
0ppp pppp current preset n umber (0-99)
000ttttt master tune (0-31)
000c ccos s = supermode on/off
o = overflow on/off
c = arpeggiator clock source:
0:internal
1: MIDI
2: 24 ppq
3: 48 ppq
4: 96 ppq
0rrr rrrr sound RAM remaining bits 0-6
0rrr rrrr sound RAM bits 7-13
000r rrrr sound RAM bits 14-18
0ppp pppp preset RAM remaining bits 0-6
0ppp pppp preset RAM bits 7-13
0000 000p preset RAM bit 14
0sss ssss software revision ASCII letter 0
0sss ssss software revision ASCII letter 15
F7 EOX

The Emax sampler responds with not only the ROM version, but also a host of other miscellaneous information - some of which could be very useful to a programmer attempting to write an editor...

Yamaha implement a similar function in some of their instruments but they call it a 'Condition Acknowledge'. It uses a format number from their bulk dumps to dump a different sort of data than the more usual voice data, namely a complete snapshot of the front panel and internal conditions:

F0, 43, 0n, 7D, be, a, m, v, r, c, e, F7

where n is the channel number, be is the byte count for the condition data, a is the classification name (the Division within Yamaha - four ASCII characters), m is the Model ID (six ASCII characters), v is the software version number, r is the software revision number, c is the condition snapshot data (length = byte count — 12 bytes), and e is the checksum.

Not all Yamaha equipment implements this, but for a DX5 synth the a, m, v and r fields could look like this:

"LM--", "8933", "2", "1"

As a further example, here is a slightly different use of the same type of exchange of information but for a Kawai K5 synth:

Machine ID Request
F0, 40, 0n, 60, F7


where n is the MIDI channel (or device number in Roland-speak) of the K5. If this matches the receive channel set on the K5's MIDI page, then the K5 will respond as follows:

Machine ID Acknowledge
F0, 40, 0n, 61, 00, 02, F7


where the 61 indicates that this is an Acknowledge message, the 00, 02 shows that it is from a K5, and n is the transmit channel as set on the K5's MIDI page. Since the two channels can be different, it is easy to set them so that normal bulk librarians or editors will be unable to communicate with the K5, but by sending requests on different channels until the K5 responds, it is possible to automatically search for a K5, and if no response is found, to report back an error.

Information request messages are easily overlooked by software authors, since it is not immediately obvious what their purpose is. Most modern MIDI instruments implement some sort of response to information requests, especially in the context of the latest version (4.1) of the official MIDI Standard, where Universal Non-Real Time message number $06 is defined as an Information Request or Inquiry. By incorporating routines to check that the connections are correct and that the right instrument is present, the resulting software can offer enhanced reliability and user-friendliness for very little added programming effort.

Bulk Dump Request

Part 3 of this series also showed a typical Yamaha Dump Request message using the $2n second sub-status byte. For Yamaha's new mega-synth, the SY77, the message is very similar:

Yamaha SY77:
Single Voice Bulk Dump Request
F0, 43, 2n, 7A, type-string, [$00x14], mt, mn, F7


where n is the device number, and type-string, mt and mn are shown in the tables below:

Dump type type-string
1 Voice "LM-8101VC"
1 Multi "LM-8101MU"
1 Pan data "LM-8101PN"
1 MicroTune "LM-8101MT"
1 System data "LM-8101SY"
mt is the memory type:
00 = Internal
02 = Preset 1
03 = Preset 2
7F = Edit Buffer
mn is the memory number (0-63 for Voices, 0-15 for Multis)

The SY77 would seem to be Yamaha model #8101 (the DX7II was #8973, the DX5 was #8933 as shown above, and the TX7 was #8950). Also notice that only single voice dumps are catered for - there is no equivalent of the 32-voice dumps of the DX7. Of course, this standard format is only applicable to Yamaha instruments, other manufacturers implement their own request formats.

Handshaking on the FB01

Because it was produced by Yamaha's Computer Division, the FB01 tone module has several peculiarities which are not found on equipment from the Musical Instrument Divisions. The FB01 has a specialised set of messages just for handshaking, even though the MIDI dumps and other MIDI input/output does not use handshaking! Instead, the three messages are used to indicate the results or progress of dumps and front panel commands, and so can be used to monitor errors or as a way to verify that dumps have been correctly received. You can therefore use an external computer to provide the missing handshaking functions!

The three messages are a subset of part of the published Yamaha System Exclusive format. The format is an evolving one, with additional parts added when necessary. For example, the SY77 dumps described above add a new dump format ($7A) to the already existing ones ($0A, SOB, $7B, $7C and $7E). The three messages are: ACK, NAK and CANCEL, which stand for ACKnowledge, Not Acknowledge, and CANCEL respectively. The messages look like this:

F0, 43, 6n, 02, F7 ACK
F0, 43, 6n, 03, F7 NAK
F0, 43, 6n, 04, F7 CANCEL


ACK indicates that an action (like a dump) has been successful. NAK shows that a checksum error (when receiving a dump), framing error, data overrun or underrun has occurred. CANCEL is used to signify a memory protection error (trying to execute a save whilst the memory protect is on) or an invalid number (out of range). By monitoring this feedback from the FB01, it should be possible to write an FB01 editor program which ensures that all operations are carried out correctly; this is something which almost every other piece of Yamaha equipment does not allow!

MIDI Sample Dumps

The MIDI Sample Dump Standard (see separate panel) can use handshaking to enable the transfer of sample data between samplers. However, some manufacturers use conventional MIDI System Exclusive messages to set up alternative higher speed modes, which they then utilise to transfer the actual data at rates considerably higher than the 31.25 Kbits per second MIDI rate. These transfers are typified by the exchange of information between the two samplers, followed by bursts of data which is usually not readable by MIDI monitors and scope programs. The initial exchanges are made in order to confirm that both ends of the transmission link are capable of operating in high speed mode.

Non-MIDI System Exclusive

As an extension of the idea of having high speed modes used for sample dumps, at least one major sampler manufacturer has used MIDI type electronics running at the standard rates, but has 'hi-jacked' the MIDI messages and redefined them for their own purposes. The Yamaha handshaking used in the FB01 might be mistaken for this misuse of MIDI, because although it would be typified by a handshaking type activity, the usual handshaking messages would not be used, instead the special Yamaha messages would be found.

As we have learned several times in this series, the obvious answer is not always the correct one - so be careful to examine MIDI data fully. It might be a 'rogue' implementation, but it might also be a legal but unpublished version, or even one which has not yet reached your reference books! For instance, the Yamaha SY77 information given in this article currently falls into this grey area of System Exclusive, but this will be rectified once the full MIDI Implementation becomes available.

CONCLUSION



Handshaking extends the ideas of requests and bulk dumps from a simple one-way data transfer to a complex interchange of messages between two devices. The price paid in complexity and in extra cables is repaid in improved integrity of the transmitted data and increased functionality - longer dumps are easier to handle and arrive intact! As the size of MIDI dumps and messages continues to grow, handshaking will become more and more important and is likely to become an essential part of MIDI systems of the future. So far, we have muddled along without it, but handshaking is destined to be the way forward.

This month's coverage of handshaking more or less completes the theory element which I planned into this series from the start. We have now turned the last comer and are accelerating along the final straight; however, there may still be an unforeseen problem on the track ahead, and to avoid it you will have to read next month's exciting episode!

DANGEROUS EDITS?

The continuing theme of the last few installments has been editing using System Exclusive messages - from simple parameter changes to involved bit-formatted decoded dumps. In the course of the descriptions I have tried to show that editing is not only possible but useful as well, since SysEx data can be incorporated into sequences or can be a useful tool for learning about MIDI. However, several people have expressed concern about the potential dangers of using programs like SYSEX to explore the hidden depths of MIDI implementations. Suppose you alter a parameter by accident and then can't restore it, thereby preventing your instrument from working properly?

Well, the first thing I will say is that it hasn't happened to me yet! But more importantly, it is exactly the sort of thing which the designers of electronic equipment 'design out' of systems. When you are working on the software (or hardware) for an electronic device, you build in a number of safeguards to aid you during debugging and testing, and also to ensure that the final release version works and keeps on working. Here are a few of the things which you may find in typical designs, together with some thoughts on why they are there.

- Reset to a known state. When you power a piece of equipment up, it goes through a Reset procedure, where the RAM is set up to a known condition. Sometimes this is a 'snapshot' of the memory at the time the power last went down (most Yamaha equipment stores the setup as it powers down), but sometimes it is a fixed default state - for example, the Kawai K5 always powers up with System Exclusive disabled and the MIDI transmit and receive channels set to 1. The method which restores the snapshot should be designed so that it checks that the instrument is left in a playable state, whilst the default system automatically ensures a sensible condition.

- Check the software. When a piece of equipment is turned on, before anything else it should check the software in the ROMs and the integrity of the RAMs, if possible. A simple checksum will suffice for the ROM, and a few reads and writes can verify the RAM. If the software has failed, then it is better to stop immediately than to let things continue.

- Accessible parameters. Part of the design process for a MIDI input routine involves tracing through its reaction to unexpected messages, since the last thing a tester wants is not knowing whether the incoming data or the on-board processing is the problem. Similarly, the areas of RAM accessed by parameter change messages are also likely to be checked (by hardware and software) by the instrument to prevent System RAM being over-written - when you are working on software, the last thing you want is for your program to corrupt the essential parts of its own memory.

- User reset. The 'panic button' for the programmer of any system is the reset function. Unsubtle switches marked 'Reset' aren't currently the fashion; instead, considerable ingenuity is employed to use obscure combinations of front panel button-presses. Similar deep thought goes into accessing the test modes used by service engineers, although these are frequently associated with the power-up routine. Published information on these reset and test modes is sparse. Although some details have been given in past pages of this magazine.

It is unlikely that you will be able to do any permanent damage to the internal workings of your instrument by sending it incorrect MIDI messages, and even if you do upset the settings, you should be able to edit them back again or else reset the machine to its factory state.


MIDI SAMPLE DUMP STANDARD

The MIDI Sample Dump Standard is one area of System Exclusive which can depend on handshaking for its operation. Here is an overview of how the standard uses handshaking.

There are three fundamental types of MIDI message defined in the standard:

1. Two Data Formats - the Dump Header
- the Data Packet
2. The Dump Request
3. Four Handshaking Flags -ACKnowledge
- NAK (not acknowledge)
- CANCEL
- WAIT

All of these messages use the Universal Non-Real Time message Identification byte: $7E. Notably, the standard says that the four flag messages can be used for other applications in the non-real time field, so you may find them used by manufacturers for general purpose handshaking.

The Dump Request message is used to initiate a sample dump; it is not acknowledged since the ACK message has a very specific meaning in the context of a sample dump, as we shall see. The two data formats are used for contrasting tasks. The first message sent in a sample dump is the Dump Header, which gives important information about the format, period, length and loop points of the sample. If all is well at the receiving end, this will be followed by one or more Data Packets, each of 120 bytes - these contain the actual sample data itself.

The four handshake messages have closely defined meanings:
- ACK means that the last packet has been correctly received, and that the next packet can now be sent.

- NAK signifies that the last packet was not received correctly, and is a request for a re-transmission of that packet.

- CANCEL indicates that the dump is to be aborted.

- WAIT is used to pause the transmission of further packets. This might be used when the receiver's memory buffer is full, or perhaps to allow time to access a disk drive. The next message received will either continue (ACK) or abort (NAK, CANCEL, WAIT) the dump.

The actual transfer process is more complex than this, because it is possible for the transfer to take place without handshaking at all: there are timing elements built into the transfer which time-out after 20 milliseconds and then assume it is a one-way non-handshaking transfer.

Complete details of the full Sample Dump Standard can be obtained from the IMA.

International MIDI Association, (Contact Details).


Series



Previous Article in this issue

Alesis MIDIverb III

Next article in this issue

Patchbay Studio


Sound On Sound - Copyright: SOS Publications Ltd.
The contents of this magazine are re-published here with the kind permission of SOS Publications Ltd.

 

Sound On Sound - Jan 1990

Donated & scanned by: Mike Gorman

>

Should be left alone:


You can send us a note about this article, or let us know of a problem - select the type from the menu above.

(Please include your email address if you want to be contacted regarding your note.)

Topic:

MIDI


Series:

System Exclusive

Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7 | Part 8 | Part 9 | Part 10 | Part 11 | Part 12


Feature by Martin Russ

Previous article in this issue:

> Alesis MIDIverb III

Next article in this issue:

> Patchbay Studio


Help Support The Things You Love

mu:zines is the result of thousands of hours of effort, and will require many thousands more going forward to reach our goals of getting all this content online.

If you value this resource, you can support this project - it really helps!

If you're enjoying the site, please consider supporting me to help build this archive...

...with a one time Donation, or a recurring Donation of just £2 a month. It really helps - thank you!
muzines_logo_02

Small Print

Terms of usePrivacy