As the official DX7 Voice Librarian for the Yamaha X Club, one of my assigned tasks has been to distribute the club voices to members and any other interested people. This was done originally for the BBC computer, and more recently for the Atari ST computer.

Yamaha's aim with these public domain voices is to make them widely available to any user of Yamaha 6-operator FM synthesizers (if only all manufacturers adopted the same policy!). The result is that the Yamaha X Club 6-operator voice library is available from SOS Software (see page 38). There are currently two 3.5" disks, each containing about 50 sets of 32 voices, making 3,200 FM voices in all. Included on each disk is a simple GEM program, which allows the voice data files to be transferred to any 6-operator Yamaha synthesizer. As more voices become available I will add further disks as necessary, if there is sufficient interest.

FORMATS

You are probably wondering why I have started by talking about distributing voices for synthesizers, and as usual there is a very good reason. I have discussed many aspects of using System Exclusive in this series, but have so far avoided any thoughts on the actual implementation of storage. When I described the Movie program, I briefly mentioned compatibility with other programs but have omitted any details so far. As often happens, following one line of enquiry leads nicely to another, and in this case, looking at the formats you can use to save System Exclusive information leads to some very useful information about 'checksums'.

Having mentioned the DX7 already, now seems like a good time to talk about Yamaha's standard 6-operator 32-voice bulk dumps, as found in the voice files on the SOS Shareware disks. This will form the background information for what follows; bear in mind that the ways in which you can store such dumps are probably common to both computer and hardware systems, irrespective of the piece of equipment supplying the System Exclusive data.

The format for these bulk dumps is the same as for any System Exclusive message, and follows Yamaha's original standard format for such transfers, as established by the first generation of DX7s. (A newer 'universal' bulk dump format is also used by the DX7 MkII and other second generation FM instruments, but all 6-operator FM synths will correctly recognise and respond to the original format dumps.) The basic format looks like this:

$FO start of System Exclusive
$43 Yamaha Identification
$00 Sub-status and channel (ssss nnnn, where n = MIDI channel)
$09 Format number
$20 Byte count MSB
$00 Byte count LSB ($2000 = 4096)
... firstdata byte
...
... last data byte
$XX checksum of data bytes
$F7 end of System Exclusive

Six bytes are added before the voice data and two bytes after it. These are called the 'header' and 'tail' of the bulk dump. The six header bytes indicate the start of System Exclusive and the manufacturer (Yamaha, in this example), whilst the two tail bytes contain just the checksum and the end of Exclusive message. (Checksums are explained in more detail in a separate panel.) The total size of the System Exclusive bulk dump is thus 4096+6+2 = 4104 bytes. The important thing to observe here is that the header is unlikely to change from one dump to another - the status, format and byte counts are fixed. The $F0, $43 and $F7 bytes are also common to all Yamaha System Exclusive messages.

The data bytes themselves are organised in a similar way to the general outline explained last month: a list of the parameters for the operators used in each sound are arranged in a fixed order. This time we do not need to know the details of the organisation of the bytes, only the structure of the message into header, data and tail.

There are basically two different and incompatible ways of storing these 6-operator FM voice dumps (and many other bulk dumps as well) in a computer. They are called by a wide variety of names, but I will use just two: 'Headerless' and 'Image'.

HEADERLESS FILES

Headerless files consist of just the voice data for each of the 32 voices in a 6-operator bulk dump. The System Exclusive header and tail information is not stored, since it can easily be added for re-transmission, and omitting it saves storage space. The checksum is also left out, since it is relatively easy to calculate (see 'Checksum' panel). The file size reflects the sum of 32 voices, each of 128 bytes: ie. 4096 bytes.

Removing just eight bytes from a total of 4104 may not seem like a worthwhile gain, as 513 lots of eight bytes need to be saved in order to equal just one extra voice dump. 513x32 equals 16,416 voices, which is more than I have in my entire collection, and I would need just over two megabytes of storage to contain them!

In practice, because of the way that most storage is designed, it is usually advantageous to have files of a particular length which matches or is slightly less than the basic unit of storage.

As an example, suppose that we store the 4096 bytes in a system which utilises basic storage blocks of 1024 bytes. 4096 bytes would occupy exactly four blocks, whilst 4104 bytes would require five - wasting most of the fifth block. This wasted storage space is not usable, because of the minimum addressable storage block of 1024 bytes. You should be able to see that after saving four lots of 4104 bytes in five blocks, the wasted space amounts to enough storage for an extra set of 4096 bytes! Losing 1/5th of the storage capacity is much more serious, and minimising the size of the file you store goes some of the way towards minimising such losses.

IMAGE FILES

Image files are so called because they are an exact image of the MIDI data which was originally sent - nothing is removed, and so they can be sent as they are to a suitable synthesizer. Thus the 4096 bytes of data have the header and tail intact, resulting in a total file size of 4104 bytes.

This is the file format used for programs like Steinberg Pro24 - .SND files are suitable for loading into it. Movie's .MVI files are also images, and so are compatible with any other image-based storage method.

CONVERSION

At this point I would normally say that I have just finished writing a program to convert between the two types of file (headerless and image), but in fact, one of the first programs offered by SOS does the job already! It can only be used for converting between .DTX and .SND formats of DX7 bulk voice files. The MIDI Bulk Dump Convertor is available from SOS Shareware (Disk 07) and converts voice files from either format to the other, using a friendly user interface. I offer no prizes as to the author!

Converting between headerless and image formats for other pieces of equipment is only a matter of deciding which format it is (use ASCHEX to look at the contents of System Exclusive data files), and then supplying the header and trailer, or removing them as appropriate.

ASCII FORMAT

The above two file types are stored directly in memory or on disk files, and so printing them out on the screen or on paper produces meaningless rubbish or upsets the printer, because the values sent to the printer/screen represent control characters rather than printable characters. This can be cured either by using a special program to view the file contents, by converting the bytes to a hexadecimal or decimal representation, or by storing the file data in a printable form in the first place. The latter type of storage is called 'ASCII format' and usually converts each byte into two hex characters followed by a space character. For example:

F0 43 00 09 20 00 32 43 54 72 44 00 00 12 23 34 76 45 00 00 46 F7

Some alternative schemes convert the data to three decimal characters with spaces:

240 067 000 009 032 000 004 057 013 064 078 068 045 000 002 102 247

In each case the resulting file is printable, although it is three or more times as long as the more compact but unprintable version. Superconductor, the ST sequencer program by Michtron (available from Microdeal), stores System Exclusive information in this format, enabling an ordinary word processor to be used to edit the information.

THIS MONTH'S PROGRAM

After the revised versions of SYSEX which have appeared in the last two installments, it may not surprise you to learn that this month's software is also another variation on the same utility. SXCHEK is almost exactly the same as SYSEX in its use, except that it has two screen areas in which to place values. The upper box contents are not included in the checksum, whilst the lower box contents are - this means that you can choose which parts of the message are to be in the header and which parts are to be in the data. The checksum byte is added to the message after the data and just before the $F7 end of Exclusive byte. SXCHEK also adds a few more manufacturer ID numbers to those found in SYSEX V0.3.

THE PRACTICAL BIT

Since this part of the series has a panel which covers the topic of checksums, the practical content this time continues the editing theme of the last few episodes and gives an example of using checksums whilst editing.

ROLAND U20 SAMPLE PLAYER

Roland have a consistently high standard of documentation, but their MIDI System Exclusive information can be difficult to interpret because it splits the explanation into two parts: first a general discussion of the System Exclusive information for all Roland instruments, and then the specific details of the instrument in question. This means that you need to correlate two different sources together in order to figure out exactly what messages are needed. To help you, here is my interpretation of what the Roland U20 documentation means...

If we wish to alter parameters we only need to send messages to the U20, and so only one MIDI cable is needed: from the ST's Out to the U20's In. This is thus a 'One-Way Transfer', and Roland define two types of message which can be used:

- RQ1 is a Request for data message, and is used for interrogating the instrument about the current value of parameters. Since you need MIDI cables to send data back in response, this should not really be part of a 'one-way' system, but it is included because the implied use of a 'One-Way Transfer' is for the transmission of small amounts of data - like single parameters. For larger amounts of data, such as bulk dumps, the 'Handshake Transfer' mode would be used. We will look closely at handshaking in a future article.

- DT1 is a Data Transfer message, and is used to send information about parameter values to and from an instrument. DT1 messages are always less than 256 bytes long (remember that 'One-Way' transfers are for short messages). It can be sent from the Atari ST to the U20 to alter a parameter value, or it can be output from the U20 in response to an RQ1 message. The basic format of the DT1 message is:

$F0 SysEx
$41 Roland ID
$dd Device ID
$mm Model ID
$12 DT1 command
$a1 Address MSB
$a2 ...
$a3 Address LSB
$vv data value
$ss checksum
$F7 EOX

The Device ID is used to uniquely identify the instrument. This means that several U20s could utilise the same MIDI channel but could each be controlled separately. The ID is factory set to 17 for the U20 and is included within messages in the standard MIDI channel form, ie. one less than the number shown. So the message byte will be 16 or $10. The Model ID is used to identify the type of instrument - the byte for the U20 is $2B. The $12 in the Command byte position shows that this is a DT1 message: a $11 here would indicate an RQ1 message.

The Address bytes are used to show which parameter inside the U20 is being accessed by the message - most Roland equipment uses three bytes for the address. The data value is always seven bits or less, and so occupies only a single un-nibbleised byte. The Checksum byte is used to ensure that the message has been correctly received by the U20. It is a number which represents the sum of the bytes sent in the message, such that the U20 can verify that the message has not been corrupted during transmission. Checksums are more normally found in bulk dumps where there are large numbers of bytes and errors might be encountered. In the case of short parameter edits, corruption is unlikely and has an unfortunate side-effect for last month's programs, as we shall see.

The U20 documentation shows the address of the parameter and other information in a series of tables, with almost no further explanation. For example:

Notice that unlike the Emu Systems table shown in Part 7, where the number of bits were explicitly stated, with the U20 table you need to examine the range of values to decide how many bits you will be changing. The Tone Number has a range of 0 to 127 and so requires all seven bits, whilst the Arpeggio Type only has four values (0 to 3) and thus only requires two bits. Taking the Arpeggio Rate first, the message would be:

F0, 41, 10, 2B, 10, 01,2A, vv, ss, F7

where vv is the value we want to send for the speed of arpeggiation, and ss is the checksum. Notice that the checksum is the last byte to be sent before the EOX ($F7) byte, and so our SXMOUSE and SXMOD programs cannot be used (the current versions of these programs cannot work out checksums 'live'). Programs like MIDIman from Hollis Research can send checksums, and some of the research for this article was carried out using MIDIman. It is still possible to use SYSEX to send suitable messages, but this means that we need to know how to calculate the checksum byte.

The checksum applies only to the address and data bytes: $10, $01, $2A and vv in the example above. Adding the address byte together gives $3B. Subtracting this from $80 gives $45, and this is the checksum for a vv value of zero. As vv increases, the checksum value will reduce to correspond: the sum of vv and the checksum is always $80. This is the two's complement of the sum. For example, at a value of $45, the checksum will be $00: the next highest value will be $46, with a checksum of $7F; and the highest value of $7F will have a checksum of $46. This checksum calculation is that used in SXCHEK, and so will work with some editing messages and not with others. Some experimentation and reading of documentation is required if you want to try this.

The U20's Key Transpose message is similar in form to the above example, but needs the vv value to be within the numbers 28 and 100 (as shown by '28...100' in the table above). The checksum is calculated in exactly the same way as just described. A value covering -36 to +36 with the numbers 28 to 100 has a value of zero at 64, which is $40 in hex, giving a sum of $10+$01 +$25+$40 = $76. Subtracting $80-$76 = $0A, finally giving $0A as the checksum for the zero value. So:

F0, 41, 10, 2B, 10, 01, 25, 40, 0A, F7

sets the transposition to zero, while:

F0, 41, 10, 2B, 10, 01, 25, 34, 16, F7

sets it one octave up, and:

F0, 41, 10, 2B, 10, 01, 2B, 1, 43, F7

turns the transposition on, with:

F0, 41, 10, 2B, 10, 01, 2B, 0, 44, F7

turning it off again.

In exactly the same way, the other parameters shown in the table can be manipulated as desired. Changing the Tone Number whilst playing (there is a slight pause on the U20 while each new PCM sample is loaded into the Timbre) can prove very dramatic as a special effect, and this concept is taken further in the Ensoniq VFX with its dynamically swept wavetables. Before anyone else reminds you, I ought to say that this technique actually first appeared in PPG synthesizers some years ago!

SUMMARY

Although the specific information described here applies only to 6-operator Yamaha synthesizers, the basic principles of headers, checksums and tails surrounding the data is the same for all System Exclusive bulk dumps. In particular, the methods for storing and displaying the files, as well as calculating the checksum, apply to all dumps.

So far, all the information on bulk dumps has avoided any detailed mention of 'handshaking'. This will be rectified in the next part of this series.

FURTHER INFORMATION

The programs mentioned are available on the SysEx Toolkit 1 and SysEx Toolkit 23.5" disks (Atari ST only) and cost £7 each inc. postage.

SOS Software, (Contact Details).