Have you noticed how convergent evolution is taking over the world? Look at any car these days and you will see the effect of wind tunnel tests reflected in rounded contours, smooth bodyshapes, etc. All cars are beginning to look much the same. Exactly the same sort of thing can happen with software - when I originally wrote Movie (described last month), it worked with most of the equipment I then owned and fulfilled the requirements of the original specification. For my own purposes I also wrote another program called EASED, which assembled together System Exclusive packets into composite bulk dumps (this month's subject, by the way) so that I could take snapshots of the internal state of all my MIDI equipment. When more complex System Exclusive packets started to appear on devices, Movie began to look underpowered - so I added support for the additional features and ended up with a program very similar to EASED. Last month ail seemed well in the world of System Exclusive packets - so what are these ongoing evolutionary additions all about?

MORE COMPLEXITY

When I introduced the concept of using System Exclusive packets to store dumps of voices or bulk memory contents, I tried to keep the explanation as simple as possible. With this in mind, I talked last month about sending and retrieving System Exclusive packets using either hardware or software sequencers and bulk librarians. Unfortunately, MIDI is evolving as I write and System Exclusive packets have been one of the areas that have changed considerably over the last five years and will probably continue to develop in the future.

In 1983, when MIDI was first proposed, synthesizers were much less complex than they are now, especially in terms of the control provided over performance features - the DX7 (Mk1) had only one performance memory. As MIDI became more understood and was designed into instruments from the initial stages, manufacturers began to exploit the potential, with the result that the TX7 (a DX7 Mk1 sans keyboard) had separate performance memories for all of its internal sounds and used MIDI to provide separate performance memories for all of a connected DX7's sounds as well.

In terms of storing sounds from a DX7, almost everyone ignored the performance memories and just used the simple System Exclusive dumps of one sound or all 32 - after all, with only one memory for the performance functions you usually set it to your own preference and then never subsequently altered it unless you were forced to. For the TX7, the situation was more complex since the performance memories were used to provide extra control over sounds - so they needed to be stored as well. Yamaha provided four separate bulk dump formats for the TX7: single sound and performance dumps, plus bulk sound and performance dumps. All of these were individually transmittable from the TX7 front panel.

Once the idea of using MIDI System Exclusive to store both performance and sound parameters had become established, equipment designers realised that this was an excellent way of providing compatibility between instruments and still retain individual differences. This is most noticeable in the vast range of 4-operator FM synths which followed the DX7. It is possible to transfer sounds between the DX100, DX21, DX27, DX27S, DX11, V2, TX81Z, YS100, YS200, DS55 and TQ5 because they have a complex format of System Exclusive packets. The basic information about the sound is common to all the instruments and is sent as one packet, whilst information about instrument-specific parameters is sent in other packets. Unlike the TX7, sending a voice from one of these instruments sends all the relevant packets - so if you send from a TX81Z to a DX100, the DX100 will listen to the basic sound information but will ignore all the extra information about alternative keyboard tunings, chord effects and pseudo-reverb, etc.

As I mentioned in Part 2, this can lead to complex groupings of packets - when the Yamaha YS200 sends a bulk dump of all its memory contents, it sends several System Exclusive packets, one after the other, just from pressing one front panel button. One of these packets contains the data from the built-in sequencer in Yamaha's 'N-seq' format, so this particular packet will be ignored by almost all the instruments except the TQ5, V50 or other compatible hardware sequencers like the QX5.

COMPOSITE BULK DUMPS

These groups of packets are called composite bulk dumps, because they are composed of several bulk dumps or packets of data. The SANE program from Part 2 was written to show exactly what is inside these dumps. If you use SANE to examine a straightforward voice dump packet from most synthesizers you will get a display which looks something like this:

Packet 1 length=165 bytes F0 40 00 01 ... F7

Here we have a single packet of 155 bytes from a synthesizer whose manufacturer ID code is $40 (Kawai). This is the sort of bulk dump which Movie was originally designed to cope with. In contrast, here is the result of analysing a dump of some of the internal memories of a Yamaha YS200 or TQ5:

Packet 1 length=21 bytes F0 43 00 7E... F7
Packet 2 length=28 bytes F0 43 00 7E... F7
Packet 3 length=41 bytes F0 43 00 7E... F7
Packet 4 length=101 bytes F0 43 00 03... F7

The first packet contains details of the effects section of the YS/TQ synth. The second packet gives details of the aftertouch routing used by the DX11/V2. The third packet contains the extra features like additional waveforms, envelope level shift and fixed frequency operators, which are not present in early 4-operator FM synths like the DX100 but are present in TX81Zs, etc. The final packet contains the basic data for the FM sound and is common to all the 4-operator synths. Notice that each of these packets is a complete System Exclusive message, with a $F0 SYSEX byte at the beginning and a $F7 EOX byte at the end.

Movie was originally designed to cope with single packets - so it would only receive the first 21 bytes of a dump and ignore the rest! The latest versions of Movie correct this by waiting after receiving a $F7 EOX byte to see if another $F0 SYSEX byte will follow. As packets are recognised, Movie displays the number of bytes in the menu bar. So the example shown above would result in this display: 21 28 41 101

When the whole dump has been received, pressing the spacebar on your Atari stops the program from listening to MIDI and will permit storing of the received packets. SANE can be used to determine the number of bytes and packets which can be expected from a particular piece of equipment - most manufacturers tend to hide the relevant information deep in the MIDI pages at the back of the user manual, where it can be difficult to find.

MIDI SNAPSHOTS

We can extend the concept of composite bulk dumps further by using the fact that instruments will usually(!) ignore System Exclusive messages which do not match with their own manufacturer ID code. In the examples shown above, the same ID code was in all the packets - this need not be the case. It is perfectly possible to assemble packets from different manufacturers into one long message: each packet has a $F0 SYSEX start byte and $F7 EOX end byte as usual. The format of these messages is thus:

$F0 SYSEX
$ii ID code for manufacturer 'i'
[data]
$F7 EOX
(there may be a brief pause here)
$F0 SYSEX
$jj ID code for manufacturer 'j'
[data]
$F7 EOX... etc.

The reason for doing this concatenation of packets is much the same as for the multiple packet dumps described above: they are grouped together to form a message which can be saved or loaded as a single file. This removes any problems of accidentally missing out an important packet and thus makes using the messages easier. In fact, the user need not know that the message is made up of smaller packets.

In the same way as with sound parameters, storing a complete set of sound dumps together enables them to be treated as a single file - so you could assemble a composite message out of dumps from each piece of equipment in your MIDI studio and save it on disk as a sort of snapshot of the contents of your instruments at that time. These snapshots can be especially useful when linked to sequencer tracks - how many times have you been unable to find the sounds you used when you originally created a sequence?

The preconditions to assembling composite bulk dumps are much the same as for using Movie - you need some means to select the MIDI Out sockets of the equipment that will be sending the dumps, and you need to be able to initiate the appropriate dumps from the front panels of the instruments whose sounds you wish to preserve. If you use SANE to determine the contents of the messages which result from sending a bulk dump, you can choose between the Movie and EASED programs - if all the messages contain only one packet then you can use EASED, otherwise you will have to use Movie. (If the convergence between Movie and EASED continues, you may see a future 'EASED/Movie Plus' program which will be able to cope with both formats.

EASED

If you had not guessed by now, EASED is the name of this month's Atari program. As already mentioned, Movie is designed to cope with one-off dumps or complex messages made up of several packets. In contrast, EASED is designed to make the assembly of composite bulk dumps easy - it works in much the same way as Movie except that you can add comments as each System Exclusive packet is received. These comments can be examined before you load in a file, so you can give meaningful names to the contents of the overall message - mine have really inventive names like the month and year: JAN89, for example.

At the time of writing the current program version is 0.3. EASED stands for 'Easy Assembler for System Exclusive Dumps' (at last, a decent acronym!) and is designed to make the grouping together of dumps from different pieces of MIDI equipment easier to manage and use.

HOW TO USE 'EASED'

You will need to connect your Atari ST and your MIDI devices together so that you can send MIDI data from the Out of the ST to the Ins of the relevant equipment. Also, you will need some sort of selector switch so that you can select which piece of equipment's Out is connected to the In of the ST. Suitable MIDI switcher boxes are available from Roland, Philip Rees, Akai, 360 Systems, Sycologic and Yamaha. There are two separate parts to EASED:

- Clicking on the BUILD option lets you assemble the separate dumps into a single disk file. You can annotate this file with a description for reference purposes.

- Clicking on the SEND option lets you search through the disk files, looking at the descriptions, and when you find the one you want, it will transmit the file to your network of MIDI instruments.

In many ways, EASED is an extension of the ideas which I started in Movie, the SysEx file utility. Movie is deliberately being kept small and simple whilst EASED aims to take the same approach a bit further... Because of the vaguaries of developing two programs in tandem, you may find that the division between Movie and EASED is very flexible - at the moment EASED is more sophisticated in its documentation of stored files but it lacks the multi-packet capability of Movie.

OTHER POSSIBILITIES

EASED is not the only way to store composite dumps. We have already seen that Movie can be pressed into service doing just that, but there are many other pieces of commercial software which can also be used. Genpatch ST from Hybrid Arts has no problems with appending dumps together into long messages. Trackman from Hollis Research has a special menu selection designed for dealing exclusively with bulk dumps - you just keep sending packets to it and press any key when you want to save the resulting file. Trackman also permits the storage of System Exclusive packets within sequence tracks, but this is best left to short, single sound patches rather than long, multi-packet messages. Microdeal's Super Conductor allows you to see how much room is left in its input buffer as you send packets to it, which is very useful for keeping things under control - unless you have a hard disk connected to your ST, there is no point in assembling composite dumps which are larger than 720K because they will not fit onto a double-sided floppy disk!

Hardware-wise, you need to be aware of the particular capabilities of your sequencer - check with the manufacturer about things like the maximum size of System Exclusive dump which can be received. Sometimes there may be problems with internal buffering inside the hardware, and you may need to pause between sending each dump in order to let the sequencer 'catch up'. Again, the manufacturer should be able to offer some hints and tips for guidance. An obvious prime candidate for storing composite bulk dumps is the dedicated Elka CR99 MIDI Disk Recorder (reviewed SOS April 1989).

PRACTICAL INVESTIGATIONS

All this talk of EASED and ST software may have deflected us away from the nitty-gritty of doing the real thing with actual hardware. When we have finished this month, you should be able to save your work at the end of a session, secure in the knowledge that you can restore it again whenever you want. In order to do this you will need to make some preparations and change the way you work slightly, but the end result should justify the effort. Let's do the preparation first.

Make sure that you have tested out the connections between the various items in your MIDI network and the Atari - especially if you have just installed a switcher box to choose the source of System Exclusive packets. Setting two pieces of equipment to the same MIDI channel and playing a few notes is a good way of testing if you have connected everything up as you thought you had. For transmission of packets of MIDI data from the ST to the rest of the network you will need a Thru box, and it is as well to test this regularly as well, especially if it is battery-powered.

It is probably best to start off by using SANE, making notes on the sort of packets of data which you get when you send dumps to the ST. A technical notebook with details of this sort can be a valuable reference source at a later date, especially if your MIDI network grows like mine has over the last few years! This is also a good opportunity to note down any peculiarities of your equipment - if you make any really interesting discoveries, drop me a line and let me know.

Once you have a good idea of what sort of packets to expect, you can plan out how you are going to assemble them together. Bear in mind that you do not really want any other MIDI data with the bulk dumps, so beware of drum machines or sequencers which transmit continuous MIDI clocks! You should already have checked out your equipment with the Channel Scope and System Scope programs from Part 2, so I am sure you know exactly what is going on. Another point to watch out for is idle tinkling on keys whilst you are waiting (DON'T!) - if you have a MIDI merger instead of a switcher, then you might end up with bulk dumps liberally interspersed with note messages!

Before proceeding any further make sure that you have a backup of the internal contents of all the instruments you will be using - you can use Movie for this. Remember that archive material like this should be copied onto two floppy disks (for safety purposes) and the two copies stored in two different places - much as you would treat a precious master tape. If you have invested time and effort in your sounds, drum patterns or sequences, then they are worth lots of money and may be virtually impossible to recreate.

You should now be ready to make a trial set of transfers from the instruments to the storage device. The details of this will depend on the hardware and software you will be using - just ensure that you are thoroughly familiar with what to do. You should already have checked for correct operation with simple dumps. In order to show the procedure, I will describe the sequence of steps using an example where dumps are sent to the ST and stored with the EASED program.

ASSEMBLY PROCEDURE

You will need to follow a set procedure to build a composite dump, so it might help to make a list of actions to tick as you proceed, especially the first time. A specific example of such a sequence of actions follows for a Yamaha DX7 Mk1, Korg M1 and Cheetah MS6:

1 Load and run EASED, then select the 'Build' option from the main dialogue box by clicking on the button. This will begin the process of assembling a composite bulk dump.

2 Choose a name to describe the contents of the whole dump - something like 'JUNE 89 SETUP' and then press the Atari's Return key (EASED will then wait for a dump).

3 Set the MIDI switcher box to the correct position to select the DX7. Press the DX7's 'Function' button, the '8' button, the '8' again, and set the display to SYS INFO AVAIL with the 'Yes' button.

4 Press the '8' button again to get a display of MIDI TRANSMIT and reply with the 'Yes' button (the DX7 then sends the dump).

5 OK/RETRY (EASED displays a dialogue box saying how many bytes have been received. You should already know how many bytes to expect from your SANE investigations earlier. If anything is wrong, click on the RETRY button, otherwise click on the OK button.)

6 Type a description of the dump, something like 'DX7 Dump Internal', and then press the Return key (EASED assembles the data as you go along. Currently there is no data assembled - so EASED will display 0 Bytes. This is correct.).

7 ADD/FINISHED (EASED displays a dialogue box asking if you wish to add another dump. Click on the ADD button. EASED will wait for the second dump).

8 Set the MIDI switcher to the position for the Korg M1.

9 Press the 'Global' button on the M1, then the '7' button to access the MIDI Data Dump page, and turn the value slider to select the 'ALL DATA' option on the display.

10 Press the 'G' button underneath the '[DUMP]' option on the M1's display to send the data dump.

11 OK/RETRY (EASED again displays a dialogue box saying how many bytes have been received. If the number received is wrong then click on the RETRY button and retransmit the dump, otherwise dick on the OK button.)

12 Describe the dump, something like 'M1 All Data', and then press the Return key. (EASED will now show two dumps on its display and should show a total byte count.)

13 ADD/FINISHED (EASED displays the dialogue box asking if you wish to add another dump. Click on the ADD button. EASED will wait for the next dump).

14 Set the MIDI switcher to the position for the Cheetah MS6.

15 Press the 'Bank' button on the MS6 until the display shows bank 8 on the left-hand side. Press the 'Edit' button and select parameter 71. The display will show '71 St' (store).

16 Pressing the MS6's 'Value Up' button will send the contents of the sound memory in a dump to the ST. (Change the parameter number to abort the data dump, if necessary.) The display will read '71 —' and then '71 dn' (done) when the dump has been completed.

17 OK/RETRY (EASED again displays the dialogue box which shows how many bytes have been received. As before, if the number received is wrong then click on the RETRY button and re-transmit the dump; if all is fine then click on the OK button.)

18 Type a description for the dump, something like 'Cheetah MS6 Tones', and then press the Return key. (EASED will now show all three dumps on its display and should show a total byte count.)

19 ADD/FINISHED [EASED displays the dialogue box asking if you wish to add another dump. Click on the FINISHED button.)

20 EASED will display the file selector dialogue box. Type in a suitable filename for the file you have assembled (eg. 'JUN89') and click on the OK button. EASED will create two files on the disk: JUN89.EZY and JUN89.IZY. These contain the dump data and the descriptive text, respectively. Always keep these two files together — EASED will not work correctly if you lose either of them!

21 EASED returns to the BUILD/SEND dialogue box. You may assemble a composite bulk dump, send a dump, or quit by clicking on the appropriate button.

Although the above procedure looks complex at first, this is a direct result of the amount of detail I have given. Once you have done this a couple of times you will become familiar with the necessary actions and it should run very smoothly. If you have different equipment, all you need to do is make a note of the required actions to send a dump and replace the relevant items above with them. The whole process can be automated if required, although the explanation will have to wait until we have covered another topic - so stay tuned!

RESTORING

This is much simpler than the assembly: you choose the Send option and then look for the file you wish to restore. This is where the descriptions you give to the stored data can become very important. You may remember what 'JohnsInt' means just after saving it, but you may be less sure a couple of months later. For serious archiving it may be worth the effort of using a separate notebook just to keep track of the contents of files - this can speed up searching through descriptions considerably. Assuming that you can find the file you want to restore, the actual process of sending is initiated with a single click of the mouse on the correct button in the 'Do you want to send this file?' dialogue box.

Unlike during assembly, sending the composite file back to your instruments does not require any switching - remember that the manufacturer's ID enables individual pieces of equipment to filter out any System Exclusive packet not intended for them. This means that a single Thru box is used to send the same set of packets to all the equipment in the network, and each instrument will listen to the data but only pick out the specific packets that are relevant to it. It does not matter that you send a bulk dump from a Yamaha DX7 to a Kawai K5, or whatever - the K5 will ignore it and will happily wait for a dump with the correct Kawai manufacturer ID byte.

Very early MIDI instruments may occasionally flag up a problem when you start sending lots of SysEx packets from other manufacturers to them, but this usually only shows a parochial attitude to System Exclusive and rarely results in any major problems. As MIDI becomes increasingly utilised for this type of multi-manufacturer networking application, assumptions which might cause problems should gradually disappear. I do not know of any piece of MIDI equipment which has been known to mistakenly try and decode a dump from another manufacturer.

SUMMARY

Assembling System Exclusive dumps into a composite form may take some time and require careful preparation, but the advantage of being able to restore a 'snapshot' of all your sounds, rhythm patterns and sequences makes it well worthwhile. The one exception to this wonderful scenario is sample dumps: Movie and EASED are both limited to a total composite packet size of 128 Kbytes. Most serious users of samplers are going to need considerably more than this and so will need to use an alternative means for backing up their sounds; hard disks may be an obvious solution to rapid access of sounds, but a tape-based backup system has more appeal for archiving of sounds.

I hope that this description of the mechanism of putting together composite bulk dumps has not been too confusing. There really is no substitute for actually trying it out yourself. Although I have concentrated on the specific example of EASED rather than advertise a particular commercial product, bear in mind that many sequencers (both hardware and software types) can also be pressed into service to perform exactly the same type of composite file storage.

Next month we change direction slightly and take a look at another completely different use for System Exclusive packets.

DISK INFORMATION
The 'SysEx Toolkit' (disk 66) contains most of the programs referred to in this article on 3.5" disk for the Atari 520/1040ST (hi-res only). It costs a mere £7. Channel Scope and EASED are available separately, price £7 each.

SOS Software, (Contact Details).