Why is it that you always feel guilty when you pass through the 'Nothing To Declare' green channel at Customs? Even with nothing to declare there is always the fear of being pounced upon by some eager uniformed Customs Officer and asked the fateful question: "And can you tell me what is in that packet, sir?" If the packet is a MIDI System Exclusive message then you could be in trouble, unless you keep on reading!

Externally, all MIDI System Exclusive packets, dumps or messages all look very similar. They start with a $F0 byte, followed by an ID code. What follows is a block of (apparently meaningless to the untutored!) data, with a $F7 byte at the end just to tidy things up. The second byte, and those which follow it in the header, are the ones which matter here - they allow us to determine exactly what the contents of the packet are.

Last month we looked at some of the different types of packet that have been defined. There are really only two major types: Manufacturer and Universal. As the names imply, Manufacturer packets contain information that is specific to the products of a particular instrument manufacturer, and often only to one instrument within the range. In contrast, Universal packets contain messages which can be widely utilised by many different products, but which do not warrant a MIDI status byte all of their own - things like Sample Dumps or MIDI Time Code.

The Universal packets are defined in the MIDI Specification Extensions and are so far restricted to just two specified uses: the Sample Dump Standard and MIDI Time Code. The Non-commercial area is not defined at all, apart from the basic outline of the format. The format of Manufacturer packets are again not formally defined by the IMA (International MIDI Association), but are designed and published by the individual manufacturers - who can therefore do anything they like!

Given a completely free rein, what have the manufacturers done? What do they use System Exclusive packets for?

FIRST STEPS

The first MIDI System Exclusive packet to enter the real world was probably a dump of the internal patch memories from the Sequential Prophet 600 - the first ever MIDI synthesizer (c. 1983). Being able to send the contents of internal patch memories between two Prophet 600s probably seems very unremarkable now, but given circumstances where the only alternative was to write settings down on a piece of paper and manually enter them on the other instrument, it was a truly amazing achievement. It also set the pattern for things to come. The voice or patch dump has become virtually a standard feature of MIDI devices - it is almost taken for granted in reviews, and would certainly get a mention if it was omitted. This ability to transfer sounds between instruments has generated a new practice - the swapping of sounds. Instead of spending hours developing your own sounds on your Yamaha CS80 and then storing them in the miniature replicas of the front panel hidden beneath a flap on top of the machine, you could now connect your synth to anyone else's and swap sounds.

This trend started with the introduction of MIDI, and it produced a wider awareness of the popularity of pre-programmed (preset) sounds, resulting in the rapid appearance of memory cartridges (and now memory cards) which could be used to store and swap sounds. These things tend to snowball, and the performance-oriented aspects of the instruments were next on the agenda. As MIDI appeared on more and more devices, so the uses multiplied. At first the exploitation of the digital transfer was restricted to the storage and retrieval of these simple voice dumps, but the next logical step was to make the front panel controls of the instrument accessible from MIDI System Exclusive messages, and eventually to map parameters inside the instruments to incoming MIDI data.

The list of possible uses for MIDI System Exclusive messages includes: voice or patch dumps in both single and bulk forms; performance dumps; drum patterns; sequencer files; effects patches; mixer parameters (DMP7 etc); maps for MIDI Program Changes to internal patches; tempo maps; Controller mapping; general information dumps; and sample data. Often neglected are the 'remote control' aspects of MIDI, and the accompanying 'Making A Request' panel shows some of the uses for this type of message.

These two major applications, storage/retrieval and editing, provide a convenient split for classification purposes. This month I will concentrate on the first of these: storage and retrieval.

STORING AND RETRIEVING

As System Exclusive is often ignored, the potential uses listed above are frequently overlooked. Most manufacturers are very happy to sell RAM cartridges, memory cards, disk drives or hard disks on which to store the relevant information whilst you are working with their products, but this typically results in a large collection of small lumps of plastic with diverse and uncatalogued bits of work on them! The cost of all these individual storage devices can also be prohibitive.

Not surprisingly, the solution involves using the MIDI System Exclusive messages for the librarian-like functions of data storage and retrieval. Storing all your files on one machine (such as an Atari computer) has several advantages:

- You can put all the separate files from many different devices (drum machine patterns, voice or patch dumps, sequencer files, etc) together into a single folder or directory, and store this on a single 3½" floppy disk.

- This single disk is easy to backup and easy to update. Try finding all the sounds, patterns and sequencer files from a song you did six months ago!

- You avoid having to buy more than the minimum quantity of cartridges, cards, etc, needed to work efficiently with the instruments (usually one per instrument, since they generally double the effective memory capacity of the machine for sounds, patterns, etc), and you therefore save money. 3½" disks are much cheaper than RAM cartridges or memory cards.

The price to be paid for all these advantages is, firstly, that you need to find out how to use System Exclusive dumps; and secondly, you need the software or hardware to be able to store and retrieve the data between MIDI and disks. Part 1 of this series gave some examples of the hardware devices and software which can perform this sort of function - further lists can be found in the MIDI Data Recorder article in the March 1988 edition. (Please note that the listed Emax sampler does not allow the storage of System Exclusive messages - some gremlins crept into my word processor!).

The first task is therefore to find out how to initiate the sending of a suitable SysEx dump from your instrument to the storage device. This means reading the manual, I'm afraid! This sort of information is usually in the MIDI section, and should involve only a few button presses in most equipment. Some manufacturers make things really easy sometimes: the DX7 (Mk1) can be set to send a voice or patch dump via System Exclusive every time you press one of the numeric buttons to select a new sound. Later on I will show you how to automate this process, but for now manual control will suffice. If you buy some white PVC tape with a low-tack adhesive, you could put scribble strips onto the appropriate instruments to remind you of which buttons, functions or jobs you need to use to initiate a suitable dump.

In addition, you will need some way to select the relevant MIDI Out and connect it to the MIDI In of the storage device. You can use MIDI cables, MIDI switching or patching units like those made by Philip Rees or DACS, or even use a MIDI merge box if you only have a couple of devices to deal with. Be careful if you decide to use a merge box, because it can be easy to set up a MIDI loop when you are using a computer to control instruments with its MIDI Out socket whilst receiving MIDI data at its MIDI In. Any hint of a software 'Thru' function, where the incoming data is duplicated at the Out, can cause problems, so beware. In normal use, where you will only need the connection to transmit a dump once a session, loose 'flying' leads will suffice. All of this is reiterated in the article on how to use MIDI Data Recorders in SOS May 1988.

SOFTWARE CHOICES

I mentioned that you would need a dedicated hardware device or suitable software in order to store and retrieve the MIDI data. The hardware has already been listed in Parts 1 and 2, so I will now look at the software available.

There are two major types of program which offer the ability to store MIDI System Exclusive messages: Sequencers and Generic Bulk Dump Librarians (a 'bulk dump' is just a shorter way of saying a System Exclusive multipatch message). Rather than examine all of the sequencers, I have chosen to look at three representative examples. If you are looking for a suitable sequencer then asking about the System Exclusive storage abilities of a program is a very good way of ensuring that (a) you are treated seriously, (b) you are allowed to talk to someone who actually understands and probably really uses the program, and (c) you are given a hands-on demonstration of this aspect actually happening.

The sequencers I have chosen are: Steinberg's Pro24, Hollis Research's Trackman and Microdeal's Superconductor. Each program description is accompanied by a composite diagram showing some of the screen dialogues and menus used - these are not screen shots but have been assembled in this fashion to give an impression of the facilities.

PRO-24

Pro24 (version 3.0) has two separate means of utilising MIDI System Exclusive messages or dumps: you can use the Dump Utility as a simple dump librarian, or you can incorporate dumps into tracks by not filtering out the System Exclusive information. Placing SysEx data into tracks in this way has the welcome advantage that a single sequence file can specify the sounds used in the song as well as the notes, although lengthy patch dumps can eat up memory and time!

The Dump Utility has two modes of operation: the default is the Normal Mode, where a dump from any instrument will be received into a buffer for subsequent storage on disk. You can create your own dump request messages and then transmit them to the MIDI instrument from which you want to request a dump (see the 'Making A Request' panel), but you cannot save them for later re-use. The second mode uses 'modules'; segments of code which are loaded into the program to specify things like the data format and dump request message. The copy of Pro24 version 3.0 which I used for these tests only had a few modules supplied, covering the Casio CZ101, Korg DW8000, Yamaha FB01, DX7, TX7, Ensoniq ESQ1 and Roland Alpha Juno 1 and 2. Unfortunately, creating and then saving modules is not possible, and writing your own does not seem to be an easy task - they consist of typically 3 Kbytes of data.

I had no difficulty in using the Dump Utility to receive, store and retrieve System Exclusive dumps ranging from single patch dumps to complex eight-part multi-packet messages of more than 16 Kbytes of data. Pro24 recognised and correctly decoded the Manufacturer ID byte for all the instruments I tested it with. The Dump Utility saves and loads dumps in the form of .SND files, and these are compatible with Movie files (see below).

TRACKMAN

In Trackman (version 1.4), a dump librarian is provided to save and load System Exclusive messages between disk and MIDI instruments and this will accept any MIDI data - there are no configurations or modules needed. This does not have a dump request facility and so you will need to manually initiate the required dumps from your instruments. It is also possible to record SysEx data within a sequence. As with all operations in Trackman, using it was intuitive and straightforward - there are two menu selections for sending or receiving data, and you interact with a couple of dialogue boxes. You can annotate the data with 16 characters of descriptive text if you wish, and this is stored with the MIDI data in the same file (useful for keeping track of the contents of messages).

I experienced no problems when using Trackman's System Exclusive data storage facilities. It also recognised the Manufacturer ID bytes and correctly decoded them, intelligently placing the name into the optional information field. Both long and short messages were presented to the Receive MIDI section and no difficulties were experienced. The feedback from the Send MIDI dialogue box was particularly clear, with a bar showing the percentage of data transmitted. Trackman uses .DMP files for these dumps, and these are again compatible with Movie (see below).

SUPERCONDUCTOR

Superconductor offers a similar system to Trackman, in that a separate part of the program is used to provide storage and retrieval of System Exclusive messages. This is perhaps the clearest of the three programs in terms of technical presentation - the only notable omission is the lack of any decoding of the Manufacturer ID byte and the absence of a dump request facility. The design uses separate receive and transmit buffers, with disk storage being made to and from the transmit buffer only. Suitable transfers can be made from the receive buffer to the transmit buffer.

Rather than store the MIDI data as a data file, Superconductor stores it as an ASCII text file of printable characters in the form of hexadecimal numbers. This enables the user to edit the MIDI data in a normal word processor and can be very useful for investigating System Exclusive - using one of the public domain Desk Accessory word processor programs should enable editing to be carried out without quitting Superconductor - something neither of the two more expensive sequencers allow! Because of this Hex-ASCII format, the SysEx files are not compatible with Movie, but Superconductor's own editing facility is probably far more useful!

Advanced sequencers often allow you to store single voice patches as part of a song, so that every time you load the song it loads the correct patch data into the appropriate synth(s). They can usually act (in a limited way) as Librarians too, but for comprehensive features you really need a dedicated Librarian program. These usually offer editing facilities as well as the ability to move patches around into convenient groups. If you don't like the idea of buying lots of separate Librarians for each of your instruments, then you need a Generic Bulk Librarian. The 'generic' aspect means that it can cope with any System Exclusive message, from any instrument, as long as you have the information about the MIDI implementation etc, but you lose such features as moving voices around within banks and editing them.

GENPATCH ST

Just about the best Generic Bulk Librarian program I have seen is GenPatch ST from Hybrid Arts (reviewed SOS March 1988). It comes ready supplied with the necessary configurations for a comprehensive selection of instruments, but programming your own is just a matter of reading the GenPatch manual and the MIDI section of your instrument manual, and then doing some simple programming to define the 'handshaking' and MIDI dump request, etc. This is no more difficult to do than setting up your video machine to record Gerry Anderson's excellent 'UFO' programme for an hour between 2am and 3am!

Since it is not specific to any particular instrument, GenPatch ST cannot provide for the editing of patches or MIDI data, or patch moving within banks, but it is excellent at providing librarian facilities for complete dumps from a wide range of equipment. The cost saving compared to separate dedicated librarians could be enormous. GenPatch ST also provides many of the tools and utilities which I will be describing in this series.

THIS MONTH'S PROGRAMS

No article in this series would be complete without some software, and this month is no exception. I don't have the time to programme anything as good as GenPatch ST, and cloning it would be pointless anyway, so instead I have developed a simple, no frills, generic MIDI System Exclusive Dump storage and retrieval program. You can't get away with ridiculous titles like that, so I have called it Movie. The companion program to this provides a means of examining the files that Movie produces, on screen, and this is called ASCHEX.

MOVIE

At last year's British Music Fair, Richard Elen (producer, past Editor of Studio Sound, Macintosh enthusiast) asked me to write a simple utility program which would save and load MIDI data on disk using the Atari ST as the computing interface. Movie is the result: it moves MIDI data to and from disk exactly as it receives it. The MIDI data is not altered or interpreted in any way, it is more of a 'snapshot'; a 'movie' of the incoming MIDI data.

Movie works in all three Atari ST screen resolutions, but the best display is obtained in medium and high resolution modes. It only works with System Exclusive packets that do not use handshaking. This means that most Casio synthesizers will not work with Movie. Roland synths and MIDI devices which use only the Roland handshake transfer mode will not work either. Most other equipment will work fine.

The .MVI files which Movie saves to disk are not MIDI Files - they are not compatible with the standard sequencer files which allow transfer of songs between sequencers, etc. These MVI files are just images of the MIDI data. To look at the data within a .MVI file, use ASCHEX.

Movie is controlled by the Atari mouse. All interaction with the program is via dialogue boxes and the file selector. The main screen gives you the choice of transferring MIDI data from the ST to your MIDI equipment, or to the ST from your MIDI equipment. Check that you understand which way the data will be going! You do not need to add the .MVI extension, Movie will automatically add it to your filename. It will also remember where you looked for the previous file and will return to that directory when you look for another file - this speeds up access to sequentially ordered files.

ASCHEX

Movie uses the Atari ST's disk drive as its storage medium. Incoming MIDI data is stored in .MVI files, and these can then be replayed via the MIDI Out socket. Unless you have used the SOS System Scope program or the SANE program on the data before you stored it with Movie, then you will have no idea how the MIDI data looks. ASCHEX solves this problem by letting you see what is inside the .MVI files which Movie produces. All MIDI system status bytes are shown as outlined characters, so you can quickly differentiate $F0 and $F7 messages, as well as any real-time clock messages, etc, inside the data. Some manufacturers put ASCII text into the headers to identify the purpose and contents of their dumps, and ASCHEX is ideal for looking at this sort of information.

When you run ASCHEX a file selector box will appear. This allows you to choose a file to examine on the screen. The Atari keyboard commands are displayed along the bottom of the screen. ASCHEX does not use the mouse, except in the file selector, but it does remember which directory you last looked in and will return there the next time you want to select a file.

PRACTICAL

Now you should have all the ingredients you need to be able to store some System Exclusive packets, dumps and messages. In order to avoid unfortunate accidents, it is wise to first try storing and loading the factory preset sounds or patterns; any data lost during the learning process will then be recoverable. Since we will be loading the data back into the memory of the instruments in question, it is also wise to make sure that you have a backup of any of your own sounds, patches or patterns - the single cartridge I mentioned earlier comes into its own here!

Having backed up your sounds, etc, you can connect the MIDI Out of the instrument to the In of the ST and try sending a voice or patch dump to the Atari. Verifying that you are receiving the right sort of packet with System Scope or SANE could be a good idea, since most Bulk Librarians give very little feedback as to what is in the file they have captured; Movie tells you the length of the packet only! This MIDI data packet should now be saved to disk, and the disk labelled and backed up.

The proof of the pudding is in the eating. So far we have sent some data to the Atari and stored it on disk, but we cannot be sure that we have done this correctly until we have retrieved the data again. To make sure, it can sometimes help to load a different set of voices, etc, into the internal memory of the instrument to which we will now return the MIDI data. Just changing the name of the first memory often suffices to indicate whether the data has been correctly received. When you are ready, send the data back from the ST to the instrument from which it originally came.

If all is OK, then you should find the instrument back as it started, with the sounds or patterns as before. If not, then something is going wrong: check what you are doing; use ASCHEX to check the file on disk; check the data coming out of the instrument with SANE; check that the instrument's memory protect function is off; check that the Receive MIDI channel is the same as the one you sent the data on in the first place; check your MIDI cables - you may have the instrument's Out connected to the ST's In, but is the ST's Out connected to the instrument's In?

Assuming all is well, you can now repeat the operation using your own sounds, patterns, etc. You should be able to store the contents of many data cassettes, RAM/ROM cartridges, or memory cards on a single 3½" disk, but remember to make a backup copy (and use good quality disks).

USING SYSTEM EXCLUSIVE DUMPS

You can use System Exclusive dumps for all manner of things, depending upon the equipment you have. I use SysEx disks to store 'snapshots' of the contents of my MIDI instruments at particular dates, so that I can restore the internal memories to their past glory. I tend to undertake this sort of 'backup' whenever I am about to test some software which uses System Exclusive messages. Whilst I was developing Movie, I suffered several cases of accidental over-writing of the internal memories when debugging the program and was very glad that I had all the original sounds stored safely away!

If you do a lot of work arranging sounds for use with sequencers, then you could make dumps of the sets of voices used for particular songs and store them on the same disk as the sequencer files - this makes it easier to return to the songs at a later date because the sounds you will be using are the same, and not the sounds from the latest ROM cartridge you happened to have loaded into the internal memory. Alternatively, you could arrange the voices into similar groups to ease the task of searching for a specific sound; you could have a disk for string sounds, another for lead sounds, another for bass sounds, etc. For live performance, you could store the sounds used for a particular set as backups for RAM cartridges, in case of accident.

So far, I have concentrated on the sound storage aspects of System Exclusive. The same principles apply to drum patterns and effects patches as well; in fact any data which could come under the general heading of 'instrument memory dumps'. I know of quite a few drum machine owners who never save their patterns - instead, they search through the presets or the existing patterns until they find one they like and edit that, or else they look for a pattern they don't like, erase it and start from scratch.

In a world where information is becoming more and more valuable, it makes a lot of sense to store everything you do so that you can access it later if you need to. By using System Exclusive packets to store data you need not lose everything forever when the power is turned off! If you have ever seen a professional synthesizer programmer turning up at a recording session, you can always recognise him by the large collection of disks, RAM and ROM cartridges, etc, he brings with him. If you have invested time in creating sounds, patterns or effects, then save them!

It is not only sounds that can be stored in SysEx format: MIDI Maps, which control which internal sound is played when a Program Change message is received, can be stored so that once you have set Program 1 to be a Piano sound, it will stay that way. The Roland D110 is an example of an instrument which lets you map its sounds to Program Changes in any way you like.

Sample dumps can also be sent in System Exclusive packets, but these tend to be very large, slow, and rather cumbersome in most cases. It is usually much better to use System Exclusive to transfer sound samples between samplers instead of to a computer. Sample editing programs use System Exclusive sample dumps to transfer the sample data into the computer so that it can be manipulated, but the dedicated storage facility in the sampler itself is often much faster and more convenient for backing up samples.

As the final thought this month, let me make a suggestion. Look at the RAM cards and cartridges you have for all your instruments and total up how much money you have spent on them. Now compare this figure against an Atari ST dedicated to acting as a storage device for everything you have now and will buy in the next few years. Which is the cheaper? Which is the most convenient? How many RAM cartridges do I have? One, actually! Everything else is on disk. I can't justify a dedicated Atari, but for some pro users it must be the best solution. Sharing a computer between sequencing and dump storage may be the cheapest option for the home studio user.

Now that I have set you thinking about using System Exclusive to store all your sounds, patches, drum patterns, sequences and effects programs, you have a whole month to get used to a new way of working. The next episode will look at more complex ways of using System Exclusive packets.

DISK INFORMATION

The programs referred to in this article are available on 3.5" disk for the Atari 520/1040ST. Channel Scope is separate and costs £7, whilst MIDI Utility, System Scope, SANE, Movie and ASCHEX have been bundled together on one special 'SysEx Toolkit' disk, which costs a mere £7.

SOS Software, (Contact Details).