• Everything You Ever Wanted T...
  • Everything You Ever Wanted T...
  • Everything You Ever Wanted T...

Magazine Archive

Home -> Magazines -> Issues -> Articles in this issue -> View

Everything You Ever Wanted To Know About System Exclusive (Part 8)

(But Were Too Afraid To Ask!)

PART 8: Martin Russ explains the circular nature of parameter editing and data dumps.


In the last installment I described the way that individual bits within bytes could be used to represent different parameters when less than seven bits were needed to represent the value. The Kawai K5 example mentioned that the same parameter table could be used to show the parameter numbers and the contents of the bulk dump. I will expand on this theme this month and present some examples of the sort of bit-formatted bytes which you may find in your own MIDI equipment.

THEORY: PARAMETERS AND DUMPS



The bit-formatted bytes shown last month looked like this:

0aaa bbbb 0ccd deee

and compress more than one value into a single byte. This makes editing such values difficult because we cannot use a program like SYSEX to change one value in isolation - each byte we alter changes two or more values. The Yamaha FB01 example below discusses some approaches to how this might be attempted, but for the purposes of explaining the theory we will concentrate on the 'why', not the 'how'...

So why do some manufacturers choose to put several parameters into the same byte? The answer probably lies in two separate and different reasons: firstly, by compressing parameters together in this way, the RAM memory needed to store them is reduced and so more sets of parameters can be stored; and secondly, it is much more convenient for the programmers to provide the same method of editing parameters via MIDI as it is from the front panel, so if the parameters are stored in a compressed way, then they are edited in that form. Unfortunately, whereas it is quite easy to manipulate individual bits in a microprocessor by a process of shifting and ANDing, it is not quite so easy via MIDI.

For example, suppose that three parameters, a, b and c, are bit-formatted in the following form:

0abb bccc

To alter the three-bit parameter bbb to a particular value xxx, we need to shift the value to the left by three positions:

original value: 0000 0xxx
shifted left: 0000 xxx0
and again: 000x xx00
and finally: 00xx x000
which matches: 0abb bccc

Notice that each x now aligns with a b in the byte. To change the value we need to replace the bs with the xs by ANDing with a mask, and then ANDing with the shifted xxx value:

byte: 0abb bccc
mask: 0011 1000
masked byte: 0a00 0ccc
AND with: 00xx x000
to give: 0axx xccc

The shifting process multiplies the value by two each time - so the original value of xxx will be the equivalent of xxx multiplied by 8 (ie. 2x2x2). Also, the masking requires access to the byte itself, and this would present problems using MIDI.

Some manufacturers convert the compressed form used internally into a more convenient form for editing via MIDI - the Yamaha DX7 is just one example of a large number of instruments which do this. The parameter tables of such devices shows a single parameter for each byte, instead of the bit-formatting. The compressed form is often also used to transmit sounds as dumps using MIDI, because it speeds the transmission time and reduces the amount of storage needed. It looks as if the expansion of the table from the compressed form to the expanded form just so that editing can be achieved easily via MIDI cannot always be justified.

One advantage of having a single table for editing and for decoding the bulk dump is that the documentation is simpler - instead of two tables with different values, one will suffice. The Kawai K5 example last month showed this. By pursuing editing to its finest detail we have arrived back at System Exclusive dumps, but at a correspondingly deeper level.

DUMP FORMATS



Part 2 of this series [SOS May 1989] looked at the typical format of a System Exclusive dump:

$F0 SysEx
$ii Identification
$ss Sub-Status
[block of data]
$cc checksum
$F7 EOX


So far we have concentrated on the parts of this format surrounding the block of data, and the complete packet of information as a whole as stored by our Movie program, but no mention has been made of the contents of the block of data in a bulk dump from a typical synthesizer, until now.

As with almost all System Exclusive formats, no true standard exists, nor does a standard need to exist! Each manufacturer has developed their own format for the data within their own packets of information, and these are generally freely available on request - the System Exclusive Book from the SOS Bookshop (see panel) contains details of some of the System Exclusive formats, but it does not go into the same depth of detail as I am about to give!

Because MIDI instruments have different numbers of sound or patch memories, and differing complexity and storage requirements, I can only outline the general form of a bulk dump for a hypothetical instrument. The SOS Vaguely Defined Hypothetical Hi-Tech Musical Instrument (or SOS VDHH-TMI, as we will call it) has sounds which are made up out of a number of parts, and has storage for several sounds. The [block of data] consists of:

[ Parameters for part 1 of sound number 1]
[ Parameters for part 2 of sound number 1 ]
[ Parameters for part n of sound number 1 ]
[ Parameters for part 1 of sound number 2 ]
.....
[ Parameters for part n of sound number 2 ]
...and so on for all the remaining sounds


The parameters are sometimes just a listing of the editable parameters, but they can be more complex. Decoding these lists is an extension of the ideas we have been pursuing so far.

DECODING DUMPS



We have already been looking at parameter tables, and the listings of dumps are much the same in format, as we have discovered. To place the two forms in context, Figure 1 contains part of the parameter table and Figure 2 the dump list for the Yamaha DX7:

Figure 1. DX7 PARAMETER TABLE

PARAMETER NO. DATA
OPERATOR 6:
Level Scaling Left Curve 11 0-3
Level Scaling Right Curve 12 0-3
Keyboard Rate Scaling 13 0-7
Amplitude Mod Sensitivity 14 0-3
Key Velocity Sensitivity 15 0-3
Output Level 16 0-99
Oscillator Mode 17 0-1
Oscillator Coarse Frequency 18 0-31
Oscillator Fine Frequency 19 0-99
Oscillator Detune 20 0-14

Each of these parameters can be accessed using a single unique parameter number, and only one parameter is changed when the value is altered. This means that simple editing messages can be used.

Figure 2. DX7 DUMP LISTING

PARAMETER BYTE FORMAT
OPERATOR 6:
Level Scaling Left Curve 11 0000 xx00
Level Scaling Right Curve 11 0000 00xx
Oscillator Detune 12 0xxx x000
Keyboard Rate Scaling 12 0000 0xxx
Key Velocity Sensitivity 13 000x xx00
Amplitude Mod Sensitivity 13 0000 00xx
Output Level 14 0xxx xxxx
Osc. Coarse Frequency 15 0xxx xxx0
Osc. Mode 15 0000 000x
Osc. Fine Frequency 16 0xxx xxxx

This time six bytes have taken the place of 10, which represents quite a saving, but there is a corresponding increase in complexity. The two, three and four-bit values are compressed together whenever possible, whilst the six-bit and one-bit values meet a similar fate. The overall effect is to change the order and binary value of the parameters, which means that they need to be decoded back into individual parameter values in order to be manipulated.

For example, suppose bit 12 of the dump was $38. The table tells us that this represents the values of the Detune and the Scaling parameters, but we need to decode the byte into the two values and extract the right values.

$38 = 0011 1000
Detune mask = 0111 1000
so Detune = 0111
Scaling mask = 0000 0111
so Scaling = 000


Ignoring the position in the byte, these values are 0111 and 000, or 7 (decimal) for Detune and 0 for Scaling. The value of 7 for the Detune represents half-way along a 0 to 14 range, and since this represents the values -7 to +7 on the DX7's LCD display, the value of 7 for the Detune parameter actually represents a Detune value of 0!

You may be asking why we need to consider the format of a dump and decode it. The answer is illustrated very nicely by the FB01 example in the practical section of this installment - if you wish to edit bit-formatted values using System Exclusive messages, then you need to be able to decode the contents of the bulk dump in order to know the values of the parameters which you do not want to edit in the bit-formatted byte. If you try to edit one value without knowing the values of the others, you will lose the other values. Remember that this only applies to bit-formatted values where several parameters share bits in the same byte.

We have now turned full circle: parameter dumps or listings led into editing of parameters, and exploration of editing leads to dumps. Two loose ends remain for future examination - checksums and handshaking, of which more soon.

Screen display from this month's program: SYSEX V0.3.


THIS MONTH'S SOFTWARE



To show that continuous development really does happen here at SOS, this month's program is an update to the SYSEX Utility (you were given a sneak preview of the program in one of last month's screen shots). Version 0.3 of the program enables values to be entered in hex and decimal, as well as converted between the two. Some additional manufacturers have also been added to the buttons. Further developments are in the pipeline!

PRACTICE



Continuing the theme of editing, but adding bit-formatting and dump listing, here are a few more examples of System Exclusive editing messages.

EXAMPLE 1: YAMAHA V50

The first true workstation from the major exponents of FM synthesis, the V50 is a powerful, flexible and somewhat underrated synthesizer. [A disk of voices and performances programmed by the author will soon be available from SOS Shareware; see yellow pages - Ed.] The supplied MIDI information is of a high quality and should pose few problems. For example, the V50's Program Change Table can be tedious to set up using the front panel controls and so could be a suitable candidate for editing via System Exclusive. The manual gives the required information in a very clear form:

1111 0000 F0
0100 0011 43
0001 nnnn nnnn = Device Number
0ggg gghh ggggg = 00100 (4), hh= 00 (0)
0ppp pppp ppppppp = 1111111 (127)
0kkk kkkk kkkkkkk = Program Change Number
0hhh hhhh hhhhhhh = high data byte
0111 1111 1111111 = low data byte
1111 0111 F7

The Program Change Number is part of the received MIDI program change and the table maps this to the Internal, Card and Preset memories for both single Voices and Performances. A table gives the details:

High byte Low byte Voice/Perf. Memory
0 0-99 Voice Internal I0-I99
1 0-99 Voice Card C0-C99
2 0-99 Voice Preset P0-P99
3 0-99 Perf. Internal PFI00-PFI99
4 0-99 Perf. Card PFC00-PFC99
5 0-99 Perf. Preset PFP00-PFP99

So to change MIDI program number 0 so that it accesses Internal Performance Number 34 (this is possible from the front panel of the V50, but you need to be very careful to press the Internal, Card or Preset buttons and hold them down, before pressing the Performance button - if you get it wrong then you go back to performance mode and need to press several buttons to get back to the program change editing mode!) you need to send:

$F0, $43, $10, $10, $7F, $0, $3, $22, $F7

Where the $3 sets the Internal Performance, and the $22 is the hex equivalent of 34 decimal. The $22 byte could be driven from either the mouse or the modulation wheel using this month's software.

Another control which is available from the V50 front panel but which can be useful to control remotely (from a sequencer, perhaps?) is the Effect Bypass switch. The V50 provides a front panel switch message which lets you alter any switch via MIDI System Exclusive messages. The format is given as:

1111 0000 F0
0100 0011 43
0001 nnnn nnnn = Device Number
0ggg gghh ggggg = 00100 (4), hh= 00 (0)
0ppppppp ppppppp = 1110110 (118)
0sss ssss sssssss = switch number (Table 1)
0ddd dddd ddddddd = switch data 0=Off, 127=On
11110111 F7

The Table 1 mentioned above details each of the numbers assigned to the front panel buttons as follows:

9 Stop
10 Play
54 Effect Bypass
56 Reset

The last switch (Reset) is similar to the Exit button on a YS-series synth and resets the machine - equivalent to turning off and then on again. The Play and Stop buttons could be used to give the same effect as the drum machine $FA and $FC MIDI messages, but would be specific to only the V50 and would not affect any other drum machines or sequencers connected to the MIDI network. The message for Effect Bypass would look like:

$F0, $43, $10, $10, $76, $36, $7F, $F7

To turn the switch on, and;

$F0, $43, $10, $10, $76, $36, $00, $F7

to turn the switch off. Or rather, this is what the Yamaha documentation implies should be sent. If you try these messages you will find that what actually happens is that sending the message with the data as $7F toggles the state of the switch every time it is sent, whilst sending the message with the data set as $0 will turn the switch off only. Experimentation has shown that the best course of action is to send an On ($7F) followed by an Off ($0):

$F0, $43, $10, $10, $76, $36, $7F, $F7, $F0, $43, $10, $10, $76, $36, $00, $F7

Each time this message is sent, the V50 Effect Bypass will change state. Because the SXMOD program (see last month) uses values greater than 120 to quit from the routine, only SXMOUSE can be used to turn the bypass on and off, by sending 0s and 127s alternatively, or by sending just 127s. SYSEX can also be used to send just $7Fs as data by sending each message twice. Discovering that the manufacturers' documentation is not always 100% correct is all part of the thrill of exploring System Exclusive!

EXAMPLE 2: YAMAHA FB01

The most pressing need for System Exclusive editing comes from instruments which have no front panel editing facility. Only two notable examples of this come immediately to mind, both of which were subsequently (and quickly) replaced (or augmented) by similar products with editing features! The two expanders in question are the Roland MT32 (the D110 is the editable version) and the Yamaha FB01 (the TX81Z is the editable version); neither makes any particular provision for the editing of the sounds with the front panel. Of these two, the Yamaha FB01 has acquired a certain notoriety as being difficult, if not impossible, to edit without using a dedicated software package. Interestingly enough, the latest generation of products designed to form part of a complete computer-based music generation system are based on similar technology to the non-editable versions; so perhaps repackaging does work in the long term after all.

The problems with the FB01 are three-fold: firstly, the manual is too detailed, but unfortunately splits the detail into a large number of tables and lists of parameter blocks etc, spaced out over several pages; secondly, the MIDI messages are both nibbleised, bit-formatted and addressed via offsets to values held in other tables, making the assembly and editing of parameters very difficult; and thirdly, the manual does not seem to give any clear idea of how the various component parts of the System Exclusive messages fit together — for example, there are seven different and incompatible bulk dumps plus an entirely non-standard way of generating notes.

Having waded through the MIDI documentation whilst writing an editor program for the FB01, I have spent a considerable amount of time working out exactly what is going on inside. So what follows is mostly given in the manual, but presented here in a (hopefully) understandable form.

There are three basic sets of parameter in the FB01:

1. Voice Data. This includes the basic information about the FM parameters used to produce the sounds - the parameters are detailed in the Voice Data block. Because the FB01 is multitimbral, sounds can be edited by the MIDI channel they are assigned to, or by the MIDI channel plus the part or instrument number. The editing messages have the value data split into 4-bit nibbles over two bytes.

2. Configuration Data. This includes the majority of the multitimbral assignment and control information - again, edited by channel or channel plus instrument number. In this case, the value data is sent as bytes instead of nibbles. Confusingly, the parameters are held in the Instrument Definition block, not the Configuration Data block.

3. System Data. This covers only a few global parameters: the master output level, the memory protect, etc. This is only Editable by MIDI channel (since it deals with all the instruments, it is not accessible by channel plus instrument number) and uses bytes to represent the values held in the Configuration Data block.

Armed with this road map of the FB01, we can now explore a few messages. The System and Configuration messages seem the best place to start, since they do not use nibbleised data values. Master Output Level seems like a useful starting point, especially since there is no front panel control for overall output level, and also the normal MIDI Volume Controller ($07) is channelised and so affects only the instruments assigned to that channel. The basic System editing message is:

$F0 Sysex
$43 Yamaha ID
$75 Sub-status
$0s s = system channel number
$10 Parameter Group
$pp Parameter Number
$dd value data
$F7 EOX

Only 10 parameter numbers are valid: $08 to $0D, and $20 to $24. The $24 represents the Master Output Level, so a suitable message would be:

$F0, $43, $75, $00, $10, $24, vv, $F7

where vv is the volume level value from 0 to 127. This could be driven from SXMOD or SXMOUSE, if you wish.

When editing an Instrument where you do not know the specific MIDI channel is part of the Configuration Data block, the basic editing message is:

$F0 Sysex
$43 Yamaha ID
$75 Sub-status
$0s s = system channel number
%0001 1iii Instrument Number (0-7 = 1-8)
$pp Parameter Number (0-$17)
$dd value data
$F7 EOX

The System channel is a special channel devoted to messages for just the FB01 with that same system number - much the same as the device number in Roland equipment. You can have several FB01s on the same MIDI channel, but access each one separately by giving them different System channels. The relevant part of the Configuration Parameter table looks like this:

Param No. Value Range Name
$00 - $0F (see Instr def block p52)
$10 0-127 LFO Speed
$11 0-127 amd
$12 0-127 pmd
$13 0-3 LFO waveform

and part of the Instrument Definition block several pages later looks like this:

$00 %0000 xxxx number of notes (0-8)
$01 %0000 xxxx MIDI channel (0-15)
$02 %0xxx xxxx high key limit (0-127)
$03 %0xxx xxxx low key limit (0-127)
$04 %0000 0xxx voice bank (0-6)
$05 %0xxx xxxx voice (0-47)
$06 %0000 0xxx detune (0-127)

(Yamaha use %0000 0000 to indicate a binary number.) Combining all these three parts together gives the following message for editing the LFO Speed of Instrument 1:

$F0, $43, $75, $00, $18, $10, vv, $F7

or the Voice Number of Instrument 3:

$F0, $43, $75, $00, $1B, $10, vv, $F7

where w is between 0 and $2F in order to access the 48 voice memories.

Editing the Voice Data is more complex. The basic parameter change by Instrument message is similar to the above Configuration.

$F0 Sysex
$43 Yamaha ID
$75 Sub-status
$0s s = system channel number
%0001 1iii Instrument Number (0-7 = 1-8)
$pp Parameter Number ($40-$7F)
$0y low nibble of value data
$0x high nibble of value data
$F7 EOX

The parameter values are given in the Voice Data table on another page of the FB01 manual, but need $40 adding to the parameter numbers shown! Here are some of the parameters:

$0A %xyyy yyyy x = LFO sync at note on mode (0=Off, 1=On)
y = Pitch Mod Depth (pmd)
$0B %0wxy z000 w = operator enable 3
x = operator enable 2
y = operator enable 1
z = operator enable 0
$0C %lrxx xyyy (l,r = Stereo output enables - this is not shown in the Yamaha documentation!)
x = feedback level
y = algorithm
S0D %0xxx 00yy x = Pitch Mod Sensitivity (pms)
y = Amplitude Mod Sensitivity (ams)
$0E %0xx0 0000 x = LFO waveform (0=saw, 1=square, 2=triangle, 3=random)

As you can see, these are in a compressed form, where the individual bits are significant to different parameters. Working out the numbers to enter into a simple editor program like SYSEX is quite difficult, since you need to know the values which are already present so that you do not alter them. For example, to change the LFO Sync you need to know what the pmd value is, since if you send a message to alter the LFO Sync, you also alter the pmd value!

In practice, this means that you will need to get a dump of the voice parameters, decode them into their component parts (the table above is also a decoding table) and then reconstruct the message you need. I recommend that you restrict yourself to editing the System and Configuration parameters unless you feel very confident with binary, hex and voice dumps.

SUMMARY



System Exclusive editing can sometimes provide access to features of an instrument which are either difficult or impossible to obtain from the front panel. More importantly, such editing messages can be embedded into sequencer tracks to provide control over instruments as they are playing, without the need for 'real time' manual changes of parameters. Sometimes, using System Exclusive is easier than pressing a dozen buttons in the right order whilst bent double trying to read the display on an expander at the bottom of the effects rack! In these days of universal and generic editors and MIDI controller software, the ability to be able to exploit System Exclusive data with these tools is becoming increasingly important.

Next month, we return to the subject of bulk dumps, and look at their storage and the calculation of checksums.

FURTHER INFORMATION

The System Exclusive Utility program referred to in this article (plus other useful programs) is available on the 'SysEx ToolKIt 2' disk, price £7.

SOS Software, (Contact Details).

SOS REFERENCE ARTICLES

Sound On Sound has published a considerable amount of material covering MIDI over the years (and not all written by myself!). Currently this is not available in a 'collected and re-edited' form, but here is a list of the articles and the magazine edition they appeared in. Back issues are available for £1.50 each (inc postage) from the usual magazine address:

MDR Bulk Dumps Mar 88
How to use MDRs May 88
Storing SPX90 Patches Oct 88

MIDI Automation Systems:
Part 1 Feb 88
Part 2 of 2 Mar 88

MIDI Effects:
Part 1 Dec 85
Part 2 of 2 Jan 86

Integrating MIDI & Studio
Sep 88

MIDI Matters:
Song Position Pointers Feb 87
Implementation Charts Mar 87
Timbre Editing Apr 87
Sample Editing Jun 87

MIDI Basics:
Part 1 Aug 87
Part 2 Sep 87
Part 3 Oct 87
Part 4 Dec 87
Part 5 of 5 Feb 88

MIDI Networks
Sep 86
MIDI Software Guide Oct 89
MIDI Software Roundup Feb 86

MIDI:
The Universal Answer? Nov 85
What's Wrong With It? Apr 86

Practically MIDI:
Part 1 Sep 87
Part 2 Oct 87
Part 3 Nov 87
Part 4 Dec 87
Part 5 Jan 88
Part 6 Feb 88
Part 7 Mar 88
Part 8 of 8 Apr 88

System Exclusive:
Part 1 Apr 89
Part 2 May 89
Part 3 Jun 89
Part 4 Jly 89
Part 5 Aug 89
Part 6 Sep 89
Part 7 Oct 89
Part 8 Nov 89

Talking MIDI:
Part 1 Dec 85
Part 2 Jan 86
Part 3 Feb 86
Part 4 Mar 86
Part 5 May 86
Part 6 Jun 86
Part 7 Jul 86
Part 8 of 8 Aug 86


USEFUL REFERENCE BOOKS

Finding reference books on MIDI which are at a suitable technical level is not easy. There are a large number of introductory books but very few definitive guides. The SOS Bookshop ((Contact Details) - see the 'yellow pages' of this magazine) is a good place to start. Here are the books to which I constantly refer when using MIDI:

- The MIDI Resource Book (SOS Code B021). The first book I reach for! An overview of MIDI containing the official MIDI 1.0 Specification, with some of the additions.

- The MIDI Implementation Book (SOS Code B022). A collection of MIDI Implementation Charts from over 200 devices. I use this for troubleshooting when I am faced with unfamiliar MIDI products. I have also built up my own collection of MIDI Implementation Charts and other manufacturer information to which I constantly refer.

- The MIDI System Exclusive Book (SOS Code B023). A very useful addition to my files, because it details the System Exclusive formats of a wide range of equipment.

These three books form a comprehensive cross-reference guide to the state of MIDI as of about mid-1987. In order to keep up with what is happening now, the best course of action is to join the International MIDI Association (or read Sound On Sound regularly, of course!)

International MIDI Association
(Contact Details)

Finally, another book which contains all sorts of interesting thoughts, hints and tips, but not in any particular order is:

- Mind Over MIDI (SOS Code B010). This is a collection of articles from America's Keyboard magazine, which have been put together under one cover and brought pretty much up to date.


Series



Previous Article in this issue

Digital Signal Processing

Next article in this issue

Studio Reproduction


Sound On Sound - Copyright: SOS Publications Ltd.
The contents of this magazine are re-published here with the kind permission of SOS Publications Ltd.

 

Sound On Sound - Nov 1989

Donated & scanned by: Mike Gorman

>

Should be left alone:


You can send us a note about this article, or let us know of a problem - select the type from the menu above.

(Please include your email address if you want to be contacted regarding your note.)

Topic:

MIDI


Series:

System Exclusive

Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7 | Part 8 | Part 9 | Part 10 | Part 11 | Part 12


Feature by Martin Russ

Previous article in this issue:

> Digital Signal Processing

Next article in this issue:

> Studio Reproduction


Help Support The Things You Love

mu:zines is the result of thousands of hours of effort, and will require many thousands more going forward to reach our goals of getting all this content online.

If you value this resource, you can support this project - it really helps!

If you're enjoying the site, please consider supporting me to help build this archive...

...with a one time Donation, or a recurring Donation of just £2 a month. It really helps - thank you!
muzines_logo_02

Small Print

Terms of usePrivacy