In the last installment I described the way that individual bits within bytes could be used to represent different parameters when less than seven bits were needed to represent the value. The Kawai K5 example mentioned that the same parameter table could be used to show the parameter numbers and the contents of the bulk dump. I will expand on this theme this month and present some examples of the sort of bit-formatted bytes which you may find in your own MIDI equipment.

THEORY: PARAMETERS AND DUMPS

The bit-formatted bytes shown last month looked like this:

0aaa bbbb 0ccd deee

and compress more than one value into a single byte. This makes editing such values difficult because we cannot use a program like SYSEX to change one value in isolation - each byte we alter changes two or more values. The Yamaha FB01 example below discusses some approaches to how this might be attempted, but for the purposes of explaining the theory we will concentrate on the 'why', not the 'how'...

So why do some manufacturers choose to put several parameters into the same byte? The answer probably lies in two separate and different reasons: firstly, by compressing parameters together in this way, the RAM memory needed to store them is reduced and so more sets of parameters can be stored; and secondly, it is much more convenient for the programmers to provide the same method of editing parameters via MIDI as it is from the front panel, so if the parameters are stored in a compressed way, then they are edited in that form. Unfortunately, whereas it is quite easy to manipulate individual bits in a microprocessor by a process of shifting and ANDing, it is not quite so easy via MIDI.

For example, suppose that three parameters, a, b and c, are bit-formatted in the following form:

0abb bccc

To alter the three-bit parameter bbb to a particular value xxx, we need to shift the value to the left by three positions:

original value:	0000 0xxx
shifted left:	0000 xxx0
and again:	000x xx00
and finally:	00xx x000
which matches:	0abb bccc

Notice that each x now aligns with a b in the byte. To change the value we need to replace the bs with the xs by ANDing with a mask, and then ANDing with the shifted xxx value:

byte:	0abb bccc
mask:	0011 1000
masked byte:	0a00 0ccc
AND with:	00xx x000
to give:	0axx xccc

The shifting process multiplies the value by two each time - so the original value of xxx will be the equivalent of xxx multiplied by 8 (ie. 2x2x2). Also, the masking requires access to the byte itself, and this would present problems using MIDI.

Some manufacturers convert the compressed form used internally into a more convenient form for editing via MIDI - the Yamaha DX7 is just one example of a large number of instruments which do this. The parameter tables of such devices shows a single parameter for each byte, instead of the bit-formatting. The compressed form is often also used to transmit sounds as dumps using MIDI, because it speeds the transmission time and reduces the amount of storage needed. It looks as if the expansion of the table from the compressed form to the expanded form just so that editing can be achieved easily via MIDI cannot always be justified.

One advantage of having a single table for editing and for decoding the bulk dump is that the documentation is simpler - instead of two tables with different values, one will suffice. The Kawai K5 example last month showed this. By pursuing editing to its finest detail we have arrived back at System Exclusive dumps, but at a correspondingly deeper level.

DUMP FORMATS

Part 2 of this series [SOS May 1989] looked at the typical format of a System Exclusive dump:

$F0 SysEx
$ii Identification
$ss Sub-Status
[block of data]
$cc checksum
$F7 EOX

So far we have concentrated on the parts of this format surrounding the block of data, and the complete packet of information as a whole as stored by our Movie program, but no mention has been made of the contents of the block of data in a bulk dump from a typical synthesizer, until now.

As with almost all System Exclusive formats, no true standard exists, nor does a standard need to exist! Each manufacturer has developed their own format for the data within their own packets of information, and these are generally freely available on request - the System Exclusive Book from the SOS Bookshop (see panel) contains details of some of the System Exclusive formats, but it does not go into the same depth of detail as I am about to give!

Because MIDI instruments have different numbers of sound or patch memories, and differing complexity and storage requirements, I can only outline the general form of a bulk dump for a hypothetical instrument. The SOS Vaguely Defined Hypothetical Hi-Tech Musical Instrument (or SOS VDHH-TMI, as we will call it) has sounds which are made up out of a number of parts, and has storage for several sounds. The [block of data] consists of:

[ Parameters for part 1 of sound number 1]
[ Parameters for part 2 of sound number 1 ]
[ Parameters for part n of sound number 1 ]
[ Parameters for part 1 of sound number 2 ]
.....
[ Parameters for part n of sound number 2 ]
...and so on for all the remaining sounds

The parameters are sometimes just a listing of the editable parameters, but they can be more complex. Decoding these lists is an extension of the ideas we have been pursuing so far.

DECODING DUMPS

We have already been looking at parameter tables, and the listings of dumps are much the same in format, as we have discovered. To place the two forms in context, Figure 1 contains part of the parameter table and Figure 2 the dump list for the Yamaha DX7:

Figure 1. DX7 PARAMETER TABLE

PARAMETER	NO.	DATA
OPERATOR 6:
Level Scaling Left Curve	11	0-3
Level Scaling Right Curve	12	0-3
Keyboard Rate Scaling	13	0-7
Amplitude Mod Sensitivity	14	0-3
Key Velocity Sensitivity	15	0-3
Output Level	16	0-99
Oscillator Mode	17	0-1
Oscillator Coarse Frequency	18	0-31
Oscillator Fine Frequency	19	0-99
Oscillator Detune	20	0-14

Each of these parameters can be accessed using a single unique parameter number, and only one parameter is changed when the value is altered. This means that simple editing messages can be used.

Figure 2. DX7 DUMP LISTING

PARAMETER	BYTE	FORMAT
OPERATOR 6:
Level Scaling Left Curve	11	0000 xx00
Level Scaling Right Curve	11	0000 00xx
Oscillator Detune	12	0xxx x000
Keyboard Rate Scaling	12	0000 0xxx
Key Velocity Sensitivity	13	000x xx00
Amplitude Mod Sensitivity	13	0000 00xx
Output Level	14	0xxx xxxx
Osc. Coarse Frequency	15	0xxx xxx0
Osc. Mode	15	0000 000x
Osc. Fine Frequency	16	0xxx xxxx

This time six bytes have taken the place of 10, which represents quite a saving, but there is a corresponding increase in complexity. The two, three and four-bit values are compressed together whenever possible, whilst the six-bit and one-bit values meet a similar fate. The overall effect is to change the order and binary value of the parameters, which means that they need to be decoded back into individual parameter values in order to be manipulated.

For example, suppose bit 12 of the dump was $38. The table tells us that this represents the values of the Detune and the Scaling parameters, but we need to decode the byte into the two values and extract the right values.

$38 = 0011 1000
Detune mask = 0111 1000
so Detune = 0111
Scaling mask = 0000 0111
so Scaling = 000

Ignoring the position in the byte, these values are 0111 and 000, or 7 (decimal) for Detune and 0 for Scaling. The value of 7 for the Detune represents half-way along a 0 to 14 range, and since this represents the values -7 to +7 on the DX7's LCD display, the value of 7 for the Detune parameter actually represents a Detune value of 0!

You may be asking why we need to consider the format of a dump and decode it. The answer is illustrated very nicely by the FB01 example in the practical section of this installment - if you wish to edit bit-formatted values using System Exclusive messages, then you need to be able to decode the contents of the bulk dump in order to know the values of the parameters which you do not want to edit in the bit-formatted byte. If you try to edit one value without knowing the values of the others, you will lose the other values. Remember that this only applies to bit-formatted values where several parameters share bits in the same byte.

We have now turned full circle: parameter dumps or listings led into editing of parameters, and exploration of editing leads to dumps. Two loose ends remain for future examination - checksums and handshaking, of which more soon.

THIS MONTH'S SOFTWARE

To show that continuous development really does happen here at SOS, this month's program is an update to the SYSEX Utility (you were given a sneak preview of the program in one of last month's screen shots). Version 0.3 of the program enables values to be entered in hex and decimal, as well as converted between the two. Some additional manufacturers have also been added to the buttons. Further developments are in the pipeline!

PRACTICE

Continuing the theme of editing, but adding bit-formatting and dump listing, here are a few more examples of System Exclusive editing messages.

EXAMPLE 1: YAMAHA V50

The first true workstation from the major exponents of FM synthesis, the V50 is a powerful, flexible and somewhat underrated synthesizer. [A disk of voices and performances programmed by the author will soon be available from SOS Shareware; see yellow pages - Ed.] The supplied MIDI information is of a high quality and should pose few problems. For example, the V50's Program Change Table can be tedious to set up using the front panel controls and so could be a suitable candidate for editing via System Exclusive. The manual gives the required information in a very clear form:

1111 0000	F0
0100 0011	43
0001 nnnn	nnnn = Device Number
0ggg gghh	ggggg = 00100 (4), hh= 00 (0)
0ppp pppp	ppppppp = 1111111 (127)
0kkk kkkk	kkkkkkk = Program Change Number
0hhh hhhh	hhhhhhh = high data byte
0111 1111	1111111 = low data byte
1111 0111	F7

The Program Change Number is part of the received MIDI program change and the table maps this to the Internal, Card and Preset memories for both single Voices and Performances. A table gives the details:

High byte	Low byte	Voice/Perf.	Memory
0	0-99	Voice	Internal I0-I99
1	0-99	Voice	Card C0-C99
2	0-99	Voice	Preset P0-P99
3	0-99	Perf.	Internal PFI00-PFI99
4	0-99	Perf.	Card PFC00-PFC99
5	0-99	Perf.	Preset PFP00-PFP99

So to change MIDI program number 0 so that it accesses Internal Performance Number 34 (this is possible from the front panel of the V50, but you need to be very careful to press the Internal, Card or Preset buttons and hold them down, before pressing the Performance button - if you get it wrong then you go back to performance mode and need to press several buttons to get back to the program change editing mode!) you need to send:

$F0, $43, $10, $10, $7F, $0, $3, $22, $F7

Where the $3 sets the Internal Performance, and the $22 is the hex equivalent of 34 decimal. The $22 byte could be driven from either the mouse or the modulation wheel using this month's software.

Another control which is available from the V50 front panel but which can be useful to control remotely (from a sequencer, perhaps?) is the Effect Bypass switch. The V50 provides a front panel switch message which lets you alter any switch via MIDI System Exclusive messages. The format is given as:

1111 0000	F0
0100 0011	43
0001 nnnn	nnnn = Device Number
0ggg gghh	ggggg = 00100 (4), hh= 00 (0)
0ppppppp	ppppppp = 1110110 (118)
0sss ssss	sssssss = switch number (Table 1)
0ddd dddd	ddddddd = switch data 0=Off, 127=On
11110111	F7

The Table 1 mentioned above details each of the numbers assigned to the front panel buttons as follows:

9	Stop
10	Play
54	Effect Bypass
56	Reset

The last switch (Reset) is similar to the Exit button on a YS-series synth and resets the machine - equivalent to turning off and then on again. The Play and Stop buttons could be used to give the same effect as the drum machine $FA and $FC MIDI messages, but would be specific to only the V50 and would not affect any other drum machines or sequencers connected to the MIDI network. The message for Effect Bypass would look like:

$F0, $43, $10, $10, $76, $36, $7F, $F7

To turn the switch on, and;

$F0, $43, $10, $10, $76, $36, $00, $F7

to turn the switch off. Or rather, this is what the Yamaha documentation implies should be sent. If you try these messages you will find that what actually happens is that sending the message with the data as $7F toggles the state of the switch every time it is sent, whilst sending the message with the data set as $0 will turn the switch off only. Experimentation has shown that the best course of action is to send an On ($7F) followed by an Off ($0):

$F0, $43, $10, $10, $76, $36, $7F, $F7, $F0, $43, $10, $10, $76, $36, $00, $F7

Each time this message is sent, the V50 Effect Bypass will change state. Because the SXMOD program (see last month) uses values greater than 120 to quit from the routine, only SXMOUSE can be used to turn the bypass on and off, by sending 0s and 127s alternatively, or by sending just 127s. SYSEX can also be used to send just $7Fs as data by sending each message twice. Discovering that the manufacturers' documentation is not always 100% correct is all part of the thrill of exploring System Exclusive!

EXAMPLE 2: YAMAHA FB01

The most pressing need for System Exclusive editing comes from instruments which have no front panel editing facility. Only two notable examples of this come immediately to mind, both of which were subsequently (and quickly) replaced (or augmented) by similar products with editing features! The two expanders in question are the Roland MT32 (the D110 is the editable version) and the Yamaha FB01 (the TX81Z is the editable version); neither makes any particular provision for the editing of the sounds with the front panel. Of these two, the Yamaha FB01 has acquired a certain notoriety as being difficult, if not impossible, to edit without using a dedicated software package. Interestingly enough, the latest generation of products designed to form part of a complete computer-based music generation system are based on similar technology to the non-editable versions; so perhaps repackaging does work in the long term after all.

The problems with the FB01 are three-fold: firstly, the manual is too detailed, but unfortunately splits the detail into a large number of tables and lists of parameter blocks etc, spaced out over several pages; secondly, the MIDI messages are both nibbleised, bit-formatted and addressed via offsets to values held in other tables, making the assembly and editing of parameters very difficult; and thirdly, the manual does not seem to give any clear idea of how the various component parts of the System Exclusive messages fit together — for example, there are seven different and incompatible bulk dumps plus an entirely non-standard way of generating notes.

Having waded through the MIDI documentation whilst writing an editor program for the FB01, I have spent a considerable amount of time working out exactly what is going on inside. So what follows is mostly given in the manual, but presented here in a (hopefully) understandable form.

There are three basic sets of parameter in the FB01:

1. Voice Data. This includes the basic information about the FM parameters used to produce the sounds - the parameters are detailed in the Voice Data block. Because the FB01 is multitimbral, sounds can be edited by the MIDI channel they are assigned to, or by the MIDI channel plus the part or instrument number. The editing messages have the value data split into 4-bit nibbles over two bytes.

2. Configuration Data. This includes the majority of the multitimbral assignment and control information - again, edited by channel or channel plus instrument number. In this case, the value data is sent as bytes instead of nibbles. Confusingly, the parameters are held in the Instrument Definition block, not the Configuration Data block.

3. System Data. This covers only a few global parameters: the master output level, the memory protect, etc. This is only Editable by MIDI channel (since it deals with all the instruments, it is not accessible by channel plus instrument number) and uses bytes to represent the values held in the Configuration Data block.

Armed with this road map of the FB01, we can now explore a few messages. The System and Configuration messages seem the best place to start, since they do not use nibbleised data values. Master Output Level seems like a useful starting point, especially since there is no front panel control for overall output level, and also the normal MIDI Volume Controller ($07) is channelised and so affects only the instruments assigned to that channel. The basic System editing message is:

$F0	Sysex
$43	Yamaha ID
$75	Sub-status
$0s	s = system channel number
$10	Parameter Group
$pp	Parameter Number
$dd	value data
$F7	EOX

Only 10 parameter numbers are valid: $08 to $0D, and $20 to $24. The $24 represents the Master Output Level, so a suitable message would be:

$F0, $43, $75, $00, $10, $24, vv, $F7

where vv is the volume level value from 0 to 127. This could be driven from SXMOD or SXMOUSE, if you wish.

When editing an Instrument where you do not know the specific MIDI channel is part of the Configuration Data block, the basic editing message is:

$F0	Sysex
$43	Yamaha ID
$75	Sub-status
$0s	s = system channel number
%0001 1iii	Instrument Number (0-7 = 1-8)
$pp	Parameter Number (0-$17)
$dd	value data
$F7	EOX

The System channel is a special channel devoted to messages for just the FB01 with that same system number - much the same as the device number in Roland equipment. You can have several FB01s on the same MIDI channel, but access each one separately by giving them different System channels. The relevant part of the Configuration Parameter table looks like this:

Param No.	Value Range	Name
$00 - $0F		(see Instr def block p52)
$10	0-127	LFO Speed
$11	0-127	amd
$12	0-127	pmd
$13	0-3	LFO waveform

and part of the Instrument Definition block several pages later looks like this:

$00	%0000 xxxx	number of notes (0-8)
$01	%0000 xxxx	MIDI channel (0-15)
$02	%0xxx xxxx	high key limit (0-127)
$03	%0xxx xxxx	low key limit (0-127)
$04	%0000 0xxx	voice bank (0-6)
$05	%0xxx xxxx	voice (0-47)
$06	%0000 0xxx	detune (0-127)

(Yamaha use %0000 0000 to indicate a binary number.) Combining all these three parts together gives the following message for editing the LFO Speed of Instrument 1:

$F0, $43, $75, $00, $18, $10, vv, $F7

or the Voice Number of Instrument 3:

$F0, $43, $75, $00, $1B, $10, vv, $F7

where w is between 0 and $2F in order to access the 48 voice memories.

Editing the Voice Data is more complex. The basic parameter change by Instrument message is similar to the above Configuration.

$F0	Sysex
$43	Yamaha ID
$75	Sub-status
$0s	s = system channel number
%0001 1iii	Instrument Number (0-7 = 1-8)
$pp	Parameter Number ($40-$7F)
$0y	low nibble of value data
$0x	high nibble of value data
$F7	EOX

The parameter values are given in the Voice Data table on another page of the FB01 manual, but need $40 adding to the parameter numbers shown! Here are some of the parameters:

$0A	%xyyy yyyy	x = LFO sync at note on mode (0=Off, 1=On) y = Pitch Mod Depth (pmd)
$0B	%0wxy z000	w = operator enable 3 x = operator enable 2 y = operator enable 1 z = operator enable 0
$0C	%lrxx xyyy	(l,r = Stereo output enables - this is not shown in the Yamaha documentation!) x = feedback level y = algorithm
S0D	%0xxx 00yy	x = Pitch Mod Sensitivity (pms) y = Amplitude Mod Sensitivity (ams)
$0E	%0xx0 0000	x = LFO waveform (0=saw, 1=square, 2=triangle, 3=random)

As you can see, these are in a compressed form, where the individual bits are significant to different parameters. Working out the numbers to enter into a simple editor program like SYSEX is quite difficult, since you need to know the values which are already present so that you do not alter them. For example, to change the LFO Sync you need to know what the pmd value is, since if you send a message to alter the LFO Sync, you also alter the pmd value!

In practice, this means that you will need to get a dump of the voice parameters, decode them into their component parts (the table above is also a decoding table) and then reconstruct the message you need. I recommend that you restrict yourself to editing the System and Configuration parameters unless you feel very confident with binary, hex and voice dumps.

SUMMARY

System Exclusive editing can sometimes provide access to features of an instrument which are either difficult or impossible to obtain from the front panel. More importantly, such editing messages can be embedded into sequencer tracks to provide control over instruments as they are playing, without the need for 'real time' manual changes of parameters. Sometimes, using System Exclusive is easier than pressing a dozen buttons in the right order whilst bent double trying to read the display on an expander at the bottom of the effects rack! In these days of universal and generic editors and MIDI controller software, the ability to be able to exploit System Exclusive data with these tools is becoming increasingly important.

Next month, we return to the subject of bulk dumps, and look at their storage and the calculation of checksums.

FURTHER INFORMATION

The System Exclusive Utility program referred to in this article (plus other useful programs) is available on the 'SysEx ToolKIt 2' disk, price £7.

SOS Software, (Contact Details).