Thanks to a Canadian invention called Mandala, digitised video images can now be used to control MIDI events, and vice versa. It could have far-reaching implications for all areas of performing art.

IT'S A JUNGLE SCENE, complete with heavy foliage. A digitised body is in the centre of it all. Suddenly two large birds appear, flying lazily overhead. The body leaps up and grabs them, causing a synthesised squeal to emanate forth as the birds turn into glowing balls. The balls seem to attach themselves like leeches to the hands of the body, following their movement - until, in one final motion, the hands wrench themselves away in a snap. The balls promptly explode, sounding a combination of sound samples. As the fragments of the exploded balls fall to the ground, each one emits a dreamlike, harmonic note as it comes into contact with the jungle floor.

Sound like a scene out of a film you'd like to see? It's just one of the limitless range of interactive video/music environments you can create with a revolutionary new video based real-time performance instrument known as Mandala. Very Vivid of Toronto Mandala's inventors, may very well have created a new artistic medium in its own right.

Mandala is a totally open-ended computer-based instrument that you can use to integrate video images and music together in a single performance system. The performer plays any combination of MIDI instruments by moving or dancing around a "musical environment" that can consist of literally any type of video image or "scene" you can imagine.

The Mandala system revolves around a customised Commodore Amiga computer and Very Vivid's own video camera/digitiser. Using the Genlock capability of the Amiga, graphic images can be combined with the live video output of the camera. The result is that by moving around in front of the camera, you cause a digitised silhouette of your body to move around "inside" the graphic images on the screen. By watching the monitor as you move, it's easy to get a feel for the position of your body inside each graphic scene.

You can create your own video scenes using one of the painting programs supplied with the package or Mandala's own custom image digitiser, which is capable of "photographing" any existing image from a book, painting, photograph, whatever. The final image will be the backdrop of your musical environment. It might be a landscape, a space scene, or whatever you can imagine.

Once you've got a visual backdrop or scene, the next step is to create a variety of icons for your image. These are graphic objects that could look like just about anything - musical instruments, buildings, projectiles, and so on. When you're satisfied with the icons or objects you've created, you can place them into various positions around the original scene you created. Icons can be moved around the screen at will, so you can adjust their positioning within the scene as much as you want.

The next step is to decide what sort of an effect you want those icons to have when you play them by allowing your body image to come into contact with them. By double clicking on any icon, you can access a menu that is designed to permit any combination of events to occur when that icon is played. These can include instructions for literally any type of MIDI event (including System Exclusive changes), a paint or animation effect, a scene change, a transposition or a total reconfiguration of the other icons on the screen.

This open-ended programming technique is referred to as "nexting" by Mandala's R&D Director Frank MacDonald, the man responsible for designing the instrument's comprehensive user interface. Mandala even lets you create complex animation events. For example, you could say: "move this object in a straight line from this point to this other point in five seconds. If the object hits something on the way, trigger a colour change and send out a program change and the following notes on MIDI channel 13."

Possibilities

NOW THE ENDLESS creative possibilities of this instrument should be apparent.

Literally any combination of events can be set up and triggered from a specific body movement. The performance environment of your creation can be set up to cause any combination of actions to happen, depending on where you move your body in relation to the screen.

Here are some examples of some of the performance environments Vincent John Vincent (Very Vivid's Creative Director) has already created for Mandala.

1) An image of the Toronto skyscape forms the backdrop, complete with skyscraper icons. The performer appears as a Godzllla-like figure towering over them. As he smashes the tops of the buildings, a variety of drum sounds are triggered via MIDI note events.

2) A wall of sound is created over an Egyptian backdrop that consists of a solid horizontal row of small circles. As the screen, complex harmonic scales emanate from the MIDI instruments connected to Mandala. One lone icon overhead is triggered occasionally to transpose the instrument to a new scale.

"Creating your own videos has never been this easy, and the idea of composing music and visuals at the same time will become a reality."

3) The performer is in the middle of an interplanetary space scene. Suddenly, some meteorite projectiles come hurling at him in three dimensions. Those that strike him change colours and create a series of complex sound-effects on contact.

4) A series of musical instruments appear on the screen: a vertical keyboard on one side, a harp on the other, two tympani drums in the middle. Each one is preconfigured to a specific scale or series of notes and is sent out on a different MIDI channel. Naturally, each of the connected sound sources is set up with the sound that corresponds to its icon.

As these examples might indicate, the applications for live performance are countless. With the advanced types of MIDI control functions Mandala lets you define, you could trigger notes or sound effects, start and stop sequences, control a lighting system or stage effects, control a mixer or signal-processing equipment - all with pre-determined body movements.

Furthermore, custom scenes could be projected to the audience with one of the arena type large-screen projection systems now available. For the first time, performance artists can literally interact live with their own video.

This is clearly the market Very Vivid are positioning the Mandala for. As David Bray (the company's Marketing Director), puts it: "We want to put Mandala into the hand of creative performance artists. With a system that's this open-ended, no two artists will use it in the same way. Everyone we've showed it to has come up with a new idea of what they could do with it."

To help artists put the system to work in their show, Very Vivid offer a consulting service on a per project basis. That way, their assistance is available to help progran Mandala for the application it's intended for. Naturally a certain amount of customisation might be necessary for certain applications, but if anything, that's a challenge that seems to excite the people responsible for creating Mandala.

Opposites

ALL THE APPLICATIONS discussed so far have used video images to control various MIDI events. But the system is also capable of working the other way - where specific MIDI events can be used to create complex animation changes.

For example, different keyboard velocities might be used to specify colour changes on the screen. A certain note might trigger an animation event. Or perhaps aftertouch might be used to move an object back and forth across the screen.

This opens up a whole new world for Mandala. After all, ever since computer animation became popular, musicians have looked for ways of tying music and video closer together. Now video animation effects may be generated directly from the composition itself. Already, Very Vivid are talking about an animated drummer that is controlled by a MIDI drum sequence...

MIDI users can now generate advanced computer animation effects for their compositions, using the wide variety of sequencing and MIDI control software available to edit the types of effects that Mandala creates.

Imagine the interactive control you could access with a Mandala and the new breed of interactive MIDI software. Creating your own videos has never been this easy, and the idea of composing music and visuals at the same time will become a reality at last. And maybe soon, no self-respecting MIDI studio will be complete without its own in-house animation system.

Mandala is available now - though as far as we know, no UK distributor has yet been appointed. The package consists of a modified Amiga with a Genlock board, a RAM expander, a custom camera/digitiser, the Mandala software, several Paint programs and Very Vivid's own custom MIDI interface. Price on the other side of the Atlantic is around $30,000, but the manufacturers intend to market a MIDI- to-visuals-only version of the instrument as well, at a significantly lower price.

Only one question remains: Which major performance artists will be the first to put this revolutionary new instrument to work in their show?

(Contact Details)