Home | Site Map | Watch | FAQ | History | Store | Contact


A Brief History of the Music Animation Machine...

...and related musical explorations of Stephen Malinowski

This material was originally prepared as a videotape (shown at a meeting of YLEM at the Exploratorium in San Francisco, 1996sep11).

Index

Origins
Performance Editor
Visualizing Tonality
Showing Interval Types
Tuning systems
Timbre and Inflection
Rhythm
Computer-assisted Performance
Shape and Motion
Most Recent Experiments
Conclusion

Origins

The idea for the Music Animation Machine came from a hallucination I had in the early seventies while listening to J. S. Bach's sonatas and partitas for unaccompanied violin ...

A few days later, I began wondering whether a complex piece of music could be presented visually in a way that would help a listener follow it. The first thing I tried was a piano reduction, and then a bar-graph scroll .

A guy I met from Hollywood suggested animating it, and I was at the point of gathering the materials to do a stop-motion animation when my friend Michael Gottlieb suggested I use a computer instead, so I bought one, and learned how to write software for it. Since the result was that I had a machine which automatically made the animated scores I'd previously planned to make by hand, I called the program The Music Animation Machine.

Performance Editor

In the first version of the software, I had to enter the music by converting it into numbers and typing them in. A single two-minute piece took me about a week to enter, and I swore I'd figure out a better way. Fortunately, MIDI was on the rise, so I built a MIDI interface card, and wrote another version of the Music Animation Machine, this time with a performance editor. Since I was the primary intended user of the software, I made it to suit myself.

Visualizing Tonality

Once the Music Animation Machine was operational, it was obvious how much was missing from it: you couldn't tell anything about timbre, very little about harmony, not much about rhythm... So I started trying to figure out how to enhance the display. The first thing I experimented with was tonality, as based on the "perfect fifth."

Showing interval types

Of course, there are other intervals besides the perfect fifth, and when a bunch of different pitches are sounding at once, the resulting chord is made up of several different interval types. My friend Michael Dalby had the idea of showing all the different interval types in a chord, as a way of visualizing the nature of the chord.

Tuning systems

Earlier, I said that the pitches used in western music were related by the interval of a perfect fifth --more or less. This "more or less" is something that musicians have been struggling with for a long time. It happens that twelve perfect fifths add up to just a little more than an octave, and three major thirds add up to almost an octave, but not quite. These discrepancies make it impossible to make instruments that can play perfectly in tune in all the different keys. To explore this, I made some software tools for studying intervals and tuning systems.

Timbre and Inflection

Although the Music Animation Machine can use color to show which instrument is playing, it can't show what an instrument actually sounds like. To do this, I developed a new kind of spectrogram, one that shows small deviations in pitch, and shows the amount of energy in the harmonics of a tone -- which determines its timbre. One idea for this is to have the scrolling Music Animation Machine display showing the notes of an instrument like the piano that can't vary its tone, combined with this spectrogram display, showing the subtleties of pitch and timbre of another instrument, or the human voice.

Rhythm

The scrolling score in the Music Animation Machine shows rhythm -- sort of. But it doesn't really give you a sense of the rhythm. In FURRY.EXE, I tried doing a Fourier transform of rhythm. Here's a regular rhythm; as you can see, a pulse waveform contains lots of harmonics. Here's an irregular rhythm; you can see that more energy accumulates in the higher frequencies than in the fundamental. Although this program works the way it ought to, what it shows is not as interesting as I'd hoped -- it doesn't show anything about the character of the rhythm ... rhythm is pretty elusive

Computer-assisted Performance

Although the Music Animation Machine editor is fast, it's not nearly as fast as playing the piece perfectly the first time. Sometimes, though, the notes are just too hard to play expressively at the right tempo. The idea occurred to me: why not separate out the pitches from the act of performing them? I could enter the notes as slowly as I wanted, and add the expression later.

It turned out that other people had come up with this idea before; it's known in the trade as the "conductor program":

  • Christopher Strangio patented it in 1974;
  • Max Mathews also invented it, and uses it in his Radio Drum (aka Radio Baton), which you can read a little about here and here;
  • Jim Plank has done perhaps the most sophisticated implementation (available as Windows freeware).

    In the Music Animation Machine implementation of the conductor program , a MIDI keyboard triggers the notes. When you press a key on the keyboard, the next note is played (or, in the case of a chord, the next group of notes that starts at the same time). You control the timing, the dynamics, and the pedaling, and the computer controls the pitches. What I like about using a piano keyboard to trigger the notes is that I can do finger patterns that fit the music. It's sort of like the violinist's bow arm being separate from the left-hand fingering; it allows better control.

    There are limitations to a keyboard implementation, however. For one thing, you have to know the rhythm of the piece perfectly. I thought: what if the conductor program were really more like conducting? So, I built a crank that you could turn once a measure, sort of like a hurdy-gurdy. Unfortunately, it turns out to be very hard to turn a crank smoothly enough ...

    ...and, it has another limitation that the keyboard version has: you don't have control of dynamics within a note. For that I made a mock-up of another interface for the conductor program; I call it the lizard trampoline.

    Shape and Motion

    At some point I realized that I was getting far away from my original experience, which had to do with the graceful gestures and motions which characterize music. Oskar Fischinger's early black-and-white films helped remind me what I was after: they have a quality of motion which seems very much like the way music moves. So I did some experiments with flowing motion, which I called Oskarettes .

    In one of my more recent animations, I've associated different geometrical shapes with different instruments, while keeping some features of the original Music Animation Machine display.

    Most Recent Experiments

    My most recent visualizations included experiments in associating motion with changes in harmony and tonality.

    Conclusion

    Do I have to conclude ? I hope not; I love doing this stuff.

    What intrigues me, and what's more interesting to me than any of my results (most of which have been achieved or superseded by others) is the process of doing this: the way the limitation of one thing leads to an idea for a new thing, and so on ...

    ...ad infinitum...