The idea for the Music Animation Machine came from a hallucination I had in the early seventies while listening to J. S. Bach's sonatas and partitas for unaccompanied violin ...
A few days later, I began wondering whether a complex piece of music could be presented visually in a way that would help a listener follow it. The first thing I tried was a piano reduction, and then a bar-graph scroll .
A guy I met from Hollywood suggested animating it, and I was at the point of
gathering the materials to do a stop-motion animation when my friend Michael
Gottlieb suggested I use a computer instead, so I bought one, and learned how
to write software for it. Since the result was that I had a machine which
automatically made the animated scores I'd previously planned to make by hand,
I called the program The Music Animation Machine.
Performance Editor
In the first version of the software, I had to enter the music by
converting it into numbers and typing them in. A single two-minute
piece took me about a week to enter, and I swore I'd figure out a better way.
Fortunately, MIDI was on the rise, so I built a MIDI interface card, and wrote
another version of the Music Animation Machine, this time with a
performance editor. Since I was the primary intended user of the
software, I made it to suit myself.
Visualizing Tonality
Once the Music Animation Machine was operational, it was obvious how much was
missing from it: you couldn't tell anything about timbre, very little about
harmony, not much about rhythm... So I started trying to figure out how to
enhance the display. The first thing I experimented with was tonality, as based
on the "perfect fifth."
Showing interval types
Of course, there are other intervals besides the perfect fifth, and when a bunch
of different pitches are sounding at once, the resulting chord is made up of
several different interval types. My friend Michael Dalby had the idea of
showing all the different interval types in a chord, as a way of visualizing
the nature of the chord.
Tuning systems
Earlier, I said that the pitches used in western music were related by the
interval of a perfect fifth --more or less. This "more or less" is
something that musicians have been struggling with for a long time. It happens
that twelve perfect fifths add up to just a little more than an octave, and
three major thirds add up to almost an octave, but not quite. These
discrepancies make it impossible to make instruments that can play perfectly in
tune in all the different keys. To explore this, I made some software
tools for studying intervals and tuning systems.
Timbre and Inflection
Although the Music Animation Machine can use color to show which instrument is
playing, it can't show what an instrument actually sounds like. To do this, I
developed a new kind of spectrogram, one that
shows small deviations in pitch, and shows the amount of energy in the
harmonics of a tone -- which determines its timbre. One idea for this is to
have the scrolling Music Animation Machine display showing the notes of an
instrument like the piano that can't vary its tone, combined with this
spectrogram display, showing the subtleties of pitch and timbre of another
instrument, or the human voice.
Rhythm
The scrolling score in the Music Animation Machine shows rhythm -- sort of. But
it doesn't really give you a sense of the rhythm. In FURRY.EXE,
I tried doing a Fourier transform of rhythm. Here's a regular rhythm; as you
can see, a pulse waveform contains lots of harmonics. Here's an irregular
rhythm; you can see that more energy accumulates in the higher frequencies than
in the fundamental. Although this program works the way it ought to, what it
shows is not as interesting as I'd hoped -- it doesn't show anything about the
character of the rhythm ... rhythm is pretty elusive
Computer-assisted Performance
Although the Music Animation Machine editor is fast, it's not nearly as fast as playing the piece perfectly the first time. Sometimes, though, the notes are just too hard to play expressively at the right tempo. The idea occurred to me: why not separate out the pitches from the act of performing them? I could enter the notes as slowly as I wanted, and add the expression later.
It turned out that other people had come up with this idea before; it's known in
the trade as the "conductor program":
In the Music Animation Machine implementation of the conductor program , a MIDI keyboard triggers the notes. When you press a key on the keyboard, the next note is played (or, in the case of a chord, the next group of notes that starts at the same time). You control the timing, the dynamics, and the pedaling, and the computer controls the pitches. What I like about using a piano keyboard to trigger the notes is that I can do finger patterns that fit the music. It's sort of like the violinist's bow arm being separate from the left-hand fingering; it allows better control.
There are limitations to a keyboard implementation, however. For one thing, you have to know the rhythm of the piece perfectly. I thought: what if the conductor program were really more like conducting? So, I built a crank that you could turn once a measure, sort of like a hurdy-gurdy. Unfortunately, it turns out to be very hard to turn a crank smoothly enough ...
...and, it has another limitation that the keyboard version has: you don't have
control of dynamics within a note. For that I made a mock-up of another
interface for the conductor program; I call it the lizard
trampoline.
Shape and Motion
At some point I realized that I was getting far away from my original experience, which had to do with the graceful gestures and motions which characterize music. Oskar Fischinger's early black-and-white films helped remind me what I was after: they have a quality of motion which seems very much like the way music moves. So I did some experiments with flowing motion, which I called Oskarettes .
In one of my more recent animations, I've associated different
geometrical shapes with different instruments, while keeping some
features of the original Music Animation Machine display.
My most recent visualizations included experiments in associating
motion with changes in harmony and tonality.
Do I have to conclude
? I hope not; I love doing this stuff.
What intrigues me, and what's more interesting to me than any of my results
(most of which have been achieved or superseded by others) is the process
of doing this: the way the limitation of one thing leads to an idea for a new
thing, and so on ...
...ad infinitum...Most Recent Experiments
Conclusion