Home | Site Map | Watch | FAQ | History | Store | Contact

The Music Animation Machine

by Jennifer Kahn (now at WIRED magazine)

Bach to the future

"Music Animation Machine" turns sound into graphics

Published in the East Bay Express, 1993

In a small room in a rambling house on unincorporated county land along the northern border of Tilden Park, Stephen Malinowski turns on three of five computer monitors, and hangs his long legs over a narrow black piano bench. His wife Lisa Turetsky joins him, balancing the heels of her bare feet on the bench's lacquered edge. Behind her, a wall bulges with cables: I/O, RCA, DB-n, stacked next to a minimalist drawing done by Malinowski on paper he salvaged from a UC Berkeley dumpster. Studying the brushstroke square with its plump diagonal crease, I quietly conclude that it represents a blank television.

"It's a pillow," Malinowski volunteers, rather telepathically adding, "It's not a TV." A mutual friend has described Malinowski as "incredibly intelligent," and telepathy does not immediately seem beyond his reach. A graceful man, with receding auburn hair and a magician's dexterous fingers, he is also possessed of an almost yogic flexibility: despite the minuscule area of the bench, he manages to fit his entire body atop it, legs bowed in a half-lotus and one arm draped languorously in his lap.

Playing a few soundless notes on a MIDI keyboard, he continues, "There's a lot of stuff you don't appreciate when you're listening to music until you learn to notice it. This helps you notice things. It definitely increases the acuity of your hearing." Turetsky nods agreement as the opening bars of Liszt's Hungarian Rhapsody scroll across the monitor like a night cityscape. While Malinowski fondles her toes, she adds brightly, "Having it visualized reifies it."

Malinowski's invention, which he calls his "Music Animation Machine," originated with "a hallucination I had in the early `70s," he says. For introduction, he has loaned me a videotape — one of his few efforts at self-promotion — "A Brief History of the Music Animation Machine." As the tape spools slowly, displaying one bar of music in suspended animation, Malinowski's precise, elegantly modulated voice creeps out of a pair of stereo speakers. "I was listening to J. S. Bach's Sonatas and Partitas for unaccompanied violin. The notes on the pages were dancing along to the music; it was so charming and graceful." Malinowski rushed to put on a Brandenburg concerto in order to get the full effect. "Unfortunately," he laments, "as a beginner at reading scores, I couldn't integrate the separate instrumental parts into a cohesive vision. It was just confusing and frustrating." But the vision had lodged in the twenty-year-old's mind. A few days later, he began wondering "whether a complex piece of music could be presented visually, in a way that would help a listener follow it."

Music animation has its roots in the drawings of Oskar Fischinger, a German-born animator who arrived in Hollywood in 1936, and who created the opening sequences of Disney's 1930s classic, Fantasia. What Fischinger excelled at representing is what Malinowski refers to as "the musical gesture," or the changing moods and voices within a composition.

In "Studie Nr. 7," Fischinger's charcoal animation that Malinowski plays for me on the second of his three monitors, music pages march, swoop, and occasionally storm to the front of the screen in time to a symphony's shifting melodic contours. As one page lilts gracefully to the top of the screen and sinks down, like a handkerchief floating from a conductor's wrist, Malinowski points excitedly. "That!" he exclaims. "That was very much the gesture of the music!"

While swooping kerchiefs may convey the beauty of music, they give very little information about, say, how long an individual note lasts, which intervals are present in a chord, or whether those intervals are harmonic or dissonant. Until recently, that information was locked inside the score. Forty years after Fantasia, the first portable music synthesizer, the Casio keyboard, appeared on the market, and Malinowski had his first encounter with digitized music. On a friend's keyboard, he typed in Rimsky-Korsakov's "Flight of the Bumblebee" using just his index finger, then played it back with the tempo cranked to superhuman speed. Although he felt no shudder of sera vu , the missing half of the Music Animation Machine (MAM) had arrived: a means for musicians to computerize a score so that its playback could be edited for rhythm, tempo, key, or instrumentation.

In the study, the Hungarian Rhapsody continues to play over the stereo, while on the monitor square windows in teal, lilac, and yellow brighten and dim as the music sweeps through them. The rhapsody is one of half a dozen pieces commissioned by the CEO of an ATM network who lives in Louisiana, and Turetsky explains with some embarrassment that "They're not what we would have picked [to animate] because, well, we're music snobs." Unperturbed by this judgment, the rhapsody keeps up its shifting Mondrian geometry, trills zippering across the screen like bared turquoise teeth.

If Malinowski' boxy visuals lack the grace of Fischinger's gestures, it's also true that they contain far more information. In the MAM program, each rectangular window represents a note or chord whose vertical position on the screen indicates its pitch, and whose horizontal position corresponds to timing. As the score scrolls from right to left, notes that are stacked vertically atop one another sound simultaneously and last for the length of the rectangle (staccato notes appear as narrow blips, and half-notes or sustained chords as horizontal bars). Color is a configuration option, and can be assigned to represent different instruments in an orchestra, harmonic distance from the home key (ominous avocado green bars plague Bach's Toccata and Fugue in D Minor, representing a slew of dissonant organ chords), dynamics (from loudest, red, to softest, dark blue), or voices within a solo piece-one might make the melody chartreuse, the accompaniment cobalt, and the descant canary yellow.

For Malinowski, a former music major turned telecommunications programmer, the MAM must be the ultimate bridge between divided loyalties. Scales ascend to skyscrapers, chords drag sweeping banners, and sustains plow brilliant furrows on the black screen-all the while tatting off more information. As an extended trill unravels like a turquoise escalator, I have to wonder if this is the future: music television a la Malinowski, the dawn of MMTV.

If so, one can only presume that the MAM program will evolve to be more user-friendly. As it is, producing something like the rhapsody requires a level of both obsession and musical expertise that would overwhelm most dilettantes. For starters, Turetsky explains, "We had an excellent pianist come over and play [the piece] into MIDI." What followed was a four-person group edit to create the "optimal recording."

What this amounted to was a note-by-note reworking of the pianist's original performance. To demonstrate the onscreen editing process, Malinowski zeroes in on one measure, enlarging it to fill most of the screen. "He came in a little early there, don't you think?" he asks, then slides the downbeat (or rectangle) in question a little bit to the right. He rewinds the tape and plays the edited version, then reenters the measure to change the duration, lengthening the downbeat until it is flush with the next sequence of notes. He replays it again, listening carefully. "This is another way of being a musician," Turetsky remarks. "It's not a technical feat of performance; it's an artistic feat."

A feat of dexterity as well. "Since I was the only one who was going to be using this program, I designed it to be perfect for me," Malinowski explains. His computer is controlled not by a standard typewriter-style keyboard, but by a piano keyboard: a mouse moves the cursor, while a sequence of notes "plays" a command. Observing Malinowski at work is both exhilarating and a little nauseating: one of those rub-your-stomach-pat-your-head short circuits that create a blizzard of neural interference.

Whether this unlikely coordination is innate or learned, in this room it seems more like a survival strategy: at a casual count I estimate-between the three keyboards, phone, fax, MIDI console, and stereo-over three hundred separate buttons, not counting a six-note novelty tape recorder with piano keys for controls. Malinowski, who also plays recorder, violin, viola, classical guitar, and "a whole passel of Renaissance instruments," according to Turetsky, argues that "The piano has a distinctive topography, so I can feel where I am without looking." More importantly, perhaps, it provides a pleasing intimacy between his twin disciplines. With a musical keyboard, Malinowski points out, "all the [computer] commands become little licks."

It's fair to say that Malinowski is something of a rare breed, and the fact that most of us wouldn't be willing to trade our Windows '96 for a MIDI/mouse combo is one indication of why the audience for the Music Animator remains both small and specialized. The way some plants are pollinated by one particular insect, Malinowski sells most of his music animation videotapes through one particular individual: Edward R. Tufte, an eminent — if somewhat obscure — graphic design scholar, makes a point of showing Malinowski's "A Brief History of the Music Animation Machine," at his seminars, and also recommends it on the dust jacket of his book, Envisioning Information. Though Tufte's books have very small runs, they go out to an audience so select that a courtship almost always ensues. "We can always tell where [Tufte] has been lecturing, because we'll suddenly get a flood of orders from one particular state," Turetsky remarks. All told, Malinowski has sold over 3,000 tapes in Tufte's wake.

Of course, by the standards of the video and software buckets, that's still not even a drop. "I would love to have more people see it," Malinowski says, noting that now that it's so much easier to do the programming, it seems inevitable that someone will market music animation software for a more mainstream audience. Whether this person would be regarded as an interloper, intellectual property thief, or savior is unclear. Malinowski seems at once dismayed by the MAM's failure to set the computer and music industries on fire, and yet strangely unwilling to light the fire himself. "It's a very complicated question," he sighs. "Having worked in the industry, I know what it would mean to bring out a piece of software, and market it, and support it, and ... no thank you."

"You're torn between wanting [your creation] to make money and be popular, and being able to just fiddle around with it," Turetsky adds sympathetically.

And Malinowski has done his share of fiddling. "Once the MAM was complete, it was obvious how much was missing," he says at one point in "Brief History." "You couldn't tell anything about timbre, very little about harmony, not much about rhythm." He experimented with applying a color wheel to the circle of fifths, a circle of the twelve chromatic half-steps (seven whites notes and five black notes) contained in an octave. Harmonious compositions appear as a narrow spectrum localized in the blue-purple area of the circle. If the performer hits a dissonant wrong note-say a B instead of a C-the harsh interval appears as it must have in Mozart's nightmares: as a noxious green flare on the wrong side of the pie.

It's this kind of visual power that makes the MAM seem like the perfect teaching tool, and Malinowski notes that many of his videos get bought by music teachers (and, oddly, architects). He also believes the MAM would be great with young children. "When I heard music at five and six years old, I didn't know where it came from, I didn't know that it was composed," he explains. "However, at that age I was aware of artistic shapes: my parents had given me markers and paper, and I knew that I could make colored shapes. I'd like to give children the idea that music is also a thing you can create."

But if children are one likely demographic, professional musicians with a penchant for programming are the most likely other — and it may be precisely the gaping lack of middle ground that has kept the music animation fire from spreading. Although Malinowski might someday get around to bridging his middle ground marketing gap, these days he's more interested in refining his original bridge: the one between animated gesture and digitized data. In one experiment, he altered his circle-of-fifths program so that intervals would be represented not by a spectrum of wedges but by colored lines drawn between points on a circle's rim. The resulting music animation was an oddly faceted shape that lurched across the screen with baby turtle earnestness, like a Tinkertoy attempting a mating dance.

Touchingly awkward, the angular form staggered through its accompanying sonata much the way the first fish must have arrived on land-shuddering and gasping with the weight of evolution. And who's to say, after all, what the Music Animation Machine will become? In the spontaneous warp of history, computers the size of bank vaults grew into laptops, Casio keyboards came to dominate the airwaves, and long ago a small fish developed the gills that breathe air.

With thanks to James Carmichael for providing additional expertise.