Stephen Malinowski wants people to see his music.
Twenty years ago Malinowski made a bar graph representing the score of Bach's Third Brandenburg Concerto on a roll of adding-machine paper with felt pens. Individual notes are shown as horizontal colored lines -- one color for each instrument, length representing duration and height representing pitch. The result is still on his desk: a simple, clear, slightly unwieldy map of the composition. It is surprisingly beautiful, with abstract patterns reminiscent of Bargello needlework. The concept gradually evolved in succeeding years. The first idea was a machine to scroll the paper along with the music; then he conceived of using an animated film, and in 1985, he developed his first computer-driven animation. At that state, the musical score had to be laboriously translated into numerical code. With the emergence of MIDI technology in the late 1980s, it became possible to translate the notes into computer code simply by playing them on a keyboard, and Malinowski developed his performance editing software, "The Music Animation Machine," that produces scrolling notation on a computer screen in bars of color that light up when the corresponding note sounds.
In 1990 he produced a beautiful demonstration videotape of his keyboard performances with their animated scores, using twelve examples of Renaissance, Baroque, Classical, Romantic and modern music, including two short compositions of his own. The music notation is in the same basic format as the original paper tape, but larger in scale and synchronized with musical sound. The demonstration video contains several experiments in the use of color: in some pieces color represents separate voices, as in the original version, but in others color represents dynamic levels, motives, or textures. This videotape was created to show off the ability of Malinowski's scrolling graphic score to make complex music scores intuitively understandable. The tape is more than a demonstration, however; it is a work of art in itself. Malinowski's refined keyboard playing and his elegant graphics style illuminate the music. Curiously, as the visual presentation makes the structure of the music more apparent, it seems to make the emotional content of the music more vivid as well.
Much of Malinowski's current work is devoted to extending his visual language to represent various qualities of music more explicitly than was possible in the original format. He has turned his attention to harmony, timbre and rhythm
Malinowski's graphics vocabulary for harmony uses a twelve-hued color wheel as a symbol for the twelve tones of a circle of fifths. A delightful property of this mapping is that consonances look consonant and dissonances look dissonant. A major triad falls on three similar colors in a closely-grouped triangle. A tritone -- diabolus in musica! -- becomes a pair of complementary colors slashing fiercely across the screen at each other like fire and ice. Higher-order chords show their main colorations in a coherent group, and each of their off-key tendencies flash off in contrasting hues and directions.
This color scheme works nicely in combination with the original bar graph format: each note is simply assigned both a color and a vertical position based on its pitch. This sounds redundant but isn't, because the vertical position is still needed to tell what octave the note is in and to view the melodic contours, while only color can show where a note lies on the circle of fifths. For more detailed study of harmony, the harmony can be displayed as a pie-chart, either by itself or along with the scrolling score on a second screen.
Malinowski's graphic representation of timbre, which he calls "Voice Tracker," selectively displays certain aspects of sound that are of musical interest: the frequency (pitch) of the fundamental, its amplitude (loudness), and the amplitudes of the first several partials in a harmonic series above the fundamental, as they vary in time. (We don't need to know the frequencies of the upper partials, since they are determined by the fundamental. These acoustical terms are explained in "Timbre," June, 1995.)
An example of the Voice Tracker display is shown here. The fundamental is represented by a horizontal band of color, its pitch by the vertical position of the band and its amplitude by the thickness of the band from bottom to top. The harmonic series of partials are shown as a stack of bands in contrasting colors, each one thicker or thinner to represent its relative prominence in the overall blend of sound. Time stretches horizontally. Note changes, vowel changes, portamento and vibrato are clearly visible in the display. Voice Tracker would seem to have great potential as a tool for voice instruction and instrument voicing, once equipment to make the system fast enough to give visual feedback to the performer becomes commercially available. This representation of timbre has not been integrated with the scrolling score display.
Malinowski's graphical vocabulary for meter and rhythm is still in the experimental stage. He is working with images of rotating wheels and interlocking gears to symbolize repeating and nesting rhythmic patterns.
Marketing his products has been a challenge for Malinowski, despite the sophistication of his software and the artistry of his demonstration tape. He had initially hoped the demonstration tape would stimulate other animators and software developers to work with the scrolling score concept. Has been disappointed with the response. In a tone of resignation he told me, "What people are interested in are the animations, not the software." He has sold a couple thousand copies of the demonstration tape through the eminent graphic design scholar Edward Tufte, who shows it in his seminars as an example of graphic excellence. He would like to make and sell additional tapes for use in music appreciation classes.
Meanwhile, the first demonstration videotape is a treasure. We can only be grateful that Malinowski is willing to throw himself into the creation of things so clear and beautiful in the world of music, and look forward to his next step in making music visible.