Use of color in the Music Animation Machine
I do not have an extraordinary sensitivity to color; my response is pretty average, I think. I don't have synesthesia to any great extent, and while I do have some vague color/music associations (e.g. G major seems brown, F major seems green), these never come to mind when I hear music, and only rarely come to mind when I play music myself or look at a score.
In my first graphical scores, I used color to differentiate instruments, selecting brighter colors for higher instruments (from among the colored markers I had on hand; I didn't have much choice).
In my first computer-animated score (1985), I selected four colors that were easily distinguished and that looked nice together. This was the main way I chose colors for the next five years (though in a few videos, I used color to show loudness).
In 1990, I had the idea of coloring notes according to their position on the circle of fifths (a technique I've called Harmonic Coloring because changes in harmony and tonality result in changes of color palette), and in 1992, my friend Michael Dalby suggested using color to indicate musical interval.
More recently, I've used color to highlight salient features of a particular piece of music (like themes or rhythmic subdivision), and I've experimented with a type of 3D that uses color to encode 3D depth. This page is a fairly complete catalog of how color and other techniques are used in my videos.
Abstract animation is composed of shape, position, and color (and their changes over time), and people (not just people with synesthesia) have strong reactions to color. In my animated graphical scores, position and motion are largely determined by the pitch and timing of notes. Only shape and color are free to be used however I choose. I studied color theory, color perception, the psychology of color, etc. looking for ideas of how to take advantage of our powerful response to color.
One important thing I realized is that an individual's response to color has elements that are universal and elements that are personal. Nearly everyone associates a brighter color with a louder note and a more vivid hue with a more timbrally-rich musical tone, but different people can find the same color (or combination of colors) either beautiful or ugly, or connoting very different things. So, I had a choice: I could use color in a way that seemed beautiful to me (or which had meaningful associations for me), or I could use it in a way that was more universal. I chose the latter, both because I want to my scores to be effective for the widest possible audience, and because I recognize that my personal responses to color are arbitrary (and not so strong that I can't ignore them).
What aspects of color perception are universal? The most important is probably color distance—that is, almost everybody thinks that red and orange are closer (more similar) than green and orange. So, when I am selecting colors for instruments in an orchestra, I will use more-similar colors within a family (e.g. all strings might be in the range of brown to red) and more-contrasting colors between families (e.g. winds might be in the blue/green range).
Another fairly universal color response is that hues in the red/yellow range are felt to be more active/happy/fast and those in the blue/green range are considered more passive/subdued/slow. I sometimes make use of this association.
There is one situation in which the selection of color is made on the basis of completely practical considerations: in my Chromadepth 3D videos, color is used to determine near/far position, so the choice of position and color go together (red is always in front, blue is always behind); here's a page describing my exploration of the Chromadepth 3D technology.
Alternate color-by-pitch approaches
For western music using 12 pitches per octave, there are two symmetrical/orthogonal ways of matching 12 colors from the color wheel to 12 pitch classes: by circle of fifths (as I do in my Harmonic Coloring method), or by "circle of half-steps" (aka "pitch height"): C, C-sharp, D, E-flat, E, F, F-sharp, G, A-flat, A, B-flat, B, C. Many people have proposed coloring music according to pitch height, on the basis of either perceptual similarity (pitches that are similar in pitch height are similar in color) or correspondence with light (the frequencies of which approximately span a 2:1 ratio).
For me, the arguments for coloring by pitch height are not persuasive. The correspondence to light frequencies seems irrelevant to me, because our perception of color is not linear (like a rainbow/spectrum), but circular. And, if anything, the colors seem to go in the wrong direction: blue is a higher frequency of light than red, but red is psychologically "higher" than blue.
Mapping color to pitch based on octave-invariant pitch class makes a little more sense to me, but even then, it seems flawed, because it ignores the aspect of pitch height that makes notes an octave apart different from each other.
But my reasons for preferring coloring by circle-of-fifth position are based less on theoretical, mathematical, rational reasons, and more on pragmatic, practical ones. Pitch height is directly visible in my graphical scores (indicated by vertical position), so coloring by pitch height is somewhat redundant. But more important: harmony and tonality are not based on pitch height, but on circle-of-fifth position. Therefore, coloring by circle-of-fifth position shows changes in harmony and tonality much more clearly that coloring by pitch height.
Let me give you a demonstration of what I mean by that.
Here's Part 1 of Stravinsky's The Rite of Spring, colored according to pitch height:
As you can see, it looks as if the entire passage uses more or less the same colors; at any point in the piece, colors from all around the circle of fifths are present.
Here's the same passage with a color gradient added (so that different octaves have different degrees of saturation):
That's pretty much the same. It looks more or less like it would look if you viewed a black-and-white score through a colored overly: it's adding color, but it's not telling you much about the way pitches are being used within this particular piece; any piece would have essentially the same appearance.
By contrast, here's what it looks like when it is colored according to circle-of-fifths pitch class:
We can see that Stravinsky is doing very different things with pitch at different points in the piece. The highly chromatic passage just before the center point (where you can see that all 12 pitches are being used—very thin strips of contrasting colors) is in marked contrast to the highly tonal passage immediately following (almost all hues are in the blue/green/yellow range). It no longer looks like a simple color overlay; the colors are revealing things about the use of pitch that are specific to this piece.
Non-uniform monotonic color/pitch mapping
In the color/pitch mapping I'd been using, I chose twelve colors that were approximately equally spaced on the color wheel so that the minimum difference between any two colors was as great as possible. Here's where the twelve pitches are positioned on the color wheel (the pitches connected by lines in the top half of the wheel are the seven notes in the diatonic scale of the key, and the ones at the bottom are the ones requiring accidentals):
For my animation of the third movement of Beethoven's opus 132 string quartet (the "Heiliger Dankgesang" — score, animation, discussion), I wanted to highlight the contrast in tonality between the slow sections (in the Lydian mode) and the fast ones (in D major). With the equally-spaced map, it looked like this:
The fast sections stood out, but I wanted the contrast the be greater, so I made this modified map:
I changed two things: the pitches in the Lydian mode section are separated by wide perceptual differences from the pitches that only appear in the D major section, and the D major section pitches are clustered together and brightened. Here's how it looks when applied to the score:
With this technique, two aspects of color perception are employed: the monotonic mapping of hue to the circle of fifths, and the use of the more bright/vivid hues for the more energetic passages of the piece.
Non-monotonic color-by-pitch approaches
These are the most personal, least universal, and are typically the result of color/pitch synesthesia—a neurological condition in which pitch classes are hard-wired to specific colors. The problem with this, from my perspective, is that a particlar mapping of color to pitch is only "correct" for one person.
In discussions of music and color, the name of composer Alexander Scriabin often comes up, and many people believe that he was a synesthete. This belief appears to be mistaken, however, and his proposed mapping of color to music is not asymmetrical, but is based on the circle of fifths (the same as mine).