Sometime around the beginning of 1998, I put together a new setup for doing animation experiments. The main differences from the previous setup were:

• Faster CPU (Pentium instead of 386/486)
• Two (faster) video adapters (Matrox Millenium)
• Combined LCD/touchscreen (Wacom PL-300)
• More comfortable programming environment (flat memory model)

I started by writing the graphics primitives: lines, circles, and so on:

After I got everything working, the first thing I did was work on my Wake visualization. The first version had four "sticks" and four "drums", to match the original score (showing here the red "explosions" when a drum hits a stick):

In the second version, I simplified the display by increasing the number of drums to sixteen, to match that structure of the piece:

V03: Pressure sensitive tablet

This was a test to make sure the Wacom's pressure sensing worked okay; the blob moved with the stylus, and changed size with pressure:

V04: Scale evenness diagnostic

About that time, I was working on my piano technique, and I was having a hard time getting my scales to sound smooth. I found that I wasn't listening objectively enough. If I tape-recorded my playing and listened to the playback, it sounded uneven, even though it sounded smooth while I was playing. So I made a display which showed the pitch class of the notes, the loudness (MIDI key velocity), and the degree of overlap. To show the overlap, I used a pseudo-random dithering so that one note could show through another.

This tool did not prove very helpful. I think the idea was basically okay, but I wasn’t good enough at recognizing the pitches by color to know which block of color represented which pitch. I think a display that was closer to standard notation (perhaps notes on a staff added to this display) would have been better. Still, it was kind of fun to be able to compose a bookshelf in real time:

V05: Text scaling and rotation

In this one, the lines of text follow the stylus...

...and then go berserk:

Next, I turned my attention to the Wacom display tablet. This is a combination of an color video display and a graphics tablet. You draw on it with a pressure-sensitive stylus. My idea was to create virtual instruments: you’d see them on the display, you’d play on them by interacting with them with the stylus, and in response to what you’re doing, the display would change.

My only experiment along these lines so far has been pretty rudimentary: using the stylus to "bow" the display -- as if it were a string instrument. With one hand, you play the pitches on a (piano/MIDI) keyboard, and with the other, you stroke the display screen; the speed of motion and the pressure of the stylus determine the dynamic level of the resulting sound (a bowed viola sample in my experiment). Since the display was pretty much superfluous, I just had it show the number of pitches that were in action (here five), the dynamic level (the size of the five circles grows and shrinks), and the speed (the little "tail" hanging off the center gets longer the faster you move).

Though simple, I liked playing on this instrument, and immediately felt comfortable doing expressive dynamic changes; it seemed much more natural and instinctive than using a MIDI controller wheel (perhaps because I already have some bowing experience – though I was using the left hand for bowing).

For the better part of a year, I forgot about my new setup and focused on my piano playing, in preparation for a concert I gave with ‘cellist Nancy Bien Souza (April 25, 1999). Then, a few weeks ago (May, 1999), I remembered (like "oh, yeah, I have this neat computer graphics system") and started playing with it again. The first visualization I did gave a picture that was like the view of the action of a harpsichord while somebody’s playing it: you can see all the pitches that are available, and when a note is being played, it looks different. In the first of these (V10), the note indicator jumps way up on the screen and gets a lot larger when the key is played (higher up and larger in proportion to the note’s dynamic level), and then drifts down as the note decays; when the key is released, the indicator goes hollow, and drifts down more quickly:

In the second (V11), I let each keystroke be represented by a separate symbol, to give more of a sense of history, and I narrowed the indicator width when two notes got closer together (an idea based on the psychoacoustic idea of "critical band" – but not a very good visualization of it):

I wasn’t very satisfied by the falling rhombi because it was hard to tell what was going on harmonically, or to discern the interval types between notes. I decided to beef up the harmonic coloring idea by adding a spatial component to the hue component: I placed the note indicators on a "harmonic staff." In V12, pitch is indicated by horizontal position (as on a piano keyboard), and pitch class is indicated vertically with a staff (using the colors from the normal harmonic coloring). To this, I added the colored interval connections from DYAD. Because I wanted chords containing the same configuration of intervals to have the same spatial relationship visually, the staff is self-adjusting so that notes that are close together harmonically (close on the circle of fifths) pull toward the center of the staff (and lines are moved from the top of the staff to the bottom or vice versa as necessary to keep the display centered).

At this point, I started to get interested in exploring where the interval types (as indicated by colored connections in DYAD) came from: the overtones in musical notes. I wanted some way of depicting a note that showed not only its pitch and pitch class, but the pitch classes of its overtones. So, I borrowed from the circle of fifths coloring idea, and created a little note indicator that showed the root, P5, M3, and m7 pitch classes (the pitch classes present in the first eight harmonics), thusly:

The root indicator (blue, pointing more or less down) is the longest, since the first, second, fourth, and eighth harmonic are the strongest; the P5 indicator (violet, at about 6:30) is shorter (since the third and sixth harmonics are less strong); the M3 indicator (purple, at about 9:30) is less strong, and the m7 indicator (should be greenish ... hmmm, something’s gone wrong in the screen capture) is the weakest.

When multiple notes play, the indicators for their common overtones get connected by lines of the same color, like this:

Also, to show where the "harmonic center" is, I added "harmonic gravity" by assigning weights to all the overtone indicators, so that they’d tend to fall to the bottom (here, the violet M3 balances the green m7):

To give a little better sense of melodic history, I added a bar-graph history display:

and a little "harmonic compass" that rotates along with the tonality angle indicators:

The tonality indicators only showed the pitch class of the harmonics, and not their absolute positions, so I tried a harmonic display. Here is a single note, with its harmonics shown (fundamental at the bottom is strongest):

Here are two notes a P5 apart (very consonant):

And here are two notes a m2 apart (very dissonant):

I don’t think this display is very successful, because it doesn’t make it obvious that the "clash" between two notes arises from their harmonics being within the "critical band" of each other. So I’m thinking about how that might be made more obvious...

In my original circle of fifths program , there were three sections of the display for each pitch: one which is illuminated when the note is sounding, one which illuminates quickly and fades quickly, and one which is illuminated and fades more slowly. In V18, I extended this idea into a continuous gradation of colors, with the edge being the "now" part and the center being the history:

Carl Lumma reminded me (1999jun26) about a way to use a lattice to show interval connections: each node in the lattice represents a pitch class, and the direction of movement from one node to another indicates the interval between those nodes.

For example, this lattice shows the perfect fifth, major third and minor third:

From a given node, movement to the right is a perfect fifth up (and thus movement to the left is a perfect fifth down), movement diagonally up to the right is a major third up, and movement diagonally down to the right is a minor third up.

This kind of chart has been done before; the things I added (1999jul01) are:

The pitch class names automatically shift when they get too far from the center (ie: when edge pitches reach triple sharps or triple flats).

V20 Snow

Hmmmm... What would it look like if there was snow falling on it ... ?