Home | Site Map | Watch | FAQ | History | Store | Contact

Conversation between Alex and Stephen (prev top next)


Alex, you wrote

I am so glad to have someone with the same level of interest in this field ...

I feel the same way. Over the years, I've gotten to know a few people a handful of people who were as interested in music visualization as I am, but your philosophy, esthetic and goals are by far and away closer to mine than any I've encountered before. I told my wife Lisa that finding you was sort of like falling in love.

>... visual narrative ... investigate whether the visual reproduction of this vast amount of aural information would lend itself to a similar narrative experience ... requires that no consideration as to emotion is given in this process, as this would impose a sense of narrative ... an appropriate analogy would be of someone applying a narrative to a musical score based entirely on what they see written there without hearing the music ...

I'm not sure I'm understanding what you think the relationships are among meaning, narrative, emotion, and formal elements (form, structure, process, etc.). For example, there are (what I think of as) emotional effects associated with different types of modulation, cadence, etc., and when I look at a score, I see those forms, and I know what the emotion associated with them might be. So I don't see how emotion could be eliminated.

This form of visual art requires a level of fluency in a visual "language". One that may be fairly easy for the academic and musical fraternity to learn, but what about the average punter on the street? This is one reason why I have supported my animation with educational material.

My take on this is that (ideally, at least) the art form is itself the educational material. The section in this


titled "Sonata as Teaching Machine" may have been my introduction to this idea (I remember being excited when I read it, but I can't remember whether it was because it reinforced an idea I already had or because it was a new idea that I could see was important). How might this work? I see Fischinger's early "Studies" to be both his own study of techniques of visual motion and examples for an audience to study to learn how to look at his films. This approach has some advantages: it provides "stepping stones" for your audience --- they don't have to learn too much at once. The "more lesson than magnum opus" pieces can be shorter, more repetitive, more obvious, which is an advantage (if it takes a year to produce a piece, there's a lot riding on each one, and less opportunity for feedback and reflection).

... my use (and yours) of up and down motion (same as notated music) to notate pitch is perfectly appropriate. It already taps into a basic level of musical education that most people have experienced at some point in their lives.

But more important, it's simple enough that even if a person has no familiarity with music notation, they can still "get it" in a short amount of time. I think one of the reasons the Toccata and Fugue scrolling score has been so popular is that its opening teaches you what's going on with the notation in such a clear way that nobody fails to get it.

The meaning of this detail is not as important as the viewing audience experiencing, probably subconsciously, "as that trumpet tone changes from the beginning of the breath to the end, I can also see a corresponding change in its visual representation."

For me, the word "corresponding" is where the rubber meets the road. There are lots of ways things can correspond. Synchrony, especially onset synchrony, is a very easy correspondence for people to notice -- so easy, in fact, that almost any animated visual stimulus will seem to be related to almost any rhythmic aural stimulus. Vertical position and pitch seem to work pretty well. So, we have pitch and time covered, for a start. Where next? You say the meaning of this detail is not as important as having the audience sense that there is a correspondence. I agree, but "importance" depends on your point of view. For example, how important are compositional decisions? How important is a particular compositional choice in a Beethoven sonata? Is it as important as, say, having your instrument in tune? Having an instrument at all? Not having your 'cello when you're about to play a Beethoven sonata is going to have a much more severe effect on the audience's experience than any compositional decision Beethoven might have made, so in that sense, you might say that having an instrument is more important than some obscure compositional detail. Of course, Beethoven didn't have to worry about whether the 'cellist had an instrument -- that was a given. So, I'd say, yes, it's essential that people recognize that there's a correspondence, but once that's a given, it's no longer something you have to consider, and other things can be seen as "important" -- and that's the direction we need to strive.

Once you have established a methodology for representing all the quantifiable variables, like volume, pitch, key etc, you are left with the unquantifiables, the "narrative". This amounts to the nuance of a phrase, for example. A melodic line which one might describe as sad, melancholic or introspective, is ultimately greater than the sum of its individual parts. So how much "detail resolution" can you rely on before you have to impose an interpretation onto it.

Actually, I don't think of expressive nuance as unquantifiable -- just "unquantified to date." I'm working on this ...
Imagine this melodic line you would also interpret as smooth and graceful, however, musically it jumps across large intervals, 5ths, octaves, etc. If you let your hand be that musical line as it plays, it has to jump these interval distances, but at the same time you wish to maintain a visual perception of smoothness and grace, which visually it does not. Already you begin to see a conflict between a literal visual interpretation of musical line with the aural experience of it.

I agree that there's a conflict, but I think it's introduced artificially.  You say "literal interpretation of the musical line" but there are many interpretations that could be considered literal.  For example, there's an octave leap between the first two notes in "Somewhere, Over the Rainbow."  If you look at the notation, it seems like an abrupt leap, and there's also a leap when you play it on the piano or the 'cello.  But if you look at the pitch curve when a singer sings it


it is much smoother (which is how it's felt).  Which one of these should we consider the "literal" interpretation?  I would argue that our perception of pitch in language has a big effect on how we interpret music, and that even when someone plays the melody of "Somewhere" on the piano, we're letting our experience of what a singer would do inform our sense of smoothness.  It makes a difference what you use as your basis for deciding what it is we're interpreting.  For example, let's say that you were trying to make something which responded to intensity in an audio signal.  A first cut at this might be to look at the waveform, and ask "what's the amplitude?"  This would track certain kinds of change just fine, but a sine tone wave at 20 Hz with a given amplitude will sound very quiet compared to a sawtooth wave at 100 Hz with the same amplitude.  Simple amplitude isn't adequate; we need to extract a feature that maps better to perception.  In other words, the interpretation needs to be based on psychoacoustics.

So we also need to investigate and understand the language of visual motion, and find a compromise to this kind of problem. This investigation may also need to address the associations we have with particular instruments, for example, the "physical mass" associations we have between a piccolo and a double bass may well influence ones interpretation of their nimbleness moving within a 3 dimension environment.


It was always my intention for the imagery to support the music, and to hopefully encourage people to return to the music itself, without the imagery.  ...  There are however questions about why certain intervals, or chord progressions have the effect that they do.

Okay, it's time for me to recommend:

        Huron, David, Sweet Anticipation, Music and the Psychology of Expectation, 2006, MIT Press

This just came out, so it's too soon for me to say for sure, but I think it's going to be the book that's the most useful to me in doing music visualization.  As a teaser, here's a chart from it:


This is the result of a survey of musicians.  Huron asks (and attempts to answer) the questions: where do these qualia come from? what accounts for them?  Here's a list of chapter titles plus selected subtitles and selected descriptions of tables and figures (which I compiled to post to Amazon):


What I am hoping to explore is using visual strategies which have similar effects of tension and release, but are based on the mathematical relationships that exist in music ... I am confident that there is some validity in establishing styles of visual consonance and dissonance based on these relationships.

I know what you referring to when you say "mathematical" and I do believe there are visual strategies based on this ... but I wouldn't call it "mathematical" ... the relationships can be described mathematically, but the aspect of them that's most important in music is only related to those relationships by happenstance.  For example, you and I would agree that an octave is consonant, relative to a major seventh or a minor ninth.  What accounts for this?  You might be tempted to say "it's the 2-to-1 ratio of the frequencies."  But we don't actually have a way of sensing that ratio directly.  If I were to give you a pair of sine-wave oscillators (that generate pure tones, with no harmonics) and ask you to tune them to an octave, you'd be able to get them relatively close, but not close enough that you'd be satisfied if a 'cello or piano were tuned to the interval in actual music.  When you add whole-number multiple harmonics, you will get beating between the harmonics of the lower and upper pitches (L2:H1, L4:H2, L6:H3, etc.) and this will let you tell very accurately when you're close to dead-on.  However, if the harmonics were out of tune, your estimate of in-tune-ness would be, too.  Some geometries of vibrating systems produce harmonics that are close to a mathematically-pure harmonic series, and we've designed instruments to take advantage of these geometries.  For example, a brass instrument produces something close to 1:2:3:4 ... times the fundamental frequency due to the shape of its bell.  Strings tend to be close to 1:2:3:4 ... if they're relatively long compared to their thickness.  The only instruments that are "natively" perfectly harmonic are ones like the human voice in which the harmonics are not produced by independent modes of vibration of multiple sine sources but are the result of filtering a waveform in which the harmonics are inherently phase-locked (because they originate in a non-sine source).  Another way to say this is: our auditory system does special things with harmonically-related sounds because sounds with those relations are present in the sounds we hear -- not because they are mathematically related.  Some of the sounds we hear contain harmonics that are mathematically related, but if they weren't, we would've learned to deal with whatever relationships were present.

Alongside my orchestra work, gig work for weddings etc, I also work at Questacon, the National Science and Technology Centre. This is essentially a fun park based on science, designed to make science fun for kids. I am a gallery assistant, a glorified child minder really. Occasionally a child or adult comes through who has a genuine thirst for knowledge with whom you can engage with, otherwise it pays bills and offers me the time versatility I require to do the other things in my life.

Sounds like you've got a nice balance.  I envy that; I need to figure out a way to get more time for art.


(prev top next)