Home | Site Map | Watch | FAQ | History | Store | Contact

My music visualization manifesto

How is it that we are able to find correspondences between an auditory experience and a visual experience?

This question leads to others:

  • Why have we developed the ability to find such correspondences?
  • What neural mechanisms make it possible for us to form them?
  • What kinds of correspondence are we able to find?
  • What effect does it have on us to find them?
  • What uses can one make of these effects?
  • What uses have I chosen to make of them?
  • What progress have I made, where am I now, and what's next?

    I have two reasons for wanting to answer these questions.  First, although I've been working on this project for more than 30 years, I've never tried to describe, clearly, what I'm doing.  I'd like to get a better grasp of that, so that I could focus more clearly on what's important.  Second, the desires and goals that motivate my work are different from those that motivate other practitioners of music visualization; I hope this article to serve as an apologia, to help my audience better align their expectations with my intent.

    Why is this the right place to start?

    Analogy Abounds

    Forming an analogical association between two experiences is not a rare occurrence; we (and, it would seem, other creatures) do it all the time; it's the basis of what we do to make sense of our environment.  (Douglas Hofstadter goes so far as to suggest that analogical thinking is the very core of cognition.)  The most obvious kind of association is between two experiences of the same kind; for example, when you see something you've seen before, you think "it's the same thing" or "it's that thing."  But associations between two different kinds of experience are also common; when you see and hear a lion roar, you know that what you're seeing and what you're hearing are both manifestations of the same event, and when you subsequently think about a lion's roar, you will likely bring to mind both the image and the sound.


    Notes ...

  • almost all things in our environment that we care about are visible
  • many things in the environment that we care about make sounds
  • sound is omnidirectional
  • making sense of our environment holistically
  • cause and effect
  • recognition
  • we want there to be correspondences
  • we form correspondences even when none are intended
  • better to see coincidental correspondences than to miss a non-coincidental one
  • we're happy to fuse things when they happen together (hand sensation cue fusion)

    The Neurology of Analogy

  • is it the same as synesthesia?
  • the output of the cochlea (or a spectral analysis of a audio waveform) is a little more raw than what we perceive consciously
  • where does grouping happen?
  • McGurk Effect
  • http://psy.ucsd.edu/chip/ramapubs.html
  • tonotopic maps

    Possible Types of Aural/Visual Analogy

  • simple: common origin, common fate, salience, importance, size, prominence
  • medium: high/low, number, articulation, alternation
  • difficult: texture, timbre, consonance/dissonance, tonality, harmony
  • impossible: ??? e.g. angle in vision has no aural analogue
  • frequency in light is not perceptible directly, and thus has no analogue
  • uniform :: variegated
  • examples LMST, LMST in both tones and lights
  • learned versus innate analogies
  • notes can be regular, irregular, on the beat, off the beat ...
  • things happen in music; what is the relationship between those things?
  • one note can be higher or lower than another, louder or softer, longer or shorter ...
  • to make a perceptual analogue of sound, you need to emulate the behavior of the auditory system ...
  • number, density
  • gesture: change, distance (relative)
  • because both exist in time, common fate can be a strong grouping cue
  • how about symmetry? no? up/down in parallel ...
  • it's very useful to be able to see something you're hearing; it often tells you more
  • causality is temporal, not spatial
  • up to a certain speed, rhythmic events can fuse
  • space: physical space, aural space, the degree to which either is filled
  • how does synesthesia fit in?

    The Psychology of Analogy

  • the novelty of music visualization cuts both ways: it's attention-getting at first, but it fades
  • when you can see music, it’s like having a new sense
  • knowledge
  • it's gratifying to form analogies
  • an analogical experience is more rich, more intense, than a uni-modal one; e.g. food can be cold, hot, have a texture, flavor, smell, appearance, and ability to satisfy hunger ... all of which are interconnected.
  • humor
  • understanding
  • http://prelectur.stanford.edu/lecturers/hofstadter/analogy.html
  • Auditory Scene Analysis, by Albert S. Bregman
  • learning

    Analogy in Art

  • listening to music evokes a pleasurable (satisfying) response
  • why is it satisfying?
  • 2003 Reith Lecture (Ramachandran)
  • in art, analogy can be used to make a point

    My Choices

  • a child can point and the parent can say “square” but with music it’s harder to know what you’re pointing at, and most parents don't know the names of things in music.
  • what things are going on in a piece of music?
  • how does this relate to the quality of music?
  • the objective versus subjective content of music you can have a passage that makes you feel a certain way but what is it in it that makes you feel that way? can a visual analogue also make you feel that way? if the visual analogue represents the feeling rather than the cause is it being too pushy?
  • the response to music is personal
  • the response to music is universal
  • the things that only music visualization can do: a performer can help you see the sense of a piece, a music theoretician can tell you what's going on in an intellectual way, music visualization can get at the space in between.
  • musicians make sounds
  • dancers move to sounds
  • the perceptual elements are what are act...
  • act of forming the piece, the whole
  • learning about music is harder than it needs to be
  • what's the best way to make use of the novelty of music visualization?
  • I'm a composer, so I'm more interested in what makes one piece of music different from another.
  • meaning

    The Hard Problems

  • rhythm
  • form
  • melody
  • what to do with words in music?
  • harmonic progression
  • what is happening when there's a chord change? A chord progression? A modulation? A return to the tonic after a modulation?