| Site Map
My music visualization manifesto
How is it that we are able to find correspondences between an auditory experience and a visual experience?
This question leads to others:
Why have we developed the ability to find such correspondences?
What neural mechanisms make it possible for us to form them?
What kinds of correspondence are we able to find?
What effect does it have on us to find them?
What uses can one make of these effects?
What uses have I chosen to make of them?
What progress have I made, where am I now, and what's next?
I have two reasons for wanting to answer these questions.
First, although I've been working on this project for more than 30 years,
I've never tried to describe, clearly, what I'm doing.
I'd like to get a better grasp of that,
so that I could focus more clearly on what's important.
Second, the desires and goals that motivate my work are different
from those that motivate other practitioners of music visualization;
I hope this article to serve as an apologia,
to help my audience better align their expectations with my intent.
Why is this the right place to start?
Forming an analogical association between two experiences is not a rare occurrence;
we (and, it would seem, other creatures) do it all the time;
it's the basis of what we do to make sense of our environment.
(Douglas Hofstadter goes so far as to suggest that analogical thinking is the very
core of cognition.)
The most obvious kind of association is between two experiences of the same kind;
for example, when you see something you've seen before, you think "it's the same thing"
or "it's that thing."
But associations between two different kinds of experience are also common;
when you see and hear a lion roar,
you know that what you're seeing and what you're hearing are both manifestations of the same event,
and when you subsequently think about a lion's roar,
you will likely bring to mind both the image and the sound.
almost all things in our environment that we care about are visible
many things in the environment that we care about make sounds
sound is omnidirectional
making sense of our environment holistically
cause and effect
we want there to be correspondences
we form correspondences even when none are intended
better to see coincidental correspondences than to miss a non-coincidental one
we're happy to fuse things when they happen together (hand sensation cue fusion)
The Neurology of Analogy
is it the same as synesthesia?
the output of the cochlea (or a spectral analysis of a audio waveform) is a little more
raw than what we perceive consciously
where does grouping happen?
Possible Types of Aural/Visual Analogy
simple: common origin, common fate, salience, importance, size, prominence
medium: high/low, number, articulation, alternation
difficult: texture, timbre, consonance/dissonance, tonality, harmony
impossible: ??? e.g. angle in vision has no aural analogue
frequency in light is not perceptible directly, and thus has no analogue
uniform :: variegated
examples LMST, LMST in both tones and lights
learned versus innate analogies
notes can be regular, irregular, on the beat, off the beat ...
things happen in music; what is the relationship between those things?
one note can be higher or lower than another, louder or softer, longer or shorter ...
to make a perceptual analogue of sound, you need to emulate the behavior of the auditory system ...
gesture: change, distance (relative)
because both exist in time, common fate can be a strong grouping cue
how about symmetry? no? up/down in parallel ...
it's very useful to be able to see something you're hearing; it often tells you more
causality is temporal, not spatial
up to a certain speed, rhythmic events can fuse
space: physical space, aural space, the degree to which either is filled
how does synesthesia fit in?
The Psychology of Analogy
the novelty of music visualization cuts both ways: it's attention-getting at first, but it fades
when you can see music, it’s like having a new sense
it's gratifying to form analogies
an analogical experience is more rich, more intense, than a uni-modal one;
e.g. food can be cold, hot, have a texture, flavor, smell, appearance,
and ability to satisfy hunger ... all of which are interconnected.
Auditory Scene Analysis, by Albert S. Bregman
Analogy in Art
listening to music evokes a pleasurable (satisfying) response
why is it satisfying?
2003 Reith Lecture (Ramachandran)
in art, analogy can be used to make a point
a child can point and the parent can say “square” but
with music it’s harder to know what you’re pointing at,
and most parents don't know the names of things in music.
what things are going on in a piece of music?
how does this relate to the quality of music?
the objective versus subjective content of music you can
have a passage that makes you feel a certain way but what is it in it that makes
you feel that way? can a visual analogue also make you feel that way? if the
visual analogue represents the feeling rather than the cause is it being too
the response to music is personal
the response to music is universal
the things that only music visualization can do: a
performer can help you see the sense of a piece, a music theoretician can tell
you what's going on in an intellectual way, music visualization can get at the
space in between.
musicians make sounds
dancers move to sounds
the perceptual elements are what are act...
act of forming the piece, the whole
learning about music is harder than it needs to be
what's the best way to make use of the novelty of music visualization?
I'm a composer, so I'm more interested in what makes one piece of music different from another.
The Hard Problems
what to do with words in music?
what is happening when there's a chord change? A chord
progression? A modulation? A return to the tonic after a modulation?