Home | Site Map | Watch | FAQ | History | Store | Contact

Conversation between Alex and Stephen (prev top next)

2007feb10

Stephen,

I am so glad to have someone with the same level of interest in this field, you are indeed asking pertinent questions which  require me to clarify my thinking.Your style of response is very effective, I shall use it myself.

Alex, you ask whether a visually emotional narrative could emerge from musical structure alone, as it does aurally, but in the next sentence you say that the purpose of your work is to develop a visual experience that places the role of music at its center.  Was the first question rhetorical, or are you straddling the fence?  Assuming it's not ...

You have identified a conflict which I had hitherto avoided. In retrospect, when I imagined a visual narrative emerging from the imagery it was one that reflected the aural narrative. The idea was to investigate wether the visual reproduction of this vast amount of aural information would lend itself to a similar narrative experience. This is tricky as the number of interpretive processes engaged in when translating from one medium to the other introduces levels of subjectivity which you could argue renders this objective unworkable. It also requires that no consideration as to emotion is given in this process, as this would impose a sense of narrative. Perhaps an appropriate analogy would be of someone applying a narrative to a musical score based entirely on what they see written there without hearing the music. But this relies on a level of musical education (either cultural or academic) which at this point in time can't be expected of general the viewing audience. This follows onto my agreement with your following statement.

I think the answer is yes, as long as you're willing to define "musical structure" in a way that's abstract enough that it could be expressed as adequately visually as it is aurally.

For example, the opening of Beethoven's fifth symphony could be abstracted as "three short ones that are all the same followed by one longer one that's different."  That's pretty small-scale, but it's structure.  Is it emotional?  I think so (to the extent that music is emotional --- which is a separate question); it's only a single atom or molecule of an emotional narrative, but it has an emotional tone, one that is distinct from, say "one long one followed by three short ones" --- more serious, for example.  Of course, a tiny motive is a different thing than a symphony.  Could a visual narrative achieve as much as an aural one?  I think the answer is "yes, but."

A musical narrative doesn't exist in a vacuum.  To understand (and feel) a Beethoven symphony, you need a certain degree of fluency in its idiom.  There are neural mechanisms that allow us to distinguish, without training, consonance from dissonance (in some contexts), but Beethoven's use of consonance and dissonance depends much more on learned meanings of pitch relations than on raw perceptions.  If there were to be, ex nihilo, a visual composition comparable in scope to a Beethoven symphony (but as foreign to us as a Beethoven symphony would be to a person who'd never heard music), we simply wouldn't have the familiarity with its idioms to understand it.  We might be amazed (and certainly awe is an emotion), but that's a different thing than understanding the narrative intent of the composer.

And so you appreciate my dilemma. This form of visual art requires a level of fluency in a  visual "language". One that may be fairly easy for the academic and musical fraternity to learn, but what about the average punter on the street? This is one reason why I have supported my animation with educational material. However this also brings me to your next point.

There's another possibility, though: using a visual idiom that already exists.  This is not a complete solution, because our contemporary visual idiom is largely representational, and does not map one-to-one to musical idiom, but it's a place to start.  The visual language needs to grow and evolve with the art that's communicated in that language.

One of my requisites in the production of music visualisation is to use already well established methods of data visualisation. So my use (and yours) of up and down motion (same as notated music) to notate pitch is perfectly appropriate. It already taps into a basic level of musical education that most people have experienced at some point in their lives.  Representation of acoustic qualities requires a greater demand on the viewing audience, however our computer savvy youth are much more exposed to these visualisation techniques than previous generations. At least for me, the important objective is to develop a type of acoustic  imagery which changes in visual detail as much as it does in aural detail. The meaning of this detail is not as important as the viewing audience experiencing, probably subconsciously, "as that trumpet tone changes from the beginning of the breath to the end, I can also see a corresponding change in its visual representation."

These type of high detail queues are particularly important, I think, for being able to absorb some of the interpretive issues concerning narrative discussed earlier. Once you have established a methodology for representing all the quantifiable variables, like volume, pitch, key etc, you are left with the  unquantifiables, the "narrative". This amounts to the nuance of a phrase, for example. A melodic line which one might describe as sad, melancholic or introspective, is ultimately greater than the sum of its individual parts. So how much "detail resolution" can you rely on before you have to impose an interpretation onto it.

Let me offer an example. Imagine this melodic line  you would also  interpret as  smooth and graceful, however, musically it jumps across large intervals, 5ths, octaves, etc. If you let your hand be that musical line as it plays, it has to jump these interval distances, but at the same time you wish to maintain a visual perception of smoothness and grace, which visually it does not. Already you begin to see a conflict between a literal visual interpretation of musical line with the aural experience of it. It also reminds us  there is already a visual language which we operate in that needs to be addressed in this process of translation.

So we also need to investigate and understand the language of visual motion, and find a compromise to this kind of problem. This investigation may also need to address the associations we have with particular instruments, for example, the "physical mass" associations we have between a piccolo and a double bass may well influence ones interpretation of their nimbleness moving within a 3 dimension environment.

However, there's more to "musical structure" than just ... structure.  A piece of music is not just a set of mathematical relations.  It is also the effect those relations have on us when they're embodied in musical sounds ... and sounds have affects.  Just as the color red can only be known by experiencing the color, musical sounds can only be known and felt by hearing them (though, of course, in both cases, the memory of the experience is also an experience).  So, visual music will always be distinct from aural music.  Of course, the distinction could be blurred: just as I can read a score an "hear" the music in my head, it should be possible for a person to watch visual music and imagine sounds.  But for these sounds to be the ones intended by the composer, the listener would have to be very fluent in the idiom.  I don't think this is impossible; for example, if you knew me personally, you could imagine my voice as you read these words.  But the average literate person's facility in verbal language is well beyond the average person's facility in musical language (being able to hear a score when reading it is the exception), and our visual language is only now being born.

It was never my intention  for anybody to use this style of music visualisation without the music it was based on. It was always my intention for the imagery to support the music, and to hopefully encourage people to return to the music itself, without the imagery. It was never my intention to engage the audience with the imagery alone. So in that regard I am not aiming to elicit those  experiences induced by sounds, using images, I am content with the music doing that itself. There are however questions about why certain intervals, or chord progressions have the effect that they do. When people listen to passages of music that move them, there are of course many factors contributing to their experience, however one of them is the choice of harmonic structures which the composer uses to create this. What I am hoping to explore is using visual strategies which have similar effects of tension and release, but are based on the mathematical relationships that exist in music, why? because I believe that the complexity we see in this area is, as is often the case else where,  a phenomenon emergent from a simpler set of functions. Embed these simple functions as an integral component of the visual structure and perhaps broader patterns may emerge. This still requires a lot or experimentation, but I am confident that there is some validity in establishing styles of visual consonance and dissonance based on these relationships. 

I don't mean to suggest that I want to develop a visual language that exists apart from music; I don't.  I'm too much a musician to be passionate about a mute art.  But even if I wasn't interested in music as a permanent companion to abstract animation, I would still chose it for now, because people's understanding and love of music provides leverage.  Was it Stravinsky who complained that when he attended concerts he didn't know what to do with his eyes?  In any case, our eyes are often free to do something else when we're listening to music; opera and ballet take advantage of this, but abstract animation has the potential for being more satisfying visually than either (though the human form can be pretty compelling).  And: music provides a context, a frame of reference in which a visual language can be taught.  If a musical event has a certain meaning (within the idiom of a certain composition) and we place a visual event in parallel with it, the audience will associate the two, and begin to expect the visual event to function the way the musical one does.  If the visual event is well-designed, this association will be satisfying, and thus more memorable than if it were arbitrary.  Using music as a platform, a springboard for developing an abstract visual musical language is very practical, because people already listen to music, and they like forming associations.  It's a natural.

P.S. Alongside my orchestra work, gig work for weddings etc, I also work at Questacon, the National Science and Technology Centre. This is essentially a fun park based on science, designed to make science fun for kids. I am a gallery assistant, a glorified child minder really. Occasionally a child or adult comes through who has a genuine thirst for knowledge with whom you can engage with, other wise it pays bills and offers me the time versatility I require to do the other things in my life.

Alex

(prev top next)