|
What am I doing?
Introduction
I made my first graphical score because I wanted to eliminate some of the things that made it
hard to follow a conventional, symbolic score in real-time: a complicated mapping of pitch
to position (multiple staves, multiple clefs, and sometimes multiple transpositions),
and the need to remember which staff corresponds to which instrument
(these are labeled at the first page of a movement, but usually not on subsequent pages,
and only within a page when there's a change of instrument on a staff), and a symbolic
system for showing timing.
In my first experiment, I put all the notes on a "grand staff" (the normal way of showing piano music: two staves,
treble and bass clef) and colored the notes to indicate instrument ...
http://www.musanim.com/mam/hist03.html
My second try was a bar-graph (aka "piano roll") ...
http://www.musanim.com/mam/hist04.html
That worked pretty well, and for the next thirty-six years,
while I did experiment with other types of notation (to highlight various musical attributes:
intervals, timbre, rhythm, tonality, etc.),
I spent most of my time making scrolling bar-graph scores,
and if you'd asked me what I was doing,
I would have explained that I was converting music notation into a form that's easier for listeners to follow.
Since the early 1900s, visual artists have been exploring with combining music and abstract animation,
but while I was a big fan of their work in this new art form,
I didn't see what I was doing as being part of that;
"I'm not an artist; I'm just doing technical work," I thought.
This began to change in 2010, when I retired from my day job and turned my full-time attention to animated graphical scores.
What had gotten me into this project in the first place was my thrill at seeing musical scores animate themselves (something that the work of Fischinger,
McLaren, Marcussen, et al. reminded me of ...
http://www.musanim.com/Inspiration/
... ), and I started experimenting with ways to share that experience with others.
This led to my first truly animated (as opposed to merely scrolling and/or highlighted) graphical score: Debussy's First Arabesque ...
https://www.youtube.com/watch?v=A6s49OKp6aE
In the years since then, my focus has been to develop new methods of representing musical notes visually (which I call renderers since they render notes as shapes), and to use these methods in animations which ...
... which what? What am I trying to do now?
That's what I'm hoping to answer here.
Since I don't have much perspective on this, I'm going to approach it by reviewing Examples what I've done recently,
examining the choices that have presented themselves, then Reflecting on how this relates to what people in related fields have been doing.
Examples
2010apr08 Debussy, First Arabesque
In previous bar-graph animations (using the Bars renderer e.g.), I'd used size and color to differentiate voices in contrapuntal music and layers in homophonic textures. In the Balls renderer, I added connecting lines and motion between the notes of a voice/layer. This brought up the question: how should the note shapes move?
I first tried moving the shapes at a constant rate,
but because this exactly cancelled out the scrolling of the score,
it looked like the notes were standing still (moving vertically but not horizontally).
It didn't feel right.
But what was "right"?
I wanted the motion of the notes to feel more like the motion of the music.
Playing around with various ways of changing the rate,
I started to see that what I was trying to depict was something you might call gestural intent,
and I found that if the shapes moved faster when the next note was imminent and slower when it was further off,
it seemed more natural, more musical.
There were also some interesting choices about which notes belonged to which layer.
I'd studied both music composition and auditory scene analysis (e.g.),
so I had a pretty good idea of what belonged together and why.
Though I didn't realize it at the time, what I'd begun doing was
embodying aspects of my perception of the music in the animation.
2010jun14 Beethoven, String Quartet No. 15, 3rd mvt,
"Heiliger Dankgesang"
This piece has a feature I'd not dealt with in any of my previous animations:
significant changes of tempo between sections.
These changes also correspond to a change of mood, between solemn and exuberant. In my previous animations,
the scrolling was at a constant rate, but that wouldn't work here: if the scrolling was slow enough to see the form of the slower passages,
it was too slow to see the detail of the fast passages. The obvious solution: multiple scrolling rates.
To make the contrast more dramatic, I used the staid Bars for the slow, hymn-like parts,
and the traversing Balls for the "feeling new strength" parts.
Again, at the time, I didn't realize it, but by having the balls intrude on the serene bar-graph scene, I was
depicting the element of surprise
(something I didn't do consciously until later; see Surprise).
2010jun17 Brahms, Piano Quartet in C minor, opus 60, 3rd mvt.
I experimented with ways of combining the Bars and Balls;
a happy result was to use Bars for piano and Balls for strings.
The made sense—the sound of the string instruments is more continuous,
the notes of the strings are more easily recognized as connected,
you can sense when a string player's next note is going to happen,
the piano's keys are rectangular, the piano's notes sound angular and uniform, etc.
In doing this, I was
mirroring perceptual differences in sound with analogous differences in visual form.
2011jan11 Gibbons, Fancie #1 for Two Treble Instruments
Here, I took the idea of "two equal instruments" and performed the two parts on pipe organ,
both with the same stop, and both played with my right hand.
I'm not sure what this counts as; I'm mirroring something ... ?
2011oct14 Beethoven, Symphony 5, 1st mvt.
In music with strong dynamic contrasts, the difference between loud and soft can go from being an expressive element
(that, for example, shapes a melodic line) to being a structural element (that creates its own distinctions).
To incorporate that into this animation, I extracted a "dynamic track" from the audio,
and varied the size of the note shapes in proportion to loudness.
Because I didn't have separate information for each instrument,
this had to be applied to all the notes sounding at the same time,
but that's pretty much how the dynamics of the piece are specified,
so it worked out okay.
2011oct17 experiments with bowing
With a solo instrument, it would not be a limitation to have a single dynamic track,
so I licensed a recording of a movement of one of Bach's suites for solo cello and started experimenting with it,
developing renderers that could display changes in dynamics continuously
(instead of just once per note, as in the Bars and Balls renderers).
As part of this, I tried showing bowing (not very successfully),
using a non-uniform time scale to show both global structure and local detail simultaneously,
wiggling a line to suggest a vibrating string (currently strikes me as silly,
but might not if it worked better).
2011nov12 continuous line ( cont...)
The bowing experiments led in an interesting direction: representing melodic motion with a continuous line.
The two most satisfying products of this exploration were the Callig(raphy) and SplineRibbon renderers.
So, what was I doing here? From my point of view at the time, I was just playing around,
but looking back, I see that what I was playing with were the
dynamic forms that emerge from timing, loudness, and pitch.
Also, the SplineRibbon renderer, built around the somewhat chaotic behavior of a cubic spline interpolation,
yielded a lovely, uncontrolled form that I was delighted to fill with a watercolor-like shading
(e.g.).
2011dec01 Bach, Contrapunctus 3, Art of Fugue
At some point, I asked myself: besides gliding (like Balls) and drawing (like Callig and SplineRibbon),
what other ways of getting from one note to the next are there?
The one I wanted to explore was walking,
but that seemed way too complicated (as most things I want to try seem), so I took a stab at rolling:
having one rectangle in a melody roll to the next one.
At this point, I had large enough collection of visually distinct renderers to use one for each subject in a multi-subject fugue,
and thereby identify thematic material by shape and style of motion.
2012jan27 Franck, Violin Sonata, mvt. 4
I'd admired the canon in this piece since the first time I heard it,
and wanted to do something special to highlight the canon, so I wrote the Canon renderer.
This was the first renderer that showed a structural relationship between two lines of music.
2012feb01 Debussy, Prelude to 'The Afternoon of a Faun'
With this animation, I felt as if I'd gotten into a new realm;
the combination of harmonic coloring (coloring notes according to their position on the circle of fifths),
shimmer effects for string tremolo,
the smooth motion of the RingWalk renderer for the solo flute,
and hollow shapes to indicate pizzicato
seemed to add up to something more than just an annotated graphical score; it was more like a tableau.
2012feb26 Malinowski, Fantasia Ostinato
This piece has a compositional feature I thought it would be interesting to highlight:
an "ostinato" figure that increased in speed every time it passed from one instrument to another.
I decided that to make it obvious that it was the same figure and that it was being played at different speeds,
I'd use different time scales at the same time for different material. This meant that the ostinato always looked the same,
but moved across the display at different speeds depending on how fast it was played.
This technique turned out to be appropriate for other pieces that feature augmentation and diminution,
such as Bach's fugues (e.g.) and Beethoven's
Große Fuge.
2012mar22 Bach, Perpetual Canon
Here, I was something of a prankster.
Bach wrote a perpetual canon that modulates continuously, so that each time around, it was one whole-tone higher.
Of course, this meant that, for practical purposes,
it wasn't truly perpetual, since it would rise up and out of the range of human hearing.
I decided to remedy that by having the pitch shift downwards at a rate that would cancel out the modulation.
In the visualization, the slow downward pitch shift is mirrored by a slow downward shift in the animation.
Depending on your pitch sense, the pitch shift can be anything from imperceptible
(or only perceptible as an unsettling feeling that something's wrong) to nauseatingly unpleasant.
2012aug28 Bach, Three-part Invention #4, D minor
Up until this point, everything shown in my animations corresponded directly to notes in the score.
In this animation, I tried showing the underlying metrical framework of the piece—sort of like barlines on steroids.
This turned out to be a lot of "visual noise" that didn't help much, and after that I pretty much stuck with normal unobtrusive lines
when I wanted to show measures.
2012nov26-2013oct25 Bach, Goldberg Variations
This was the first large collection of pieces that I completed. Several of these turned out especially well,
but I did the most innovative work for the canons. In the canons in inversion (e.g.,
I put the dux and comes in the same horizontal position, so that there were two "now" points in the display and the inversion was obvious.
In the 25th variation I used one of the spline renderers for the melody, and RingWalk for the accompaniment,
a nice contrast.
2014sep12 Purcell, Chacone
I'd tried something like this before, and it hadn't worked very well,
but for this piece, showing the repeating structure of the ostinato made more sense.
2015apr27 Rivera, Cumba-Quin
This piece was unusual in that three styles of playing (normal, "pizzicato," and drumming) were used structurally.
Because the four guitar combined to form a single mega-instrument, I was able to ignore instrumentation and focus on pattern.
2015jun15 Sousa, Semper Fidelis, march, ver. 2
Sousa's marches are characterized by a relatively simple structure fleshed out with generous orchestration.
In my first version of the animation, I used my standard approach,
but this made the piece look more complicated than it sounded (and, I thought, more complicated than it actually is).
To address this, I tried something different: I sorted the score into groups, based on what melodic material was being used.
That is, all the notes of the melody would be in one group, regardless of what instrument was playing it or what octave it was playing in.
Then, I compressed the ranges so that all the notes in a group were at the same pitch
(whichever pitch seemed to me like the one you'd most notice when the piece was played).
And then, I used different colors and shapes to show which instruments, at which octaves,
were participating in a note group.
This resulting in a compressed/simplified score in which all the notes were shown,
but information about instrument and octave were compressed into a single location.
2016mar03 Mozart, Variations on "Ah vous dirai-je, Maman"
I'm indebted to my wife Christine (a Mozart lover) for coming up with the idea of
layering each variation on top what had happened before, giving a very clear picture of the structure.
2016mar10 Pachelbel, Canon in D
This one was especially fun. Pachelbel's Canon has two rules which govern the entire piece:
the bass repeats the same eight notes continuously, and the three violin parts play the same music
(each at a delay from the previous corresponding to the length of the repeating bass).
I was not the first person to think of using a circular representation for this; artist/animator David McCutchen's 1983
Pachelbel's Canon does the same thing.
2016may28 Bach, Herr Christ,
der einge Gottes-Sohn from Das Orgel-Büchlein, BWV 601
The experiments that led to this are described here.
What's significant about the Voronoi renderer is that a Voronoi tessellation "owns" the entire screen,
in a way that's arguably analogous to the way any piece of music "owns" the entire acoustic space.
A score can be very sparse, but music always seems to be, in some sense, everywhere:
you can't turn away from music, and "holes" in a musical texture only become obvious when something exits precipitously.
2016jun25 Bach, Fugue in B-flat major, WTC I, BWV 866
My second "major set" of pieces was
Chopin's Etudes
(most based on recordings by Paul Barton, but some performed by me);
the third was the first
book of Bach's Well-Tempered Clavier (based on public domain recordings by Kimiko Ishizaka).
In this particular fugue, I tried something I hadn't before: not showing most of the notes in the score.
For some reason, I found this (which was an unintended effect that came up when I was experimenting with various things
while designing the animation) almost paralyzingly funny (and still do), so I left it as-is, as a kind of Duchampian joke.
2017jun07 Chopin, Scherzo No. 3 in C-sharp minor, opus 39
Music has lots of surprises, and sometimes what happens in my animations is surprising,
but this piece was the first place where I deliberately hid notes until they were played to create a visual surprise
that mirrored the effect in the music. (I don't want to be a spoiler, so I won't say anything about where this happens,
other than that it's not right at the beginning.)
2017jun14 Bach, 4th mvt. , Trio from A Musical Offering
Continuing the exploration of ways to use surprise, in this piece,
I hid the notes in the future so as to mirror the effect of a deceptive cadence ...
2017jul20 Mussorgsky/Rimsky-Korsakov, Night on Bald Mountain
... and in this, I went a step further and
removed notes in the future that the listener might expect but which didn't happen.
2017nov04 Cummings, love is more thicker than forget
In this, I tried my tools on non-music, a poem by E. E. Cummings.
2017dec17 Bach, Fuga a 3 Soggetti, Art of Fugue
My fourth big set was based on Kimiko Ishizaka's recording of Bach's Art of Fugue.
I explored some interesting techniques with its canons (which I plan to discuss here).
The new thing I did in the final fugue was to
use various techniques to show how Ishizaka's completion fit into Bach's unfinished score.
2018jul10 Ravel, La valse
In this piece, there is a surprise of a kind I'd not encountered before: in the middle of a repeating passage,
the repetition is interrupted by the insertion of some material from earlier in the piece, after which the
repetition resumes. My animation mirrors this. (SPOILER: It's toward the end of the piece so if you just want to see the effect,
start at 12:00. But I recommend you watch the whole video, since it's one of my best to date.)
Reflections
I've clearly gone beyond "converting music notation into a form that's easier for listeners to follow," but how far beyond, and in what direction?
Is this a new art form? My current answer to that is: sort of, but not mostly.
To explain that, let me compare what I'm doing to other things people have been doing.
Ballet. What I'm doing is similar to ballet in that it provides something to look at while listening to music,
it moves in time to the music, and it has a structure that's related to the structure of the music. Beyond that, not so much.
A ballet can have its own structure that operates somewhat independently of the structure of the music.
And, of course, many ballets have a narrative structure that's not abstract like music's.
And, although a ballet choreographed to a musical score is less pleasing without the music, most ballets make a good deal of sense when viewed silently,
and while we might not go so far as to say that they "stand on their own,"
it's not true that they "would make little sense without the music," which is more true of my animations.
Abstract animation. I don't think there's much of a comparison here, either.
Abstract animation, even abstract animation with music on its soundtrack, is about what you are seeing,
whereas my animations are about the music.
Music theory. Many viewers report that my animations help them understand the music,
and many music theory/history/appreciation teachers use my animations in their classes,
but are the animations themselves "explaining" the music a way that's similar to how a music theory/appreciation teacher would?
No, I don't think so. For one thing, it's not using words to name, describe, and categorize musical elements.
It's not telling you what's going on, it's not explaining what the music is supposed to mean.
At its most effective, it can help you notice things that you might have noticed on your own if you had more experience as a listener.
Orchestration. This might be the closest analogue to what I'm doing.
A person orchestrating a piece of music is designing "clothes" for the composition to wear.
I was trained as a composer, and to me, what I'm doing feels a lot like orchestration.
I even use some of the same tools: my animations are partly designed in music notation software;
that's where I specify which notes are to be realized by which visual effects
(the specifics of those effects are specified elsewhere).
Like an orchestrator, I'm responding to what's in the music,
and I'm trying to make something that enhances that.
An orchestrator can "make the music work better,"
and I don't think that my animations do that—they don't make the music better—
but I do think that they can help the music work better for some people.
Guidance. One difficulty beginning listeners have is that they don't know what to listen for.
This is one of the main things music appreciation classes are for: to direct the students'
attention to aspects of the music that are important to notice and attend to.
This is definitely something my animations can do, and they are an aspect that I've only started to focus on recently.
When I was making the last animations of the Art of Fugue set,
I was very consciously noticing the flow of my attention through the music, and asking:
does the animation help the listener notice that?
I was, in a way, "choreographing"—not choreographing the animation,
but choreographing the audience's attention—ideally,
helping them attend to the things they'd attend to if the "knew what to look for."
Candy. When I started making graphical scores, I was insulted if people referred to my animations as "eye candy,"
but I no longer feel that way. If they give a listener something pleasing to look at while listening to music,
and don't detract from the listening experience, I think that's fine. If they help the listener appreciate the music,
so much the better.
Because I have a very poor memory and have made several hundred (probably more than a thousand by now) animations,
I can have the experience of encountering my work almost as it would be if I'd never seen it before.
The first time this happened was not with an animation, but with a still image.
My colleague Etienne Abelin presents my work in live-synchronized performances around the world, and for many projects,
he works almost completely independently: I design the animation and then forget about it,
and he goes off and performs it with an orchestra somewhere.
So, I was surfing the web, and I came across a display, a scene in some foreign country,
with an orchestra in the foreground and in the background ... what? ... it seemed somehow familiar, but I couldn't place it ...
and after a few surreal moments, I realized that it was a frame from one of my animations.
I was stunned, both by the fact that I hadn't recognized it, and by the effect of the image itself—it was overwhelming,
magical, magnificent, mysterious. (The effect was short-lived because my innate humility kicked in.)
More often, though, my "see as if for the first time" experience comes when I go back and review videos
that I published several years ago. Sometimes I remember them, but sometimes they catch me completely by surprise:
I don't remember doing them, and I don't remember what happens in them. This happened recently with the
video of Beethoven's opus 131.
I'd completely forgotten that I'd made it, and I had no idea what was coming.
I watched it with awe and delight. If this were an experience I had in a vacuum, I'd write it off as being an example of
"only you appreciate the smell of your own farts," but my reaction was similar to what some viewers have described,
so I do think there's a "there there."
For some people, anyway. During a Q&A following a live performance of my animations,
a woman in the audience asked me what I would say to someone who,
if they knew my animations were to be presented as part of a musical event, would choose not to attend.
I've encountered many people like that, ranging from ones who don't "get" what I'm doing
("is there some correspondence between what I'm hearing and what I'm seeing") to ones who actively resent and dislike it.
Would this change if I were doing a better job? I don't know. I do know that just as it takes experience to appreciate
complex music fully, my more complex animated graphical scores can take some getting used to. Some viewers find anything
more complicated than a simple bar-graph score bewildering. So, it wouldn't surprise me if, in the future, we had
different audiences for different genres of visualization.
| |