Method

How I make animated graphical score videos

I've written a few explanations of how I work (see this page for an index to them); here,
I go into some detail of how this went with Michael Barry's wind ensemble piece Boomba.

Materials

After Michael and I agreed on the details of this project, he sent me an audio recording,
and human-readable and machine-readable versions of the score (PDF and Sibelius files).

Click track

The first step in synchronizing the audio recording with the score was to record a click-track with
one note per beat (or, in some cases, more frequently than that). Since the piece moves at a reasonable clip,
I used a half-speed copy of the recording for this (made with the Amazing Slow Downer software).
Then, I edited the clicks to match the timing more accurately. (These steps were done in Cubase.)

Initial score preparation

Because the music has some pauses and cadenzas which interrupt the regular flow of beats,
the score needed to have a sync track indicating the location of the synchronization points
in the click-track. I added these (and tracks showing the location of measures) to the Sibelius file.

If you're curious about what the click-track is like, here's the score for another piece I'm working on
(a pavan by William Byrd) with the click-track (labeled 'sync') edited to roughly match the recording.
Here's what the animated graphical score looks like at this stage:



The short lines at the top show the location of the clicks in the click-track. (As you can see,
the score is very rough at this point, because I haven't added the ornaments and rolled chords.)

For Boomba, there was little need to edit the score, since most of the ornamentation was
already written out. Here's what it looks like (in Sibelius) at this point in the process:

First review

Once the score was in sync with the recording, I did a first draft of the animation, using a basic
piano-roll style renderer, with diagnostics for sync points, measure lines, and measure numbers:


After watching this a few times, I felt like I was hearing the piece better, but I wasn't able
see what was going on. I thought that bringing the 3+3+2 eighth-note rhythm to the foreground
would clarify things, but I tried that and it it didn't help, so I set that idea aside.

I thought it might help to differentiate the instrumental families, so I assigned colors and
nesting shapes to the groups (listed here back to front):
  • percussion: fuzzy dots (gray)
  • brass: octagon (yellow/green)
  • woodwind: ellipse (blue/green)
  • saxophone: rhombus (reddish)
  • piano/bass: star (violet)
I also knew that I was going to "stop time" (halt the horizontal motion for some notes) in the
passages where the continuous pulsing rhythm paused, so I separated out the "free" (more
improvisational) notes so that I could exempt them from that pause. The result was this draft:


Okay, that's better. Next changes ...
  • Depict the swoopy woodwind lines with a faster-moving continuous-line renderer
  • Patch the joints in the continuous rhythm so that breaks are invisible until they happen
  • Depict the triangle with a suitable icon (at high pitch level)
  • Add temporary diagnostics for"stop time"
  • Depict the shakers with a tambourine-like icon, and snare drum with vertical ellipse aura
... which result in this draft:


From there, I did these ...
  • Change the horizontal warping from linear to cylindrical (to make it feel like time is rolling by)
  • Fade out at the edges
  • Move long sustained notes into the background
  • Explode the final chord
  • Voronoi for tuba fart
  • Clean up flute runs
... which resulted in this draft:


At this point, my main dissatisfaction was that a lot of the foreground material looked the same:
the notes which stood out aurally—which sounded different, or louder, or more expressive—didn't
stand out visually. My plan at this point was to do two things: make the strong, regular rhythmic
elements stand out more, and make the expressive, quirky, irregular elements more distinctive.

To make my view of the irregular things in the score simpler, I segregated these parts:
  • marimba, vibraphone, piano, and string bass notes that weren't essential foreground notes
  • notes of the predominant 3+3+2 eighth-note pattern (and things that work like that)
Here's what those parts look like by themselves (I had to check this to make sure I didn't include foreground stuff) ...


... and here's what the unsorted remainder looks like ...



At this point, I separated what remained into groups according to instruments, rhythms, function, etc.
There were 20 groups. In this video, I show examples of each group (the piece is about 200 seconds long,
so it switches to a different group every 10 seconds or so):


The way I specify the design of the animation is by editing an XML document that tells which
renderer is used with which part in the score. Here's what that document looks like (at this point).

I have to admit: I don't feel like I'm doing this piece justice. There's a lot going on, and I don't have
a clear idea of how to organize it visually so that it makes as much sense as it does aurally. I want to
make the layers distinct, but I also want to show how they interact. To make the layers distinct, I decided
to have them move at slightly different rates, layered slow to fast correponding to back to front.
Here's what that looks like:


That helped, but I still felt like I don't have the chops to do the things for the piece that is deserves.
I did a few things to make some elements stand out more from the background, resulting in this version ...


... which I'm going to consider "final" — for now, at least.

Maybe after learning more and developing better tools, I'll be able to come back and do a better job.

Stephen Malinowski — December 27, 2020