BWV 1007, first movement (experiments)

In the series of recent remakes of the first movement of Beethoven's Fifth Symphony, I experimented with incorporating dynamics extracted from the recording. This was rather crude, both because a single dynamic level was used for each note (regardless of changes of loudness within a note), and because the dynamics for the entire orchestra were appled to all the instruments. To get a more detailed picture of dynamics, I licensed Vito Paternoster's recording of the first movement of Bach's first violoncello suite from Magnatune, and extracted a fine-grained dynamics track from it (using a simple MATLAB script). My hope was to get more of the feeling of bowing into the video.

The first experiment using this new data had a "caterpillar" walking between notes. In this video, the notes were colored using "harmonic coloring" (descirbed here and here), and the color of the caterpillar was a mixture of the color of the current note and the note that was just previously played; the width of the caterpillar corresponded to the loudness.

In the caterpillar animation, the dynamic information was so subtle that I wasn't even sure it was working right, so I replaced the caterpillar with articulated dynamics track with one rhombus per loudness measurement. To highlight the "now" point, I colored the current rhombus red.

The changes in dynamics showed up pretty well in 2011oct17b, but the display seemed a little spare, so I smoothed the dynamics track and increased the scope of the view so that the full score could be seen. To show more detail about the music currently playing, I added a fish-eye effect to magnify the time period in the near past and near future.

It still looked relatively static, and I thought "to make it feel more like bowing, I should show how the bow moves." I didn't know what bowing Vito Paternoster was using, so I watched the video of Mischa Maisky playing the movement and transcribed the bowing into the score (D=up-bow, B=down-bow):

My original idea was to have the bow move the way a 'cellist would, but I found that when the notes move vertically, it's hard enough to just get the bow to the point where the note is, so instead I simply had the bow move up and down. The width of the bow corresponds to the loudness of the note (and can change during a note). Note: the bowing shown in the score is not Maisky's, so it doesn't match the video.

I found that it was hard to follow which note was being bowed, so I put the dynamics track back in, and had it wiggle when a note was playing.

I found that because the ends of the bow faded to nothing, it was sometimes hard to tell that the bow was moving, so I added knobs at the ends.

I asked Vito Paternoster whether he might be able to send me his bowings so that the video could match what he did in the performance, and he agreed. (The bowings shown in the score above are his, and match the videos from here on).

There were some mistakes in that video, so I fixed them and re-did the video in 1080p.

I was unsatisfied with the way it felt to go through a long note, because the pitch track was moving even though the pitch was staying the same. To address this, I tried curving the pitch track so that for short notes, it moved fairly directly toward the next note, but for a long note, it stayed in place for a while before moving.

That seemed better, but I found I was sometimes losing track of the pitches; to remedy this, I added the standard bar-graph score underneath. I also had the "now" point widen to emphasize it.

The pitch-track bending sometimes looked right, but sometimes didn't, because it used the same calculations regardless of the pitch distance. I changed the algorithm to incorporate both pitch and time in the calculation.

2011oct26a

Of course, I'd abandoned the bowing. I decided to try a completely different approach: instead of having the bow as a distracting object, I'd show it with cross-hatching the background.

This was mostly a failure. If you stopped the video and studied it, you could see what was going on, but while it was playing, the cross-hatching was pretty hard to see. I'd originally planned on having the cross-hatching move vertically to emphasize which direction the bow was moving, but that didn't work at all. Also, the addition of cross-hatching made the video so big that it wouldn't play without jerking, and was looking like it would take ten hours to upload (which is why it's not available on YouTube).

As a substitute for the cross-hatching, I tried just adding an arrow(head) on top of the note bar.

The note bar now seemed superfluous, so I removed it, and changed the arrowhead position so that the long edge was always centered on the pitch of the note (I'm not sure I like this). Also, I tweaked the way the note path curved.

I'd noticed how slurs in music notation seemed very intuitive, and had been trying to think of a way to do something like slurs in my animation. I'd been thinking along the lines of adding slurs, but hadn't come up with a way of doing that which was uncluttered and simple. Then, Vito Paternoster suggested that I break the pitch-track itself. Good idea!

With the broken-pitch-track bowing indication, I noticed a bowing I'd gotten wrong, so I fixed that. I also adjusted the timing of the dynamics to be better aligned.

Carl Lumma sent me a link to this (be sure to watch it for a minute or so before deciding you're not interested).

Vito gave me a few more corrections to the bowing, and made some suggestions. This led me to try emphasizing the bowing groups by eliminating the pitch-track on the last note of each bowing group. I also removed the "cursor" and put back the caterpillar.

I was looking at one of my test programs that draws a bunch of random shapes, and the conjunction of an ellipse and a "filament" caught my eye, and it occurred to me that when a filament went through an ellipse, a note could sound. Since I had information about bow direction, I connected the filaments together to show notes that were bowed in a single stroke, and showed the bow direction by the direction the filaments moved. The result isn't very effective, but that's probably because it's just a first experiment; I think the idea (that two things meet to cause a note to sound) has possibilities.

I added a "ellipse ring" primitive (color filled in between two ellipses) to my graphics collection. To try it out, I had the ellipse walk from one note to the next. Not very interesting.

As a first step toward showing multiple tiers of dynamics, I added an "aura" to the "egg" display. The aura effect is nice and soft, but what I'm doing with it in this first trial is quite far from what I had in mind. Lots of stuff to be worked out ...

One of the first "big bang for the buck" graphics effects I ever saw was shaded circles drawn being drawn on a path. Besides looking smooth and sexy, it also has a couple of nice features: it shows order (the most-recently-drawn circle is on top), and even though each circle takes up a lot of space on the screen (and covers up a lot of the circles drawn previously), because each circle is drawn to a different center, partially-covered circles tend to be visible. The combination of a moderate time-warp effect and the regular rhythm of the piece resulted in a nice rhythmic undulation.

2011nov06a

I experimented more with yesterday's effect, and was impressed enough by it to spend some time trying it with another recording: Lara St. John's performance of the Siciliano from Bach's G minor Sonata for unaccompanied violin.

I'm pretty sure that the first time I saw an "animated calligraphy" effect was in the closing sequence of the "Fractured Fairy Tales" segment of The Rocky and Bullwinkle Show.

I was wondering whether I could do a similar thing but have the shape of the calligraphic swirl be determined by the note positions. I licensed Lara St. John's recording of the Preludiofrom Bach's third partita for unaccompanied violin from Magnatune, and started in on it; the first thing was to just connect the note dots with straight lines (of varying width corresponding to the loudness of the recording):

Last night, in a talk about 20th-century innovations in music notation, Charles Boone reminded me about Paul Klee's wonderful Fugue in Red:

I always find Klee's work thought-provoking, and I put this in my notebook to remind me to think about it some more:

After drawing the triangles, I realized that it might be possible to use them in place of circles in 2011nov06a, and that this change might make it more palatable for viewers who were reminded of intestines, worms, condoms, maggots, etc. by the circle version.

It doesn't work as well as the circles (doesn't feel as tangible), but I may be able to do something about that.

In the circles version (2011nov06a) the layers of circles were much clearer than in today's triangle version (2011nov17a). I realized that this was because the distinctive outline of a circle is visible even when the circle is almost completely obscured by another. To achieve this with triangles, I rotated them 180 degrees:

This meant that the "arrow" of the triangle pointed in the wrong direction (ouch), but several things worked better. Not only were the individual triangles more distinct and the colored triangle at the start of each note more clearly part of a triangle, but it gave the center of the triangle path a clear "spine."

2011nov18a

After several days of trying to remember my high school algebra and geometry, I completed the first draft of the "calligraphy" version of Lara St. John's performance of the first movement of the E Major partita. I was thinking that the simple approach I tried first would be too angular, but it turned out smoother than I expected.

2011nov18b

I was curious to see how the calligraphy animation worked in a contrapuntal piece. I had most of what I needed to re-do the first movement of Bach's 3rd Brandenburg concerto, so I tweaked to code enough to do a passable job with that. It's much harder to follow than the solo violin piece, but some things worked pretty well. Since it's polyphonic, I couldn't easily extract the dynamics for each part, so I just used the fat track for the tutti and the thin tracks for the solos.

2011nov23a

I've known about spline curves for years, but had never played with them, so today I added a very simple spline curve renderer to the software. This is just a test to make sure it's working; nothing fancy.

2011nov25a

I started today's experiment with the idea of making a "ribbon" formed from a twisting spline curve, but before I put the twists in (and when there were still lots of random behaviors caused by the interaction between changes in direction and changes in dynamic level), it looked funky in a nice almost hand-drawn way that I liked, so I saved a version at that state to document it.

2011nov25b

The "ribbon" version still looks pretty funky.

2011nov26a

I liked that version enough that I decided to do another piece with it: another fast movement from one of Bach's pieces for unaccompanied violin.

2011nov28c

I wondered what it would be like if the spline curves could interact with time-warping. It turned out that the two effects combined to emphasize the present moment, making the animation seem more alive.

2011dec16a

Yesterday I realized that one way to expand the scope of the "immediate future" would be to combine multiple renderers on the same melodic line, and have them operate with different time windows. Today I combined the copperplate calligraphy, ball track, and spline ribbon:

Since these three renderers were not designed to go together, the combination is fairly ugly.