(n.b. here is a complete playlist of my videos using the Voronoi tessellation)
How is hearing similar to vision, and how is it different? This is a question I've struggled with.
You can notice an "empty space" visually, but sounds always seem to fill the auditory space—to exist everywhere. (I'm not talking about binaural hearing; yes, we have two ears, and we can tell what direction a sound comes from, but what I'm referring to is the sense in which all sounds exist "in the same space"—that sounds from multiple sources overlap, mix, combine, interact.) By contrast, my early scores were mostly empty space with a few thin bars in them; these were clear, but were a poor psychological match for the "space filling" effect of the music:
Over the years I learned how to make better use of visual space, but I rarely came close to filling the visual space (for a rare exception, see Tallis, Spem in alium).
At some point, I posed the question this way:
If all the visual space in a graphical score were filled, what would it look like?
For example, what would it look like if each point in the visual space were be "owned" by whatever note was closest to it?
The general form of that idea (dividing space into regions based on proximity) was explored by the Russian mathematician Georgy Voronoi, and the Voronoi tessellation seemed like something I should experiment with.
Okay, so here's a bar-graph score of a piece of music:
There's an obvious mismatch between this representation and what's required for a Voronoi tessellation: the score is made up of horizontal lines (vertical height equals pitch and length equals duration in time), but a Voronoi diagram is based on points. So, in my first approximation, I used the points corresponding to the time a note starts (the left end of the line).
Here's what a note-start-position version of the bar-graph score looks like:
And here's what that looks like when the Voronoi tessellation is applied (with dots at the note-start points):
I found the note cells to be indistinct; to remedy that, I faded them to the background color at their edges:
Not bad ... here's what that looks like as an animation. The "now" moment stands out more if the animation is done with a fisheye bulge, but this is at the expense of the structure being as clear.
After trying out the technique with a few pieces ...
... it became clear that this technique had strong and weak points, and worked better for some pieces than others:
Many modifications suggested themselves—some as remedies for shortcomings, some as enhancements.
Show loudness. While the area occupied by a note is fixed, the degree to which that area is filled can be varied, a technique I used in this animation of Beethoven, Symphony 5, 1st movement.
Show rests. Aaron Andrew Hunt suggested that silences could be shown by adding invisible "rest notes" to the input of the Voronoi calculation. Here's the Bach 6-voice ricercar (same as above) without rests, and with rests. The effect would doubtless be more striking with a better example.
Show note duration. To do this, I chopped each note into segments, generated the Voronoi diagram, then joined the segments for each note back together. Here's a passage in a Bach fugue
... and here's the video of that. With the notes chopped into segments and rejoined, it looks like this (and here's the video) ...
The notes look quite irregular in that; part of that is due to the segmenting/joining technique itself (and can't be avoided); part is due to the way shading works (and which could be improved upon if I wanted to take the trouble); here's the unshaded version (in which individual notes are not clear, as before):
Regularize note shapes. The shapes of Voronoi cells can be very irregular ...
... and I wondered what they'd look like if I limited them so that all of the "radii" (lines from the seed to the vertices where the edges intersect of the edges) were the same length; here's what that's like ...
... (and here's the corresponding animation).
Of course, once you add a knob like that to the software, it's fun to play with it; here's what it looks like if you first limit the length of the radii, then expand them ...
... (and here's the corresponding animation for that).
This suggests lots of possibilities. Incidentally, the radii lines themselves look pretty neat too (video) ...
Shrink cells to match note decay. As in my GenAlpha renderer (example). Also as in GenAlpha, move, and morph the shape from one note to the next. And, fade the color so that the onset/brightening is a more vivid event. One thing I noticed when I added these features was that it made the small errors in note timings much more obvious, so when I did a video combining these effects (traversing, morphing, shrinking, fading), I re-did the timing to make it more accurate.
Reverse colors. It turns out that the cells can be differentiated pretty well by having their innards dark and their edges colored. This is suggestive of cells, so as a test, I tried it with the Björk song Virus from her Biophilia album (sorry, no video link for this one; I did animations for the app, but I don't have permission to post videos):
White background. Softer look (video):
Animate segments. Treating the segments of the Voronoi cells separately is a possibility; here (video), the vertices of the segments twist around ...
... here (video), the segments assemble, and here (video), they explode.
Because the Voronoi diagram fills the space completely, the most natural way to combine it with the graphics from other renderers is to draw it first, as a background, and then draw other things on top of it, as I've done with this Chopin prelude (here's the video of that) ...
Here's one (Chopin nocturne) in which the Voronoi part doesn't fill the entire space (video) ...
Another approach to the "background" idea is to render everything with a Voronoi diagram, then draw everything with some other renderer (video) ...
It's also possible to layer one Voronoi rendering on top of another; this could get messy if they're both solid, but by using just the radii on top and the fill underneath, it can work (video) ...
By limiting the horizontal and/or vertical range of the Voronoi cell, different parts can be further differentiated (video) ...
Other pieces I've experimented with (not a complete list)
Harrington, Blue Strider This turned out remarkably well, and I'm trying to figure out why that is. Harrington wrote "...the changing geometric contexts reflect, somehow, the evolving motivic development, so you see these color shapes symbolizing the competition between musical dramas at a local level," and I would agree with that. Even the debugging displays of this piece looked nice:
Mozart, Symphony 40, 1st movement Although this loses a lot in terms of detail, I like the way the contrasts between strings (darker greens) and other instruments (reds, yellows, violet, etc.) show up. However, sometimes the non-string parts are not shown (because they're in unison with the strings); I could remedy that by having the software select among unisons based on, say, duration and/or dynamics (then, I could make the winds louder in the score).
Tallis, Spem in alium In this, I'm using the fact that unisons are represented by a single cell as an advantage: to simplify the view of this 40-voice piece. The simple chord structure can be seen easily. To preserve the information about how the individual voices move, those are shown with lines between the cells (drawn as the note sounds, to make the motion more apparent):
Bach, Brandenburg Concerto No. 2 in F major, first movement I'm including this as an example of something that did not work very well. The problem was: how to get the proper balance between the solo parts and the orchestral tutti? If I made the tutti visible enough to be clear, it interfered with the solo parts: here, you can see it having both problems at once:
Stravinsky, The Rite of Spring This is an amazing piece, and nearly any visualization of it looks fascinating:
To try (future)
There are other variations I'm planning to explore, including ...
Note: if you'd like to support this work, please consider becoming a patron.