Voronoi experiments


(n.b. here is a complete playlist of my videos using the Voronoi tessellation)

Background

How is hearing similar to vision, and how is it different? This is a question I've struggled with.

You can notice an "empty space" visually, but sounds always seem to fill the auditory space—to exist everywhere. (I'm not talking about binaural hearing; yes, we have two ears, and we can tell what direction a sound comes from, but what I'm referring to is the sense in which all sounds exist "in the same space"—that sounds from multiple sources overlap, mix, combine, interact.) By contrast, my early scores were mostly empty space with a few thin bars in them; these were clear, but were a poor psychological match for the "space filling" effect of the music:

Over the years I learned how to make better use of visual space, but I rarely came close to filling the visual space (for a rare exception, see Tallis, Spem in alium).

At some point, I posed the question this way:

If all the visual space in a graphical score were filled, what would it look like?

For example, what would it look like if each point in the visual space were be "owned" by whatever note was closest to it?

The general form of that idea (dividing space into regions based on proximity) was explored by the Russian mathematician Georgy Voronoi, and the Voronoi tessellation seemed like something I should experiment with.


First try (experiments starting 2016may28)

Okay, so here's a bar-graph score of a piece of music:

There's an obvious mismatch between this representation and what's required for a Voronoi tessellation: the score is made up of horizontal lines (vertical height equals pitch and length equals duration in time), but a Voronoi diagram is based on points. So, in my first approximation, I used the points corresponding to the time a note starts (the left end of the line).

Here's what a note-start-position version of the bar-graph score looks like:

And here's what that looks like when the Voronoi tessellation is applied (with dots at the note-start points):

I found the note cells to be indistinct; to remedy that, I faded them to the background color at their edges:

Not bad ... here's what that looks like as an animation. The "now" moment stands out more if the animation is done with a fisheye bulge, but this is at the expense of the structure being as clear.

Initial observations

After trying out the technique with a few pieces ...

  • Bach, Herr Christ, der einge Gottes-Sohn from Das Orgel-Büchlein, BWV 601
  • Bach, Prelude in D major, WTC I, BWV 850
  • Bach, Fugue in D major, WTC I, BWV 850
  • Bach, 6-voice Ricercar (from The Musical Offering)

    ... it became clear that this technique had strong and weak points, and worked better for some pieces than others:

    Pros

  • Dramatic/striking/novel   One thing is for sure: these animation look very different from anything I've done before, and are quite striking—sometimes even pretty. Many people have commented on the "stained glass" look (which seems especially appropriate for organ music, which often gives the feeling of "filling every space with sound").
  • Colors more visible   The spatial resolution of human vision is high for contrasts in brightness, but relatively low for contrasts in color; because of this, having larger shapes makes the colors more distinct. This is especially helpful when my "harmonic coloring" technique is used.
  • Note salience parallels perception   When a musical note is distant in time and pitch from other notes, it is more easily noticed—more salient; conversely, when notes are close together in pitch and time, they are harder to discern individuall. The Voronoi diagram allocation of space nicely parallels this, with the result that important/salient notes are bigger.

    Cons

  • Onset times obscured   The position of the left edge of the Voronoi cells in this display is based on the position of the corresponding note onset and the onset position of the previous note. This means that it can be either very close to the onset position of the corresponding note, or very distant (if the previous note is distant—or for the first note of the piece). This means that the time the edge of a cell meets the "now line" (the center of the display) is only loosely correlated to the time the note sounds, which obscures the rhythm of the music.
  • Offset times not shown by size/position   Offset (release) times of notes are not used in the Voronoi calculation, and are therefore not represented spatially. Cells are highlighted (brightened) when the corresponding note sounds, but the size/shape of a short note is identical to that of a long note.
  • Pitches obscured   This occurs for the same reason, though to a lesser extent, that onset times are obscured.
  • Irregular note sizes   Although more salient notes tend to be larger than notes which less salient, this is not always the case, and notes of equal aural prominence are often of very different size.
  • Silences are not shown   While it's true that music tends to fill the available aural space, there are times in music when the absence of notes is noticeable (for example, when an instrument stops playing suddenly); such "holes" are missing from this display.

    Modifications/additions/enhancements

    Many modifications suggested themselves—some as remedies for shortcomings, some as enhancements.

    Show loudness.   While the area occupied by a note is fixed, the degree to which that area is filled can be varied, a technique I used in this animation of Beethoven, Symphony 5, 1st movement.

    Show rests.   Aaron Andrew Hunt suggested that silences could be shown by adding invisible "rest notes" to the input of the Voronoi calculation. Here's the Bach 6-voice ricercar (same as above) without rests, and with rests. The effect would doubtless be more striking with a better example.

    Show note duration.   To do this, I chopped each note into segments, generated the Voronoi diagram, then joined the segments for each note back together. Here's a passage in a Bach fugue with the Voronoi technique I'd been using (included empty spaces for rests) ...

    ... and here's the video of that. With the notes chopped into segments and rejoined, it looks like this (and here's the video) ...

    The notes look quite irregular in that; part of that is due to the segmenting/joining technique itself (and can't be avoided); part is due to the way shading works (and which could be improved upon if I wanted to take the trouble); here's the unshaded version (in which individual notes are not clear, as before):

    Regularize note shapes.   The shapes of Voronoi cells can be very irregular ...

    ... and I wondered what they'd look like if I limited them so that all of the "radii" (lines from the seed to the vertices where the edges intersect of the edges) were the same length; here's what that's like ...

    ... (and here's the corresponding animation).

    Of course, once you add a knob like that to the software, it's fun to play with it; here's what it looks like if you first limit the length of the radii, then expand them ...

    ... (and here's the corresponding animation for that).

    This suggests lots of possibilities. Incidentally, the radii lines themselves look pretty neat too (video) ...

    Shrink cells to match note decay.   As in my GenAlpha renderer (example). Also as in GenAlpha, move, and morph the shape from one note to the next. And, fade the color so that the onset/brightening is a more vivid event. One thing I noticed when I added these features was that it made the small errors in note timings much more obvious, so when I did a video combining these effects (traversing, morphing, shrinking, fading), I re-did the timing to make it more accurate.

    Reverse colors.   It turns out that the cells can be differentiated pretty well by having their innards dark and their edges colored. This is suggestive of cells, so as a test, I tried it with the Björk song Virus from her Biophilia album (sorry, no video link for this one; I did animations for the app, but I don't have permission to post videos):

    White background.   Softer look (video):

    Animate segments.   Treating the segments of the Voronoi cells separately is a possibility; here (video), the vertices of the segments twist around ...

    ... here (video), the segments assemble, and here (video), they explode.

    Combinations

    Because the Voronoi diagram fills the space completely, the most natural way to combine it with the graphics from other renderers is to draw it first, as a background, and then draw other things on top of it, as I've done with this Chopin prelude (here's the video of that) ...

    Here's one (Chopin nocturne) in which the Voronoi part doesn't fill the entire space (video) ...

    Another approach to the "background" idea is to render everything with a Voronoi diagram, then draw everything with some other renderer (video) ...

    It's also possible to layer one Voronoi rendering on top of another; this could get messy if they're both solid, but by using just the radii on top and the fill underneath, it can work (video) ...

    By limiting the horizontal and/or vertical range of the Voronoi cell, different parts can be further differentiated (video) ...

    More pieces

    Other pieces I've experimented with (not a complete list)

    Harrington, Blue Strider    This turned out remarkably well, and I'm trying to figure out why that is. Harrington wrote "...the changing geometric contexts reflect, somehow, the evolving motivic development, so you see these color shapes symbolizing the competition between musical dramas at a local level," and I would agree with that. Even the debugging displays of this piece looked nice:

    Mozart, Symphony 40, 1st movement    Although this loses a lot in terms of detail, I like the way the contrasts between strings (darker greens) and other instruments (reds, yellows, violet, etc.) show up. However, sometimes the non-string parts are not shown (because they're in unison with the strings); I could remedy that by having the software select among unisons based on, say, duration and/or dynamics (then, I could make the winds louder in the score).

    Tallis, Spem in alium    In this, I'm using the fact that unisons are represented by a single cell as an advantage: to simplify the view of this 40-voice piece. The simple chord structure can be seen easily. To preserve the information about how the individual voices move, those are shown with lines between the cells (drawn as the note sounds, to make the motion more apparent):

    Bach, Brandenburg Concerto No. 2 in F major, first movement    I'm including this as an example of something that did not work very well. The problem was: how to get the proper balance between the solo parts and the orchestral tutti? If I made the tutti visible enough to be clear, it interfered with the solo parts: here, you can see it having both problems at once:

    Stravinsky, The Rite of Spring    This is an amazing piece, and nearly any visualization of it looks fascinating:



    More recently (added 2024nov06) ...

    The above examples are all from my first round of experiments (in 2016).
    Since then, I've used the Voronoi renderer in over 200 videos (playlist).
    In one of the most recent ones, I showed how the music moved between registers
    on the harpsichord by having bands of the Voronoi move at different times (video).