Conversation between Alex and Stephen (0023)

Conversation between Alex and Stephen (prev top next)

2007mar24

Alex, you asked:

Have you considered rotating the keyboard vertically to the right so the position of the notes on the keyboard corresponds with the bars in the visualisation?

Yes, but I haven't tried it yet. This particular piece has a pretty small range for keyboard music, only about two and a half octaves, so that approach is feasible; for most pieces, the keyboard would have to get pretty small to fit vertically, which would limit the amount it contributed to the scene ... though I could include more of my arms and body, which might be interesting.

When the keyboard and the score are oriented in the same way, it's tempting to try to make the two relate even more closely (as is done in Klavarscribo). There's a problem with that, though: the "now" on the keyboard is at the right edges of the screen, but the "now" in the score is at the middle, so there's a big hunk of future score separating the two. I've been thinking about ways to tie the two together. A simple way would be to use lines, connecting the currently-playing notes with the corresponding keys on the keyboard. Another fairly simple way would be to color the notes on the keyboard to match the notes being played. (I was also thinking about coloring my fingers.) A less-simple way would be to show the effect of the future notes on the hands; having the motion of notes presage (or perhaps even appear to drive) the motions of the hands.

I meant the "experience" of the spacings growing smaller as we go up the fingerboard on string instruments is inverse to what we know about the actual measure of frequency as we go up in pitch.

For me, the knowledge about increasing frequency intervals as you go higher in pitch works the way "symbolic" relationships work: it's a learned thing, and seems arbitrary. That is, I feel that if I'd learned the opposite, I could have just as easily believe that. Also, since it's a learned correspondence, it doesn't work for people who haven't learned it.

It is not my intention to represent roughness and smoothness in the spectrographic content of a musical note, through the structure of geometric entity designed to represent that instrument. I wish to have that quality mapped through the spectrogram itself. This would then be texture mapped onto the geometry of the instrument.

Okay, I think I'm following you: the structure (which is to say, shape of the instrument and the way it moves) wouldn't be related to the timbre, but the "skin" of it would ... right? That sounds cool. (Or hot, depending on the texture.)

So considering that modelling the spectral envelope is most effective for modelling timbre, how can this be applied to different geometric entities? The modelling shown by Jensen is useful for seeing the overall timbrel envelope, however in a dynamic temporal environment, such as the actual performance of a note within a phrase, that spectral information can only be presented as it happens (this isn't strictly true, but to visually present all the spectral information before it occurs introduces a whole host of other issues). Maintaining a history of the spectral energy as to show the full envelope as it evolves may be inconsistent with the discreet experience of hearing the sound itself at any given moment. This warrants further investigation.

I agree, timbre has enough dimensions that trying to represent its progression over time in a comprehensive way is a challenge. Maybe there's a middle ground, though ... I heard a talk last week given by Hynek Hermansky about the significance of a certain range of time interval in speech communication: 50 mS to 500 mS --- that is, 1/20 second to 1/2 second. His point was that what's happening in the spectral shapes (timbres) that carry speech information isn't completely localized—isn't completely instantaneous—that you have to look at what's happening through a certain interval. When you go from sound to image, you need to take this into account, because hearing and vision have different time scales. A sound that lasts 1/30 of a second may be significant, but an image that lasts only 1/30 of a second is almost unnoticeable. So, for a timbral event that's very short to have the appropriate visual significance, it needs to be scaled somehow.

One could argue that there is nothing gained if one does not need to exert some effort in understanding an artwork. But here we can drift into an endless void of philosophical discussion about art... which I would rather like to avoid right now. I have to go and practice.....

... and I should try to write some software ... more later.

Stephen