Home | Site Map | Watch | FAQ | History | Store | Contact

Conversation between Alex and Stephen (prev top next)

2007mar18

Alex,

You wrote

... my requirements for spectrographic representation are different in that I only wish to show the acoustic content of a single instrument. ...

Are you sure? If you're only considering the acoustic content of a single note in isolation, that is, its timbre, you have a lot of freedom! For most single-note instruments, the harmonics fuse into a single sound object, and the fact that they're related by whole-number ratios is not something you have to express, since it's more or less constant (doesn't change much from note to note within an instrument, seldom changes from instrument to instrument). What is important is where in the spectrum the energy lies, that is, spectral envelope. Here's a recent paper with a proposal for how to model timbre.

However, when you have two notes, there's the question of how they sound together (that is, how consonant they are with each other), and that has to do with the relationships between their harmonics: when harmonics are close enough together that they affect the same region in the cochlea (within a single critical band), they interact. If you want to provide a visual explanation for why an octave sounds very different from a minor ninth even though they're about the same size, one way is to show where the harmonics in the two notes are. There are lots of ways to do this. The simplest is just to show a spectrogram:





There, you can see that when the harmonics cross, they interact. The most dramatic ones of these are at about T=36, where the perfect fifth happens, and at the end, where the unison happens. A display that emphasized these "beating" points would correspond well to the sound in this example. But this display is suboptimal in other ways; the most serious problem is that it's very hard to see that there are just two pitches. Once you know what to look for, you can see that there are equally-spaced parallel lines that run the entire width of the screen, but it's not something that jumps out at you, even in this very simple example.



... the issue of acoustic/spatial relationships being skewed by depth perception are less problematic if you are moving around the object in a 3D space. ...

Yes, it's true, when you move an object with respect to the rest of the scene, its visual features group together, and it's easy to see what belongs to it. But in the case of the harmonics on the spiral, that kind of grouping would happen regardless of whether the harmonics were equally spaced. And if you're using something other than the equal spacing to show the harmonic grouping, why bother with the spiral?



... the fact that as you climb in pitch on the finger board on a string instrument your spacings get smaller, is an example of the inverse relationship we experience in the physical realm. ... 

I don't know if I'd use the word "experience" here. I'd agree that what you can measure is contradictory. That is, if you have a frequency counter and you calculate the frequency differences between pitches on a fingerboard, you would find that the semitone spaces got larger as you went up. But I don't think we experience that, any more than we experience the fact that blue light has a higher frequency than red light. You can know, intellectually, that the frequencies are more widely spaced, but it doesn't seem intuitively obvious. In fact, I'd go the other way: I would say that my intuition is that higher intervals are smaller, if anything. Where does such an intuition come from? It's hard to say. Maybe because animals that make high-pitched sounds are smaller? I'm just guessing.



One of my criteria for generating my imagery is that the fundamental aspect of an instrument's sound production is represented through the visual entity, hence in DFR each entity looked like a string ... the most important role of spectrographic content for my needs is figuring out which style of representation will best visually discriminate between the qualities of smooth tones and rough tones, for want of a better description.


This is a place where it's tricky to distinguish between symbolic/arbitrary and structural/inherent associations. For example, bowed string instruments have an indented place in their bodies to make room for a bow, so a shape that is indented like that is suggestive of a string instrument, and could be used associatively with the sound of a string instrument. However, I would call this a symbolic association, since the indentation is not related to the sound; a string instrument without the indentation would sound more or less the same (it would just be harder to play). On the other hand, larger resonators are required to produce lower notes, so the association between physically bigger and lower in pitch is not arbitrary (if you assume that sounds are made by the vibrations of physical objects).

There are lots of differences between symbolic and structural associations. Symbolic associations, being ad hoc and arbitrary, can be invented at will. You can assert "yellow means timbrally rough" and people can learn to associate yellowness with roughness. So they're very flexible. On the other hand, structural associations, being inherent, don't have to be taught (though you might have to learn to become sensitive to them), and I think they have better "staying power." On the other hand, you can't just make up a structural association; you have to find it, and that takes work. If we want a structural association for roughness, we need to figure out what causes roughness, what we sense as roughness, and try to find analogies between physical roughness and timbral roughness.

Why is it tricky to distinguish between symbolic and structural associations? Let's consider your example: the string as the analogue of the sound of a bowed string instrument. Would a person whose never seen a bowed string instrument know from the sound that it came from a string? They could probably guess that the sound of the low notes on a 'cello came from a bigger instrument than the sound of the high notes of a violin. And there is definitely something audible about the change of direction of the bow. So for me, the left-right motion of the DFR instrument entity seems structural, as does its reaction to changes in timbre, but the fact of it being a string seems symbolic --- I recognize it as being like string instruments I've seen, and I "get" the reference.

S.

(prev top next)