Everybody is an electronic musician. Some of us push only one button in a performance, and some of us push many buttons.
— Vladimir Ussachevsky
There are many ways people share the responsibility of making music: they can collaborate in composing and performing music, a composer can leave performing to other musicians, a group of musicians can hire a conductor, a performer can rely on an instrument maker to build a machine capable of turning musical gestures into sound. A less commonly used way to share musical responsibility is for a machine to be responsible for storing the notes of the score and playing back those notes in the proper order, leaving a human performer with only the responsibilities of the conductor — rhythm, tempo, dynamics, expression, interpretation. With the advent of computers and computer music, development of such an approach (which, when implemented with computers, is sometimes called the conductor program) has taken off, with several people experimenting with its possibilities. In this article, I describe some of this work, including my own implementation of the conductor program, Tapper (which is available as freeware). My goal is for the article to provide an introduction to the idea of the conductor program and for Tapper to provide an introduction to the experience of doing computer-mediated musical performance. Neither is intended to be comprehensive, but both are works in progress, and I'm open to suggestions for simplifications, corrections and additions.
First let's go over some of the concepts. What is going on when a performer is using the conductor program? How is using the conductor program different from other things musicians commonly do? How is one implementation of the conductor program different from another? There are a lots of ways to think about these questions. You can ask: What kinds of choices are musicians making? What roles do the computer and human performers play? What aspects of musical expression are performers responsible for? Where does the acoustic energy come from? Where does the performance energy come from? How tightly is the performance coupled to the performer's gestures? What is the relationship between pitch choice, pitch memory, and pitch responsibility? Is a person using the conductor program really performing? ...
Considered abstractly, the responsibility of a musician is to make (and act upon) three types of choice about the allocation of acoustic energy
These three types of choice are seldom considered separately, and they can interact in lots of ways:
All musicians make choices about time, frequency and dynamics, but musicians in different roles are responsible for different kinds of choices; these fall somewhat loosely into four groups:
Which choices are made by musicians in which roles? This is summarized in the following table.
For each group, a given type of choice is colored according to when whether the musician in that role is responsible for that decision always, usually, sometimes, rarely , or never:
As you can see, choices made by the user of the conductor program fall somewhere in between those made by a conductor and those made by a performer. A live performer can make real-time choices about pitches; the user of the conductor program cannot, and is in that way more like a conductor. A conductor cannot control the fine details of dynamics, rhythmic microstructure and articulation in real time; the user of the conductor program can, and is in that way more like a live performer. Of course, the extent of the effect of these choices varies from one implementation of the conductor program to another. For example, if the conductor program is controlling a pipe organ synthesizer, the performer will not have any control of expressive timbre. Or, if the implementation of the conductor program is only tracking the user's beats, then the user will have little or no control over the music's rhythmic microstructure.
In addition to the responsibility for making musical choices, there's responsibility for (or, at least, the labor of) doing the work to make the music sounds. All musical performers make use of "sound reinforcement technology" in one form or another (if only in that their instrument has a resonator), but there's always the question: who's doing the work of creating the sound, who is supplying the acoustic energy? A singer, or the player of any non-amplified instrument is doing all the work (and, in addition to supplying the acoustic energy, brass players and singers are also responsible for supplying the primary vibrating object — lips and larynx — singers also supply a primary resonator). An organist isn't doing any of the work of creating the sound (except perhaps for the sound of feet hitting the pedals, or the sound of keywork on a tracker organ), nor are conductors.
Apart from creating the sound energy, there's the energy required to control it. An organist may be creating none of the acoustic energy, but the physical demands of playing the organ (requiring activity of all four limbs) are certainly as great as those of any other performer. A musical performer is, in some sense, an athlete, and a conductor is perhaps the most athletic of musical performers — there's no question that a conductor is doing physical work. We might call this performance energy — energy required to control and modulate the sounds.
Like organists or conductors, users of the conductor program are supplying none of the acoustic energy (at least, not in any modern implementations), but they are supplying a certain amount of performance energy. Would it be advantageous for them to supply more? If you've done the work, have you performed the piece? If you haven't done any work, have you not performed the piece? I don't think there's a single answer to this ...
For any choice a musician makes, there's the question: how tightly is the performer's choice coupled to the sound that results? A singer's interpretive ideas can expressed immediately as sound, with a minimum of intermediate steps (just neurons and muscles); a singer is tightly coupled to the sound. A pianist's ideas take longer to effect the sound, because motions of the pianist's fingers must move the keys of the piano, which move the action, which moves the hammers, which move the strings. A conductor's interpretive ideas must additionally go through another performer, and a composer's ideas ... well, you get the idea. One way to quantify the tightness of the coupling is by the minimum time delay that's introduced; these are something like this (just my back-of-the-envelope approximations for now):
I've put in two extreme examples of the conductor program: my Tapper and Max Mathews' Radio Baton. Tapper can have a minimum coupling time that's faster than a piano because of two factors: a synthetic piano can respond faster than a real piano, and Tapper doesn't require the performer to move her hands as far. The coupling delay I've given for a composer is somewhat arbitrary, based only on personal experience (it takes a couple of minutes to make a truly compositional change to a piece, write it down, get the parts in front of the musicians, and have them play it). There are, of course, contexts in which a composer is working closely with the performers, and can say "make that an e-flat" and have a coupling delay that's only a few seconds ... but it's hard to say that they're composing, exactly ...
You can see that, in terms of coupling delay, using the conductor program is similar to other kinds of music performance.
All musicians need to know about pitch, but musicians in different roles have different areas of pitch-related responsibility. A composer typically decides what the pitches (at least, some of the pitches) will be. A performer needs to know what the pitches are, either by sight-reading during the performance or by memory. Does a conductor need to know what all the pitches are? The answer to that depends on the breadth of the conductor's role. If the conductor is merely beating time to keep the performers, a detailed knowledge of the score is not necessary, but to the extent that the conductor is responsible for interpreting the music, she should be familiar with the score; after all, how can you know what the notes mean if you don't know what they are?
Beyond knowing what the pitches are, there's the matter of keeping track of which ones are about to occur and causing the corresponding notes to come into being. Learning how to do this is a big part of what musicians work on when they are developing technique and practicing — and with good reason: if you can't produce the right note at the right time, the other estimable features of your performance will be hard to appreciate.
A conductor or the user of the conductor program is typically not responsible for causing pitches to come into being. There are a lot of ways to look at this. If there is virtue in being responsible for something, then conductors are less virtuous than other performers, but if effort spent being responsible (and learning how to be responsible) for producing notes of the correct pitches is unnecessary (or better spent on some other aspect of musical performance), then conductors are spending their time better, which is perhaps also a kind of virtue. If you're responsible for producing the notes, then you are certain to know what the pitches are, likely to know the piece better, and thus likely to be in a better position to understand it and perform it expressively. This is not guaranteed, however (as audiences who've listened to renditions performed by memory that were nonetheless dull will attest).
Related to the skill required to remember a sequence of pitches and render them as notes is the ability to subtly modulate the dynamic level, pitch, timbre, and articulation of those notes. This kind of practice is intimately tied to musical interpretation; as a performer is learning a piece, he will be having ideas about how the notes should sound, and will be developing and refining the techniques that allow them to sound that way. To the extent that an implementation of the conductor program allows the user to control these aspects of the performance, the expressive technical side of the conductor program user's practice will be similar to that of a "normal" performer.
Next, let's examine how these considerations come into play in various implementations of the conductor program (and its precursors) ...
Machines capable of controlling the playback of music date back to the middle ages, when pinned barrel mechanisms driven by pre-wound clockwork were used to control pipe organs (and later, harmoniums and the plucked-comb instruments we now think of as "music boxes"). One might dismiss such devices as being outside the realm of “artificially assisted performance,” but I think that’s a mistake: the decision that "now, there will be music" is a musical decision; causing musical sounds to come into existence by force of will is inherently gratifying.
The 1870s saw the introduction of hand-cranked, paper-roll-controlled "organettes." These offered the player control of tempo and, in some cases, volume. The device shown above is a pin-barrel organette.
The player piano took the idea of expressive control of a mechanically-reproduced performance much further than the music box. It's been long enough since player pianos were replaced by phonographs that few people remember the details of how a player piano was used. It wasn't just a matter of pumping the pedals (though this was demanding enough to give you the feeling that you'd "done the work") — there were also buttons for applying accents and levers for controlling tempo and dynamics. These are described in the 1925 "On Playing the 'Pianola'", by Reginald Reynolds, and demonstrated by Julian Dyer here. Some rolls had indications of when an accent should be applied (in addition to lyrics for sing-along). These controls worked pneumatically, so the coupling wasn't as tight as it would be if you were playing the piano directly, but it was still tight enough that you felt you were controlling the instrument, and responsible for what it was doing.
The expressive capabilities of the music box and player piano were not extended until more than a half-century later, when Christopher Strangio invented and patented his "COMPUTER-AIDED MUSICAL APPARATUS AND METHOD" (U.S. Patent 4,022,097), which described a method of electronically storing data corresponding to notes of a musical composition so that a human performer could subsequently trigger them and thus cause them to sound. His implementation (described in detail in his own words here) was hand-built from discrete logic (remember, no PCs yet!) ...
... using a custom-modified reel-to-reel tape recorder to store a digital representation of the notes. The keys of a 1960's electronic organ were the input device (five keys for each hand), and the organ's oscillators provided the output sound. Two separate paths of storage, control and playback were provided, which allowed the score to be divided into to two parts, each of which could be independently controlled by the performer.
In Strangio's implementation, individual key-presses on the organ keyboard were translated with a very short delay (less than 30 milliseconds) into individual notes, thus the coupling was very tight.
It's commonly believed that Max Mathews invented everything about electronic music, single-handedly, in the 1950s. While this is not strictly true, it's not too much of an overstatement to say that he has been everywhere (Bell Labs, IRCAM, CCRMA...) and done a little bit of everything (music synthesis, speech synthesis, human learning, acoustics, robotics, drawing, typography, conducting ...) related to electronic music. Mathew's work on the conductor program began in response to Pierre Boulez's request for a method for controlling the speed of playback of tape recordings. Mathews tried several kinds of interfaces, settling finally on a radio-frequency-based design (the Radio Drum and Radio Baton) first developed by Bob Boie at Bell Labs (a version of which Mathews describes in U.S. patent 4,980,519).
In Mathews' implementations,
the absolute position of the batons could be sensed in a 3D space extending a few feet above the receiver (the rectangular object in these two photos). The positions of the batons could be interpreted in two different ways: their absolute position could be mapped to various parameters, and information could be derived from their relative motion. So, for example, close and far positions could correspond to loud and soft, while left and right would correspond to staccato and legato, with up and down motion indicating the beats. Since there were two batons, each one could control a separate instrument or group of instruments, for example, the string instruments and wind instruments of an orchestra.Where Strangio followed a "performance" paradigm (one keypress equals one chord), Mathews' implementation was really a "conductor" program; the performer was not responsible for individual notes per se; instead, there was a "conductor track," separate from the notes of the pitch, which specified the positions of the beats the conductor was expected to indicate with gestures. Notes which fell between these beats would sound according to the tempo established by the preceding beats. Thus, the coupling was fairly loose — the performer's intention to change tempo had to wait until the next beat in the conductor track.
Casio re-invented the conductor program idea and
licensed its use from Strangio, including it in such devices as the PT-1 (which allowed
you to record a piece and then play it back by pressing either of the two blue
"one button play" buttons at the right). Because it was monophonic (with
instruments that make early cell phone ring tones sound good), had a limited
melodic range, and had a memory of only 100 notes, it was hard to take the PT-1
seriously as a musical instrument, and it and the other Casio devices
incorporating the one button play feature never made much of an impact.
Still, I was intrigued when I saw one; I programmed the first 100 notes of Flight of the Bumblebee into it then played it back as
fast as I could wiggle two fingers ...
Morton Subotnick began using computers for music during his
1985 residency at MIT. There, using David Levitt’s software program Hookup (and with the assistance of
Miller Puckette), he began developing algorithms that could follow the rhythmic gestures
of a conductor. He continued this work the following year at Cal Arts, where Mark Coniglio joined
him, helping first with programming (writing Macintosh versions of his Hookup algorithms) and later, with
software design. The result of
their collaboration eventually became the Mac program Interactor. In Subotnick’s Hunger (which premiered at the 1987 Los Angeles Festival), the conductor program
participated in control of both the musical composition itself and audio-visual
effects synchronized with the music: camera zooms, lighting, equalization
adjustments, etc. In And the Butterflies Begin to Sing
(premiered the Santa Fe Chamber Music Festival in 1988), the computer’s
interaction with the performers was extended to include modifications to pitches
in response to the details of the pianist’s performance). Subotnick’s went the furthest with the conductor program idea
with a piece for orchestra and computer sound called A Desert Flowers (premiered by the Cleveland Chamber
Orchestra in 1989). In it, the conductor uses a MIDI baton built by Pat Downs called the Air Drum. It measured movement in several
directions; there were six movements of the wrist it could recognize. Interactor knew the overall metric
organization of the piece as well as specific notes in the piano and percussion
parts and therefore could, like an orchestral player, base its timing on both
the conductor’s beats and the notes it was seeing in the MIDI from the piano and percussion players. Andrew Schloss started using the Radio Drum at IRCAM,
and has subsequently developed software to use with it and performed with it a
lot. His use of the Radio Drum extends beyond using it to control the
expression of existing compositions into the realm of improvisation. For
example, a gesture with a baton could control the release of a certain set of
pitches In the late 1980s, when I was working on the performance
editor for the Music
Animation Machine software, I was frustrated with the amount
of time it took to edit a passage to make it more expressive. Some pieces
were easy enough that a recording of my own performance needed only minor
touch-ups, but others (especially, pieces for large numbers of instruments or
ones with fast passagework) needed to be recorded a part at a time or at a
substantially reduced tempo; this resulted in passages in which the notes were
right but the fine structure of the dynamics, timing and articulation
was not. Since the editor operated on one attribute (time, pitch, dynamic
level) of one note at a time, this editing was very laborious. It occurred
to me that while I couldn't necessarily play a passage perfectly, I could
possibly play the rhythm more expressively, or do the accents more expressively
— and that since the computer already knew the pitches, I could add an
editing mode in which a single attribute, that I would perform in real-time,
would be applied to the notes already in the score. I started
experimenting with this by writing the first version of Tapper, a very
simple DOS program with no user interface to speak of. It was incredibly
fun to play! — and since I was interested in patents at the time, I did a patent search for related
ideas, which led me to Christopher Strangio's work. We
corresponded, and he became one of the first (and remains one of the most
enthusiastic) Tapper users. A few years later, I incorporated the Tapper
functionality into the Music Animation Machine software. Functionally, the
current version is the M.A.M.
version (minus the rest of the M.A.M.), with modernized GUI. In it, the
score is presented in bar-graph form, with vertical lines indicating the
positions of the beginnings of notes (and thus serving as a rudimentary type of
rhythmic notation): In Tapper, the gestures that control playback are single
keystrokes on either a MIDI keyboard or the ASCII (typewriter) keyboard on the
computer. Notes are grouped into alternating sets of play groups (which
are explicit and indicated in the display) and release groups
(which are implicit and not shown). Play groups are groups of note starts (MIDI
NOTE ON events) that happen at the roughly same time, for example, notes in a
chord that is played on a certain beat. Release groups are all the notes ends (MIDI NOTE OFF
events) that happen between one play group and the following play group. When you press a
key on the MIDI keyboard, Tapper plays any note ends in the upcoming release
group (there may be none) and all the note starts in the next play group. When
you release a key and there are no more keys pressed, Tapper ends all the notes
in the next release group. The dynamic level of the notes being played
depends on both the score and the velocity of the MIDI keystroke
keypress that triggers them: for each group of notes started by a given
keypress, the loudest note in that group will be played at the level dictated by
the velocity of the keypress; the other notes in that group will have dynamic
levels that maintain their dynamic proportion to the loudest note.
If you use the ASCII (computer) keyboard
instead of a MIDI keyboard, only the press actions happen (you have no explicit
control of release groups except those between play groups or at the end of the
piece), and the dynamic levels are those in the score. Notes and chords that come out of Tapper correspond
one-to-one to MIDI events produced by the performer's actions and happen with
minimal delay; the coupling of Tapper is therefore very tight. This is a
sword that cuts both ways: it offers the performer good control of rhythmic, dynamic and articulatory
nuance, but it requires that the performer know the rhythmic structure of
the piece perfectly (or be able to infer it from the display of the score); a
rhythmic mistake often results in a musical catastrophe.
Here are
two demonstration movies that show the kind of control Tapper gives the performer. First, a passage from Brahms' Capriccio, opus 76 #2,
played on Tapper without much expression:
Then, the same passage played with
expression:
The full Tapper performance of the Brahms Capriccio can be heard
here;
there is also a
YouTube playlist
with all my Tapper performances. The first public Tapper performance was by Liza Dalby in
October, 1994, at an all-Brahms concert. She played the G-minor
Rhapsody, Opus 79, #2 (there were no catastrophes). Some thoughts on the possible future development of Tapper were stimulated by an email from
David Christian. Mathematician and pianist Harvey
Friedman independently invented the conductor program in the late 1980s, and
released his Macintosh implementation of it, Instant Pleasure, under license from
Strangio, in 1992. Herewith, a brief tour of the user interface of
the software: Clicking on the Instant Pleasure icon takes you to the copyright screen, and thence to the file selection screen .
Instant Pleasure supported both standard MIDI files and
files in its own proprietary format. When files in the proprietary format
were used, some extended features of the software were available; for example
chords could be controlled several ways: Controlled meant
that the performer had complete control of the chords, and had to be responsible
for triggering every note in every chord. Divided was similar to that, except that the
chords and melodies had to be triggered by the hand to which they were assigned
in the score (allowing the player to do a rubato performance). Automatic meant that the chords were played
automatically, in sync with the performer (so, for example, if the performer
stopped, the chords would stop too), and Independent meant that the performer was only
responsible for the melody, and the chords would be played independently (like
an accompaniment, not necessarily in sync with the performer if the performer
deviated too far from the correct performance). The control of legato could work two ways: Controlled meant
that the articulation was completely controlled by the performer; Automatic meant that the articulation from the
score would be used (that is, the performer would control the times the notes
started, but the software would control when they ended). The GUI (ah, Macintosh ...) could show either the rhythm
...
or the lyrics. Instant Pleasure supported touch sensitive MIDI
keyboards, which meant that the user could control timing, phrasing
and dynamics. The controllable "thresholds" were ... Volume (this
prevented keys which you brushed against accidentally from causing notes to be
triggered), Time (notes played within this
time window would be considered "part of the same trigger" and wouldn't causes
an extra note to play), and Chord (same as
Time, except for chords). Instant Pleasure was briefly marketed commercially, and
was reviewed in some music journals. Interestingly, various people
have used it in psychological studies (if you're curious about this, Google for
variations on these search terms: "instant pleasure" "software program" music
study). Around 1990, Friedman licensed his work to Kawai, who included it
(still under the name "Instant Pleasure") in their GMega synthesizer (shown here).
Later, similar functionality was also included in some of their pianos, under the name Concert Magic.
In these implementations, the scores were limited to those supplied by Kawai, stored permanently in ROM (read-only memory).
Thomas Zimmerman and Samuel Wantman invented and
patented a system for applying performance gestures to a pre-stored score.
Zimmerman and Wantman dug into many of the juicy issues, and their U.S. patent
5,488,196 is a great read. Let me touch on a few
of the high points. An important advance of this work beyond Strangio's is
that of making a distinction between two aspects of the performer's
control: finger control (the type of control
typified by wiggling your fingers, characterized by speed of change and precise
timing) and energy control (the type of control
typified by gestures of the arms, or by breathing, characterized by slower,
continuous change). In the implementation of this that I saw a video
of Zimmerman using, the interface was in the shape of a violin; the finger
control was composed of two push-buttons on the fingerboard, and the energy
control was a bow, able to sense pressure, position, and velocity. The place where Zimmerman and Wantman did what I
consider the nuts and bolts of the conceptual work was in figuring out exactly
how to use these two types of data. Their approach goes like
this: look at the two data strings in combination, and calculate a gesture output and an expression output. The gesture
output corresponds roughly to "when do the notes happen" and the expression
output describes the dynamic changes (that are free to be asynchronous to the
gestures). For example, a change of finger would typically result in a
gesture output, but so could a change of bow direction. Then, convert the
gesture output into note timings, and send them plus the expression data to the
synthesizer. A second important piece of work by Z&W was to
develop the method by which the gesture output gets converted into note
timings. In Mathews' implementation, this task was simplified by the fact
that there was a conductor track for the software to follow, and the conductor's
gestures were expected to be well-defined and not too closely space. But
if the performer's gestures can cover the gamut from quick finger wiggles to
large, less distinct motions of the player's limbs, the task of mapping gestures
to notes is harder. Z&W's design not only addresses this, but also
allows for variability in the performer's level of expertise, rubato, multiple
performers, synchronizing an accompaniment, etc. Here's a block diagram
from the patent: One distinctive feature of this implementation is that
allowed a degree of expressive control not found in any other implementation: if
the score contained a trill, the performer could trill as fast and for as long
as desired, meaning that there wasn't a one-to-one correspondence between notes
in the score and notes that were produced. In this sense, the
implementation allowed a small degree of freedom that might be called
improvisatory. It was with a mixture of pleasure and dismay that I
learned that the patent covers both the use of this technology for music
"re-performance" (the same application as Strangio, Mathews, et al.) and for
editing (the idea which was the source of my interest in the invention) —
pleased because it meant that the idea is "out there" in the world; dismayed
because I'd been planning to use it myself. Ah, well — that's patents
for you. Teresa Anne Marrin-Nakra's work focuses on the
activities of a conductor, and her implementations of the conductor program
follow the conducting paradigm the most closely. In her 1996 Master's thesis, Marrin-Nakra describes the
Digital Baton
(a device designed and implemented by her, Joseph
Paradiso, Tod Machover, Christopher Verplaestse and Margaret Orth), a
device which records conductors' gestures (see U.S. Patent 5,875,257)
for details). The first version of this device contained accelerometers to
detect the orientation, position and motion of the device, plus five finger
pressure sensors. A second version added optical position sensing
(infrared light emitted by the baton was received by a detector positioned
a few meters away from the performer). The software that used the
information from these sensors was fairly simple, translating measurements (and
triggers and threshold crossings based on those measurements) directly into
control signals used in interactive musical compositions; gestures per se were
not extracted from the sensor data. In the concluding sections of her master's thesis,
Marrin-Nakra discusses possible directions for future research, many of
which she pursued subsequently. Extending the capabilities of the
Digital Baton was her Conductor's Jacket, a device that measured a
conductor's muscle tension, breathing, heart rate, skin conductance, and
temperature. The analysis of the data collected from the jacket was
followed by the development of a system which could extract expressive features
(e.g. beats, tempi, pitch choices, accents) from the real-time output of the
jacket. This work was the subject of her PhD thesis. These gestural interfaces have been used to control
various types of performance, including the conductor program (seeYou're the Conductor: A Realistic
Interactive Conducting System for Children and YoureTheConductor.com for a description of one
installation) and other, more improvisatory compositions. Marrin-Nakra went on to found Immersion Music, a
nonprofit organization in the Boston area, to develop and present this kind of
thing. James Plank's piano-oriented implementation of
the conductor program, MusPlay (described in
this article) In 2003, James Fallgatter assembled a team (including
James Plank and others) called Aqwertian Music to
develop and market a QWERTY keyboard-based version of the conductor program
based on Plank's work. This implementation, called Music At Your Fingertips, relies on the performer's
ability to touch-type; its graphical user interface presents a piano-roll style
display in which the notes are assigned letters, which are typed to play the
music. A cursor moves through the bars as they are being played, and waits
when the program is waiting for you to press the next key; therefore, you can
infer the rhythm of the piece by how long it takes the cursor to move through
the note; if you're playing it at the correct tempo, the cursor will move
smoothly. Here is the display for the first few measures of Beethoven's
Für Elise: Plank's ideas (as used in his MusPlay) for how to
convert keypresses to notes when the performer doesn't get it exactly right have
been incorporated into Aqwertian's software, and thus the coupling spans a range
of tight to moderately loose.
photo by brian morri In the implementation of the conductor program conceived by pianist, engineer and professor Elaine
Chew, the performer controls the tempo and dynamic level of the music by
driving through a virtual environment (here, with Chew at the wheel): This implementation was developed at USC's Integrated Media Systems Center (IMSC) in collaboration with professor Alexandre François
(Chew's husband) and students Jie Liu and Aaron Yang
as part of the Expression Synthesis Project (ESP), and is described in more
detail in this article from the Proceedings of the
2005 International Conference on New Interfaces for Musical
Expression. François was the designer of IMSC's Software Architecture for
Immersipresence (SAI), which
formed the basis for ESP's interactive software, which Liu and Yang helped
realize.
There are several notable innovations in this system. The interface takes
advantage of the user's experience in driving a car in several
ways. First, the function of the accelerator (gas) and brake pedals is
familiar. Second, the structure of the musical composition being performed
is reflected in the geometry of the road. So, for example, a passage where
the music slowed down might correspond to a stretch of road with sharper curves;
the performer wouldn't be forced to slow down, but would be
encouraged to. In this way, the terrain is a kind of graphical
notation for tempo. There's a third, more subtle concept involved here:
tempo in music is related (in a way which musicians understand intuitively but
which has not been well described theoretically) with physical motion,
especially motions involving the whole body, such as gait and dance. When
musicians make tempo changes, the rates of change are not arbitrary;
experimenters' first attempts to create accelerandos and ritardandos
algorithmically resulted in tempo changes that did not sound musical or
natural. So, by explicitly mapping tempo to physical motion, Chew's
implementation takes advantage of the performer's experience of moving
physically in the world. Another innovation in Chew's implementation is that tempo and dynamic
level are controlled by the speed and acceleration of the virtual car. This works
because changes in dynamic level are well-enough correlated with changes in
tempo that one can be used to estimate the other. Batuhan Bozkurt has implemented
iOS-based
and
web-based versions of his Touch Pianistca. 1985 Casio
1987 Subotnick / Coniglio
ca. 1988 Schloss
1988 Malinowski
1989 Friedman
1990 Kawai
1993 Zimmerman/Wantman
1996 Marrin-Nakra
1999 Plank
2003 Fallgatter
2004 Chew
2015 Bozkurt
Aaron Andrew Hunt took Malinowski's design as a starting point for his full-featured, professional-grade MIDITapper.
There are almost as many types of interface to the conductor program as there are implementations. Strangio's version used Push-Buttons, as did Casio's (though Strangio's controlled two independent streams of music and Casio's only controlled one). Max Mathews' Radio Drum and Radio Baton measure position in space, as does Teresa Marrin-Nakra's; hers also measured orientation; she also developed a Jacket which measures many aspects of the performer's position and muscle tension. Plank, Friedman and I all developed versions using a MIDI Piano Keyboard, and Fallgatter and I also developed versions using the ASCII Typewriter Keyboard. My Lizard Trampoline explored the feel of a Flexible Surface (and I'm hoping that somebody will develop a version using an interface like this one). Zimmerman and Wantman developed a Violin-Style interface which generated very different kinds of control from the fingers and the bow. A while back, there was a device called the Video Harp which showed promise (and I believe they did a conductor-program-like thing with it, but I've yet to dig up the details), and there have been various approaches to Sensing Hand and Body Position, including computer image tracking, radio, and capacitance. Just as a violin is a very different instrument than a piano, each implementation of the conductor program is its own thing. A few important differences between implementations are summarized in this table:
implementation |
coupling |
note |
tempo |
dynamics |
articulation |
pedal |
tracks |
interface |
improv |
score |
music box |
loose |
no |
yes |
no |
no |
n/a |
1 |
crank |
no |
graph |
player piano |
loose |
no |
yes |
yes |
no |
yes |
1 |
pump, buttons and lever(s) |
no |
graph |
Strangio |
tight |
yes |
yes |
no |
yes? |
n/a |
2 |
buttons |
no |
no |
Mathews |
loose |
no |
yes |
yes |
no |
n/a |
1 |
baton(s) |
no |
no |
Casio |
tight |
yes |
yes |
no |
? |
n/a |
1 |
buttons |
no |
no |
Schloss |
? |
? |
yes |
? |
? |
n/a |
1 |
? |
yes |
no |
Malinowski |
medium |
no |
yes |
no |
no |
no |
1 |
crank |
no |
graph |
Malinowski |
tight |
yes |
yes |
yes |
yes |
yes |
1 or 2 |
MIDI (keyboard) |
no |
graph |
Friedman |
tight |
yes |
yes |
yes |
yes |
? |
1? |
MIDI (keyboard) |
no |
symbol |
Zimmerman / Wantman |
variable |
yes |
yes |
yes |
yes |
n/a |
1 |
violin-like |
trills |
no |
Marrin-Nakra |
loose |
no |
yes |
yes |
no |
n/a |
1 |
baton or jacket |
no |
no |
Plank |
medium |
yes |
yes |
yes |
yes |
yes? |
multi? |
MIDI (keyboard) |
trills |
graph |
Fallgatter |
tight |
yes |
yes |
no |
yes |
yes |
1 |
ASCII keyboard or MIDI (keyboard) |
no |
graph/ |
Chew |
loose |
no |
yes* |
yes* |
no |
no :-) |
1 |
steering, gas, brake |
no |
road |
Bozkurt |
tight |
yes |
yes |
yes |
yes/no |
no |
1 |
touch/ASCII keyboard |
no |
graph |
Hunt |
tight |
yes |
yes |
yes |
yes |
yes |
1 |
MIDI (keyboard) or ASCII keyboard |
no |
graph |
*In Chew's implementation, tempo and dynamics are controlled in tandem (not independently).
The only feature that all implementations share is that they allow you to control the tempo of the music.
"Okay, so it's an interesting idea. But what's it good for?" A fair question, and one for which I have only a partial answer.
Personally, the main way I've used Tapper is the same way I use a piano: to play music for my own enjoyment. I can play the piano pretty well without computer assistance, and Tapper has many limitations, so I tend to use it for pieces that run afoul of my limitations more than they do Tapper's. At the top of the list are pieces that are not written for the piano. I've gotten a great amount of satisfaction playing movements from Bach's Brandenburg Concerti and organ music. For pure thrill, the two pieces tied for first place are the first movement of Mozart's sonata for two pianos (K. 448) and Sousa's The Stars and Stripes Forever.
Then, there are pieces which are written for piano, but which present technical challenges that I'd rather avoid. I played Grieg's Butterfly before I ever used Tapper, and thought I did a pretty decent job with it, but with Tapper, my interpretation changed completely — especially the rubato. When I play it "for real" now, it is much closer to how I play it with Tapper, but never quite as fluid. I could play some of Chopin's etudes, but they are a lot of work, and when I want to play them at the tempo I want (rather than what I can manage), out comes Tapper. Occasionally, a piano piece presents a combination of articulatory and technical problems, such that I can get the notes right, or the articulation, but not always both. One of these is Brahms' opus 76 #2 Capriccio which, with Tapper, I can play with the right kind of playful delicacy (as shown in the movie above).
Harvey Friedman named his implementation of the conductor program Instant Pleasure. I wouldn't say that my pleasure with Tapper has been instant — I still have to practice pieces to play them well with Tapper. But it is certainly pleasurable. How does it compare to playing music on the piano "for real"? All things being equal, and with the current implementation of Tapper, I'd prefer to play a piece on the piano. But all things are seldom equal, and there's a wide range of overlap between the amount of pleasure I get from playing the piano and from playing Tapper. At its best, playing a piece on Tapper is completely engaging, thrilling, expressive — everything you'd want, and is certainly preferable to playing something badly on the piano. However, a piano sound coming out of a loudspeaker is not the same as a piano sound coming out of a piano, tapping keys on a MIDI piano keyboard is not as gratifying as playing a real piano, and Tapper doesn't give you as much control as a real piano does, so with the present implementation, it's at a disadvantage.
Besides pleasure and gaining new insights into pieces, there is another use I have for Tapper: when I want to make renditions of pieces to use as a basis for music visualization, I sometimes use a combination of Tapper performing and manual editing. This was how I made the rendition of Bach's Toccata and Fugue in D-minor that's on the Second Demonstration Reel: I started with a version of the score that had the trills removed, which I performed with Tapper; then I added the trills and cleaned up a few places where I'd slipped up. You can listen to (and watch) the result here. If the conductor program were incorporated more fully into a performance editing tool, this approach could be extended by alternating between "by-hand" editing (e.g. changing a single note timing/dynamic/articulation by dragging it with a mouse) and "re-perform" editing in which some attributes of a new performance through the conductor program would be applied to the pitches in the score (this has been done to some extent in the "NTEMPO" feature of Virtuoso Works' NOTION software). Another piece I used the conductor program with is Debussy's Clair de lune (listen/watch here ); this piece fit the Tapper paradigm well, so there was very little editing after the fact.
Other people have used the conductor program for pleasure in the way I have. Some of these people have enough technical skill on "real" instruments that the conductor program is a means to extend their musical experience into areas that ability or practice time would otherwise keep them from entering. For some, Tapper is their preferred way of playing music. Age is not a critical factor — James Plank's 4-year-old daughter learned to play his MusPlay software before learning a "real" instrument. Some have used it in public performances.
There are some uses of the conductor program which seem likely to be effective, but which I don't have any experience, either direct or indirect. For example, it seems likely that a singer might accompany herself by playing the piano part to a song with the conductor program. Song accompaniments often fall in the category of "music I'd rather not have to spend time practicing," so pianists too might want to use the conductor program (especially in rehearsals). And I'd certainly like to try playing the piano part to Schubert's Der Erlkönig with the conductor program — my right hand always runs out of steam.
I showed Tapper to guitarist James Edwards . He told me that because it's so hard to learn guitar music, guitar students have heard themselves play a piece badly, haltingly, and with poor interpretation for a long time before they have the control to play it expressively — by which time they are no longer able to imagine the piece sounding any way other than how they've heard themselves play it. He said he'd like his students, before learning a piece on the guitar, to first learn it with the conductor program, so that they could develop their expressive concept of the piece and have that firmly in mind while they were practicing.
There are also some more prosaic uses for the conductor program. For example, if you're studying a piece, it can be useful to step through it at a reduced tempo to hear what's going on more clearly. The step-through feature can also be used when proofreading (proofhearing?) a score.
The following section is an unsorted catch-all for subjects worth mentioning but for which my thoughts are too under-developed to fit into this article in a more structured way.
Musicians have to learn lots of things. Some of these things are directly related to musicianship: being able to do rhythmic gestures, to recognize what's going on in music, to perform while simultaneously listening closely to other musicians performing, etc. Some, though, are incidental: the player of a modern flute has to learn very different fingerings than the player of a baroque flute, yet a singer doesn't have to learn fingerings at all; a singer has to learn how to produce a pleasing tone but an organist does not, etc. The user of the conductor program is relieved of some responsibilities that other musicians must bear. To the extent that these responsibilities are incidental or unnecessary, their reduction is a good thing, but if they are things that put the performer into more direct contact with the essential aspects of music, then something is being lost. Which responsibilities are essential and which are incidental? I'm sure there is a lot of difference of opinion about this; personally, I believe that ...
... it is incidental that musicians have to learn ... | ... but that it is essential that they learn ... |
...fingerings. Because there are seven pitches in diatonic scale but only five fingers on the human hand, most instrumentalists are saddled with the chore of learning how to associate finger motions with notes in a scale. This task is multiplied by twelve because a scale can start on any of the available chromatic pitches, and again some other number to include all the scale variants (major, various forms of minor, modal, blues, whole-tone, etc.). Keyboard players must additionally learn how to play with both hands, how to alternate fingers in such a way that there is always a finger available at the right time, and how to play multiple melodies with one hand. | ...scales. Every pitch is unique; every note is different from every other note. The quality of a particular pitch, the distance between one scale degree and another — these things are as basic to a musician as knowing what blue looks like is to a painter. You can't know about music and not know where the notes are, how they relate to one another, and what they sound like. Also, it's essential for musicians to be able to articulate fast notes with ease and control It's reasonable to try to "design out" some of the more onerous manual demands of musical instruments, but it is not possible to eliminate the requirement that a musician be familiar with the elements of musical composition, or keep track of notes happening too fast to conceive of individually. |
...how to play chords and arpeggios. As with scales, there are lots of different types of chords, and each type of chord can start on any pitch, so there are lots of them to learn. Moreover, chords and arpeggios are more difficult than scales because there are more varieties of distance between chord notes, and on most instruments, moving by a scale step is easier than moving by a larger interval. |
...harmony. Just as every pitch is unique, so is every chord and every chord progression. |
...physical motions peculiar to a particular instrument. Have you ever noticed what a bassoonist has to do with her thumb? It's pretty unnatural. Organ players need to learn to play melodies with their feet (and with one foot while controlling a swell pedal with the other foot) and to push and pull stop pistons without interrupting their player. Even something as simple as turning pages (which singers can do without much special training) can be elevated to an acrobatic feat when combined with the requirement that you not interrupt the music and have enough of the score memorized that you don't have to be looking at it while the page is turned. Not to mention the demands of embouchure, holding an instrument under your chin, etc... |
...physical motion that corresponds to musical motion. Music is all about motion: melodic motion, harmonic motion, rhythmic motion. We are sensitive to motion in music precisely because we are gestural creatures; we understand musical gestures innately. A musician's body knows all about gesture; if the musician can learn to transfer that knowledge into sound, all will be well. A musical instrument that requires a musician to do a smooth gesture to produce a smooth sound is requiring that the musician have the feeling of a smooth gesture in her body; this feeling, and the resulting gesture, reify the musical idea. A brain without a body would make a very inhuman music! Dancers, singers, and conductors have a certain advantage over instrumentalist; their physical/musical demands are inherently more natural. |
...ultra-precise finger positions. On a violin, a difference of a few hundredths of an inch can be the difference between beauty and ugliness, between pleasure and pain, between Heifetz and ... well, me. Keyboard playing is less demanding, but accuracy is still an issue: you can have your hand in the right position, have the right finger ready at the right time, and bring it down with the right velocity and the right amount of muscular support behind it, but if it falls quarter of an inch too far to the left or right, the result can be anything from the note being too loud or soft, to the wrong note being played; and even if the note still happens to sound correctly, the sensation of contacting a key in a manner other than what you expected is a distraction. |
...intonational nuance. |
...awkward breathing. The oboe requires such a small volume of air that the air in an oboist's lungs goes stale before it is used up; the performer must therefore breathe out at the end of a phrase before breathing in for the next phrase. This is an extreme example, but few wind instruments place no unusual demands on the player's lung capacity. | ...breathing. Pianists and players of string instruments don't have to breathe in any particular way to make sounds on their instruments, but musical phrasing is so closely related to breathing that most musicians breathe musically as part of their playing, and would find it harder to express the music properly if they were prevented from doing it. |
...practicing to produce a good tone. What determines whether the notes a performer produces are, in themselves (that is, apart from the musical context in which they appear), pleasing? For instrumentalists, the quality of the instrument plays a part. Practice of course makes a big difference. For singers, it is something like physical beauty: there are things one can do, but some people are simply more gifted. | ...practicing to modulate the tone. What's the right way to move from one note to another? On most instruments, you can tell whether a skilled musician or a beginner is playing by listening to how they play a single note; and on most of the rest, you can tell from two notes. A single note can be played in a wide variety of ways, as can the connection between two notes. How a note should sound depends on context, and musicians must learn what those contexts are, and how to make the notes sound the right way at the right time. |
...physical endurance. Perhaps the most treacherous example of this is the brass player's embouchure. It takes years to develop a good embouchure, but a few days without practice is enough for it to deteriorate, and a few hours of playing can be enough to use it up for the day. | ...mental endurance. Even if a musician's skills are so well-developed that playing music is physically effortless, keeping one's focus through a long piece of music, and keeping it well enough that the piece makes sense as a whole, is a feat. |
...finger patterns associated with playing contrapuntal keyboard music. I can play most of Bach's Well-Tempered Clavier tolerably well, and I have to say that it is one of the hardest things I've ever learned to do. That would be okay, I supposed, except that learning it has very little to do with learning how to conceive of how to perform contrapuntal music. Rather, it is a collection of a thousand and one tricks, stratagems, heuristics and an equal number of only partially successful compromises. If I had six hands, none of this would be necessary. | ...controlling multiple parts in contrapuntal music. Attention is unary, but counterpoint involves multiple objects of attention. How can a performer do it? Playing contrapuntal music is something like juggling; though there are several balls, you're usually only throwing one at a time; the skill is to shift your attention from one ball to another. And with contrapuntal music, you need to pay attention to a musical line at the point where it is changing from doing one thing to doing another, and switch your attention rapidly enough to never "drop the ball." |
How does the conductor program fare in this light? This varies significantly from one implementation to another. In general, compared to real performing, the incidental responsibilities are fewer and some of the essential responsibilities are still present, but some are not. It's not too difficult to imagine a way to make the conductor program more like real performing in any given aspect, but as responsibilities are added, the difficulty increases, and the advantage over real performing decreases. A possible way out of this corner is suggested by the way a keyboard player manages to do several things at once. Human attention is limited, and the number of things going on when playing multi-part music exceeds this limit, so keyboard players learn to perform various actions without having to attend to them continuously. This approach could be extended to the conductor program: just as a real conductor can reduce her effort when the performers are doing the right thing without attention, a user of the conductor program could, for example, let the tempo go on "auto-pilot" while attending to articulation, then control the balance, then punch up a few accents, then take control of the tempo, etc. In such a scheme, the performer can chew as much as he cares to bite off.
Musical gesture is related to physical gestures. With conventional musical instruments, the structure of music is related to the structure of the gestures that are required to produce it — a big gestures make a bigger sound than a smaller gesture, a fast physical gesture is required to make a fast musical gesture. Every instrument has its own mapping from gesture to sound; a quick motion with your right hand has a very different effect depending on whether you're playing a violin, a trombone, a gong, or the Theremin. One reason to learn to play more than one instrument is that each has its own lessons about gesture to teach. Singing involves gestures with the lungs that are not integral to piano playing, but a pianist can learn a lot about phrasing by learning to sing expressively.
For conventional instruments, the gestures required to produce sounds are a compromise between what feels right to a performer and what's necessary from an acoustical or mechanical standpoint. Presumably this compromise means that these gestures are not optimal for the performer. When you put a computer between the performer and the sound-production apparatus, the instrument's acoustical and mechanical constraints are removed, and any mapping between the performer's gestures and the musical result is, theoretically, possible (though some mappings, like wiggling your tongue, might require an interface that has not yet been developed).
What do we want to do with this freedom? What are the optimal gestures for producing music? There are various ways to approach this question. One way is very pragmatic and direct: ask people. What gesture would you like to do to make such-and-such a sound? Another way is to study the gestural aspect of music, and develop a theory for how this relates to the human system of nerves, muscles, bones, and masses. I believe that both approaches have merit.
With a real instrument, an imperfect performance gesture can lead to an imperfect sound. How imperfect the sound is varies from instrument to instrument — pressing a key too hard on an organ has relatively little effect, but having your finger position wrong by an eighth of an inch when you're playing a high note on the violin can make the difference between bliss and blech. In general, though, the greater the imperfection of the gestures, the greater the imperfection of the result. This is a good thing, because one performer's mistake is another's ornament; letting imperfection through is a natural side-effect of letting expression through.
This issue comes up when designing an implementation of the conductor program: what kind of freedom does the performer have to deviate from the most-likely gestures, and what effect do such deviations produce? You don't want a little mistake to turn into a big disaster (like it does in Tapper). The art here is to figure out a good mapping between the range of possible gestures and the range of possible outcomes. Making it impossible to make a sound you'd never want to produce might seem like one reasonable goal, but it begs the question: what are those sounds? Where is the borderline between "I might sometimes want that sound" and "I will never want that sound"? There probably isn't an answer to that.
A useful principle is: the more unlikely a sound is, the more effort it requires to produce it. Since what is likely and what is unlikely will change from one context to another, a mapping that could change is probably a good idea ...
I began by discussing some of the ways musicians divide up responsibility for making and enjoying music. There are lots of different ways, and they vary a lot in terms of ease of performance, ease of learning, quality of the result, pleasure in performance, pleasure in listening, pleasure in learning, extra-musical pleasures, etc. Musical activities could be placed on a "spectrum of control" like this
In this spectrum, there are various "sweet spots" — mixes of freedom and involvement that are in some way optimal. Listening to music and dancing to music are very popular, and certainly qualify as sweet spots. As you increase control and responsibility, the level of gratification goes up, but so does the degree of (for want of a better term) work. The fact that the conductor program lies in the region between a very popular type of involvement (dancing) and a more gratifying but more demanding type of involvement (performing for real) suggests that it might have the potential for becoming a popular activity.
As a pianist, my first approach to using a piano keyboard to control the conductor program (with Tapper) was to follow the melodies, to do finger patterns that reminded me of the finger patterns I'd use if I were playing some part of the piece on the piano. Sometimes, though, I would encounter a passage for which the first fingering that came to mind was awkward (or downright difficult), and I started trying other finger patterns (the more effective of which are described here). What I found is that Tapper is not just a new use for a piano keyboard; it's actually a new kind of instrument. The finger patterns, divorced from the constraint that they relate to pitches, become like the finger patterns of a guitarist's right hand, or the violinist's bow. In fact, the feeling of playing Tapper reminds me, in a certain way, of bowing. The most effective finger pattern for a passage is one which follows the phrasing. I've found that this way of approaching the keyboard has transferred to my "normal" playing.
If the score is in the computer, what about improvisation? Is there any room for this? The answer is "yes, but with freedom comes responsibility." And complexity. Of those described here, only the Zimmerman/Wantman implementation allows flexibility in which pitches are played: if the score indicates a trill, the player controls how fast it goes and how long it lasts, and thus how many notes the trill contains. Four things are necessary for this to work: the score must be encoded in such a way that the trill is represented as a trill (as opposed to being individual notes), the software must be written to respond to a trill in the score differently than for untrilled notes, the performer needs to know where in the score the trills are, and the performer needs to know what to do to make a trill happen. There are other ways this could happen; for example, there could be a special gesture a performer could do that would cause a trill to happen regardless of what was in the score. Or, the notes of the trill could be controlled by some other aspect of the performer's gestures, like how fast they're playing or how loudly (though this takes it a little out of the realm of improvisation).
There are other simple types of improvisation that performers of notated music might want: the ability to roll chords at will, add extra notes to chords, play a note or chord more or less often than written, play a chord with an unusual voicing (bringing out one note, for example). The design issue is: how would these things be controlled? If you develop a system of special gestures that have special effects, you reduce the amount of "gesture space" that's available for non-special effects.
A possible way around this would be for a performer to play some parts of the score with the conductor program (say, the accompaniment) and other parts "for real." Such a feature could be selectable in real-time during the performance: for example, pushing a pedal could put the right hand part into "for real" mode to allow the performer to ornament or improvise.
Another possibility is to divide the controller into separate sections: the "conducting" section and the "improvising" section.
Where does the conductor program end and some other type of application, a more improvisation-oriented one, begin? There's no clear dividing line. A score could be represented not by pitches, but by some description of the piece that was looser (e.g. chords changes, or tonalities, or melodic motives), with the performer's gestures mapped to pitches conforming to that description. At some point, we're not really talking about "conducting" anymore ...
An implementation of the conductor program may present information about what's in the score to the performer. Some of the implementations described here do not present it at all (Strangio, Mathews, Casio, Schloss, Zimmerman/Wantman, Marrin-Nakra). Friedman's implementation uses a symbolic notation for rhythms; mine, Plank's and Fallgatter's use variations on bar-graph (piano-roll) notation. Conventional notation is an option, but some people want to use the conductor program because they don't know how to read conventional notation, so it is perhaps not ideal.
No musical notation specifies everything about a performance; any notation is only an approximate indication of what is to be played, a reminder. What is the most appropriate, most effective reminder for a user of the conductor program? There are a lot of ways to go with this question. Could there be a notation that expressed elements of rhythmic microstructure ? Or the kinds of timbral, amplitude and pitch modulations that are possible on string and wind instruments?
Sadly, the part of conventional notation that's most obscure to non-musicians, rhythm, is the part that would be most valuable to a user of the conductor program. This suggests that a new, more graphical, more intuitive rhythmic notation might be appropriate. Conventional notation evolved to satisfy many constraints, many of which are relaxed or removed in the context of the conductor program implemented on a personal computer. For example:
One possibility is to make a rhythmic notation in which the rhythmic hierarchies are made more explicit, perhaps with a graphical notation; for example, this rhythm
might be indicated something like this
to show how the measure is subdivided.
Depending on the implementation, various amount of information about a composition's performance is stored in the score used by the conductor program. If the performer is responsible for everything except pitches, only the pitches need to be stored; if the responsibility for the performance is more equally shared between the performer and the computer, the stored score many also contain information about dynamics, tempo changes, timbre, etc. This raises the question: how do these details get into the score? An approach which has been discussed a fair amount (but not, as far as I know, implemented in any commercial software) is for the computer to learn these details from the performer.
A possible scenario would go like this: the stored score would start out very straight, containing more or less what's in a conventional score. Each time the performer conducted the piece, information about note timings, dynamics, articulation, etc. would be extracted from the performer's gestures and added to the score as probabilities. These probabilities would shape the computer's playback during subsequent performances. The degree to which this information is used would depend on the implementation: if the computer were only responsible for playing notes falling between beats indicated by the performer, it would only adjust those notes, but if the paradigm were more like real conducting, the computer could anticipate ritards (and not play notes too early), diminuendos, etc.
This approach could also be used to create renditions of pieces to be played back by the computer alone, in which case the performer assumes the role of the director of a master-class. In this scenario, the performer might choose to make "suggestions" about only a certain aspect of the performance. For example, if the tempo in a ritard needed work, the performer could tell the computer to incorporate only timing information into its stored score, then conduct a performance of the coda, paying special attention to the tempo of the ritard, and ignoring dynamics.
One interesting design challenge here would be to create a "coaching language" for a performer to communicate the nature of the changes to be incorporated into the score; for example, if they performer wanted the computer to play a passage "with more delicacy," what features of a sample rendition by the performer should the computer incorporate into its stored score and which should it ignore?
It's been suggested that removing too many obstacles from the path to musical performance would make people less interested in it, that the challenges of musical performance provide an essential motivation. Since it was never part of my motivation, I was surprised when I first heard this idea, but I know that I'm not any more "average" than anybody else, so I made an informal survey of the musicians I knew personally. Since the responses were in the form of long essays and often did not answer the question of motivation directly, the results presented here involve a certain amount of my judgment. However, I tried to err in the direction opposite my preconception. If it seemed like the person might be saying that the challenges of playing a musical instrument provided some motivation, I counted it as "moderate effect." If they spoke a lot about challenges, I counted it as "essential" even if they didn't say that the challenges were essential. Only in the cases where people explicitly said it had no effect (or an anti-effect) did I score them that way (by "anti-effect" I mean that the person said that if there had been less challenge, they would have been more motivated). Here are the results:
I'm not sure how to interpret these results. Since it is a cross-section of active musicians (at various levels from skilled amateur to professional), I would expect it to be skewed in favor of people for whom musical challenges were welcomed or at least not a big negative factor. Does the fact that the majority of responders said it had a small effect, no effect, or a negative effect mean that, on balance, musical challenges are not important?
Another unanswered question is how people who are not musicians would respond. I've never known anybody who said "oh, I tried learning a musical instrument once, but it was too easy, not enough of a challenge, so I quit" and I've met people who'd studied music as a child and regretted having given it up, presumably because it would be too much work to learn it as an adult.
So, it's still an open issue. All I can say for sure is: it's not yet clear to me that musical challenges are an important part of what motivates most musicians.
It's certainly satisfying for a single performer to use the conductor program. Thomas Zimmerman's patent describes a system in which several performers play computer-assisted instruments together. What is the point of diminishing returns? Would you want to have a hundred musicians all playing computer-assisted instruments together in an orchestra? I think not ... The optimal number depends on the music. A piece of solo piano music probably only requires the attention of a single musician to perform it adequately, but a complicated piece could certainly take more; after all, there are orchestral pieces which require two conductors. There is a practical upper bound, though, because the complexity of music is limited by how much a person can hear. If a hundred people are all playing different things, a single listener can't hear them all separately — and this applies to both the performers and the audience. We can follow a single musical idea, or two, or three ... but beyond three or four distinct musical things, our attention breaks down and we hear groups of things, not individual elements. How many distinct musical ideas can a person control expressively? The limit imposed by human attention is further limited by the nature of the interface. A flute player seldom plays more than one melody at a time, a guitarist more than two, a pianist more than four, an organist more than five. One of the reasons that organs have pedalboards but pianos do not is that human attention limits a keyboard player's ability to play multiple lines expressively, but the expressive demands of an organ, which does not include control of dynamic level by key velocity, are somewhat less than those of the piano. None of the current implementations of the conductor program explore the question of higher-bandwidth user interfaces, but are instead based on existing models of music instruments and conducting. Could a person with the right user interface control the conductor program well enough to play a late Beethoven string quartet as expressively as four string players could? I believe the answer is either "yes" or "pretty darn close," but that it would require a user interface very different from what's been developed so far ...
It's called the "conductor program" but that doesn't mean that a person using it is doing everything a conductor does — or vice versa. There are some parallels, though. For example:
as a conductor, you... |
as a user of conductor program, you... |
...choose the repertoire (sometimes). | ...choose the repertoire (usually). |
...study the score. | ...study the score. |
...develop an idea of how the music should be performed. | ...develop an idea of how the music should be performed. |
...practice. | ...practice. |
...find gestures that best communicate your intent to the performers. | ...find gestures that best communicate your intent to the software. |
...decide how you look to the audience. | ...decide how you look to the audience (if any). |
...decide how you look to the performers. | N/A |
...direct rehearsals, talk to the performers. | N/A |
...interact with all the people who make an orchestra happen. | N/A |
Conducting as a social, interpersonal activity is quite different than conducting as an abstracted, music- and information-centric activity ...
There are lots of interfaces that might be good for use with the conductor program. Here are a few I know about
name (and link) |
picture |
description, discussion |
Collects signals from up to 12 EMG (electromyographic) sensors and sends them wirelessly to a receiver, where they are mapped to MIDI. | ||
|
If the conductor is going to control lots of musical effects, one approach is for each effect to have its own button. | |
A continuous controller with three dimensions (left/right, forward/back, and pressure) presents a useful gesture space; this one can sense multiple fingers in that space. | ||
The hope for this was that a rotating gesture would map well onto the repeating nature of musical beats, phrases, etc. It turned out not to work very well, because circular motion is hard to do smoothly, and the interface provided no tactile feedback about where the beats and/or phrases were, so it was hard to make them happen at the right time. Haptic feedback might improve this. | ||
Digital agility a plus; absence of haptic feedback is a minus. | ||
This system includes 14 sensors to detect the angles of the performer's limbs. | ||
Large-scale interaction a plus. | ||
Immersion focuses on haptic feedback, a plus for music interfaces. | ||
Senses the vertical and horizontal positions of two hands; performer is free to move around (no wires). | ||
The Lizard Trampoline provides multi-finger control of both multi-finger timing (from contact of conductive glove fingertips with screen) and pressure (displacement/tension of screen). Too bad it's only a concept so far. | ||
Like a normal marimba, but with more axes of control. | ||
Your basic MIDI wind controller; good for continuous control of dynamics. | ||
O'Hearn |
This interface (invented by industrial design student Stephen O'Hearn), was described in the "Ask Mr. Moog" column of Keyboard Magazine, January 1989, page 106. It has many axes of large-gesture control (pressure of thumbs, rotation of hands, bending of main staff) plus a two-dimensional finger controller (each black/white dot is a trackball), and a "function-key" pedalboard. | |
Nice large surface for expansive, full-body gesture. | ||
Senses position of hands, butt. Good for audience; lack of haptic feedback a minus. | ||
The cross-bars of the Tone Ladder are mapped to notes of a scale. | ||
Includes both sensing and haptic feedback for the four degrees of freedom of the performer's bow. | ||
The VideoHarp had some nice features: a large area for gestures, sensing of multiple fingers, and easy for the audience to watch. | ||
Zipper |
The zipper on the front of a sweater or jacket is in the perfect position for a hand-operated musical interface, and allows many of the kinds of gestures that bowing affords string players: two directions of stroke that can be alternated, three dimensions of control (stroke, horizontal, tension); a zipper controller could also have a pressure sensor in the handle. |
In the conductor program, the computer plays music described in a score in response to a performer's gestures. In computer accompaniment, the performer performs "for real" and the computer provides the rest of the parts of the piece. Computer accompaniment is similar to the conductor program in many ways (the software collects gestures from the performer, analyzes them, and plays back notes from a score, etc.) and is typically considered a more "practical" (and hence marketable) application, since musicians need and use accompanists already.
How could the conductor program be improved?
Zimmerman, Wantman and Marrin-Nakra have begun several avenues of exploration of the physical part of the interface that seem ripe for further development: differentiating different kinds of control (e.g. trigger versus modulation), capturing more information from the performer's gestures and physical state, analyzing this information to extracting "gestural intent," and mapping this intent to the control of the performance. The visual part of the user interface needs work too: for the conductor program to be most effective, music notation needs to be redesigned from the ground up so that the information most important to a conductor/interpreter is presented naturally and efficiently.
Ideally, the interface to the conductor program would eliminate the distinction between the performer's physical control of the music, the gestures necessary to perform the music, and the visual score that represented the music. The score would be represented as an evolving collection of dynamic sound objects, with which the performer would interact directly. The intent of the composer, the performer in rehearsal, and the performer in real-time would all be made manifest both visually and aurally.
But what about non-technical considerations? Is there a place in our society for the conductor program? Will people want to use the conductor program? Will something else take the place of the conductor program? I have high hopes for the future of the conductor program, but I'm faced with this piece of evidence: the idea has been around for more than thirty years, and there has been readily-available hardware, adequate to show the idea's potential, for more than half that time, and yet it's still failed to get past the "interesting idea" point for all but a handful of practitioners (and some lucky children in Boston). Hopefully this article will help more people know about its potential.
I extend my thanks to all the implementers and users of the conductor program mentioned in this article and to David Pihl, George Pontis, and Ed Severinghaus. A special thanks to Mervin Lane, who provided the motivation for pushing this project to the top of my to-do list.
If any part of this article is incorrect, confusing, or unclear, please let me know.
An inadequate list of references.
Study using Friedman's Instant Pleasure to examine the musical responses of children with disabilities.
"Technology-Assisted Research in Music Cognition: Enhancing Instruction in Expressive Music Performance," Robert H. Woody
Richard Boulanger, user of the Radio Baton
"Abstract Cinema and Light-Music," Bulat Makhmudovich Galeev, who is affiliated with the Prometheus Institute
Bert Bongers, "Physical Interfaces in the Electronic Arts"
"Music-Driven Motion Editing: Local Motion Transformations Guided By Music Analysis"
"Considering Human Memory in Designing User Interfaces for Computer Music," Otto Laske, Computer Music Journal, Volume II, Number 4, pages 39-45
"Electronic Music Interfaces ,"
Joseph A. Paradiso, IEEE Spectrum, December 1997, pages 18-30
"The Double Revolution of the Theremin," Patrick Rashleigh