Home | Site Map | Watch | FAQ | History | Store | Contact

Synchronization of Animation and Audio

There is no single way that I synchronize the audio with the animation. This chart shows the steps that may be involved:


It may be easiest to understand the process in reverse ...

  • The final movie is generated by video editing software; I use Adobe Premiere. The inputs to this step are an audio recording (a .wav file), animated image files (one file for each frame of the movie), and static image files (screenshots of the conventional notation score); these are all shown in green in the diagram. In this step, I align the audio with the images, do transitions (wipes and fades), and add titles.
  • The animated image files are generated by the frame-rendering software I wrote. This software is similar to the MAMPlayer, but instead of generating live animation, it writes each frame (24, 30 or 60 frames per second) to a file.
  • The input to the frame-rendering software is a MIDI file; the notes in this MIDI file must be in sync with the audio. If the MIDI came from recording a performance (from a MIDI instrument), it is already in sync with the audio; it is also in sync if the audio and MIDI were both exported from the digital score.
  • If MIDI file and the audio came from different sources, they are not synchronized; in this case, I listen to the synchronized audio and record a MIDI click-track: typically one click/note per quarter-note or eighth-note of the piece. After recording this click-track, I edit it to get the timing more precise. The click-track and the unsynchronized MIDI score are then fed into synchronization software which adjusts the notes in the MIDI score to match the timings of the click-track (and therefore, to the timings of the audio recording).
  • Often, the recorded audio differs from the digital score; in this case, I edit the score to more closely match the audio (before exporting the unsynchronized MIDI file).
  • In rare cases, I have a performance-quality MIDI file but no corresponding audio; in this case, I will play the MIDI file through a digital piano and record the playback as an audio file.