Watch full video on youTube
0:00 - 0:41: Introduction
0:42 - 1:34: First Attempt: STFT
1:35 - 2:24 CQT TimbreTron
2:25 - 2:42 Rainbowgram Visualization
2:43 - 3:42 Tempo Manipulation
3:43 - 4:16 Pitch Manipulation
4:17 - 6:41 Final results
Roger B. Grosse1,2
University of Toronto1,   Vector Institute2,   Dalhousie University3
This is the project website accompanying this paper on
TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer.
We encourage you to watch our video first as it will give you a general idea of this work.
In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness. In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high- quality waveform generation. We introduce TimbreTron, a method for musical timbre transfer which applies "image" domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer. We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance. Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.
Website maintained by: Sheldon Huang / Last updated on: November 15, 2018: