Sicong Huang1,2, Qiyang Li1,2, Cem Anil1,2, Xuchan Bao1,2, Sageev Oore2,3, Roger B. Grosse1,2
University of Toronto1,   Vector Institute2,   Dalhousie University3

This is the project website accompanying this paper on
TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer.
We encourage you to watch our video first as it will give you a general idea of this work.

Abstract

In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness. In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high- quality waveform generation. We introduce TimbreTron, a method for musical timbre transfer which applies "image" domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer. We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance. Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.

Website maintained by: Sheldon Huang / Last updated on: November 15, 2018: