Voice Conversion towards Arbitrary Speakers With Limited Data

Voice conversion towards a specific speaker requires a large number of target speaker's utterances, which is expensive in practice. This paper proposes a speaker-adaptive voice conversion (SAVC) system, which accomplishes voice conversion towards arbitrary speakers with limited data. First, a multi-speaker voice conversion (MSVC) model is trained to learn the shared information between speakers and build a speaker latent space. Second, utterances of a new target speaker are used to fine tune the MSVC model aiming to learn the voice of the target speaker. In the two steps, phonetic posteriorgrams (PPGs), a speaker-independent linguistic feature, and speaker embeddings such as i-vector or x-vector are encoded to train the model. In order to achieve better results, two different adaptive approaches are explored: adaptation on the whole MSVC model or additional linear-hidden layers (AHL). As the results show, both adaptive approaches significantly outperform the MSVC model without adaptation. Besides, the whole adapted model based on x-vector gets a higher similarity to target speaker within 10 utterances.