awesome by NasionaMarihuany

Real-Time Voice Cloning

This repository is an implementation of Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (SV2TTS) with a vocoder that works in real-time. This was my master's thesis.

SV2TTS is a deep learning framework in three stages. In the first stage, one creates a digital representation of a voice from a few seconds of audio. In the second and third stages, this representation is used as reference to generate speech given arbitrary text.

Video demonstration (click the picture):

Toolbox demo

Papers implemented

URL Designation Title Implementation source
1806.04558 SV2TTS Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis This repo
1802.08435 WaveRNN (vocoder) Efficient Neural Audio Synthesis fatchord/WaveRNN
1703.10135 Tacotron (synthesizer) Tacotron: Towards End-to-End Speech Synthesis fatchord/WaveRNN
1710.10467 GE2E (encoder) Generalized End-To-End Loss for Speaker Verification This repo

News

10/01/22: I recommend checking out CoquiTTS. It's a good and up-to-date TTS repository targeted for the ML community. It can also do voice cloning and more, such as cross-language cloning or voice conversion.

28/12/21: I've done a major maintenance update. Mostly, I've worked on making setup easier. Find new instructions in the section below.

14/02/21: This repo now runs on PyTorch instead of Tensorflow, thanks to the help of @bluefish.

13/11/19: I'm now working full time and I will rarely maintain this repo anymore. To anyone who reads this:

  • If you just want to clone your voice (and not someone else's): I recommend our free plan on Resemble.AI. You will get a better voice quality and less prosody errors.
  • If this is not your case: proceed with this repository, but you might end up being disappointed by the results. If you're planning to work on a serious project, my strong advice: find another TTS repo. Go here for more info.

20/08/19: I'm working on resemblyzer, an independent package for the voice encoder (inference only). You can use your trained encoder models from this repo with it.

Setup

1. Install Requirements

  1. Both Windows and Linux are supported. A GPU is recommended for training and for inference speed, but is not mandatory.
  2. Python 3.7 is recommended. Python 3.5 or greater should work, but you'll probably have to tweak the dependencies' versions. I recommend setting up a virtual environment using venv, but this is optional.
  3. Install ffmpeg. This is necessary for reading audio files.
  4. Install PyTorch. Pick the latest stable version, your operating system, your package manager (pip by default) and finally pick any of the proposed CUDA versions if you have a GPU, otherwise pick CPU. Run the given command.
  5. Install the remaining requirements with pip install -r requirements.txt

2. (Optional) Download Pretrained Models

Pretrained models are now downloaded automatically. If this doesn't work for you, you can manually download them here.

3. (Optional) Test Configuration

Before you download any dataset, you can begin by testing your configuration with:

python demo_cli.py

If all tests pass, you're good to go.

4. (Optional) Download Datasets

For playing with the toolbox alone, I only recommend downloading LibriSpeech/train-clean-100. Extract the contents as <datasets_root>/LibriSpeech/train-clean-100 where <datasets_root> is a directory of your choosing. Other datasets are supported in the toolbox, see here. You're free not to download any dataset, but then you will need your own data as audio files or you will have to record it with the toolbox.

5. Launch the Toolbox

You can then try the toolbox:

python demo_toolbox.py -d <datasets_root>
or
python demo_toolbox.py

depending on whether you downloaded any datasets. If you are running an X-server or if you have the error Aborted (core dumped), see this issue.

MIT License

Modified & original work Copyright (c) 2019 Corentin Jemine (https://github.com/CorentinJ) Original work Copyright (c) 2018 Rayhane Mama (https://github.com/Rayhane-mamah) Original work Copyright (c) 2019 fatchord (https://github.com/fatchord) Original work Copyright (c) 2015 braindead (https://github.com/braindead)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Project Meta

  • Registered on LibreCores 2 months ago
  • Project started 3 years ago
  • Last commit 2 months ago

Commits

{"labels":[2018,2019,2020,2021,2022],"series":[[46,195,36,12,5]]}

Commits per year

Contributors

{"labels":[2018,2019,2020,2021,2022],"series":[[1,9,5,6,2]]}

Unique contributors per year

Releases

Data not available

Languages

{"labels":["Others","Python","Markdown"],"series":[0,57,3]}

Share of languages used

Data Sheet
Project Web Site
https://github.com/CorentinJ/Real-Time-Voice-Cloning
Last activity 2 months ago
Primary language: Python
294 commits by 18 contributors
Corentin Jemine blue-fish Alex Peattie Matheus Fillipe Chris Van Pelt
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,1,0,0,0,4,0,3,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0

activity over the last year

LibreCores data updated 2 months ago