ChatTTS 是一个面向日常对话场景的生成式语音合成模型，主要服务于中英文 LLM 助手的朗读需求。它用十万小时以上的中英音频训练，开源版本为四万小时预训练权重，在韵律自然度上优于多数开源 TTS。模型能预测并控制笑声、停顿、语气词等细粒度特征，支持多说话人交替，可用于合成有来有回的对话语音。仓库提供推理代码、流式生成、WebUI 与命令行示例，30 秒音频约需 4GB 显存。需要注意的是，模型权重以 CC BY-NC 4.0 发布，仅限学术与研究用途。
pip install ChatTTS[!Tip]
For the extended end-user products, please refer to the index repo Awesome-ChatTTS maintained by the community.
You can find a diagram visualization of the codebase here.
ChatTTS is a text-to-speech model designed specifically for dialogue scenarios such as LLM assistant.
You can refer to this video on Bilibili for the detailed description.
[!Important] The released model is for academic purposes only.
2noise org is welcomed)The code is published under AGPLv3+ license.
The model is published under CC BY-NC 4.0 license. It is intended for educational and research use, and should not be used for any commercial or illegal purposes. The authors do not guarantee the accuracy, completeness, or reliability of the information. The information and data used in this repo, are for academic and research purposes only. The data obtained from publicly available sources, and the authors do not claim any ownership or copyright over the data.
ChatTTS is a powerful text-to-speech system. However, it is very important to utilize this technology responsibly and ethically. To limit the use of ChatTTS, we added a small amount of high-frequency noise during the training of the 40,000-hour model, and compressed the audio quality as much as possible using MP3 format, to prevent malicious actors from potentially using it for criminal purposes. At the same time, we have internally trained a detection model and plan to open-source it in the future.
GitHub issues/PRs are always welcomed.
For formal inquiries about the model and roadmap, please contact us at [email protected].
Join by clicking here.
git clone https://github.com/2noise/ChatTTS
cd ChatTTS
pip install --upgrade -r requirements.txt
conda create -n chattts python=3.11
conda activate chattts
pip install -r requirements.txt
pip install safetensors vllm==0.2.7 torchaudio
[!Warning] DO NOT INSTALL! The adaptation of TransformerEngine is currently under development and CANNOT run properly now. Only install it on developing purpose. See more details on at #672 #676
[!Note] The installation process is very slow.
pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable
[!Warning] DO NOT INSTALL! Currently the FlashAttention-2 will slow down the generating speed according to this issue. Only install it on developing purpose.
[!Note] See supported devices at the Hugging Face Doc.
pip install flash-attn --no-build-isolation
Make sure you are under the project root directory when you execute these commands below.
python examples/web/webui.py
It will save audio to
./output_audio_n.mp3
python examples/cmd/run.py "Your text 1." "Your text 2."
pip install ChatTTS
pip install git+https://github.com/2noise/ChatTTS
pip install -e .
import ChatTTS
import torch
import torchaudio
chat = ChatTTS.Chat()
chat.load(compile=False) # Set to True for better performance
texts = ["PUT YOUR 1st TEXT HERE", "PUT YOUR 2nd TEXT HERE"]
wavs = chat.infer(texts)
for i in range(len(wavs)):
"""
In some versions of torchaudio, the first line works but in other versions, so does the second line.
"""
try:
torchaudio.save(f"basic_output{i}.wav", torch.from_numpy(wavs[i]).unsqueeze(0), 24000)
except:
torchaudio.save(f"basic_output{i}.wav", torch.from_numpy(wavs[i]), 24000)
###################################
# Sample a speaker from Gaussian.
rand_spk = chat.sample_random_speaker()
print(rand_spk) # save it for later timbre recovery
params_infer_code = ChatTTS.Chat.InferCodeParams(
spk_emb = rand_spk, # add sampled speaker
temperature = .3, # using custom temperature
top_P = 0.7, # top P decode
top_K = 20, # top K decode
)
###################################
# For sentence level manual control.
# use oral_(0-9), laugh_(0-2), break_(0-7)
# to generate special token in text to synthesize.
params_refine_text = ChatTTS.Chat.RefineTextParams(
prompt='[oral_2][laugh_0][break_6]',
)
wavs = chat.infer(
texts,
params_refine_text=params_refine_text,
params_infer_code=params_infer_code,
)
###################################
# For word level manual control.
text = 'What is [uv_break]your favorite english food?[laugh][lbreak]'
wavs = chat.infer(text, skip_refine_text=True, params_refine_text=params_refine_text, params_infer_code=params_infer_code)
"""
In some versions of torchaudio, the first line works but in other versions, so does the second line.
"""
try:
torchaudio.save("word_level_output.wav", torch.from_numpy(wavs[0]).unsqueeze(0), 24000)
except:
torchaudio.save("word_level_output.wav", torch.from_numpy(wavs[0]), 24000)
Example: self introduction
inputs_en = """
chat T T S is a text to speech model designed for dialogue applications.
[uv_break]it supports mixed language input [uv_break]and offers multi speaker
capabilities with precise control over prosodic elements like
[uv_break]laughter[uv_break][laugh], [uv_break]pauses, [uv_break]and intonation.
[uv_break]it delivers natural and expressive speech,[uv_break]so please
[uv_break] use the project responsibly at your own risk.[uv_break]
""".replace('\n', '') # English is still experimental.
params_refine_text = ChatTTS.Chat.RefineTextParams(
prompt='[oral_2][laugh_0][break_4]',
)
audio_array_en = chat.infer(inputs_en, params_refine_text=params_refine_text)
torchaudio.save("self_introduction_output.wav", torch.from_numpy(audio_array_en[0]), 24000)
male speaker
female speaker
For a 30-second audio clip, at least 4GB of GPU memory is required. For the 4090 GPU, it can generate audio corresponding to approximately 7 semantic tokens per second. The Real-Time Factor (RTF) is around 0.3.
This is a problem that typically occurs with autoregressive models (for bark and valle). It's generally difficult to avoid. One can try multiple samples to find a suitable result.
In the current released model, the only token-level control units are [laugh], [uv_break], and [lbreak]. In future versions, we may open-source models with additional emotional control capabilities.
内容由编辑整理，部分信息来自 GitHub 与项目官网，版权归原作者所有。发现错误？报告问题