| Model | APS (%) | DSD (%) | RP (%) | AVG (%) |
|---|---|---|---|---|
| Gemini 2.5-Flash* | 88.2 | 90.9 | 77.3 | 85.4 |
| Gemini 2.5-Pro* | 89.0 | 90.1 | 75.5 | 84.8 |
| GPT-4o-Mini-TTS* | 54.9 | 52.3 | 46.0 | 51.1 |
| ElevenLabs* | 42.8 | 50.9 | 59.1 | 50.9 |
| VoxInstruct | 47.5 | 52.3 | 42.6 | 47.5 |
| MiMo-Audio-7B-Instruct | 70.1 | 66.1 | 57.1 | 64.5 |
| VoiceSculptor | 75.7 | 64.7 | 61.5 | 67.6 |
Note
- Models marked with
*are commercial models.- InstructTTSEval — InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems. arXiv preprint arXiv:2506.16381.
final.mov
- [2026-1-18] We have released the RAG and WebUI code, and the WebUI supports vLLM.
- [2026-1-16] We have released the technical report! VoiceSculptor
- [2026-1-8] We have released the Demo Page and Demo Video! VoiceSculptor-Demo-Page
- [2026-1-2] We opened the repository and uploaded the voice design models! VoiceSculptor
Follow the steps below to clone the repository and install the required environment.
# Clone the repository and enter the directory
git clone https://github.com/ASLP-lab/VoiceSculptor.git
cd VoiceSculptor
# Create and activate a Conda environment
conda create -n VoiceSculptor python=3.10 -y
conda activate VoiceSculptor
# Install dependencies
pip install -r requirements.txtgit lfs install
git clone https://huggingface.co/ASLP-lab/VoiceSculptor-VD
git clone https://huggingface.co/HKUSTAudio/xcodec2For detailed instructions on how to design high-quality voice prompts,
please refer to Voice Design Guide or Voice Design Guide EN.
You need to specify the local paths to the voice-design model and the xcodec2 model in the infer.py file.
python infer.pyThis project provides a simple workflow to build and deploy a private text vector database for Retrieval-Augmented Generation (RAG). You can create your own database from text files and run a lightweight server to query it via a client.
Use the create_database.py script located in the rag/ directory.
Before running the script, make sure to:
- Replace the model path
- Replace the input text file path
Input Text File Format
Each line in the input file should follow this structure:
utt_file_name \t wav_path \t text_command <|endofprompt|> target_text
Example
ZH_B00074_S00400_W000029 Emilia/ZH/ZH_B00074/ZH_B00074_S00400/mp3/ZH_B00074_S00400_W000029.mp3 这是一位中年男性的中低音有声书朗读，嗓音浑厚略带粗砺，以标准普通话清晰咬字，通过多变的语调 动态的语速和戏剧化的停顿，生动演绎充满张力的动作场景<|endofprompt|>而就在此时，邵飞忽然露出坏笑，他一脚踹在赵和的窝锅子上。赵和顿时扑通一下，跪了下去。
Run the Database Creation Script
python rag/create_database.pyOnce completed, your private RAG vector database will be generated.
After building the database, you can launch the query service using the server script in the rag/ folder.
Before starting the server:
- Update the database path
- Update the model path
bash run_server.shTo connect to the running service, modify the IP address and port in rag/client.py so they match the server configuration.
Run the Client
python client.py| Step | Action |
|---|---|
| 1 | Prepare and format your text dataset |
| 2 | Build the vector database |
| 3 | Configure and start the RAG server |
| 4 | Query and retrieve results via the client |
python gradio/webui.pypython gradio/webui.py- 🌐 Demo website
- 🔓 Release inference code
- 🤗 Release HuggingFace model
- 🤗 HuggingFace Space
- 📝 Release Technical Report
- 💡 Release RAG code
- ✨ Release gradio code
- 🚀 Release vLLM code
- 🔓 Release both the text generation code and a sample dataset
- 🔓 Release the code for voice cloning
- 🔓 Release training code
@misc{hu2026voicesculptorvoicedesigned,
title={VoiceSculptor: Your Voice, Designed By You},
author={Jingbin Hu and Huakang Chen and Linhan Ma and Dake Guo and Qirui Zhan and Wenhao Li and Haoyu Zhang and Kangxiang Xia and Ziyu Zhang and Wenjie Tian and Chengyou Wang and Jinrui Liang and Shuhan Guo and Zihang Yang and Bengu Wu and Binbin Zhang and Pengcheng Zhu and Pengyuan Xie and Chuan Xie and Qiang Zhang and Jie Liu and Lei Xie},
year={2026},
eprint={2601.10629},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2601.10629},
}
@misc{ye2025llasascalingtraintimeinferencetime,
title={Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis},
author={Zhen Ye and Xinfa Zhu and Chi-Min Chan and Xinsheng Wang and Xu Tan and Jiahe Lei and Yi Peng and Haohe Liu and Yizhu Jin and Zheqi Dai and Hongzhan Lin and Jianyi Chen and Xingjian Du and Liumeng Xue and Yunlin Chen and Zhifei Li and Lei Xie and Qiuqiang Kong and Yike Guo and Wei Xue},
year={2025},
eprint={2502.04128},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2502.04128},
}We use the Apache 2.0 license. Researchers and developers are free to use the codes and model weights of our VoiceSculptor. Check the license at LICENSE for more details.
Additional Notice on Generated Voices
This project provides a speech synthesis model for voice design, intended for academic research, educational purposes, and legitimate applications, such as personalized speech synthesis, assistive technologies, and linguistic research.
Please note:
Do not use this model for unauthorized voice cloning, impersonation, fraud, scams, deepfakes, or any illegal or malicious activities.
Ensure compliance with local laws and regulations when using this model and uphold ethical standards.
The developers assume no liability for any misuse of this model.
Important clarification regarding generated voices:
As a generative model, the voices produced by this system are synthetic outputs inferred by the model, not recordings of real human voices.
The generated voice characteristics do not represent or reproduce any specific real individual, and are not derived from or intended to imitate identifiable persons.
We advocate for the responsible development and use of AI and encourage the community to uphold safety and ethical principles in AI research and applications.
If you are interested in leaving a message to our work, feel free to email jingbin.hu@mail.nwpu.edu.cn or lxie@nwpu.edu.cn
You’re welcome to join our WeChat group for technical discussions, updates.
