Cosyvoice2 GRPO RL training recipe

[yuekaizhang]

Hi, we would like share the GRPO RL training recipe based on cosyvoice2 llm.

Recipe: https://github.com/nvidia-china-sae/mair-hub/tree/main/rl-tutorial/cosyvoice_llm

Here are the initial training results:

Model	Seed-TTS test_zh CER	Cosyvoice3 zero_shot_zh	Comment
Official CosyVoice2 LLM	1.45 %	4.08%	See the paper
+ GRPO	1.37%	3.36%
SFT (initialized from Qwen2-0.5B-Instruct)	1.81 %	4.83%	See PR #1887
+ GRPO	1.06 %	4.03%

We will add more experimental results as we continue refining the recipe.

[zhangyike]

Hi, I've reproduced your work and obtained the following results：

Model	Seed-TTS test_zh CER	Cosyvoice3 zero_shot_zh	Comment
Official CosyVoice2 LLM	1.45 %	4.08%	See the paper
+ GRPO	1.43%	3.39%	100 steps

However, as training steps increased beyond this point (100 steps), ​WER gradually rose on both test sets. At 700 steps, the WER reached ​9.05% on zero_shot_zh​.

Upon analyzing error transcripts in errs-sherpa_onnx.txt, I observed some synthesis failures where utterances contained ​unintelligible speech​. This primarily manifested as high ​deletion errors​ in the output.

Has this issue occurred in your experiments? Could you specify the checkpoint (training steps/epochs) used for the results in this issue?

Here's an example of synthesized audio containing unintelligible speech: zero_shot_uttid_125.wav.zip

[xiaoyangnihao]

@yuekaizhang @zhangyike hi, using "pip install "vllm==0.8.5.post1" "torch==2.6.0+cu126" "torchaudio==2.6.0+cu126"", when importing vllm , I met this error :'vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb', can you give me some suggestions? thanks

[zhangyike]

@yuekaizhang @zhangyike hi, using "pip install "vllm==0.8.5.post1" "torch==2.6.0+cu126" "torchaudio==2.6.0+cu126"", when importing vllm , I met this error :'vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb', can you give me some suggestions? thanks

It seems a problem related to torch version. I didn't meet this problem with cuda 12.4.

[xiaoyangnihao]

@yuekaizhang @zhangyike hi, using "pip install "vllm==0.8.5.post1" "torch==2.6.0+cu126" "torchaudio==2.6.0+cu126"", when importing vllm , I met this error :'vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb', can you give me some suggestions? thanks

It seems a problem related to torch version. I didn't meet this problem with cuda 12.4.

solved, I update torch version to 2.7.0 and vllm to 0.9.0, thanks

[zhangyike]

@yuekaizhang @zhangyike hi, using "pip install "vllm==0.8.5.post1" "torch==2.6.0+cu126" "torchaudio==2.6.0+cu126"", when importing vllm , I met this error :'vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb', can you give me some suggestions? thanks

@zhangyike 使用token2wav_asr_server.py部署服务时，gpu数量大于1或者实例数大于就会在初始化阶段失败，有遇到过吗？等于1时正常

我没有遇到过这个问题。你的启动命令是什么？我用了8个GPU，每个GPU一个实例，export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" && python3 token2wav_asr_server.py --number-of-devices 8

[xiaoyangnihao]

Hi, I've reproduced your work and obtained the following results：

Model Seed-TTS test_zh CER Cosyvoice3 zero_shot_zh Comment
Official CosyVoice2 LLM 1.45 % 4.08% See the paper

• GRPO 1.43% 3.39%  100 steps
  However, as training steps increased beyond this point (100 steps), ​WER gradually rose on both test sets. At 700 steps, the WER reached ​9.05% on zero_shot_zh​.

Upon analyzing error transcripts in errs-sherpa_onnx.txt, I observed some synthesis failures where utterances contained ​unintelligible speech​. This primarily manifested as high ​deletion errors​ in the output.

Has this issue occurred in your experiments? Could you specify the checkpoint (training steps/epochs) used for the results in this issue?

Here's an example of synthesized audio containing unintelligible speech: zero_shot_uttid_125.wav.zip

@yuekaizhang @zhangyike 在token2wav_asr_server中，保存token2wav之后的中间音频，每个音频均是20s，然后末尾是大段的静音段或者噪音，感觉和data.max_response_length=512是对应上的，不知道符不符合预期？中间模型，即使是100步，也会有不少音频不可懂的问题。

[github-actions]

This issue is stale because it has been open for 30 days with no activity.