My RTF is always around 0.12 with TRT-LLM docker on L20, never match the claimed benchmark result

[qiulang]

Checks

• This template is only for usage issues encountered.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

nvidia L20, soar97/triton-f5-tts:24.12, just as https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/README.md

But my RTF is always around 0.12

⚡ Real-Time Factor:
   Mean: 0.120x
   Median: 0.112x
   Min/Max: 0.097x / 0.152x

My result is much slower than the claimed benchmark result.

What did I do wrongly ?

Steps to Reproduce

1. find a clean L20 machine, clone the project
2. start the triton docker MODEL=F5TTS_v1_Base docker compose up as https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/README.md said
3. test the RTF

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[SWivid]

Hi @qiulang , the benchmark results reflects the 16 NFE inference speed.

We change the default setting of NFE from 16 to 32 in previous commit (to match with default pytorch imple. setting).
You could change it back to 16 NFE as mentioned here: #1179 (comment)

[yuekaizhang]

@qiulang Also, for reproduce, I suggest to use the same client setting here https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/README.md#using-client-server-mode.

[qiulang]

@SWivid actually I figured that out and had changed nfe to 16, my benchmark for RTF , which I listed in the beginning , Median: 0.112x is already using nfe 16, if it is 32, it is aroung 0.15, so much higher than your result

[SWivid]

I have tested on an RTX4090, RTF is ~0.03 for NFE16 with Offline TRT-LLM.

Median: 0.112x is already using nfe 16, if it is 32, it is aroung 0.15

It would be weird, cuz the RTF of NFE16 should be almost the half of NFE32 case.

[qiulang]

@SWivid by Offline TRT-LLM you meant running thisdocker-compose.yml, https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/docker-compose.yml, right ? I run it on a clean L20 machine.

I notice every time F5 generates an audio, it seems to load the refer audio. But if I just use the same refer audio, is there any "cache" F5 will use to speed up the generated time ?

[SWivid]

@qiulang The benchmarking guidance is here:
https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/README.md#benchmarking

Which code have you run for benchmarking?

[qiulang]

@SWivid I realize when I change nfe to 16 I need to rebuild my docker image. Because I modified your https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/runtime/triton_trtllm/docker-compose.yml (otherwise I need to do git clone and run.sh every time and yes, of course I read benchmarking guidance carefully and many times )

# I modify this, so my docker-compose run run.sh 4 4 instead of the following 0 4
command: >
      /bin/bash -c "pip install vocos && rm -rf F5-TTS && git clone https://github.com/SWivid/F5-TTS.git && cd F5-TTS/src/f5_tts/runtime/triton_trtllm/ && bash run.sh 0 4 $MODEL"

Now I got a satisfied result, with NFE 32 on L20 machine

================================================================================
COMPARISON WITH OFFICIAL BENCHMARK (L20 GPU)
================================================================================

                     Official      Your System    Ratio
                     --------      -----------    -----
Latency (ms):           253.0            705.7    2.79x
RTF:                   0.0394           0.0540    1.37x

With NFE 16

================================================================================
COMPARISON WITH OFFICIAL BENCHMARK (L20 GPU)
================================================================================

                     Official      Your System    Ratio
                     --------      -----------    -----
Latency (ms):           253.0            362.7    1.43x
RTF:                   0.0394           0.0278    0.70x

0.0278 是用 basic_ref_en.wav 虽然是英文，但是生成的中文没问题

我改用一段我自己找的中文， 剪裁到5s, RTF 是 0.0318

================================================================================
COMPARISON WITH OFFICIAL BENCHMARK (L20 GPU)
================================================================================

                     Official      Your System    Ratio
                     --------      -----------    -----
Latency (ms):           253.0            259.1    1.02x
RTF:                   0.0394           0.0318    0.81x

[qiulang]

@SWivid 下一步我想测试 F5的并发处理能力，有没有什么建议？

[SWivid]

测试 F5的并发处理能力

直接测就行

[qiulang]

@SWivid 试过了，好像不行，只能一个并发

[SWivid]

像之前说的 你不说你跑的啥具体代码/脚本 我们怎么帮你解决？

[qiulang]

@SWivid

cat ~/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/config.pbtxt
instance_group [
  {
    count: 1
    kind: KIND_GPU
  }

是不是这个值造成，一次只能处理一个并发？

我的测试代码是自己写的，就是同时发几个请求，看RTF会变成多少

[SWivid]

参考我们readme里写的来测
如果你自己写脚本测，我们肯定没法帮你保证是对的啊

[qiulang]

readme 的benchmark 写的就是 Concurrency 1 所以我才问的

Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE.

Model	Concurrency	Avg Latency	RTF	Mode
F5-TTS Base (Vocos)	2	253 ms	0.0394	Client-Server
F5-TTS Base (Vocos)	1 (Batch_size)	-	0.0402	Offline TRT-LLM
F5-TTS Base (Vocos)	1 (Batch_size)	-	0.1467	Offline Pytorch