fish_speech: 500 Internal Server error

[alexivai]

fish_speech: 500 Internal Server Error - Something went wrong: Failed to generate semantic tokens

whats wrong i do?

[alexivai]

UPD. This error is related to the use of CUDA. Question! How to fix it without switching to CPU?

[alexivai]

Resolve:
Change in main.rs for cuda BF to F16

[EndlessReform]

ah yes, sorry about this. The original Fish Speech 1.5 weights are natively in BF16, so I'm pretty hesitant to allow casting to FP16 as an option, conversion would cause precision loss. I can pass through an option in the server CLI to load weights as FP32 as that's lossless, but inference would be slower.

• Which GPU are you using?
• Would you mind pasting a screenshot of the error log from the CLI?

[alexivai]

RTX 2080 and i have solution in this Change in main.rs for cuda BF to F16

[EndlessReform]

Just to clarify—did you modify server/main.rs yourself and get it working, or are you asking for this to be upstreamed?

If you’re requesting an upstream change, the only option I’d support is a CLI flag to load weights as FP32. That’s lossless and ensures compatibility with CUDA on RTX 2080 (Turing).

I’m not adding FP16 loading as an option because these weights were pretrained by the Fish Team in BF16 natively, not FP16. Casting them down to FP16 introduces precision loss that could degrade inference quality. Since your 2080 doesn’t support BF16, FP32 is the only supported format. If FP32 is too slow, you’d need different weights (FP16-trained) or different hardware.