CUDA error: device-side assert triggered

[MayVid6]

Help a question, please.😥😥

Traceback (most recent call last):
File "/root/music-service/task/task_music.py", line 41, in process status, msg, data = main_run(current_task)
File "/root/music-service/main.py", line 133, in main_run music_info1 = generate_music(pipe, task_info, param_info)
File "/root/music-service/main.py", line 50, in generate_music model_demo(
File "/root/music-service/src/heartlib/pipelines/music_generation.py", line 349, in call model_outputs = self._forward(model_inputs, **forward_kwargs)
File "/root/music-service/src/heartlib/pipelines/music_generation.py", line 275, in _forward prompt_tokens = model_inputs["tokens"].to(self.mula_device)

RuntimeError: CUDA error: device-side assert triggered

What caused this error? How to fix？

[MayVid6]

I keep running in a python thread to generate music. After running for a while, it will appear.