run demo_video.py failed.

[yangyuya]

Thanks for your job！

I don't load whisper and set all ars empty, then I try to run demo_video.py just using upload_vid as follows：

video = "examples/videos/v_-EIsT868Trw.mp4"
text_input = "What is the woman doing?"
input_split = "0 3.1\n3 11.5\n11.5 24.2\n24.2 45"
chat_state = CONV_VISION_MS.copy()
upload_vid(video, text_input, chat_state, temperature=1.0, input_splits=input_split)

But I failed:
1、I get error:

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

2、If I set do_sample=False in "self.model.llama_model.generate"，the error above can be solved. But the "summary" will be:

examples/videos/v_-EIsT868Trw.mp4 <unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>

I can't solve this problem, could you help me test my own video without whisper？

[youthHan]

Hi so you want to run the demo without ASR and even without loading whisper model?

[youthHan]

Please change the demo.yaml config to disable ASR,

Shot2Story/code/lavis/projects/blip2/eval/demo.yaml

Line 20 in 7ddced2

asr_audio: True

to False and

Shot2Story/code/lavis/projects/blip2/eval/demo.yaml

Line 27 in 5bc6cf8

question_prompt: "The audio transcripts are: {asr}. Please describe the video in detail."

to Describe this video in detail.

[yangyuya]

Hi so you want to run the demo without ASR and even without loading whisper model?

Yes, because the videos in activitynet caption don't have ASR, and if I load whisper model I get oom on a 4090 24G GPU.

I have already change the demo.yaml config to disable ASR, then I still get the same problem above.

[youthHan]

@yangyuya Hi have you solved this issue on 4090 GPU? Would you mind providing more details of you environment, errors? I fail to find a 4090 GPU to test.

[lujuncong2000]

hi! can you solve this problem now? i have the same problem.

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
output = await app.get_blocks().process_api(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
result = await self.call_function(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/gradio/blocks.py", line 1025, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2364, in run_sync_in_worker_thread
return await future
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 864, in run
result = context.run(func, *args)
File "demo_video.py", line 117, in upload_vid
summary = chat.answer(conv=chat_state,
File "/hy-tmp/Shot2Story-temp/code/lavis/conversation/conversation.py", line 524, in answer
outputs = self.model.llama_model.generate(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/transformers/generation/utils.py", line 1485, in generate
return self.sample(
File "/usr/local/miniconda3/envs/shot2story/lib/python3.8/site-packages/transformers/generation/utils.py", line 2560, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0