Fix error on loading audio of the input video, as described in issue #163.

README.md

@@ -96,16 +96,17 @@ The following checkpoints are the full weights (visual encoder + audio encoder +
 ## Usage
 #### Environment Preparation 
 
-First, install ffmpeg.
-```
-apt update
-apt install ffmpeg
-```
-Then, create a conda environment:
+First, create a conda environment:
 ```
 conda env create -f environment.yml
 conda activate videollama
 ```
+Second, install `ffmpeg` and `ffmpeg-python`.
+```
+apt update
+apt install ffmpeg
+pip install ffmpeg-python
+```
 
 
 ## Prerequisites

video_llama/conversation/conversation_video.py

@@ -10,6 +10,7 @@
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaTokenizer
 from transformers import StoppingCriteria, StoppingCriteriaList
+import ffmpeg
 
 import dataclasses
 from enum import auto, Enum
@@ -252,7 +253,9 @@ def upload_video(self, video_path, conv, img_list):
 
         try:
             audio_flag = 1
-            audio = load_and_transform_audio_data([video_path],"cpu",  clips_per_video=8)
+            audio_path = video_path[:-3] + 'wav'
+            ffmpeg.input(video_path).output(audio_path).run(overwrite_output=True, quiet=True)
+            audio = load_and_transform_audio_data([audio_path],"cpu",  clips_per_video=8)
             audio = audio.to(self.device)
         except :
             print('no audio is found')