All outputs are 20.0s-21.0s: !!!!!!!!!!!!!!!! 21.0s-22.0s: !!!!!!!!!!!!!!!!

[FurkanGozukara]

I just locally installed and all outputs are

How can I debug the reason?

CMD has no errors

[FurkanGozukara]

@chenjoya i hope you can help me to fix this issue. what could be cause? how to debug?

[chenjoya]

Its weird. I just clone and run that, no bugs:

I notice that you are running in windows OS. I also see another issue that related to windows OS #1

Could you please try hands-on inference first to see if its normal? https://github.com/showlab/livecc/blob/main/inference.md
Many thanks!

[FurkanGozukara]

@chenjoya thanks just tested

[chenjoya]

Thanks. Could you provide your python environment, pip list to me?

[FurkanGozukara]

Thanks. Could you provide your python environment, pip list to me?

sure @chenjoya

python 3.11.9

(venv) E:\LiveCC_v1\livecc>pip freeze
absl-py==2.2.2
accelerate==1.6.0
aiofiles==23.2.1
aiohappyeyeballs==2.6.1
aiohttp==3.11.18
aiosignal==1.3.2
albucore==0.0.23
albumentations==2.0.5
annotated-types==0.7.0
ansicon==1.89.0
anyio==4.9.0
attrs==25.3.0
av==12.0.0
beautifulsoup4==4.13.4
blessed==1.20.0
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
colorama==0.4.6
coloredlogs==15.0.1
contourpy==1.3.2
cycler==0.12.1
Cython==3.0.12
datasets==3.5.0
decord==0.6.0
deepspeed @ https://files.pythonhosted.org/packages/0b/78/9a87d137f83f53dd57f425ac174723c746c53029bf6cdce22eb9e805e3f0/deepspeed-0.16.5-cp311-cp311-win_amd64.whl#sha256=8a08ffb4710094b9897f71068179298794dce0886d051b302273a8ab14ba2dc0
dill==0.3.8
distro==1.9.0
easydict==1.13
einops==0.8.1
fastapi==0.115.12
ffmpy==0.5.0
filelock==3.13.1
flash-attn @ https://huggingface.co/MonsterMMORPG/SECourses_Premium_Flash_Attention/resolve/main/flash_attn-2.7.4.post1-cp311-cp311-win_amd64.whl#sha256=290e9cd5b5c5cefa324105183ea76bca37e81677c417ed7a92c4266235a97e64
flatbuffers==25.2.10
fonttools==4.57.0
frozenlist==1.6.0
fsspec==2024.6.1
gdown==5.2.0
gpustat==1.1.1
gradio==4.44.1
gradio_client==1.3.0
groovy==0.1.2
grpcio==1.71.0
h11==0.16.0
hf_transfer==0.1.9
hjson==3.1.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.30.2
humanfriendly==10.0
idna==3.10
imageio==2.37.0
importlib_resources==6.5.2
insightface==0.7.3
Jinja2==3.1.4
jinxed==1.3.0
jiter==0.9.0
joblib==1.4.2
kiwisolver==1.4.8
lazy_loader==0.4
liger-kernel @ https://huggingface.co/MonsterMMORPG/SECourses_Premium_Flash_Attention/resolve/main/liger_kernel-0.5.8-py3-none-any.whl#sha256=fdf15819ede04d05948a755a5f2329ace1bcd109d1b4b436311b3cdae9a6d046
livecc-utils==0.0.1
Markdown==3.8
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.10.1
mdurl==0.1.2
mpmath==1.3.0
msgpack==1.1.0
multidict==6.4.3
multiprocess==0.70.16
networkx==3.3
ninja==1.11.1.4
numpy==2.1.2
nvidia-ml-py==12.570.86
onnx==1.17.0
onnxruntime-gpu==1.21.1
openai==1.76.0
opencv-python==4.11.0.86
opencv-python-headless==4.11.0.86
orjson==3.10.16
packaging==25.0
pandas==2.2.3
peft==0.15.2
pillow==10.4.0
pillow_heif==0.22.0
prettytable==3.16.0
propcache==0.3.1
protobuf==6.30.2
psutil==7.0.0
py-cpuinfo==9.0.0
pyarrow==19.0.1
pydantic==2.10.6
pydantic_core==2.27.2
pydub==0.25.1
Pygments==2.19.1
pyparsing==3.2.3
pyreadline3==3.5.4
PySocks==1.7.1
python-dateutil==2.9.0.post0
python-multipart==0.0.20
python_speech_features==0.6
pytz==2025.2
PyYAML==6.0.2
qwen-vl-utils==0.0.11
regex==2024.11.6
requests==2.32.3
rich==14.0.0
ruff==0.11.7
safehttpx==0.1.6
safetensors==0.5.3
scikit-image==0.25.2
scikit-learn==1.6.1
scipy==1.15.2
semantic-version==2.10.0
sentencepiece==0.2.0
shellingham==1.5.4
simsimd==6.2.1
six==1.17.0
sniffio==1.3.1
soupsieve==2.7
starlette==0.46.2
stringzilla==3.12.5
sympy==1.13.3
tensorboard==2.19.0
tensorboard-data-server==0.7.2
threadpoolctl==3.6.0
tifffile==2025.3.30
timm==1.0.15
tokenizers==0.21.1
tomlkit==0.12.0
torch==2.7.0+cu128
torchaudio==2.7.0+cu128
torchvision==0.22.0+cu128
tqdm==4.67.1
transformers==4.51.3
triton-windows==3.3.0.post19
typer==0.15.2
typing-inspection==0.4.0
typing_extensions==4.12.2
tzdata==2025.2
urllib3==2.4.0
uvicorn==0.34.2
wavfile==4.7.2
wcwidth==0.2.13
websockets==12.0
Werkzeug==3.1.3
xxhash==3.5.0
yarl==1.20.0
yt-dlp==2025.3.31

(venv) E:\LiveCC_v1\livecc>

[chenjoya]

Hi! I guess it should be numpy==2.1.2 problem:

Despite this, I also upgrade some key packages following your environments:

torch==2.7.0
torchvision==0.22.0
transformers==4.51.3
flash-attn==2.7.4.post1
liger_kernel==0.5.8

Could you try to degrade numpy==1.24.4?

[FurkanGozukara]

@chenjoya thanks for checking. still same sadly. how can i debug?

it gives first sentence then the rest not

[chenjoya]

No worries, let's debug. Before you downgrade numpy, can it output the first sentence normally?

[FurkanGozukara]

No worries, let's debug. Before you downgrade numpy, can it output the first sentence normally?

yes both cases first sentence output

[chenjoya]

Ok, then I guess it should be the kv cache problem. Have you used the most recent code? I have fixed some bugs previously.

[FurkanGozukara]

Ok, then I guess it should be the kv cache problem. Have you used the most recent code? I have fixed some bugs previously.

thanks just tested after git pull

tested with numpy 2 and 1 both errors

tested on rtx 5090 and 3090 ti both same errors

first sentence parsed then it is error

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

If you get error during downloads with Hugging Face, edit this file with any text editor and set HF_HUB_ENABLE_HF_TRANSFER=0
Hugging Face cache folders set to: E:\LiveCC_v1\livecc\models
[2025-04-25 19:59:42,591] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
test.c
LINK : fatal error LNK1181: cannot open input file 'aio.lib'
test.c
LINK : fatal error LNK1181: cannot open input file 'cufile.lib'
W0425 19:59:45.739000 34096 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
livecc_utils.video_process_patch: FORCE_QWENVL_VIDEO_READER='decord+', FPS_MAX_FRAMES=480, VIDEO_MIN_PIXELS=78400, VIDEO_TOTAL_PIXELS=19267584
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:06<00:00,  1.55s/it]
* Running on local URL:  http://127.0.0.1:7860
* Running on public URL: https://7bcf1a6be055493541.gradio.live

This share link expires in 1 week. For free permanent hosting and GPU upgrades, run `gradio deploy` from the terminal in the working directory to deploy to Hugging Face Spaces (https://huggingface.co/spaces)
No query provided, use default_query=Please describe the video.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Traceback (most recent call last):
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\queueing.py", line 715, in process_events
    response = await route_utils.call_process_api(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\route_utils.py", line 322, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\blocks.py", line 2136, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\blocks.py", line 1674, in call_function
    prediction = await utils.async_iteration(iterator)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\utils.py", line 729, in async_iteration
    return await anext(iterator)
           ^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\utils.py", line 723, in __anext__
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\anyio\to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 2470, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\utils.py", line 706, in run_sync_iterator_async
    return next(iterator)
           ^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\gradio\utils.py", line 867, in gen_wrapper
    response = next(iterator)
               ^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\demo\app.py", line 120, in gr_for_streaming
    for (start_timestamp, stop_timestamp), response, state in gradio_backend(message=query, state=state, mode=mode, hf_spaces=hf_spaces):
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\torch\utils\_contextlib.py", line 57, in generator_context
    response = gen.send(request)
               ^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\demo\infer.py", line 156, in live_cc
    outputs = self.model.generate(
              ^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\utils.py", line 2465, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\utils.py", line 3422, in _sample
    while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\utils.py", line 2616, in _has_unfinished_sequences
    elif this_peer_finished:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[chenjoya]

I see ...

1. How much the GPU memory you have?
2. Could you try https://github.com/showlab/livecc/blob/main/inference.md ? It seems that Gradio async make some problems.
3. What if CUDA_LAUNCH_BLOCKING=1 python demo/app.py?

[FurkanGozukara]

I see ...

1. How much the GPU memory you have?
2. Could you try https://github.com/showlab/livecc/blob/main/inference.md ? It seems that Gradio async make some problems.
3. What if CUDA_LAUNCH_BLOCKING=1 python demo/app.py?

just tested again - i have rtx 5090 - 32 gb, rtx 3090 ti - 24 gb , it is fitting into VRAM

readme generates output but same issue, tested both numpy 2 and 1

first sentence generated rest is !!!!!!!

did you update readme code too? i see same code

(venv) E:\LiveCC_v1\livecc>python a.py
[2025-04-25 20:15:32,295] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
test.c
LINK : fatal error LNK1181: cannot open input file 'aio.lib'
test.c
LINK : fatal error LNK1181: cannot open input file 'cufile.lib'
W0425 20:15:35.769000 27524 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
livecc_utils.video_process_patch: FORCE_QWENVL_VIDEO_READER='decord+', FPS_MAX_FRAMES=480, VIDEO_MIN_PIXELS=78400, VIDEO_TOTAL_PIXELS=19267584
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.46s/it]
E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\configuration_utils.py:631: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.1` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\configuration_utils.py:636: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.001` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
E:\LiveCC_v1\livecc\venv\Lib\site-packages\transformers\generation\configuration_utils.py:653: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `1` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
0.0s-3.0s: Hey guys, welcome to another video. ...
3.0s-4.0s: !!!!!!!!!!!!!!!!!!!!
4.0s-5.0s: !!!!!!!!!!!!!!!!!!!!
5.0s-6.0s: !!!!!!!!!!!!!!!!!!!!
6.0s-7.0s: !!!!!!!!!!!!!!!!!!!!
7.0s-8.0s: !!!!!!!!!!!!!!!!!!!!
8.0s-9.0s: !!!!!!!!!!!!!!!!!!!!
9.0s-10.0s: !!!!!!!!!!!!!!!!!!!!
10.0s-11.0s: !!!!!!!!!!!!!!!!!!!!
11.0s-12.0s: !!!!!!!!!!!!!!!!!!!!
12.0s-13.0s: !!!!!!!!!!!!!!!!!!!!
13.0s-14.0s: !!!!!!!!!!!!!!!!!!!!
14.0s-15.0s: !!!!!!!!!!!!!!!!!!!!
15.0s-16.0s: !!!!!!!!!!!!!!!!!!!!
16.0s-17.0s: !!!!!!!!!!!!!!!!!!!!

[chenjoya]

Current readme has been correct. Let's do an experiment to see if it is the problem related to multi-turn. Could you try the multi-turn conversation part in https://github.com/showlab/livecc/blob/main/inference.md? Thanks!