HELP REQ - 'use_fast is unset' resulting in very slow loading

[ColumbusAI]

I have tried the Qwen2 VL 72B AWQ and Qwen2.5 VL 72B weights taken straight from the Qwen HF repos. I keep getting this warning and I have tried the environment variable, "HF_TOKENIZER_USE_FAST=1", but that does not fix the issue.

Where can I add the 'use_fast' setting to the source code in order to speed up loading?

2025-03-23 18:21:46 Using a slow image processor as use_fastis unset and a slow processor was saved with this model.use_fast=Truewill be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor withuse_fast=False.

[matatonic]

I'm not able to test at the moment, but it should be here:

openedai-vision/backend/qwen2_5_vl.py

Line 26 in 13a9c8d

self.processor = AutoProcessor.from_pretrained(model_id)

The line would be:

        self.processor = AutoProcessor.from_pretrained(model_id, use_fast=True)   

[matatonic]

Please let me know if this improves anything, I just saw this may be a transformers bug (fixed in the latest release):

huggingface/transformers#36890

[ColumbusAI]

sorry for the delay! will be testing this out tomorrow I've been out of town and get back in the morning. Thanks again!

[ColumbusAI]

@matatonic thanks again for the help. I can confirm that the fix you provided is working for fast tokenizers. Yet now there is a cuda error below:

openedai-vision  | 2025-04-07 01:58:51.157 | INFO     | __main__:<module>:161 - Loading VisionQnA[qwen2_5_vl] with /app/models
openedai-vision  | [2025-04-07 01:58:51,996] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
openedai-vision  | df: /app/.triton/autotune: No such file or directory
openedai-vision  | Traceback (most recent call last):
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1967, in _get_module
openedai-vision  |     return importlib.import_module("." + module_name, self.__name__)
openedai-vision  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
openedai-vision  |     return _bootstrap._gcd_import(name[level:], package, level)
openedai-vision  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
openedai-vision  |   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
openedai-vision  |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
openedai-vision  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 43, in <module>
openedai-vision  |     from ...modeling_utils import PreTrainedModel
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 158, in <module>
openedai-vision  |     import deepspeed
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/deepspeed/__init__.py", line 25, in <module>
openedai-vision  |     from . import ops
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/deepspeed/ops/__init__.py", line 15, in <module>
openedai-vision  |     from ..git_version_info import compatible_ops as __compatible_ops__
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/deepspeed/git_version_info.py", line 29, in <module>
openedai-vision  |     op_compatible = builder.is_compatible()
openedai-vision  |                     ^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/deepspeed/ops/op_builder/fp_quantizer.py", line 35, in is_compatible
openedai-vision  |     sys_cuda_major, _ = installed_cuda_version()
openedai-vision  |                         ^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 51, in installed_cuda_version
openedai-vision  |     raise MissingCUDAException("CUDA_HOME does not exist, unable to compile CUDA op(s)")
openedai-vision  | deepspeed.ops.op_builder.builder.MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s)
openedai-vision  |
openedai-vision  | The above exception was the direct cause of the following exception:
openedai-vision  |
openedai-vision  | Traceback (most recent call last):
openedai-vision  |   File "/app/vision.py", line 162, in <module>
openedai-vision  |     backend = importlib.import_module(f'backend.{args.backend}')
openedai-vision  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
openedai-vision  |     return _bootstrap._gcd_import(name[level:], package, level)
openedai-vision  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
openedai-vision  |   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
openedai-vision  |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
openedai-vision  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
openedai-vision  |   File "/app/backend/qwen2_5_vl.py", line 1, in <module>
openedai-vision  |     from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
openedai-vision  |   File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1956, in __getattr__
openedai-vision  |     value = getattr(module, name)
openedai-vision  |             ^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1955, in __getattr__
openedai-vision  |     module = self._get_module(self._class_to_module[name])
openedai-vision  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
openedai-vision  |   File "/usr/local/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1969, in _get_module
openedai-vision  |     raise RuntimeError(
openedai-vision  | RuntimeError: Failed to import transformers.models.qwen2_5_vl.modeling_qwen2_5_vl because of the following error (look up to see its traceback):
openedai-vision  | CUDA_HOME does not exist, unable to compile CUDA op(s)

[matatonic]

This is more complex, you're missing a pre-compiled package. I' not able to test anything like this for probably another week at least. If you can, try rebuilding the docker image with the changes rather than an install somewhere else. versions and dependencies are very complex and problematic with transformers projects.