vLLM not working with qwen3.5 27B

[surapuramakhil]

logs

(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=4161)     args.dispatch_function(args)
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 111, in cmd
(APIServer pid=4161)     uvloop.run(run_server(args))
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=4161)     return __asyncio.run(
(APIServer pid=4161)            ^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=4161)     return runner.run(main)
(APIServer pid=4161)            ^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=4161)     return self._loop.run_until_complete(task)
(APIServer pid=4161)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=4161)     return await main
(APIServer pid=4161)            ^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 457, in run_server
(APIServer pid=4161)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 476, in run_server_worker
(APIServer pid=4161)     async with build_async_engine_client(
(APIServer pid=4161)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=4161)     return await anext(self.gen)
(APIServer pid=4161)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 96, in build_async_engine_client
(APIServer pid=4161)     async with build_async_engine_client_from_engine_args(
(APIServer pid=4161)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=4161)     return await anext(self.gen)
(APIServer pid=4161)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 122, in build_async_engine_client_from_engine_args
(APIServer pid=4161)     vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=4161)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1410, in create_engine_config
(APIServer pid=4161)     model_config = self.create_model_config()
(APIServer pid=4161)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1264, in create_model_config
(APIServer pid=4161)     return ModelConfig(
(APIServer pid=4161)            ^^^^^^^^^^^^
(APIServer pid=4161)   File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
(APIServer pid=4161)     s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=4161) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=4161)   Value error, The checkpoint you are trying to load has model type `qwen3_5` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
(APIServer pid=4161) 
(APIServer pid=4161) You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git` [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
(APIServer pid=4161)     For further information visit https://errors.pydantic.dev/2.12/v/value_error

[model-ui] Waiting for API at http://localhost:18000 ...
[model-ui] Waiting for API at http://localhost:18000 ...
grep: write error: Broken pipe
Ray process detected - continuing
[model-ui] Waiting for API at http://localhost:18000 ...

(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]        █     █     █▄   ▄█
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.16.0
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-27B-GPTQ-Int4
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:287]
(APIServer pid=4317) INFO 03-06 01:53:38 [utils.py:223] non-default args: {'model_tag': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'api_server_count': 1, 'host': '127.0.0.1', 'port': 18000, 'model': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'max_model_len': 8192, 'enforce_eager': True, 'download_dir': '/workspace/models'}

(APIServer pid=4317) Traceback (most recent call last):
(APIServer pid=4317)   [... Previous trace repeating for PID 4317 ...]
(APIServer pid=4317) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=4317)   Value error, The checkpoint you are trying to load has model type `qwen3_5` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[model-ui] Waiting for API at http://localhost:18000 ...
grep: write error: Broken pipe
Ray process detected - continuing

(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]        █     █     █▄   ▄█
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.16.0
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-27B-GPTQ-Int4
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:287]
(APIServer pid=4472) INFO 03-06 01:53:49 [utils.py:223] non-default args: {'model_tag': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'api_server_count': 1, 'host': '127.0.0.1', 'port': 18000, 'model': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'max_model_len': 8192, 'enforce_eager': True, 'download_dir': '/workspace/models'}

(APIServer pid=4472) Traceback (most recent call last):
(APIServer pid=4472)   [... Previous trace repeating for PID 4472 ...]
(APIServer pid=4472) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=4472)   Value error, The checkpoint you are trying to load has model type `qwen3_5` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[model-ui] Waiting for API at http://localhost:18000 ...
grep: write error: Broken pipe
Ray process detected - continuing

[robballantyne]

Hi

You'll need to select a more recent vLLM version from the template editor. You'll also need an updated transformers version, but you can automate in the template directly with the PROVISIONING_PIP environment variable and the image will handle it.

Version tag vastai/vllm:v0.19.0-cuda-12.9

PROVISIONING_PIP : "transformers==5.5.4"

will work for this model