DIFF by sagearc · Pull Request #1 · sagearc/vllm

sagearc · 2026-03-04T09:49:00Z

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Co-authored-by: HyunKyun Moon <mhg5303@gmail.com> Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Signed-off-by: hallerite <git@hallerite.com>

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

…llm-project#31025) Signed-off-by: Szymon Reginis <sreginis@habana.ai> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

… using prefix caching (vllm-project#35442) Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ct#35754) Signed-off-by: jiang1.li <jiang1.li@intel.com>

…#35604) Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

vllm-project#34307) Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>

…LA (vllm-project#34552) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>

Signed-off-by: Anshika Ojha <anshikao@nvidia.com> Co-authored-by: Anshika Ojha <anshikao@gb-nvl-059-compute09.nvidia.com>

Signed-off-by: Jason Ozuzu <jasonozuzu@cohere.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>

…lm-project#35882) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

Signed-off-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: Robert Shaw <rshaw@neuralmagic.com> Signed-off-by: Robert Shaw <robertgshaw2@gmail.com> Co-authored-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>

) Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

Signed-off-by: Micah Williamson <micah.williamson@amd.com>

…oject#35813) Signed-off-by: Bill Nell <bnell@redhat.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

…ect#35917) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>

…roject#35912) Signed-off-by: Amr Mahdi <amrmahdi@meta.com>

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>

Signed-off-by: Nick Hill <nickhill123@gmail.com>

…P mode (vllm-project#35916) Signed-off-by: Jaewon Lee <jaewon@meta.com>

…determinism (vllm-project#35710) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

Signed-off-by: xjx <493337577@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com>

…ject#33753) Signed-off-by: shen-shanshan <467638484@qq.com>

Signed-off-by: zixiao <shunli.dsl@alibaba-inc.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…llm-project#35711) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

Signed-off-by: charlifu <charlifu@amd.com>

…35835) Signed-off-by: ZhongsJie <zhongsjie@gmail.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

Signed-off-by: damaozi <1811866786@qq.com>

…ithout penalties (vllm-project#35654) Signed-off-by: kkt-cohere <komal@cohere.com>

…oject#35539) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com> Signed-off-by: Andrii Skliar <askliar@nvidia.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: Andrii <askliar@nvidia.com> Co-authored-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com> Co-authored-by: Andrii Skliar <askliar@oci-nrt-cs-001-vscode-01.cm.cluster> Co-authored-by: Andrii <askliar@nvidia.com> Co-authored-by: root <root@pool0-03748.cm.cluster> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: root <root@pool0-02416.cm.cluster> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com> Co-authored-by: root <root@pool0-04880.cm.cluster>

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

github-actions · 2026-03-04T09:49:09Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

sagearc and others added 30 commits February 24, 2026 22:18

decouple base engine client logic from engine client

33173b0

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

split openapi serving class

ef863e0

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

render client

cc71837

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

remove inheritance between engine client and render client

a762964

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

remove OpenAIServingInference

e331104

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Split EngineClient into RendererClient + EngineClient sibling ABCs

559ba40

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

AsyncRenderer

f38e0bc

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Merge branch 'main' into base-engine-client

6d46f31

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

fix after merge

4afc924

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

revert disable_frontend_multiprocessing

cb3d2a8

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

revert disable_frontend_multiprocessing flag in benchamrks

7d267c2

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

fix get_world_size

9fb01a7

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

merge main

0e8de93

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

asyncllm backward compatibility

f516e9b

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

render client optional arg in init_app_state

b976c3c

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

fix io_processor init

2adc96c

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

Add co-author

8dc756f

Co-authored-by: HyunKyun Moon <mhg5303@gmail.com> Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

build_async_clients_from_engine_args

1952ce9

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

add regression test (vllm-project#35834)

b8401cd

Signed-off-by: hallerite <git@hallerite.com>

decouple AsyncRenderer to a separate file

fd02984

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

import fix

8d3f4ee

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

[CI/Build][Intel] Add new performance benchmarks for Intel Gaudi 3 (v…

4beebfd

…llm-project#31025) Signed-off-by: Szymon Reginis <sreginis@habana.ai> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

[Perf] [Hybrid] Copy num_accepted_tokens in non-blocking way when not…

ad9d09e

… using prefix caching (vllm-project#35442) Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

[CI] And PPL test for Qwen3.5. (vllm-project#35853)

fd4a90f

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

[Bugfix] Avoid src/dst as None in irecv/isend_tensor_dict (vllm-proje…

440f0e7

…ct#35754) Signed-off-by: jiang1.li <jiang1.li@intel.com>

[Frontend][1/n] Improve pooling entrypoints | classify. (vllm-project…

ea46397

…#35604) Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[ROCm] [CI] Add new fusion test cases that are relevant to vLLM IR Ops (

fb7fdc4

vllm-project#34307) Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>

[BugFix] Add support for MTP num_speculative_tokens > 1 with sparse M…

28ef9ba

…LA (vllm-project#34552) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>

TRTLLM gen-full attn Test Coverage (vllm-project#34986)

e05cb3b

Signed-off-by: Anshika Ojha <anshikao@nvidia.com> Co-authored-by: Anshika Ojha <anshikao@gb-nvl-059-compute09.nvidia.com>

fix: Ensure invalid audio files return 400 error (vllm-project#34715)

ae88468

Signed-off-by: Jason Ozuzu <jasonozuzu@cohere.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>

MatthewBonanni and others added 27 commits March 3, 2026 09:26

[CI] Bump num_speculative_tokens to 3 in nightly DeepSeek tests (vl…

8e1fd5b

…lm-project#35882) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

[CI] Temporarily Disable Llama4 MoE Refactor Test (vllm-project#35870)

881a6b0

Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

[ROCm][Bugfix]: Disable AITER Triton ROPE by default (vllm-project#35601

3a8eef5

) Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

[ROCm][CI] Fix TP size issue for test_gpt_oss (vllm-project#35887)

e721300

Signed-off-by: Micah Williamson <micah.williamson@amd.com>

[Bugfix] Fix misnamed parameter in compressed_tensors_moe.py (vllm-pr…

a9b8b13

…oject#35813) Signed-off-by: Bill Nell <bnell@redhat.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

[Model Runner V2] Fix inputs_embeds=None bug for MM models (vllm-proj…

467886a

…ect#35917) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>

[CI/Build] Allow mounting AWS credentials for sccache S3 auth (vllm-p…

12b38c0

…roject#35912) Signed-off-by: Amr Mahdi <amrmahdi@meta.com>

[Model Runner V2] support dp & ep for spec decoding (vllm-project#35294)

97286a2

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>

[Core] Move save_tensorized_model logic to Worker (vllm-project#35825)

d15c3b9

Signed-off-by: Nick Hill <nickhill123@gmail.com>

[Bugfix] Fix coord_socket assertion in DPEngineCoreProc for offline D…

f22ff29

…P mode (vllm-project#35916) Signed-off-by: Jaewon Lee <jaewon@meta.com>

[ROCm][CI] Support async weight transfer example with platform-aware …

f7da9cd

…determinism (vllm-project#35710) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

Enable bnb for multiple indices weight (vllm-project#35838)

9a9d442

Signed-off-by: xjx <493337577@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Bugfix] Fix EVS implementation for Qwen3 VL (vllm-project#33607)

70c73df

Signed-off-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com>

[PluggableLayer][MM] Add PluggableLayer for RelPosAttention (vllm-pro…

77e6dcb

…ject#33753) Signed-off-by: shen-shanshan <467638484@qq.com>

[model] support FireRedASR2 (vllm-project#35727)

c1d9634

Signed-off-by: zixiao <shunli.dsl@alibaba-inc.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Chore] Remove debug code in model implementation (vllm-project#35883)

6e9f21e

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Refactor] Clean up processor kwargs extraction (vllm-project#35872)

e379396

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Guard mm_token_type_ids kwarg in get_mrope_input_positions (v…

edba150

…llm-project#35711) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[Rocm][CI] Fix ROCm LM Eval Large Models (8 Card) (vllm-project#35913)

3c85cd9

Signed-off-by: charlifu <charlifu@amd.com>

[BugFix] Support tool_choice=none in the Anthropic API (vllm-project#…

7cdba98

…35835) Signed-off-by: ZhongsJie <zhongsjie@gmail.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

[Bugfix] Improve engine ready timeout error message (vllm-project#35616)

097eb54

Signed-off-by: damaozi <1811866786@qq.com>

[cohere][fix][spec-decode]: fix crash when allowed_token_ids is set w…

9e0f44b

…ithout penalties (vllm-project#35654) Signed-off-by: kkt-cohere <komal@cohere.com>

Merge branch 'main' into base-engine-client

4bb967b

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

merge 34551 and resolve conflicts

8f6e033

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

make engine client opetional for OpenAIServing

0acf167

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

sagearc closed this Mar 4, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DIFF#1

DIFF#1
sagearc wants to merge 57 commits intovllm-online-fastapifrom
base-engine-client

sagearc commented Mar 4, 2026 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

sagearc commented Mar 4, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions bot commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

sagearc commented Mar 4, 2026 •

edited by github-actions bot

Loading