optimized dsv3.2 missing underscore#847
optimized dsv3.2 missing underscore#847gronsti-amd wants to merge 9 commits intoganyi/optimized_dsv3.2from
Conversation
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: ganyi <ygan@amd.com>
ROCMAiterMLASparseMetadataBuilder sets cudagraph_support, but the actual checked name is _cudagraph_support. This causes the default value AttentionCGSupport.NEVER to be used instead of the intended AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
|
It's critical that DS3.2 has full graphs on decode. Some perf numbers: Single decode step went down from 59ms to 29ms. This is pretty much 100% speedup in this particular case on this particular node (MI300X) with a slow CPU . There are some additional warnings in the logs so the real PR may need to retest and change things now that the graphs actually work, but they simply must work before it goes in as it's so critical for performance. |
e68d9a5 to
85ef10d
Compare
716f54f to
9edcf5e
Compare
5111929 to
ee0a366
Compare
ROCMAiterMLASparseMetadataBuildersetscudagraph_support, but the actual checked name is_cudagraph_support.This causes the default value
AttentionCGSupport.NEVERto be used instead of the intendedAttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODEPurpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.