Moe int8w8a8 support #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

HandH1998 merged 9 commits into dsv3-w8a8-channel-int8 from moe-int8w8a8-support

Feb 26, 2025

Collaborator

yych0745 commented Feb 20, 2025

Motivation

Modifications

Checklist

Format your code according to the Contributor Guide.
Add unit tests as outlined in the Contributor Guide.
Update documentation as needed, including docstrings or example tutorials.

yych0745 requested a review from HandH1998 as a code owner

February 20, 2025 09:27

yych0745 and others added 2 commits

February 20, 2025 18:12


          support moe int8-w8a8

bc8e18d

Co-authored-by: HandH1998 <1335248067@qq.com>
Co-authored-by: thomas-zhu-2006 <thomas-zhu-2006  thomas.zhu.2006@gmail.com >


          clean code

fb3f603

yych0745 force-pushed the moe-int8w8a8-support branch from befa2ec to fb3f603 Compare

February 20, 2025 10:15

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/layer.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/models/deepseek_v2.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/models/deepseek_v2.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/models/deepseek_v2.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/test/test_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/test/test_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/test/test_int8.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved


          clean code

75d6b61

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py Outdated Show resolved Hide resolved

HandH1998 reviewed

View reviewed changes

python/sglang/srt/layers/quantization/w8a8_int8.py Outdated Show resolved Hide resolved

yych0745 and others added 5 commits

February 25, 2025 17:42


          clean code

452cfbd

Co-authored-by: sleepcoo <sleepcoo@gmail.com>
Co-authored-by: b0urnee <2769086541@qq.com>


          fix bug

385eb0a


          resolve conflicts

67da0d5


          resolve confict

6929fdd


          adjust code

Co-authored-by: yych0745 <1398089567@qq.com>
Co-authored-by: sleepcoo <sleepcoo@gmail.com>


          fused_moe kernel fall back to fp32 accumulator to reduce precision loss

b39b462

Co-authored-by: yych0745 <1398089567@qq.com>
Co-authored-by: sleepcoo <sleepcoo@gmail.com>

HandH1998 approved these changes

View reviewed changes

HandH1998 merged this pull request into dsv3-w8a8-channel-int8

1 check passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet