Add Support for msamp FP4 Quantization and Unit Test #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

guoshzhao merged 10 commits into Azure:main from Mr-Philo:ruizhe/fp4

Dec 1, 2025

Contributor

Mr-Philo commented May 22, 2025 •

edited

Loading

This PR is for new features regarding to FP4 quantization for MS-AMP library.

Working items:

Custom FP4 quant function in CUDA
Custom Differentiable Gradient Estimation for weight update
Forward and Backward pass for simulated FP4 quantization
Unit Test for FP4 quantization


          add support for msamp fp4 quantization and unit test

d0361f1

Mr-Philo mentioned this pull request

Add Example Training Script in FP4 Format Azure/MS-AMP-Examples#26

Merged

yzygitzh reviewed

View reviewed changes

msamp/operators/fp4_quant/setup.py Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/operators/fp4_quant/quant.cu Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/common/dtype/floating.py Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/megatron/layers.py Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

Makefile Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/megatron/layers.py Outdated Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/operators/fp4_quant/fp4_quant.py Show resolved Hide resolved

yzygitzh reviewed

View reviewed changes

msamp/operators/fp4_quant/fp4_quant.py Outdated Show resolved Hide resolved

Mr-Philo added 5 commits

July 11, 2025 02:51


          Rename fp4_quant to fp4_quantize

70dd974


          revert change for msamp/common/dtype/floating.py

19406e1


          Rename FP4 based env variables

526b016


          Rename quantize_simulate_fp4_in_bf16 function

4d45ea1


          Remove unused lines and add end of new line

32d686b

yzygitzh reviewed

View reviewed changes

msamp/operators/fp4_quantize/quantize.cu Show resolved Hide resolved

Mr-Philo added 3 commits

July 11, 2025 03:25


          add MIT License

eb3825a


          Add doc for paper link

6faa426


          update spelling

ecc9fe4

Mr-Philo changed the title ~~[WIP] Add Support for msamp FP4 Quantization and Unit Test~~ Add Support for msamp FP4 Quantization and Unit Test


          Update ubuntu version from 20.04 to 22.04 in workflow/lint.yaml

0dcd7cf

guoshzhao approved these changes

View reviewed changes

Mr-Philo mentioned this pull request

fp4 is going to open? #204

Open

michaelroyzen commented Nov 29, 2025

Can this be merged @Mr-Philo @guoshzhao? It's been approved for months.

Contributor

guoshzhao commented Dec 1, 2025

Can this be merged @Mr-Philo @guoshzhao? It's been approved for months.

Will setup the runner for pipelines and merge soon.

guoshzhao merged commit 9746ea8 into Azure:main

5 checks passed

Contributor

guoshzhao commented Dec 2, 2025

Can this be merged @Mr-Philo @guoshzhao? It's been approved for months.

Have merged. @michaelroyzen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet