megatron-LM support?

[mhqmhy]

Great job！ I saw the example only FSDP backend, is megatron-LM backend supported (like TP, PP parameter segmentation)?

[MingjiHan99]

ByteCheckpoint supports ByteDance’s internal Megatron-LM, also known as MegaScale, for both model and distributed optimizer saving/loading checkpoint with arbitrary TP/PP/DP/EP configurations.

However, the overall solution is highly integrated with MegaScale—encompassing modifications and improvements to MegaScale as well as a highly customized checkpoint component in ByteCheckpoint—which cannot be directly used by open source users.

We are exploring better ways to benefit open source Megatron-LM users.

[xichengpro]

+1