Segmentation fault when try to transfer megatron distcp checkpoint to hf format

[Xialinxuan]

Thanks for the great works!
I use megatron as the backend in verl, and I need to change the saved distcp checkpoint to hf format.
I referred the code you provided in the following link (dist2hf.py in my case):
volcengine/verl#3057 (comment)
But I got a segmentation fault:

python dist2hf.py --model_path model/QwQ-32B --dist_weight_path checkpoints/SFT-GRPO-lr-2e-6-tp-8-pp-1-batch-256/global_step_150/actor --save_path checkpoints/QwQ_SFT_GRPO_step150
 > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 32763876352
Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid: 625371) ====
 0 0x0000000000042520 __sigaction()  ???:0
=================================
Segmentation fault (core dumped)

In fact, I met the same issue when I use verl.model_merger, I can locate the problem happened when executing:

## verl/utils/megatron/dist_checkpointing.py
from megatron.core import dist_checkpointing, mpu
from megatron.core.dist_checkpointing.serialization import (
    get_default_load_sharded_strategy,
)
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
    FullyParallelSaveStrategyWrapper, 
)

def load_dist_checkpointing(sharded_state_dict, ckpt_dir):

    load_strategy = get_default_load_sharded_strategy(ckpt_dir)
    load_strategy = FullyParallelLoadStrategyWrapper(
        load_strategy, mpu.get_data_parallel_group(with_context_parallel=True)
    )
    #### Segmentation Fault happend when executing load####
    state_dict = dist_checkpointing.load(sharded_state_dict, ckpt_dir, sharded_strategy=load_strategy)

    return state_dict

Could you please provide any suggestions to fix this bug?
I tried megatron-core version 0.12.1, 0.12.2, 0.13.1. This error always occurs.

[ISEEKYAN]

Hi Xialinxuan, could you please share more traceback so I can locate where the error is from.
I suspect you are using 32B model, It might be because 1 GPU's memory is not enough, you could test with this example: https://github.com/ISEEKYAN/mbridge/blob/main/example/2.load_model_and_export_multiple_gpus.py

[Xialinxuan]

could you please share more traceback so I can locate where the error is from

I'm also frustrated, but that's all the error information I have. Do you have any idea about how to get more traceback?
In my environment, using verl model_merger also encounters this fault when instantiating the FullyParallelLoadStrategyWrapper class from Megatron, as I mentioned at first:

In fact, I met the same issue when I use verl.model_merger, I can locate the problem happened when executing

Thus I suspect this maybe a error about megatron. The command I use:

python -m verl.model_merger merge --backend megatron --local_dir checkpoints/SFT-GRPO-lr-2e-6-tp-8-pp-1-batch-256-n-16_2025-08-15-14-01-09/global_step_150/actor --target_dir checkpoints/QwQ_SFT_GRPO_step150/ --use_cpu_initialization

This error ocurs even when I don't use GPU

It might be because 1 GPU's memory is not enough

Additionally, I'm using an H20 GPU with a total memory of 96GB. I observed that when running dist2hf.py, the GPU memory usage is around 64GB before the segmentation fault occurs. If your code doesn't require additional memory later on, it probably isn't a memory issue causing this error.

[Xialinxuan]

I conducted further tests using verl model_merger. I apologize for the earlier incorrect information—I've now found that the segmentation fault actually occurs during the dist_checkpointing.load process.

[ISEEKYAN]

I would recommend you to use the same version of megatron when you saved that dist_checkpoint weights.

[Xialinxuan]

Thanks for your reply. But that's exactly what's confusing me—on my training environment, I encounter this segmentation fault whether I use verl.model_merger or your provided dist2hf.py. Since you've developed mbridge and have deeper knowledge of Megatron, I'm reaching out to ask if you might have any insights into this issue.
Finally I locate the error occurs when executing:

The code above is at the function in your/python/path/dist-packages/megatron/core/dist_checkpointing/validation.py L532