AssertionError: no_sync context manager is incompatible with gradient partitioning logic of ZeRO stage 2

[yunqingliu1996]

It seems related to using accelerator.accumulate() together with DeepSpeed’s ZeRO Stage 2 (also with 1).

might be due to version incompatibilities between accelerate and deepspeed?

Could you plz share the specific versions of accelerate, deepspeed?

[Hongyuan-Tao]

Hello! I’m really sorry for the inconvenience caused. I have checked and found that I used accelerate==1.3.0 and deepspeed==0.16.4 in one of my environments. You may refer to this for troubleshooting.

[yunqingliu1996]

Hello! I’m really sorry for the inconvenience caused. I have checked and found that I used accelerate==1.3.0 and deepspeed==0.16.4 in one of my environments. You may refer to this for troubleshooting.

thank you for your helpful input! however, this error persists... no_sync context manager is incompatible with gradient partitioning logic of ZeRO stage 2
And if revising the train_step by removing with self.accelerator.accumulate(model), training can start, but will terminate due to OOM if long data occurs...

[Hongyuan-Tao]

Sorry to hear the error persists—I'm not entirely sure what's causing it either, so you might need to search online for possible solutions. I'll continue investigating this issue.
As for the OOM issue, try reducing the max_length value in configs/experiment/distill_stage_mmMamba.yaml to lower peak GPU memory usage during training. This adjustment wouldn't significantly impact loss convergence.

[yunqingliu1996]

Sorry to hear the error persists—I'm not entirely sure what's causing it either, so you might need to search online for possible solutions. I'll continue investigating this issue. As for the OOM issue, try reducing the max_length value in configs/experiment/distill_stage_mmMamba.yaml to lower peak GPU memory usage during training. This adjustment wouldn't significantly impact loss convergence.

really appreciate that! and reducing the max_length is definitely helpful! big thx!

[rodosingh]

Hello! I’m really sorry for the inconvenience caused. I have checked and found that I used accelerate==1.3.0 and deepspeed==0.16.4 in one of my environments. You may refer to this for troubleshooting.

thank you for your helpful input! however, this error persists... no_sync context manager is incompatible with gradient partitioning logic of ZeRO stage 2 And if revising the train_step by removing with self.accelerator.accumulate(model), training can start, but will terminate due to OOM if long data occurs...

Also having the similar issue. Any solution yet?
Found this: deepspeedai/DeepSpeed#6793 (comment)