Integration with PyTorch Lightning #179
Description
What would you like to be added: Integrate MS-AMP with PyTorch Lightning
Why is this needed: MS-AMP shows huge gains in throughput when training in FP8. That's very exciting. Adoption and experimentation would easier, if it could be used in common libraries like PyTorch Lightning. The DeepSpeed folks wrote an integration into PyTorch Lightning a little while back -- so as similar collaboration would be awesome to see.
Without this feature, how does current msamp work：You can use DeepSpeed for DDP training.
Components that may involve changes: Not sure
Brief description of your proposal if any: NA