PyTorch DTensor #15
Description
PyTorch now has native support for distributed tensor, might be a better way to do TP than megatron's MPU.
{{ message }}
PyTorch now has native support for distributed tensor, might be a better way to do TP than megatron's MPU.