Predictions during training are zero #8
Description
In both Megatron and Megatron LoRA my gradient norm during training is 0, all predictions are zeros. Maybe some problem with weights initialization or gradient accumulation?
{{ message }}
In both Megatron and Megatron LoRA my gradient norm during training is 0, all predictions are zeros. Maybe some problem with weights initialization or gradient accumulation?