How many GPU-hours did training require?

[jmliu206]

Hi, great work! Could you share your training setup—how many GPUs did you use and how long did the training take?

[csuhan]

Hi, the 1.5B model takes around 1 week with 64 A100 GPUs, 7B takes around 10 days with 128 H100 GPUs. Since the current code is not well optimized for multinode, the traing speed is limited.