About experimental setup #2
Description
I trained with a batch size of 2048 for 32k steps, but the performance is still poor. The paper mentioned training on 80 A800 GPUs for two months. Approximately how many steps are needed to see significant improvements?