Distribute Training

• Training Loss
• Get Local Rank
• Worker in DataLoader

Collective Communication(Gather, All-Gather, Reduce-Dict)

• Check-Dist
• Gather
• All-Gather
• Reduce-Dict

Data Loader in CIFAR-10

• Dataset for CIFAR-10
• Iteration Sampler
• data builder
• loader Test