Multi gpu training

kaoutar55
Thu, 25 Feb 2021 21:15:36 +0000

It seems that the hugging face implementation still uses nn.DataParallel for one node multi-gpu training.
In the pytorch documentation page, it clearly states that " It is recommended to use DistributedDataParallel instead of DataParallel to do multi-GPU training, even if there is only a single node. Could you please clarify if my understanding is correct? and if your training support DistributedDataParallel for one node with multiple GPUs.