riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-01 18:51:14 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Accumulate opt state dict on do_rank 0 (#11481)

Browse Source
This commit is contained in:
Sylvain Gugger 2021-05-03 13:18:27 -04:00 committed by GitHub
parent 1e8e06862f
commit f4c9a7e62e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
src/transformers/trainer.py
View File

@ -1420,6 +1420,7 @@ class Trainer:
xm.save(self.lr_scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt")) xm.save(self.lr_scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
reissue_pt_warnings(caught_warnings) reissue_pt_warnings(caught_warnings)
elif is_sagemaker_mp_enabled(): elif is_sagemaker_mp_enabled():
if smp.dp_rank() == 0:
# Consolidate the state dict on all processed of dp_rank 0 # Consolidate the state dict on all processed of dp_rank 0
opt_state_dict = self.optimizer.state_dict() opt_state_dict = self.optimizer.state_dict()
# Save it and the scheduler on the main process # Save it and the scheduler on the main process