Rooba
/
lightning
mirror of https://github.com/Lightning-AI/lightning.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Run on_validation_end only on main process in DDP (#1125) 

Co-authored-by: xingzhaolee <xingzhaolee@users.noreply.github.com>
This commit is contained in:
Xing Zhao LEE 2020-03-12 22:50:00 +08:00 committed by GitHub
parent b29613b758
commit b4d4e489bf
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
pytorch_lightning/callbacks/model_checkpoint.py
View File

@ -175,6 +175,10 @@ class ModelCheckpoint(Callback):
return filepath return filepath
def on_validation_end(self, trainer, pl_module): def on_validation_end(self, trainer, pl_module):
# only run on main process
if trainer.proc_rank != 0:
return
metrics = trainer.callback_metrics metrics = trainer.callback_metrics
epoch = trainer.current_epoch epoch = trainer.current_epoch
self.epochs_since_last_check += 1 self.epochs_since_last_check += 1