This function has the if statement `if (train_dataloader or val_dataloaders) and datamodule:`.
The issue is similar to that in https://github.com/PyTorchLightning/pytorch-lightning/pull/1560. The problem is that the `if(dl)` translates to `if(bool(dl))`, but there's no dataloader.__bool__ so bool() uses dataloader.__len__ > 0. But... dataloader.__len__ uses IterableDataset.__len__ for IterableDatasets for which __len__ is undefined.
The fix is also the same, the `if dl` should be replaced by `if dl is not None`.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix missing return statement. Do not normalize remote paths
* Update pytorch_lightning/utilities/cloud_io.py
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* Add some documentation that we now support s3 and hdfs paths
* suggestion from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add initial tracking of states in Trainer.
* Add INTERRUPTED state, improve tests, move state switching from callback to a trainer.
* Move part of a trainer state switching to a decorator.
* Add documentation.
* Fix docs, rename state enum, restore state to previous on exit if None, add tests for decorator only.
* Fix callback typing.
Co-authored-by: William Falcon <waf2107@columbia.edu>
commit 29fb0506cd38a15c359e369cc8bc4435916b0c78
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 19:35:30 2020 +0000
fix checking for version for docs to build
commit 467fd640db02275972c7111af031c86bb59333e9
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 18:56:05 2020 +0000
remove no local test
commit a7cc9f88de00feec1a5406874d05313c42bd004c
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 18:46:44 2020 +0000
fix
commit 3fdbb729da79ae9348c83410a138666bad467951
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 18:23:30 2020 +0000
revert requirements
commit 9b8686bd83e2bc243cf329e26f1c667c6949cf67
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 18:16:42 2020 +0000
make it a fixture
commit eec74953d24c8b25268d3b6dde3cc4affdd5cb8f
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 18:01:32 2020 +0000
fix up the testing
commit 896d94a0e60083d52c81db2a036b7f1e015cad11
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 17:47:28 2020 +0000
fix some tests
commit 6d22bde19767bf2b71dfd44839b01efdf6888f83
Merge: 6175d4e2 6ebe0d72
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Sat Aug 8 10:20:47 2020 +0000
Merge remote-tracking branch 'origin/master' into tb_use_gfile
commit 6175d4e26b15a43c412c26d501762cd0b570616a
Author: Brendan Fahy <bmfahy@gmail.com>
Date: Fri Aug 7 10:16:36 2020 +0000
Use tensorboard.compat.gfile to support remote writing
* Add support to Tensorboard logger for OmegaConf hparams
Address https://github.com/PyTorchLightning/pytorch-lightning/issues/2844
We check if we can import omegaconf, and if the hparams are omegaconf instances. if so, we use OmegaConf.merge to preserve the typing, such that saving hparams to yaml actually triggers the OmegaConf branch
* avalaible
* chlog
* test
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
* Override the default gather method to support scalars
* add computing average of a list
* bug: change if to elif
* add some tests
* change style
* change documentation
* use apply_to_collection in DP gather
* use apply_to_collection in DP gather
* fix warning msg
* override gather method in DP
* add tests for python scalars
* add python scalars to docstring
* Update message
* override gather method in DP
* formatting
* chlog
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
* Fix shuffle for distributed sampler
* add test
* test
* chlog
* update test
* update test
* update test
* assertions via callback
* define callback outside for pickling
* skip ddp test on windows
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* pt 1.6
* don't use the new zipfile serialization for now
* quick flake8 fixes
* remove unnecessary f
* coalesce strings
* remove comma
* remove extra commas
* Apply suggestions from code review
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
* set _use_new_zipfile_serialization to False only for pytorch 1.6.0
* remove unnecessary comments
* flake8 fixes
* use pkg_resources instead of packaging
* readme
* format
* version
* chlog
Co-authored-by: Peter Yu <peter@asapp.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
The speed up is achieved by:
- Moving the "where" out of the loop (and replacing with min for simplicity).
- Replacing manual sum and pow with torch.norm. Even though this results
in unnessecary computation (computing pow(root)) this is still a lot
faster.
- Preallocating the output gives a slight speed up.
Note that calling .to for all parameters results in a small speed
penalty (~4 ms in my case) but allows parameters on different devices.
Overall this reduces the time used for gradient clipping from 206ms to
74 ms for my model (Resnet50 + few additional vars, all vars on GPU).
* Fix fast_dev_run to run for all val_dataloaders
* fast_dev_run check
* changelog
* explicit
* limit_batches with fast_dev_run in init
* add test
* whitespace and comment fix
* comment and assertion
* added tests
* Fix fast_dev_run to run for all val_dataloaders
* fast_dev_run check
* changelog
* explicit
* limit_batches with fast_dev_run in init
* add test
* whitespace and comment fix
* comment and assertion
* added tests
* added tests
* added tests
* added tests
* update rtol
* Revert "update rtol"
This reverts commit 4320329540.
* added tests
Co-authored-by: William Falcon <waf2107@columbia.edu>
* fix weights_save path and drop ckpt_path
* add tests
* unused import
* update docs
* changelog
* pep8
* fix horovod test
* make backward compatible
* perform same test for all loggers
* fix for when logger=False and weights_save_path is set
* update changelog
* update docs
* update tests
* do not set save dir dynamically
* remove duplicate test
* remove duplicated tests
* update tests
* update tests
* remove remaining ckpt_path references
* move defaults to init as suggested by @Borda
* test deprecation
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* reactor into gpu accelerator
* 🎨 warn instead of error out on loaders
* 🐛 test misconfiguration should still fail
* 🚧 .
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
* updated docs with new result obj
Co-authored-by: William Falcon <waf2107@columbia.edu>
* fix setup call while testing
* changelog
* drop if condition
* add test to check setup call
* flake8
* update test to check model stage
Co-authored-by: William Falcon <waf2107@columbia.edu>
* Horovod: Adjust base LR used by schedulers to match that of the optimizer after scaling by number of workers
* Added unit test
* Removed debug statements
* Updated changelog
* Apply suggestions from code review
Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* r
* r
* r
* patched optimizer closure with sr
* patched optimizer closure with sr
* patched optimizer closure with sr
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added train step structured result
* added autoreduce for train step
* added auto reduce on train
* added auto reduce on train
* added auto reduce on train
* added auto reduce on train
* added auto reduce on train
* added auto reduce on train
* added hooks
* added hooks
* added hooks
* added hooks
* added hooks
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* cache
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* Update pytorch_lightning/callbacks/early_stopping.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update pytorch_lightning/callbacks/early_stopping.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update pytorch_lightning/callbacks/early_stopping.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update pytorch_lightning/callbacks/model_checkpoint.py
* Update pytorch_lightning/core/step_result.py
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* Apply suggestions from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
* simple
* finished tests for structured results on train epoch
* simple
* simple
* revert
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* Update tests/base/deterministic_model.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* finished tests for structured results on train epoch
* docstring typos
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* finished tests for structured results on train epoch
* Update pytorch_lightning/core/step_result.py
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* Update pytorch_lightning/overrides/data_parallel.py
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
* Fix local rank zero casting
The environment variable 'LOCAL_RANK' can be a string, causing the `if rank_zero_only.rank == 0` check to fail
* Update distributed.py
address comment
* add tests for single scalar return from training
* add tests for single scalar return from training
* add tests for single scalar return from training
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* fixing val step only
* add tests for single scalar return from training
* add tests for single scalar return from training
* add tests for single scalar return from training
* add tests for single scalar return from training
* add tests for single scalar return from training
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* remove grad scaling tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* fix deprecation warnings
* added base tests for tpu
* added base tests for tpu
* Update pytorch_lightning/trainer/trainer.py
Co-authored-by: Jeremy Jordan <13970565+jeremyjordan@users.noreply.github.com>
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
* added base tests for tpu
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jeremy Jordan <13970565+jeremyjordan@users.noreply.github.com>
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix tpu hang
* fix and test for ddp block logging rank > 0
* rename
* use the dummy logger
* dummy logger test
* set the logger in model
* decorator for rank zero experiment
* simplify check
* simplify
* fix problem with None in checkpoint path
* revert configure logger
* unused import
* offline
* try rank 0 decorator in checkpoint
* try fix test
* imgs
* add asserts to make sure log zero only saves checkpoints
* add asserts to make sure log zero only saves checkpoints
* add asserts to make sure log zero only saves checkpoints
* add asserts to make sure log zero only saves checkpoints
* add asserts to make sure log zero only saves checkpoints
* fix tpu tests
* fix tpu tests
Co-authored-by: William Falcon <waf2107@columbia.edu>
* Adding importing ipywidgets before importing tqdm.auto to make sure ipywidgets is installed.
* Updated CHANGELOG.md
* Updated ipywidgets importing checks to @awaelchli comments.
Co-authored-by: William Falcon <waf2107@columbia.edu>
* add state_dict for early stopping
* move best attr after monitor_op defined
* improve early stopping and model checkpoint callbacks
* fix formatting
* fix attr init order
* clean up setting of default_root_dir attr
* logger needs default root dir set first
* reorg trainer init
* remove direct references to checkpoint callback
* more fixes
* more bugfixes
* run callbacks at epoch end
* update tests to use on epoch end
* PR cleanup
* address failing tests
* refactor for homogeneity
* fix merge conflict
* separate tests
* tests for early stopping bug regressions
* small fixes
* revert model checkpoint change
* typo fix
* fix tests
* update train loop
* cannot pass an int as default_save_path
* refactor log message
* fix test case
* appease the linter
* fix some doctests
* move config to callback
* fixes from rebase
* fixes from rebase
* chlog
* docs
* reformat
* formatting
* fix
* fix
* fixes from rebase
* add new test for patience
* Update pytorch_lightning/callbacks/model_checkpoint.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update pytorch_lightning/callbacks/model_checkpoint.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update tests/callbacks/test_early_stopping.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* fix formatting
* remove enable_early_stop attribute
* add state_dict for early stopping
* move best attr after monitor_op defined
* improve early stopping and model checkpoint callbacks
* fix formatting
* fix attr init order
* clean up setting of default_root_dir attr
* logger needs default root dir set first
* reorg trainer init
* remove direct references to checkpoint callback
* more fixes
* more bugfixes
* run callbacks at epoch end
* update tests to use on epoch end
* PR cleanup
* address failing tests
* refactor for homogeneity
* fix merge conflict
* separate tests
* tests for early stopping bug regressions
* small fixes
* revert model checkpoint change
* typo fix
* fix tests
* update train loop
* fix test case
* appease the linter
* fix some doctests
* move config to callback
* fixes from rebase
* fixes from rebase
* chlog
* docs
* reformat
* formatting
* fix
* fix
* fixes from rebase
* add new test for patience
* Update pytorch_lightning/callbacks/model_checkpoint.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update pytorch_lightning/callbacks/model_checkpoint.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update tests/callbacks/test_early_stopping.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* fix formatting
* remove enable_early_stop attribute
* fix test with new epoch indexing
* fix progress bar totals
* fix off by one error (see #2289) epoch starts at 0 now
* added missing imports
* fix hpc_save folderpath
* fix formatting
* fix tests
* small fixes from a rebase
* fix
* tmpdir
* tmpdir
* tmpdir
* wandb
* fix merge conflict
* add back evaluation after training
* test_resume_early_stopping_from_checkpoint TODO
* undo the horovod check
* update changelog
* remove a duplicate test from merge error
* try fix dp_resume test
* add the logger fix from master
* try remove default_root_dir
* try mocking numpy
* try import numpy in docs test
* fix wandb test
* pep 8 fix
* skip if no amp
* dont mock when doctesting
* install extra
* fix the resume ES test
* undo conf.py changes
* revert remove comet pickle from test
* Update CHANGELOG.md
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update weights_loading.rst
* Update weights_loading.rst
* Update weights_loading.rst
* renamed flag
* renamed flag
* revert the None check in logger experiment name/version
* add the old comments
* _experiment
* test chckpointing on DDP
* skip the ddp test on windows
* cloudpickle
* renamed flag
* renamed flag
* parentheses for clarity
* apply suggestion max epochs
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jeremy Jordan <jtjordan@ncsu.edu>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jeremy Jordan <13970565+jeremyjordan@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: William Falcon <waf2107@columbia.edu>
* no cov
* no cov
* ReduceOp
* group
* reduce_op.sum
* Update sklearns.py
* formatting
* horovod
* Apply suggestions from code review
* horovod
* horovod
* horovod
* horovod
* ci
* print
* ci
* timeout
* timeout
* time
* fix
* distributed cpu
* pipes
* time
* cpu
* spawn
* spawn
* spawn
* tp
* separate
* os
* os
* npm
* Fix load_from_checkpoint() not working with URL on Windows
* Update CHANGELOG
* Update CHANGELOG.md
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
* Apply suggestions from code review
* fix
* fix meta tags creating empty lines
* pyright
* node
* fix httpserver address
* drop tutils.default_trainer_options
* imports
* Better fix for load_from_checkpoint() not working with absolute path on Windows (#2294)
* Fix load_from_checkpoint() not working with URL on Windows
* Update CHANGELOG
* Update CHANGELOG.md
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
* drop duplicate
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: airium <airium@outlook.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: AIRIUM <38249940+airium@users.noreply.github.com>
* added tpu params test
* added tests
* removed xla imports
* added test cases for TPU
* fix pep 8 issues
* refactorings and comments
* add message to MisconfigurationException
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* test if device is set correctly
* added TPU device check
removed mark.spawn
* removed device selection
* remove xla_device call
* readded spawn due to test failures
* add TODO for tpu check
* Apply suggestions from code review
* Apply suggestions from code review
* flake8
* added tpu args to cli tests
* added support for tpu_core selection via cli
* fixed flake formatting
* replaced default_save_path with default_root_dir
* added check for data type for tpu_cores
* fixed flake indent
* protected
* protected
* added tpu params test
* added tests
* removed xla imports
* test if device is set correctly
* added support for tpu_core selection via cli
* replaced default_save_path with default_root_dir
* added check for data type for tpu_cores
* chlog
* fixed tpu cores error
* rebased with latest changes
* flake fix
* Update pytorch_lightning/trainer/distrib_parts.py
added suggesstion
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
* Init fix num_batches
* Fix num_batches in case of multiple dataloaders
* Apply suggestions from code review
* Changes based on suggestions
* Flake8
* Add test to check num_batches
* generalize dataloader percent check test
* fix formatting
* remove hparams
* tests
* CHANGELOG
* Update CHANGELOG.md
* max_batches can be int
* conflict and rebase
* add back the test
fix
fix message
0.0 works
Revert "fix message"
This reverts commit 839cacf8b8610f4e697e654ef6f3d2501bf23984.
* update changelog
* Update CHANGELOG.md
* Fix num batches in case of multiple dataloaders and percent_check (#1920)
* git conflict
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* missing union
* doc update suggestion by @rohitgr7
* extend test
* changelog
* docs add note about multiple loaders
* update changelog
* remove unused variable
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Revert "deprecated: epoch indexing from 1 (#2206)"
This reverts commit f94b919b
* chlog
* grad index
* Apply suggestions from code review
* tests
* fix
* test
* deal with NotImplementedError raised by torchtext
* deal with NotImplementedError raised by torchtext
* Added tests for dataloader which raise NotImplementedError in __len__()
* Fixed some typos
Co-authored-by: Thomas Schaaf <tschaaf@cs.cmu.edu>
* move backward
* refactor backward to remove 16 bit from user override
* refactor backward to remove 16 bit from user override
* Update pytorch_lightning/core/hooks.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
* init the port using a seed that matches process id for ddp
Co-authored-by: Zhaofeng Wu <zfw7@cs.washington.edu>
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* drop train_percent_check
* chlog
* deprecated
* deprecated
* deprecated
* tests
* tests
* Apply suggestions from code review
* tests
* hydra support
* tests
* hydra support
* hydra support
* hydra support
* tests
* typo
* typo
* Update test_dataloaders.py
* docs
* docs
* docs
* docs
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* fixed percent check for val/test
* fixed percent check for val/test
* fixed percent check for val/test
* fixed percent check for val/test
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* overfit_pct now uses train loaders for val and test and does not shuffle
* add on fit_start on fit_end hooks
* add on fit_start on fit_end hooks
* add on fit_start on fit_end hooks
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* First attempt at auto-moving data for inference
* Correct my copypaste errors
* Correct for if device is CPU
* Get rid of the WIP code I accidentally added
* Add tests
* Make tests more foolproof
* Make sure we stick with pep8 formatting
* Clarify docs a little
* Apply suggestions from code review
* Get everything working again hopefully
* refactor and added hook
variant a
variant b
add test
revert rename
add changelog
docs
* move changelog entry to top
* Move data transfer to utilities
* Add back in warnings for autotransfer
* Get rid of the test code I ended up accidentally commiting again
* Add docs any changelog
* Correct PR number in Changelog
* Correct changelog
* Update data.py
* Update test_cpu.py
* make a decorator
* type hint
* changelog
* changelog
* remove old function
* import
* test for decorator
* fix test
* remove old test
* doctest
* apply decorator directly
* convert doctest to code block
* prevent side effects in tests
* fix merge
* update forward docs
* update docs
* added docs in section "deployment / prediction"
* update changelog
Co-authored-by: Hengjian Jia <henryjia18@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: William Falcon <waf2107@columbia.edu>
* Add ckpt_path option to LightningModule.test()
If ckpt_path is "best" (default), it loads the best weights saved by ModelCheckpoint for the test loop.
If ckpt_path is a path to a checkpoint file, it loads the weights from the file for the test loop.
If ckpt_path is None, it uses the weights from the end of training for the test loop.
If model parameter is set, ckpt_path is ignored.
* Update test_set.rst
Co-authored-by: William Falcon <waf2107@columbia.edu>
* log row might be a bottleneck depending on network. 50 unblocks this and is small enough for small datasets
* log row might be a bottleneck depending on network. 50 unblocks this and is small enough for small datasets
* past checkpoints
* omegaConf save
* enforce type
* resolve=True
Co-authored-by: Omry Yadan <omry@fb.com>
* test omegaconf
* tests
* test past
Co-authored-by: Omry Yadan <omry@fb.com>
* Fix pyright member access errors in training module
* Fix Trainer instantiation error due to inheritence order
* Add GH workflow for pyright
* Fix more pyright errors in trainer module
* Add pyrightconfig and setup python environment in type-check workflow
* Exclude pyrightconfig.json
* suggestions
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
* allow loading checkpoints from urls
* tmpdir_server fixture
* test cases for loading checkpoints from url
* dir => root_dir
* default map_location to None
* test case for resume_from_checkpoint
* changelog
* doc update
* monkeypatch TORCH_HOME to avoid caching
* Use a threading server with random ports so that it is easier to clean up
* test fixes
* pep8 fix
* ThreadingHTTPServer support in 3.6
* pep8 fix
* fix changelog
* separate tests for urls
* typo
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* training batch clean up
* adding spawn
* adding spawn
* adding spawn
* adding spawn
* adding spawn
* adding spawn
* adding spawn
* adding spawn
* refactor and added hook
variant a
variant b
add test
revert rename
add changelog
docs
* resolve merge duplication
* overridden typo
* fix test
* tpu id
* raise if TPU not available
* re-use apply_to_collection function for parsing collections
* comment
* make utility function available to user
* documentation
* move changelog entry to top
* fix tpu transfer call
* fix call
* remove hardcoded string
* improve test
* call model hook by default
* Apply suggestions from code review
* rename utility function
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Raise an error when lightning replaces an existing sampler
Currently, Trainer replaces the existing sampler with DistributedSampler
if running distributing training and `replace_sampler_ddp=True` (default
behaviour). If a user has configured an existing sampler, this would
lead to widely different results if running a distributed vs
non-distributed training.
This PR fixes this by raising an Error if user has configured a sampler
and uses `replace_sampler_ddp=True`. The recommended behavior from now
on is to either remove the sampler or set `replace_sampler_ddp=False`
* Fix tests
* Simpler fix
* Fix tests
* Make inner method protected
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* fix grad norm formula
* grad-norm tracker test
* fixed seed and explicit rtol in grad norm tracking test
* a docstring for grad-norms and forced cast to float of norm_type
* support for inf-norm
* renamed the grad norm test
* docs
* fixed language in docstring
* Apply suggestions from code review
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* use parallel loader
* Revert "use parallel loader"
This reverts commit ed6e7583
* select tpu id for pl
* condition if tpu_id is None
* added info to changelog
* Revert "condition if tpu_id is None"
This reverts commit 1fb6e586
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* replace ddp spawn with subprocess
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* hot fix
* Add an additional attribute to ModelCheckpoint to keep track of the best model's path
Currently, only the best metric value is directly tracked. This new attribute will help in uses cases where the trained model needs to be used or tracked right after training.
* Add small description and usage example to docs
* Fix PEP8 issues
* Fix doctest example
* Fix expected output in doctest
* Apply suggestions from code review
* Show example as code block instead of doctest
* Apply suggestions from code review
* Update CHANGELOG.md
* Rename `ModelCheckpoint.best` to `ModelCheckpoint.best_model_score`
Also rename `ModelCheckpoint.best_model` (added in this PR) to `ModelCheckpoint.best_model_path`, for consistency, and `kth_best_model` to `kth_best_model_path`.
* Update pytorch_lightning/trainer/training_io.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Add warning when loading checkpoint from an old version
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* filter valid args
* error on unknown manual args
* added test
* changelog
* update docs and doctest
* simplify
* doctest
* doctest
* doctest
* better test with mock check for init call
* fstring
* extend test
* skip test on 3.6 not working
Co-authored-by: William Falcon <waf2107@columbia.edu>
* FixesPyTorchLightning/pytorch-lightning#490
`EarlyStopping` should check the metric of interest `on_validation_end` rather than `on_epoch_end`.
In a normal scenario, this does not cause a problem, but in combination with `check_val_every_n_epoch>1` in the `Trainer` it results in a warning or in a `RuntimeError` depending on `strict`.
* Highlighted that ES callback runs on val epochs in docstring
* Updated EarlyStopping in rst doc
* Update early_stopping.py
* Update early_stopping.rst
* Update early_stopping.rst
* Update early_stopping.rst
* Update early_stopping.rst
* Apply suggestions from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* Update docs/source/early_stopping.rst
* fix doctest indentation warning
* Train loop calls early_stop.on_validation_end
* chlog
Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
* Allow dataloaders without sampler field present
Sometimes we have a custom dataloader that doesn't have a sampler, better to check that the field is there before reading it.
* chlog
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
* Add flag to `dump_checkpoint` for only including weights
`ModelCheckpoint` then passes `self.save_weights_only` to the save function.
* Fix tests and add changelog entry
* Add check and descriptive message when training state is restored from a weights only checkpoint
Also add a test for making sure `ModelCheckpoint.save_weights_only` works as expected.
* Fix weights-only test to properly match expected exception
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Fix test configuration check and testing
* Fix test configuration check and testing
* Remove check_testing_configuration during test
* Fix docstring
* fix function name
* remove conflicts
The intention of the code is to output a warning message when `hparams`
is null or not set. Instead the code now fatals when
`model.hparams = None`. Prevent that.
The changes are quite local and limited in nature -- viz., checking for
some indicator environment variables. We check for (SLURM_LOCALID,
NODE_RANK, GROUP_RANK) in order. If multiple are found set, a warning is
logged.
This patch also fixes a minor bug with comparing the `WORLD_SIZE`
environment variable. This can be a string type.
* Fixed typing annotation by adding boolean type. After that Profiler flag will be added to argparse.
* Updated CHANGELOG.md
* Updated git_init_arguments_and_types() to pass doctests.
* Added doctest example to add_argparse_parser()
* Option to provide seed to random generators to ensure reproducibility
I added small function in utilities which imports torch, numpy, python
random and sets seed for all of the libraries to ensure reproducibility
of results.
* Apply recommendations from core contributors on seeding
1. Moved the seeding code to another file
2. Make deterministic as a parameter for trainer class
3. Add assertions for seeding numpy
4. Added warnings
5. torch.manual_seed should be enough for seeding torch
* Revert "Apply recommendations from core contributors on seeding"
This reverts commit a213c8e6882eec8a9e7408b9418926d2db7c5461.
* Revert "Revert "Apply recommendations from core contributors on seeding""
This reverts commit 59b2da53c62878de7aab0aa3feb3115e105eea06.
* Change in test, for correct seeding
* Allow seed equal to 0
* Allow seed to be uint32.max
* Added deterministic to benchmarks
* Cuda manual seed as in benchmark seeding
* Seeding should be done before model initialization
* cuda manual_seed is not necessary
* Fixing seed test_cpu_lbfgs
On some seeds seems like lbfgs doesn't converge.
So I fixed the seed during testing.
* rebasing issue with old reproducibility.py
* Improved documentation and ability to seed before initializing Train
class
* Change in docs
* Removed seed from trainer, update for documentation
* Typo in the docs
* Added seed_everything to _all_
* Fixing old changes
* Model initialization should be earlier then Trainer
* Update pytorch_lightning/trainer/__init__.py
From Example to testcode
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Fixing according to the contributors suggestions
* Moving horovod deterministic to Trainer class
* deterministic flag affects horovod docs update
* Improved static typing
* Added deterministic to test runners of horovod
It is failing on some versions, not very predictable
* static seeds for horovod tests
* Change for reset_seed function in tests
* Seeding horovod using reset_seed from tutils
* Update pytorch_lightning/trainer/__init__.py
* chlog
* Update trainer.py
* change "testcode" to "Example" in trainer init documentation
* Update pytorch_lightning/trainer/seed.py, first line in comment
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: William Falcon <waf2107@columbia.edu>
* Join Horovod workers at the end of trainer.fit() to prevent race conditions following training
* flake8
* flake8
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
* update prog. bar metrics on train epoch end
* changelog
* wip test
* more thorough testing
* comments
* update docs
* move test
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
* removed if dl from _reset_eval_dataloader()
* changed to if dl != None to be more safe
* hints from pep8speaks
Co-authored-by: ybrovman <ybrovman@ebay.com>
* Fix Horovod backend to disable progress bar on all ranks except 0
* Add join barriers
* Added changelog
* Make protected and add verbosity
* Refactor to disable progress bar callback in train
* Removed vebose setting
* Add cache check for Horovod
* Test run again
* Updated comment
* Always skip cache for Horovod
* Only reinstall when necessary
* Added separate step
* Fixed spacing
* Skip Python 3.8
* params
* drop acc
* Fix Horovod distributed backend to set the root_gpu
* Fixed test
* Fixed tests
* Fixed lint
* Set root_gpu during initialization
* chlog
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
* diable val and test shuffling
* diable val and test shuffling
* diable val and test shuffling
* diable val and test shuffling
* log
* condition
* shuffle
* refactor
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
William Falcon