60850ef510
fix overfit_batch sampler replacement logic ( #10486 )
...
Co-authored-by: thomas chaton <thomas@grid.ai>
2021-11-15 22:31:45 +00:00
01cf7a2ac5
Deprecate `DistributedType` in favor of `StrategyType` ( #10505 )
2021-11-15 17:10:08 +00:00
8d810d6144
Enable distributed training with CombinedDataLoader and max_size_cycle ( #10374 )
...
* solve combinedloader
* update
* update changelog
* update on comments
* resolve iterable dataset support
* update test description
* update
* update on comments
* update
* Accelerator auto
* Address review
* Refactor
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-11-09 20:06:10 +00:00
f1882aa69f
[Fault Tolerance] Don't check the len of a dataset, but its instance. ( #10432 )
2021-11-09 20:21:53 +05:30
ce3e63262a
Fix failure when `DataLoader(batch_size=None)` is passed ( #10345 )
...
* add test, + add change to data loading batch sample method
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refactor and CHANGELOG
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-11-04 12:46:57 +01:00
f6ed0bd8ca
introduce has_len_all_ranks() to check the length of dataloader across ranks ( #9827 )
...
* introduce , udpate tests
* update CHANGELOG.md
* change staticmethod and hook attribute naming
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* remove non-essential comment
* fix merge error and comment format
* try to fix test_tpu.py failure
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update on comments
* chlog
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chlog
* update
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* try fix
* Revert back TPUSpawn changes
* Update test
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
Co-authored-by: Kaushik B <kaushikbokka@gmail.com>
2021-11-02 13:22:58 -04:00
9d136a9fc5
Lightning Lite core and tests ( #10175 )
2021-10-29 21:46:39 +00:00
1f7bd6650c
Mark accelerator connector as protected ( #10032 )
2021-10-25 19:24:54 +00:00
e94dcf6936
Mark `trainer.data_connector` as protected ( #10031 )
...
Co-authored-by: tchaton <thomas@grid.ai>
2021-10-25 12:29:09 +01:00
56bc55db71
Update strategy flag in docs ( #10000 )
...
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-10-20 21:02:53 +05:30
2c16f1d6b9
remove dataloader patching on the LightningModule ( #9764 )
...
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-10-20 15:23:20 +02:00
4dc32ad7db
Fix logic to check for spawn in worker_check ( #9902 )
...
* fix
* update tests
* chlog
* skip windows
2021-10-18 13:02:46 +00:00
19008ce98f
IPU hotfix for #9721 ( #9759 )
2021-09-29 15:36:39 +02:00
fa44dbcd9e
[Refactor] Simplify data loading logic around replacing sampler to prevent confusion ( #9721 )
...
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-09-28 17:04:02 +00:00
f74eb58493
remove `InternalDebugger` ( #9680 )
...
* wip
* reset _notebooks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* reset _notebooks
* testing with mock
* update test with mock
* update test
* update tests
* update test
* remove track_load_dataloader_calls
* update last test
* remove unused imports
* remove InternalDebugger
* update changelog
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-09-27 09:33:45 -04:00
15cd6ad45b
Call TrainingTypePlugin collective functions directly instead of going through the Accelerator ( #9677 )
...
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: ananthsub <ananth.subramaniam@gmail.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2021-09-27 14:52:57 +02:00
d67aff7494
remove `InternalDebugger.track_load_dataloader_call` ( #9675 )
...
* wip
* reset _notebooks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* reset _notebooks
* testing with mock
* update test with mock
* update test
* update tests
* update test
* remove track_load_dataloader_calls
* update last test
* remove unused imports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-24 15:37:36 +02:00
f239b96320
Fix `replace_sampler` missing the batch size under specific conditions ( #9367 )
2021-09-08 12:27:59 +02:00
6e124e7207
CI: precommit - docformatter ( #8584 )
...
* CI: precommit - docformatter
* fix deprecated
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-06 12:49:09 +00:00
e1442d247e
Always use `trainer.call_hook` ( #8498 )
2021-08-20 18:22:03 +02:00
522df2b89b
3/n integrate new LightningDataFetcher into loop ( #8953 )
...
Co-authored-by: tchaton <thomas@grid.ai>
Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com>
2021-08-17 21:42:22 +00:00
19136ac847
[Feat] 2/n Add Fault Tolerant Training to LightningFetcher ( #8891 )
...
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com>
2021-08-17 16:32:43 +00:00
93ab24d1ee
Replace DataLoader sampler once for IPUs ( #8858 )
2021-08-16 11:28:05 +02:00
a1264a6850
Automatic string fixes ( #8886 )
2021-08-13 14:28:14 +00:00
ed13040729
Connect the model to the training type plugin at the start of run ( #8536 )
2021-08-04 17:43:34 +02:00
567e905ead
update logic to inject FastForwardSampler / CaptureIterableDataset 2/n ( #8366 )
...
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-08-02 20:52:06 +00:00
a64cc37394
Replace `yapf` with `black` ( #7783 )
...
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-07-26 13:37:35 +02:00
6dbdf438e8
Support `DataLoader`s with missing arguments in `replace_sampler` ( #8519 )
...
* Support `DataLoader`s with missing arguments in `replace_sampler`
* Fix for multiprocessing context
* Fixes and test improvements
* Fixes and test improvements
* Fixes and test improvements
* Test any variadic name
* Update CHANGELOG
* Make sure extra attributes can be present
* Skip on old Windows
* Update pytorch_lightning/trainer/data_loading.py
* Update pytorch_lightning/trainer/data_loading.py
* Check is dataloader
* Typo
2021-07-26 10:04:21 +02:00
f7027a8701
Remove `torch >= 1.6` checks ( #8523 )
2021-07-23 04:03:20 +00:00
e345469a87
Delete legacy dataloader processing utility ( #8439 )
2021-07-19 11:25:04 +00:00
374fae59ef
[Feat] Add utilities for CombinedLoader state dict and dataloader state dict 1/n ( #8364 )
...
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-07-19 09:56:57 +00:00
cbf71d0a14
Remove unnecessary comprehension ( #8405 )
2021-07-19 08:30:24 +00:00
24db914093
Support state restoration of logged results 2/2( #7966 )
...
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
Co-authored-by: Ethan Harris <ewah1g13@soton.ac.uk>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2021-06-25 19:16:11 +00:00
971908a1aa
Loop Refactor 1/N - Training Loop ( #7871 )
...
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: tchaton <thomas@grid.ai>
Co-authored-by: Justus Schock <justus.schock@rwth-aachen.de>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
Co-authored-by: Ethan Harris <ewah1g13@soton.ac.uk>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
2021-06-15 12:55:06 +00:00
20f37b85b6
add warning when Trainer(log_every_n_steps) not well chosen ( #7734 )
...
* add warning
* update changelog
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* logger check
* add docstring for test
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2021-06-07 12:40:43 +00:00
6388c29e87
[IPU] Add reset dataloader hooks to training type plugin 3/n ( #7861 )
...
* Add hooks
* Add tests for hooks
* Add changelog
* Test changes, add typing
2021-06-07 10:37:09 +00:00
7e2f7e956b
fix: improve UserWarning message ( #7685 )
...
* fix: improve UserWarning message
when both overfit and training dtaloader shuffling are enabled
fixes issue: #7656
* chore: update changelog
* Polish userwarning msg in pytorch_lightning/trainer/data_loading.py
Co-authored-by: ananthsub <ananth.subramaniam@gmail.com>
* shuffling typo
* Update CHANGELOG.md
Co-authored-by: ananthsub <ananth.subramaniam@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-05-25 17:35:15 +00:00
6e6e29af49
remove trainer hidden state | sanity refactor [2 / n] ( #7507 )
2021-05-17 08:57:15 +01:00
8c0ea92af2
`TrainerState` refactor [5/5] ( #7173 )
...
* `TrainerState` refactor
* flake8
* Update finished check
* Test cleanup
* Fix tests
* Fixes
* Reorder
* flake8
* Update CHANGELOG
* Better docs
* Better docs
* Remove default
* Update tests
* Bad merge
2021-05-04 12:50:56 +02:00
b7a444883c
Remove model.trainer call inside of dataloading mixin ( #7317 )
...
* Update data_loading.py
* Update data_loading.py
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2021-05-03 13:53:54 -07:00
e147127c0e
[feat] Add better support for predict + ddp 2/3 ( #7215 )
...
* wip
* update
* update
* update
* update
* update
* typo
* update on comments
* update
* update
* update
* update
* update changelog
* update
* Fix merge
* Fix merge
* move code
* resolve test
* add extra test
* add an extra test
* update on comments
* add typing
* resolve flake8
* Refactor and Docs
* Fix tests
* Fix tests
* Fix tests
* Duplicate
* Fix tests
* resolve bug
* update
* update on comments
* update
* update changelog
* update
* update
* remove tpu
* resolve flake8
* update on comments
* update on comments
* update on comment
* resolve flake8
* add a cpu test for predict
* add None test
* update
* Update CHANGELOG.md
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
* resolve tests
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-04-27 08:46:45 -04:00
9beec26c3e
[bugfix] Add support for CombinedLoader in validation with ddp ( #7102 )
...
* add test
* add changelog
* resolve flake8
* remove print
2021-04-20 08:22:02 +00:00
60c1c8fe83
Auto-set `DataLoader.worker_init_fn` with `seed_everything` ( #6960 )
...
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: ananthsub <ananth.subramaniam@gmail.com>
2021-04-19 16:28:37 +02:00
e9fca760ac
Set `DistributedSampler` seed if `seed_everything` was called ( #7024 )
...
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-04-19 14:50:31 +01:00
f645df5e9a
Add typings for evaluation_loop.py and remove some dead code ( #7015 )
2021-04-15 07:36:04 +00:00
b85cfbe8f3
Run ddp_spawn dataloader checks on windows ( #6930 )
2021-04-09 19:06:47 +00:00
e35192dfcd
Update `DataLoader.persistent_workers` warnings in ddp_spawn ( #6762 )
...
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-04-09 11:38:13 +02:00
ac60536818
Follow E231 [flake8] ( #6110 )
...
* Remove E231 from ignore list
* Follow E231
* Update pytorch_lightning/trainer/data_loading.py
2021-03-24 12:50:50 +01:00
0995d30fab
Flash predict step ( #6577 )
...
* add predict_step
* Update predict_loop.py
* Update trainer.py
* Update trainer.py
* resolve bugs
* update
* update
* update
* resolve bug
* resolve some failing tests
* udpate tests
* update
* resolve tests
* add a test
* remove typo
* add a test for attachement
* update
* changed to on_train_dataloader
* remove __flash_special_attr__
* resolve tests
* update
* update
* update
* update on comments
* Update pytorch_lightning/trainer/data_loading.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2021-03-23 11:13:13 -04:00
dcd9dd8338
Update docs for limit_predict_batches ( #6507 )
...
* add docs and minor updates
* docs
* fraction
2021-03-14 09:09:58 +00:00
Rohit Gupta