Commit Graph

49 Commits

Author SHA1 Message Date
Carlos Mocholí 6309a59c3c
Do not prefetch when possible (#12101) 2022-02-28 18:31:18 +00:00
Carlos Mocholí db1c709519
Clean loop fetching usage (#12103) 2022-02-28 10:51:33 +00:00
ananthsub 5d040e7fc6
Add support for `dataloader_iter` to validate and test steps (#11546)
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2022-02-24 18:01:35 +00:00
Carlos Mocholí 963adc7857
Small cleanup when dataloader states are saved (#11843) 2022-02-16 20:57:21 +00:00
Carlos Mocholí 1515ef90ee
Revert saving the dataloader and result collection by default (#11842) 2022-02-10 20:43:35 +01:00
Rohit Gupta 4d72110b51
Deprecate `on_batch_start/on_batch_end` callback hooks (#11577) 2022-02-03 19:51:56 +00:00
Carlos Mocholí 79a3ff690b
Add typing to data fetching (#11515) 2022-02-02 20:53:50 +01:00
Carlos Mocholí 623dc974f5
Construct the hook kwargs inside each loop (#11511) 2022-01-22 15:57:12 +00:00
Carlos Mocholí 9cf9ded73b
Simplify data fetching (#11466) 2022-01-17 14:46:55 +00:00
Danielle Pintz f95976d602
rename _call_ttp_hook to _call_strategy_hook (#11150) 2021-12-18 17:53:03 -08:00
Carlos Mocholí 75d96d9897
Reset the current progress tracking state during double evaluation (#11119) 2021-12-17 19:20:11 +01:00
Danielle Pintz 06c5903600
Simplify several profile calls (#11031) 2021-12-14 19:49:19 +00:00
Carlos Mocholí 99adc45af1
Follow-up changes to #10575 (#10957)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2021-12-07 15:27:52 +01:00
Danielle Pintz 6043179931
Re-design `call_hook` interface (#10575) 2021-12-04 16:39:55 -05:00
Carlos Mocholí a28b4cd0c0
Sort out the dataloader idx logic for evaluation (#10923)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2021-12-03 20:01:46 +00:00
four4fish 6fe3211573
Unroll dict input before call Accelerator X_steps (#10908)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2021-12-03 17:00:52 +00:00
Carlos Mocholí 0061619e0a
Improve typing for loops (#10780) 2021-11-30 20:28:55 +00:00
four4fish 8bf7f9cce7
1/n Move Accelerator into strategy - move batch_to_device to strategy (#10649)
* 1/n Integrate Device Specific Accelerator Logic with strategy - move batch_to_device to strategy

* add changelog

* add model is not none check

* Apply suggestions from code review

Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

* Update CHANGELOG.md

* Update test_datamodules.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update test_hooks.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update dp.py

Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-29 12:11:21 -08:00
Carlos Mocholí 724a92b065
Mark outputs as protected in the evaluation loops (#10781)
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2021-11-28 20:09:30 +00:00
Carlos Mocholí 31bb6e69ca
Avoid optional instances in Loops (#10735)
* Avoid optional instances in Loops

* More cleanup
2021-11-26 18:00:18 +00:00
thomas chaton 3d6262b7a9
Fault Tolerant Manual: Add support for DDP (#10638) 2021-11-25 18:31:53 +01:00
thomas chaton b28ab34ff5
Fault Tolerant Manual: Add loading to reload the states (#10699)
Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com>
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-23 17:18:36 +00:00
Carlos Mocholí a6dedcf492
Fix `move_metrics_to_cpu` with evaluation (#10631) 2021-11-22 15:58:21 +00:00
Danielle Pintz e94dcf6936
Mark `trainer.data_connector` as protected (#10031)
Co-authored-by: tchaton <thomas@grid.ai>
2021-10-25 12:29:09 +01:00
Carlos Mocholí e44921ee21
Fix `self.log(on_epoch=True, reduce_fx=sum)` on_batch_start (#9791) 2021-10-20 01:56:37 +02:00
Carlos Mocholí c69a79c86f
Fix `self.log(on_epoch=True)` on_batch_start (#9780) 2021-10-18 14:02:16 +02:00
thomas chaton 5841ca9782
[Feat] Add auto_restart for fault tolerant training (#9722) 2021-10-01 16:37:17 +00:00
thomas chaton fa44dbcd9e
[Refactor] Simplify data loading logic around replacing sampler to prevent confusion (#9721)
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-09-28 17:04:02 +00:00
Carlos Mocholí d02fc2b728
Rename `reset_on_epoch` to `reset_on_run` (#9658) 2021-09-25 04:27:54 +02:00
thomas chaton 89ab2470c1
[Refactor] 1/2 Move reset_on_restart within the loop reset (#9561)
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-09-17 16:11:32 +00:00
Adrian Wälchli 5a846d48ce
mark several methods in evaluation loops as protected (#9516)
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-09-15 14:12:27 +00:00
ananthsub c963bf6568
[loops] Reset reference to dataloader iterator on run end (#9386)
* [loops] Reset reference to dataloader iterator on run end
2021-09-10 04:18:58 +00:00
Jirka Borovec 6e124e7207
CI: precommit - docformatter (#8584)
* CI: precommit - docformatter
* fix deprecated

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-06 12:49:09 +00:00
Carlos Mocholí f745aa9ce1
Move tracking epoch end outputs logic to the `EvaluationEpochLoop` (#9261) 2021-09-03 15:02:34 +02:00
Adrian Wälchli c0bd658354
Remove calls to internal dev debugger in training- and eval loop (#9188) 2021-08-30 17:16:59 +02:00
Adrian Wälchli 02612e14ef
remove redundant iterator call to data fetcher in loops (#9117)
Co-authored-by: tchaton <thomas@grid.ai>
2021-08-26 11:36:29 +02:00
thomas chaton f959b13ab9
3/n inter batch parallelism (#9052)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-08-24 18:45:54 +00:00
thomas chaton 92c7eec966
2/n inter batch parallelism (#9047)
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
2021-08-23 19:30:44 +00:00
Adrian Wälchli 522df2b89b
3/n integrate new LightningDataFetcher into loop (#8953)
Co-authored-by: tchaton <thomas@grid.ai>
Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com>
2021-08-17 21:42:22 +00:00
ananthsub 037a86c873
Remove write_predictions from LightningModule (#8850)
* Remove write_predictions from LightningModule
2021-08-14 02:00:23 +00:00
Adrian Wälchli 1bc052c290
Remove dead code in eval loop output tracking (#8625) 2021-07-30 14:04:51 +02:00
Carlos Mocholí a64cc37394
Replace `yapf` with `black` (#7783)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-07-26 13:37:35 +02:00
Adrian Wälchli 7d93d70110
Loop specialization (#8226)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: tchaton <thomas@grid.ai>
Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2021-07-19 15:08:53 +02:00
thomas chaton 7bb810f143
Add progress tracking on Loops - 2/n (#8362)
* resolve issues

* update

* update

* update

* add more exceptions

* resolve bug

* update

* update

* update changelog

* resolve bug

* resolve comments

* update

* update

* update changelog

* update

* update

* remove space

* update

* add progress tracking to loops

* validate json

* update

* convert to dict for better readability

* validate reload

* update

* update

* update on comments

* remove deadcode

* clean changelog

* clean changelog

* update

* update on comments

* CHANGELOG

* CHANGELOG

* Update pytorch_lightning/loops/base.py

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

* whitespace suggestions

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* make fault_tolerant_enabled protected

* whitespace fixes around Args

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* typo it's -> its

* fix copy-paste typo in progress docstring

* Delete classes

* Minor change

* docs

* protected get_loops_state

* merge restore_loops with restore_progress

* Fix tests after removals

* explicit save with trainer.save_checkpoint()

* handle optimization restart based on optimizer_idx

* update increments

* update val batch progress and remove iteration count

* update progress tracking for dataloader loops

* remove self.dataloader_idx from eval_epoch_loop

* add batch progress to predict loop

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* incorporate progress tracking for current_epoch

* Fix test

* Actually remove it

* Remove unused TrainingEpochProgress

* Fix optimization progress - missing scheduler

* Restarting changes

* Scheduler progress

* Unused property, reset on epoch

* Resolve FIXME

* Remove FIXME

* fix test_progress (wip)

* fix batch_progress.current.reset

* Hold off on split progress. Out of scope of this PR

* Unnecessary if

* fix structure in test_progress

* structure

* clean up unused variables in test_progress

* refactor naming and organization in test_progress

* Unnecessary variable

* Remove unnecessary diff

* Improve comment

* Undo typing change to avoid polluting everything with mypy fixes

* Fix and improve test_loops.py

* Fix and organize `test_loop_state_dict`

* Remove unnecessary checks in test

* Update test after disallowing updates on None attributes

* Typing

* Minor test cleanup

* Fix and move loop test

* Move test from progress to loops

* Reset the scheduler progress

* SchedulerProgress fix

* Consistent whitespace

* Fix final test

* Minor test changes

* One test to rule them all

* Formatting

* Rename and clean variables

* Shorter names

* Shorter scheduler name

* Fix optimizer step calculation for stop_batch=2

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove empty connects

* Update CHANGELOG

* Holy shit finally got the formula right

* Fix final thing!!!

* Do not check state dicts

* parametrize multiple_dataloader progress test

* Update CHANGELOG.md

Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
2021-07-19 08:31:45 +00:00
Carlos Mocholí 176df202e4
Mark evaluation epoch loops attributes as protected (#8420)
* Mark evaluation epoch loops attributes as protected

* Fix pre-commit
2021-07-15 20:32:44 +02:00
Carlos Mocholí 3379477242
Connect progress tracking dataclasses to loops (#8244)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2021-07-05 13:33:12 +02:00
Adrian Wälchli ea5cfd2005
move batch to device before sending it to hooks (#7378)
* update train step

* test

* x

* limits

* val

* typeo

* x

* x

* step

* min gpus

* run all loops

* x

* limit test

* profiler

* clean up accelerator code

* move files

* rename

* move tests

* changelog

* reorder callbacks and model hooks

* add test description

* replace unneccessary method

* fix chlog

* adjust batch_to_device for DP Plugin

* update tests for dataloader idx

* unused imports

* hook change

* switch None

* clear memory

* change to None

* None

* None

* memory savings

* remove redundant todo

* hack

* cheat

* Revert "cheat"

This reverts commit a8433bd0b4.

* Revert "hack"

This reverts commit 43a6d1edeb.

* update new epoch loop

* remove from old loop code

* update chlog

* update hook test

* changelog

* teardown

* integrate changes in new eval loop

* fix hook calls

* add prediction step

* bad merge

* Revert "bad merge"

This reverts commit 488080863c.

* fix train batch hook test

* rm -rf _notebooks

* update chlog

* release memory

* fix type

* notebooks mess

* debug

* Revert "debug"

This reverts commit eec4ee2f77.

* teardown

* fix teardown bug

* debug

* x

* debug

* Revert "debug"

This reverts commit a6e6101946.

Revert "debug"

This reverts commit 5ddeaec069.

debug


debug


Revert "debug"

This reverts commit 605be746f7daedf265b2c05a1c153ce543394435.

Revert "Revert "debug""

This reverts commit a7612d5410409ed886cfb609457349ecf44cbfa8.

debug


x


x


x


s


tol


x


tol

* Fix changelog

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2021-07-05 09:31:39 +01:00
Carlos Mocholí 0e19d16ca6
Move result teardown to loops (#8245)
* Move result teardown to loops

* Update CHANGELOG

* Remove teardown from run

* Move previous teardown to on_run_end

* Add comment

* Merge 8250

* Remove stage set to None where it shouldnt
2021-07-02 14:36:14 +01:00
Adrian Wälchli 55a90af7fc
`pytorch_lightning.loops` file structure: group by dataloader, epoch, and batch loop (#8077) 2021-06-24 23:40:46 +02:00