Commit Graph

151 Commits

Author SHA1 Message Date
Jirka Borovec d4491bb14a
update PR template (#2965)
* template

* typo

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-08-14 00:13:23 +02:00
Jirka Borovec 4354690e55
add apex test (#2921)
* add apex test

* rename

* level

* events

* wrap

* evt

* miss

* apex

* apex

* apex

* apex

* apex

* apex

* Update tests/models/test_amp.py

Co-authored-by: William Falcon <waf2107@columbia.edu>

* notes

* notes

Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-08-13 10:03:13 -04:00
Nathan Raw 118bd14d16
Update CONTRIBUTING.md (#2927)
* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-12 12:59:36 +02:00
zcain117 580a5bd1df
Use kubectl to get logs from TPU CI instead of gcloud logging. (#2918)
* Use kubectl to get logs from TPU CI instead of gcloud logging.

* Update Github Action to read logs from kubectl rather than gcloud logging.
2020-08-11 19:30:56 -04:00
Jirka Borovec aa6fe25436
Fix install setup - push pypi (#2872)
* fix setup install

* fix setup install

* 📝 edit docs install command

Co-authored-by: nateraw <nxr9266@g.rit.edu>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-08-07 19:02:38 -04:00
Jirka Borovec 91b0d46cd5
do not fails all dockers (#2861) 2020-08-07 09:10:35 -04:00
Jirka Borovec ad956b5ed9
do not fails all dockers (#2860) 2020-08-07 14:14:22 +02:00
Rosario Scalise b37c35a8ff
Update CONTRIBUTING.md (#2855)
* Update CONTRIBUTING.md

Added docker option to testing section.

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-07 11:53:29 +02:00
Jeff Yang 4307dd9dc6
fix PR link (#2858) 2020-08-07 09:43:38 +02:00
Jirka Borovec ea658e300c
Tests/install pkg (#2835)
* add install matrix

* nb tests

* win

* cfg

* torch

* link

* Update .github/workflows/install-pkg.yml

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* try

* try

* try

* try

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
2020-08-07 03:08:23 -04:00
edenlightning 1dc411fc53
Update contributing guide (#2830)
* update contributing.md

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Apply suggestions from code review

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* Update .github/CONTRIBUTING.md

* suggestion from code review

* minor changes

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-08-06 22:46:45 +02:00
Nathan Raw 9ab071588b
Setup extras (#2831)
* 🎨 use package extras

* 🎨 get extras from reqs

* 🎨 .

* 📝 docs

* 🎨 .
2020-08-06 07:12:47 -04:00
Jirka Borovec 3772601cd6
update CI testing with pip upgrade (#2380)
* try pt1.5

* cpu

* upgrade

* tpu

* user

* [blocked by #2380] freeze GPU PT 1.4 (#2780)

* freeze

* user
2020-07-31 14:50:06 -04:00
Jirka Borovec bc7a08fbe0
test dockers & add AMP in pt-1.6 (#1584)
* exist images

* names

* images

* args

* pt 1.6 dev

* circleci

* update

* refactor

* build

* fix

* MKL
2020-07-31 08:23:13 -04:00
Jirka Borovec b88fc43871
re-enable skipped tests (#2762)
* re-enable skipped

* timeout
2020-07-31 07:52:17 -04:00
Jirka Borovec fcfdb4df13
conda speedup (#2546)
* conda speedup

* cache

* add pip cache

* suggestion

* cache

* cache

* req
2020-07-31 06:31:23 -04:00
Jirka Borovec 06e8910f06
pytorch 1.6 (#2745)
* pt 1.6

* don't use the new zipfile serialization for now

* quick flake8 fixes

* remove unnecessary f

* coalesce strings

* remove comma

* remove extra commas

* Apply suggestions from code review

Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>

* set _use_new_zipfile_serialization to False only for pytorch 1.6.0

* remove unnecessary comments

* flake8 fixes

* use pkg_resources instead of packaging

* readme

* format

* version

* chlog

Co-authored-by: Peter Yu <peter@asapp.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
2020-07-31 11:18:32 +02:00
Jirka Borovec bc833fbf52
Horovod & py3.8 (#2764) 2020-07-30 23:39:07 +02:00
Jirka Borovec 40337cce58
freeze PT 1.5 for Horovod issue (#2744)
* freeze pt 1.5

* torchtext

* Apply suggestions from code review

Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>

* timeout

Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
2020-07-28 15:52:23 -04:00
Jirka Borovec 0fe933e23d
fixing TPU tests (#2632)
* init

* rename

* tpu_core_idx

* idx 8

* idxs

* @pl_multi_process_test

* assert

* assert

* deamon

* no close

* imort

* msg

* use_single_gpu

* dataset

* idx

* fix idx

* dataset

* format

* add pickable

* typo

* apex

* typo

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* docs

* typo

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* docs

* docs

* Apply suggestions from code review

Co-authored-by: Ethan Harris <ewah1g13@soton.ac.uk>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* Apply suggestions from code review

Co-authored-by: Ethan Harris <ewah1g13@soton.ac.uk>

* docs

* Apply suggestions from code review

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

Co-authored-by: Ethan Harris <ewah1g13@soton.ac.uk>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-07-27 19:07:09 -04:00
zcain117 d0b8e850a4
integrate with CircleCI (#2486)
* add circleCI

* wip

* CircleCI setup that worked on my private repo. Use a working pytorch-lightning commit

* Fix the orb imports

* Update circleci header comment

* Try to pull the GITHUB_REF from the CI_PULL_REQUEST

* Use null instead of space for 'sed'

* Add TODO for codecov

* Remove echo of GKE_CLUSTER since it will be redacted by CircleCI.

* Try running codecov upload.

* Try using codecov orb

* Use pip install codecov

* Use codecov orb again since it should be approved

* dockers/tpu-tests/Dockerfile

* action

* suggestions

* drop suggestion

* suggestion

Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-07-23 12:13:10 -04:00
Rohit Gupta 5025be7860
Fix missing docs (#2659)
* dataloader_idx typo

* typo

* update test_step docs

* missing optimizer_idx
2020-07-22 09:54:14 -04:00
William Falcon 6d10ac2ac8
Structured results (train loop only. val loop separate PR) (PR 2/5) (#2615)
* r

* r

* r

* patched optimizer closure with sr

* patched optimizer closure with sr

* patched optimizer closure with sr

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added train step structured result

* added autoreduce for train step

* added auto reduce on train

* added auto reduce on train

* added auto reduce on train

* added auto reduce on train

* added auto reduce on train

* added auto reduce on train

* added hooks

* added hooks

* added hooks

* added hooks

* added hooks

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* cache

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* Update pytorch_lightning/callbacks/early_stopping.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update pytorch_lightning/callbacks/early_stopping.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update pytorch_lightning/callbacks/early_stopping.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update pytorch_lightning/callbacks/model_checkpoint.py

* Update pytorch_lightning/core/step_result.py

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* simple

* finished tests for structured results on train epoch

* simple

* simple

* revert

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* Update tests/base/deterministic_model.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* finished tests for structured results on train epoch

* docstring typos

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* finished tests for structured results on train epoch

* Update pytorch_lightning/core/step_result.py

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* Update pytorch_lightning/overrides/data_parallel.py

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

Co-authored-by: Jirka <jirka@pytorchlightning.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
2020-07-20 19:00:20 -04:00
Adrian Wälchli ee6f618aa4
Fix CI crash on coverage upload timeout (#2548)
* fix ci crash on codecov timeout

* Update .github/workflows/tpu-testing.yml

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-07-09 07:08:23 -04:00
William Falcon 69cbb62774
Finish #2549 (#2557)
* removed spawns for test_converters and verified tests

Co-authored-by: Ananya Harsh Jha <ahj265@nyu.edu>
Co-authored-by: zcain <zcain@google.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-07-08 20:33:48 -04:00
Jirka Borovec fb85d493d0
use XLA base image for TPU testing (#2536)
* drop py3.6

* use base image

* typo

* skip extra

* drop cache
2020-07-07 07:05:17 -04:00
Jirka Borovec 977df6ed31
Docker: building XLA base image (#2494)
* refactor

* add TPU base

* wip

* builds

* typo

* extras

* simple

* unzip

* rename
2020-07-06 14:21:36 -04:00
Jirka Borovec 39a6435726
Revert "Revert "join coverage (#2460)" (#2499)" (#2500)
This reverts commit 355918af8d.
2020-07-04 11:31:12 -04:00
William Falcon 355918af8d
Revert "join coverage (#2460)" (#2499)
This reverts commit 944ffba305.
2020-07-04 10:29:50 -04:00
Jirka Borovec 944ffba305
join coverage (#2460)
* join coverage

* full TPU test

* codecov

* typo

* report

* docker

* timeout

* base

* show

* cd dir

* req

* docker

* docker

* docker

* coverage

* upload

* drop main

* report

* report

* python

* upload

* drone

* drone

* drone

* drone

* drone

* drone

* drone

* drone

* drone
2020-07-04 10:22:58 -04:00
zcain117 1a40963d1d
Add Github Action to run TPU tests. (#2376)
* Add Github Action to run TPU tests.

* Trigger new Github Actions run.

* Clean up more comments.

* Use different fixed version of ml-testing-accelerators and update config to match.

* use cluster in us-central1-a

* Run 'gcloud logging read' directly without 'echo' to preserve newlines.

* cat coverage.xml on the TPU VM side and upload xml on the Github Action side

* Use new commit on ml-testing-accelerators so command runs fully.

* Preserve newlines in the xml and use if: always() temporarily to upload codecov

* Use pytorch_lightning for coverage instead of pytorch-lightning

* Remove the debug cat of coverage xml

* Apply suggestions from code review

* jsonnet rename

* name

* add codecov flags

* add codecov flags

* codecov

* codecov

* revert codecov

* Clean up after apt-get and remove old TODOs.

* More codefactor cleanups.

* drone

* drone

* disable codecov

* cleaning

* docker py versions

* docker py 3.7

* readme

* bash

* docker

* freeze conda

* py3.6

* Stop using apt-get clean.

* Dont rm pytorch-lightning

* Update docker/tpu/Dockerfile

* Longer timeout in the Github Action to wait for GKE to finish.

* job1

* job2

* job3

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-07-01 21:44:19 -04:00
Jirka Borovec 7f1eab4cad
try adding coverage (#2441)
* add coverage, test failing

* fix test

* badges

* typo

* freeze conda
2020-07-01 08:00:36 -04:00
Llannelongue fa2233f56f
Corrected typo `python -m pip pre-commit install` (#2447) 2020-07-01 07:02:02 -04:00
Jirka Borovec e268061614
Pure package & base tests (#2418)
* base tests

* pil

* wip

* wip

* wip

* ignore

* ignore

* win

* link

* win

* cpu

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-06-30 19:35:54 -04:00
Jirka Borovec d4a02e3bd8
tests: drop CircleCI (#2412)
* drop CircleCI

* add PT testing

* fix

* cpu

* conda

* conda

* req

* base

* conda

* conda

* conda

* conda

* conda

* conda

* conda

* name

* req

* info

* tests

* pt 1.6

* drop 1.6

* info
2020-06-30 10:56:05 -04:00
Jirka Borovec dec074c2e7
typo (#2415) 2020-06-29 07:36:56 -04:00
Adrian Wälchli d910cc5200
docs: dont mock imports when running sphinx doctest (#2396)
* skip if no amp

* dont mock when doctesting

* install extra
2020-06-27 23:31:06 -04:00
Jirka Borovec 51711c265a
fix loading model with kwargs (#2387)
* test

* fix

* fix
2020-06-27 16:38:03 -04:00
Jirka Borovec 73a78a13c7
CI: partial move from CircleCI (#2378)
* move from CircleCI

* req

* tex

* tex

* sudo

* extra

* recom

* pic

* dvipng
2020-06-27 16:25:33 -04:00
Jirka Borovec 6673fc9a0b
fix docker builds (#2383) 2020-06-27 08:49:19 -04:00
Jirka Borovec 2f739f5977
fix key typo (#2374) 2020-06-26 21:46:08 -04:00
Jirka Borovec f1c96930b1
repair CI for Win (#2358)
* no cov

* no cov

* ReduceOp

* group

* reduce_op.sum

* Update sklearns.py

* formatting

* horovod

* Apply suggestions from code review

* horovod

* horovod

* horovod

* horovod

* ci

* print

* ci

* timeout

* timeout

* time

* fix

* distributed cpu

* pipes

* time

* cpu

* spawn

* spawn

* spawn

* tp

* separate

* os

* os

* npm

* Fix load_from_checkpoint() not working with URL on Windows

* Update CHANGELOG

* Update CHANGELOG.md

Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>

* Apply suggestions from code review

* fix

* fix meta tags creating empty lines

* pyright

* node

* fix httpserver address

* drop tutils.default_trainer_options

* imports

* Better fix for load_from_checkpoint() not working with absolute path on Windows (#2294)

* Fix load_from_checkpoint() not working with URL on Windows

* Update CHANGELOG

* Update CHANGELOG.md

Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>

* drop duplicate

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: airium <airium@outlook.com>
Co-authored-by: Peter Yu <2057325+yukw777@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: AIRIUM <38249940+airium@users.noreply.github.com>
2020-06-26 21:38:25 -04:00
Jirka Borovec a5f45787ea
fix get dataloader size (#2375)
* get dataloader size

* pyright
2020-06-26 15:38:48 -04:00
edenlightning f8c10fbff0
Change PR template (#2224)
* Change PR template

* Update .github/PULL_REQUEST_TEMPLATE.md

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-06-18 17:32:02 +02:00
Jirka Borovec c52497e859
template label typo (#2195) 2020-06-15 17:08:06 -04:00
Jirka Borovec 206245ef69
Docs & Changelog (#2176)
* missed

* format

* math

* req

* notes

* fix CI

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-06-13 19:29:21 -04:00
Jirka Borovec 10c643f162
Docs/template (#2152)
* typo

* typo
2020-06-13 10:15:44 -04:00
Jirka Borovec bfaabd7b7f
clean requirements (#2128)
* clean requirements

* missing

* missing

* req

* min

* default >> base

* base.txt
2020-06-13 10:15:22 -04:00
Jirka Borovec 2674976f2c
remove deprecated API for v0.8 (#2073)
* remove deprecated API

* chlog

* times

* missed

* formatting check

* missing

* missing

* miss

* fix docs build error

* fix pep whitespace error

* docs

* wip

* amp_level

* amp_level

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-06-12 14:37:52 -04:00
Udit Arora 08573d0f7e
Fix some pyright member access errors in training module (#2121)
* Fix pyright member access errors in training module

* Fix Trainer instantiation error due to inheritence order

* Add GH workflow for pyright

* Fix more pyright errors in trainer module

* Add pyrightconfig and setup python environment in type-check workflow

* Exclude pyrightconfig.json

* suggestions

Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-06-12 17:23:18 +02:00