Commit Graph

34 Commits

Author SHA1 Message Date
Jirka Borovec 7f1aa7229c
ci: hotfix for doctests (#17841) 2023-06-15 15:25:59 +00:00
Noha Alon ca30fd7752
Lightning Dataset (including optimized dataloading of s3 buckets) (#17743)
* Lightning DataLoader

* lightning dataloader

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init

* example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* env var

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/__init__.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* remove unused functions

* extra reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/fileio.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports work now! yay

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* missing import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* error handling

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update creds for local use case

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* recursive get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update imagenet example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* example cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* changelog

* reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* requirements

* expose LightningDataset too

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expost LightningDataset at top level

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused private methods from init

* remove private imports

* upper bound on extra requirements

* review comments

* loosen req

* deps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test updating fabric base req

* remove version pin on s3fs to test

* recover missing function

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* update

* random

* torchdata >= 0.3.0

* update torchdata version

* remove torchdata version to test

* try rem torch version pin

* req

* update bucket in test

* req

* skips

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* import

* update structure to lightning.data

* base.txt for data reqs

* fix imports

* rename to LightningS3Dataset

* new workflow

* dont need to test warnings

* reqs

* req

* revert data folder in pytorch

* test import

* tests

* req

* req

* req

* torch version

* req

* req

* open dep

* reformatted

* pin strict

* pin strict extra

* req

* modify workflow, no cache

* try

* patch

* import

* fix

* dataset test

* update getattr

* pin everything to test

* remove torch preinstall from workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* print

* skip test for now

* update path join

* revert app dep version bump

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow updates

* app base req

* req

* window test failure

* add data req to assistant

* try

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing comma

* updates

* update

* typo

* requirements

* try widening req

* older torch version

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* update

* update

* update

* cleanup tests

* typo again

* update

* remove unnecessary line

* Update .github/CODEOWNERS

* Discard changes to requirements/pytorch/base.txt

* Discard changes to requirements/fabric/base.txt

* Discard changes to requirements/app/base.txt

* requirements

* requirements

* one line

* app workflow pick only app reqs

* rename package

* undo

* don't use cache

* examples CI

* pytorch and fabric CI

* try remove cache

* Apply suggestions from code review

* jirka playing

* jirka playing

* jirka playing

* blah

* flatten LightningDataset

* cleans up dataset class

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* extra

* fix dataset test

* update checkgroups

* Luca's review comments

* val error fix

* unskip test

* min

* fix precommit warning

* cpu

* docstrings

* req

* 2.0.1

* add return type

* typing errors

* req

* return types with quotations

* import for type-checking

* no botocore in cloudagnostic code

* exit args

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* backends typing

* remove oldest from data tests

* typing

* typing

* typing

* types

* type

* typing

* typing

* typing

* import fix

* Changelog

---------

Co-authored-by: Noha Alon <nohaalon@Nohas-MacBook-Air.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
2023-06-13 11:44:41 +01:00
Jirka Borovec 4413e98e4e
ruff: enable & fixing RET (#17540) 2023-05-05 09:34:40 +00:00
Jirka Borovec df97141781
add & apply flake8-simplify (#17386) 2023-04-24 21:57:08 +00:00
Jirka Borovec e04d4fd25e
ci: separate integrations (#17170)
* integrations

* Rename

* Apply suggestions from code review

* cut out

* skip _

* fix

* setup

* 1e-2

* uninstall

* revert

* note
2023-03-24 09:46:45 +01:00
Jirka Borovec f697fff5db
docs: rename source-app (#16863)
* docs: rename source-app

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci

* group check

* trigger

* param

* fix

* cleaning

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-02-28 10:04:43 +01:00
Jirka Borovec 770b792925
copyright Lightning AI team (#16647)
* copyright Lightning AI team

* more...
2023-02-06 15:26:51 +01:00
Jirka Borovec 710ca4b096
cleaning as standalone==mirror (#16601) 2023-02-02 14:10:29 +00:00
Jirka Borovec cc49e4a31f
tests: switch imports for apps (#16554) 2023-02-01 11:07:00 +00:00
Jirka Borovec 34140c0603
move lightning_app >> lightning/app (#16553)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-02-01 06:29:16 +01:00
Carlos Mocholí b40b2ed2c9
Improve requirements parser (#13912) 2023-01-12 16:25:29 +00:00
Adrian Wälchli 2b6ec90ec1
Lightning 1.9.0 Release Candidate 0 (#16264)
* release

* Apply suggestions from code review

* cleaning

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2023-01-05 15:35:56 -05:00
Carlos Mocholí 15ef52bc73
Rename LightningLite to Fabric (#16244)
* Rename LightningLite to Fabric

* Fix introspection test

* Fix deprecated Lite tests

* Undo accidental Horovod removal

* Fixes
2023-01-04 10:57:18 -05:00
Jirka Borovec 1f1c71ad93
unify extras & minor CI cleaning: move env. var (#15942)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2022-12-22 13:54:44 +00:00
Jirka Borovec 186b799a62
ci: adjust version in all requirements (#16100) 2022-12-22 06:21:52 +00:00
Carlos Mocholí ca75e497a2
Merge setup_tools into assistant (#15847)
* Merge setup_tools and assistant
* Project root
* Fix PROJECT_ROOT

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
2022-12-13 14:47:35 +00:00
Jirka Borovec c06ea41056
Include images with the mirror package (#15659)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2022-11-12 19:27:13 +00:00
Carlos Mocholí e33d09a1a8
Reuse assistant code to create the mirror package (#15365) 2022-11-08 18:52:24 +00:00
otaj 537da82059
Replace oldest versions also for Lite (#15337) 2022-10-26 18:22:46 +00:00
Carlos Mocholí 7b3de1215f
Remove examples and loggers from develop dependencies (#15282)
* Remove examples and loggers from develop dependencies

* remove more references

* Fix mypy

* Keep logger file for docs mocking

* Simpler fix

* Fix docs build

* Global testsetup

* Matching files

* Undo change

* loggers as info

* Clarify

* Update requirements/pytorch/loggers.info

Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: otaj <6065855+otaj@users.noreply.github.com>
2022-10-25 09:23:26 -04:00
Jirka Borovec 56ca4aafee
CI: fix example imports for App as standalone (#15260)
* cleaning

* adjust examples

* fix some imports

* fix imports

* ci

* fixing

* cmd_install._install_app

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bloody queues

* queue

* mock

* queue

* scope

* win

Co-authored-by: otaj <ota@lightning.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-10-24 09:52:04 -04:00
otaj 099580cf2b
Assistant fixes (#15221)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2022-10-20 18:23:47 +00:00
Justus Schock 775e9ebc0f
Assistant for Unified Package (#15207)
* Update assistant and workflow files
* Update .actions/assistant.py

Co-authored-by: otaj <6065855+otaj@users.noreply.github.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: otaj <ota@lightning.ai>
2022-10-20 14:17:27 +00:00
otaj e0974a1536
[LAI] Bump minimal requirements (#15203)
* bump minimal requirement
* include app requirements in oldest
2022-10-20 08:10:44 +00:00
Jirka Borovec d9be959bc2
fix: keep pre versions (#14752) 2022-09-16 18:11:00 +00:00
otaj ced94874b2
Use valid pypi versions for install in assistant (#14750)
Co-authored-by: Kushashwa Ravi Shrimali <kushashwaravishrimali@gmail.com>
2022-09-16 13:26:04 +00:00
Jirka Borovec 7cbf153332
CI: hotfix last version (#14627)
LooseVersion was not correctly evaluation RC and set it as last even though the full release is out...
2022-09-09 18:38:03 +00:00
Jirka Borovec aa62fe36df
add testing PT 1.12 (#13386)
* add testing PT 1.12
* Fix quantization tests
* Fix another set of tests
* Fix check since https://github.com/pytorch/pytorch/pull/80139 is only going to be available for 1.13
* Skip this test for now for 1.12

Co-authored-by: SeanNaren <sean@grid.ai>
2022-07-15 19:41:23 +02:00
Jirka Borovec 8ba5ad9dce
cd: releasing packages (#13489)
* assist

* split

* need

* inverse

* checkout

* json

* include

* unzip

* mirror

* fire

* twine

* for tar
-xf

* clean

* 3.8

* ls
2022-07-11 17:46:10 -04:00
Carlos Mocholí ad87d2cad0
Future 5/n: Move requirements (#13306)
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2022-06-21 17:11:33 +02:00
Jirka Borovec fec9a09672
add freeze for development and full range for install (#12994)
* freeze versions

* unfreeze

* dependabot

* Apply suggestions from code review

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

* fix all req

* ...

* use base

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix refs

* Apply suggestions from code review

Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>

* Apply suggestions from code review

* dockers

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
2022-05-12 09:14:18 -04:00
Jirka Borovec bb51e2a55b
Merge pull request #12723 from PyTorchLightning/req/strategies
Separate strategies' requirements
2022-05-04 10:06:02 -04:00
Jirka Borovec fe940e195d
CI: update prune_pkgs (#12382) 2022-03-21 12:50:50 +00:00
Jirka Borovec bc8172856f
aggregate multiple helper scripts to single CLI (#11147)
* nightly release
* min version
* fire
2022-03-11 11:13:43 +00:00