Commit Graph

308 Commits

Author SHA1 Message Date
Andrei-Aksionov 6b88ddc6d0
Code-style changes via pre-commit (#20483)
Pre-commit
2024-12-09 17:00:57 +01:00
Luca Antiga caa9e1e594
Remove deprecated distutils (#20481)
* Remove deprecated distutils

* Fix format

* Fix package name

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-12-09 16:00:04 +01:00
Luca Antiga 38971a0cae
Patch jsonargparse for Python >= 3.12.8 (#20479)
* Patch argparse _parse_known_args

* Add patch to test

* Avoid importing lightning in assistant

* Fix return type
2024-12-09 14:35:44 +01:00
Luca Antiga 87565cbca0
FP8 + FSDP2 + torch.compile examples for PyTorch Lightning and Fabric (#20440)
* Minimal transformer examples

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add tests for compile after fsdp2/tp

* Add README's

* Add docs

* Rename folder, add cross-reference

* Fix link

* Newline after code-block directive

* Update section name

* Fix reference

* Half standalone tests batch size

* Fix integration tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-11-26 15:29:40 +01:00
Jirka Borovec 1e88899a51
bump python 3.9+ (#20413)
* bump python 3.9+

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* --unsafe-fixes

* contextlib.AbstractContextManager

* type: ignore[misc]

* update CI

* apply fixes

* apply fixes

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Antiga <luca.antiga@gmail.com>
2024-11-25 09:20:17 +01:00
Alan Chu cd2bd3ce53
Fix configure_optimizers example (#20420)
Co-authored-by: Alan Chu <alanchu@Alans-Air.lan>
2024-11-14 23:41:44 +01:00
Jirka Borovec cf24a190ce
fix(docs): remove dead link from readme (#20170) 2024-08-06 11:05:00 -04:00
pre-commit-ci[bot] a40affb953
[pre-commit.ci] pre-commit suggestions (#20035)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: awaelchli <aedu.waelchli@gmail.com>
2024-07-05 12:17:15 -04:00
awaelchli 330af381de
Remove the lightning app code (#20039)
* remove source, tests, docs, workflows

* update checkgroup

* update codeowners

* update workflows

* package setup

* config files

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove entry point

* docs

* __main__

* remove store

* leftover store removals

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-07-05 13:56:29 +02:00
liambsmith 709a2a9d3b
Updated Fabric trainer example to not call `self.trainer.model` during validation (#19993) 2024-06-21 10:43:30 -04:00
SW Yoo d3a0ada4ff
Fix dtype for MPS in reinforcement learning example (#19982) 2024-06-21 10:36:10 -04:00
liambsmith 394c42aaf6
Fix callback call in Fabric Trainer example (#19986) 2024-06-18 13:14:32 -04:00
Bhavay Malhotra a611de0c15
Removing numpy requirement from all files in examples/pytorch/domain_templates (#19947) 2024-06-06 11:02:01 -04:00
awaelchli 98005bbed0
Add Studio badge to tensor parallel docs (#19913) 2024-05-28 09:04:55 -04:00
awaelchli c09356db1e
(10/10) Support 2D Parallelism - Port Fabric docs to PL (#19899) 2024-05-23 08:55:52 -04:00
awaelchli 987c2c4093
(7/n) Support 2D Parallelism - TP Fabric Docs (#19884)
Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2024-05-22 06:20:40 -04:00
awaelchli d76feef0d6
Enable loss-parallel in example (#19882) 2024-05-20 13:19:38 +02:00
awaelchli c8059d7bfd
(6/n) Support 2D Parallelism - Trainer example (#19879)
* Add 2D parallel example

* replace with torchtitan code
2024-05-18 20:35:58 -04:00
awaelchli 9455871c93
(2/n) Support 2D Parallelism - Distributed Checkpoints (#19852)
* distributed checkpoints

* use decorator

* refactor if-strict

* update example

* filter non-persistent buffers (todo, add test)

* simplify checkpoint loading for model
2024-05-15 08:19:08 -04:00
awaelchli 0c8a193d3c
(1/n) Support 2D Parallelism (#19846) 2024-05-07 17:02:58 -04:00
Inyong Hwang a6ab0d1486
replace bolts url on example's readme (#19677) 2024-03-20 10:23:12 -04:00
Jirka Borovec fe535970a9
lint: switch `pyupgrade` with Ruff's `UP` rule (#19638)
* ruff
* configure
* update

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-03-15 20:48:17 +01:00
awaelchli 8549a932f7
Remove orphaned docs pages (#19555) 2024-03-12 21:19:05 -04:00
awaelchli ea89133c65
Rename `fabric run model` to `fabric run` (#19527) 2024-02-27 11:36:46 -05:00
Jirka Borovec 99fe6563ef
precommit: ruff-format (#19434)
* precommit: ruff-format

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* manual update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* manual update

* order

* mypy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* mypy

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-02-15 13:39:17 -05:00
awaelchli e950bb4828
Remove the Graphcore IPU integration (#19405)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2024-02-12 16:16:02 -05:00
pre-commit-ci[bot] f120c91e9f
[pre-commit.ci] pre-commit suggestions (#19229)
* [pre-commit.ci] pre-commit suggestions

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0)
- [github.com/asottile/pyupgrade: v3.14.0 → v3.15.0](https://github.com/asottile/pyupgrade/compare/v3.14.0...v3.15.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.3 → v0.1.9](https://github.com/astral-sh/ruff-pre-commit/compare/v0.1.3...v0.1.9)
- [github.com/psf/black: 23.9.1 → 23.12.1](https://github.com/psf/black/compare/23.9.1...23.12.1)
- [github.com/pre-commit/mirrors-prettier: v3.0.3 → v4.0.0-alpha.8](https://github.com/pre-commit/mirrors-prettier/compare/v3.0.3...v4.0.0-alpha.8)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update .pre-commit-config.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* drop unused

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2024-01-10 13:11:48 -05:00
thomas chaton 792cb73fc6
Remove the LightningDataset relying on un-maintained torchdata (#19019)
Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
2023-11-16 16:08:15 -05:00
thomas chaton cb23fc2dd4
Update auto_encoder.py to accomodate torchvision breaking change (#18996)
Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
2023-11-13 15:32:16 -05:00
Adrian Wälchli e0ba4d46e1
Fix spelling errors (#18773)
fix spelling errors
2023-10-30 17:14:52 -04:00
pre-commit-ci[bot] c0ec0decec
[pre-commit.ci] pre-commit suggestions (#18697)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
2023-10-03 22:07:21 +02:00
Adrian Wälchli c631726b5c
Set the generator seed in `random_split` usages in the examples (#18651) 2023-09-27 14:09:05 +02:00
Jirka Borovec 830a62a722
ruff: replace isort with ruff +TPU (#17684)
* ruff: replace isort with ruff

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing & imports

* lines in warning test

* docs

* fix enum import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing

* import

* fix lines

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* type ClusterEnvironment

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-26 11:54:55 -04:00
Jirka Borovec 358336268f
enable codespell for docs & fixing +TPU (#18629)
* precommit/codespell

* run

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* disable

* more fixing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Apply suggestions from code review

* more fixing

* json

* note

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-26 11:54:44 -04:00
Adrian Wälchli f83ad093e5
Utility function to check shared filesystem (#18586)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2023-09-25 15:49:52 -04:00
Jirka Borovec d579cfed57
precommit: unify formatting with prettier (#18605)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-25 14:34:41 +02:00
Jirka Borovec 3fbf1540b0
docs: 3/3 enable Sphinx nitpicky [app] (#18603)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-21 22:44:59 +02:00
Jirka Borovec b4b21e0152
docs: 2/3 enable Sphinx nitpicky [pytorch] part 1/n (#18484)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-20 19:09:34 +02:00
Adrian Wälchli 01cd89fd04
Set up dataloaders in k-fold cross validation example (#18582) 2023-09-18 11:45:20 -04:00
Jirka Borovec 5f5d99eae6
Relax app dependencies for `lightning` | breaking imports from `L` (#18386)
* make App conditional import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci: min req.

* data + store

* typo

* module_available

* extra

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* envs

* doctest

* more

* docs

* **

* rm

* -f

* Apply suggestions from code review

* docs

* Revert "docs"

This reverts commit 20e8e6be98.

* Apply suggestions from code review

* warnings

* breaking

* ...

* fixing source

* fixing examples

* fixing tests

* fixing docs

* fixing source

* fix mock

* fix trainer

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-08-28 11:28:58 -04:00
Jirka Borovec efa7b2f9ef
docformatter: config with black (#18064)
* docformatter: config with black

* additional_dependencies: [tomli]

* 119

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-08-09 10:44:20 -04:00
pre-commit-ci[bot] 834bd61164
[pre-commit.ci] pre-commit suggestions (#17983)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka B <j.borovec+github@gmail.com>
2023-08-08 16:26:06 +02:00
Nripesh Niketan 4e3087951d
Add GPU Acceleration apple silicon examples (#18127)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2023-07-27 19:52:51 -04:00
Carlos Mocholí a9269aecdb
Add nightly testing to CUDA CI (#18078) 2023-07-17 20:53:29 +02:00
Carlos Mocholí 3a55f0c0a1
Minor miscellaneous fixes (#18068) 2023-07-13 06:01:58 -04:00
jere357 28007bd076
Consistent early exit logic for validation loop in Fabric Trainer example #17962 2023-07-03 08:26:50 -04:00
jere357 6329caa876
slightly changing the behavior of val_loop specifying the limit_val_batches or reaching should_stop (#17873) 2023-06-20 00:30:34 +02:00
Noha Alon ca30fd7752
Lightning Dataset (including optimized dataloading of s3 buckets) (#17743)
* Lightning DataLoader

* lightning dataloader

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init

* example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* env var

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/__init__.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* remove unused functions

* extra reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/fileio.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports work now! yay

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* missing import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* error handling

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update creds for local use case

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* recursive get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update imagenet example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* example cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* changelog

* reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* requirements

* expose LightningDataset too

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expost LightningDataset at top level

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused private methods from init

* remove private imports

* upper bound on extra requirements

* review comments

* loosen req

* deps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test updating fabric base req

* remove version pin on s3fs to test

* recover missing function

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* update

* random

* torchdata >= 0.3.0

* update torchdata version

* remove torchdata version to test

* try rem torch version pin

* req

* update bucket in test

* req

* skips

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* import

* update structure to lightning.data

* base.txt for data reqs

* fix imports

* rename to LightningS3Dataset

* new workflow

* dont need to test warnings

* reqs

* req

* revert data folder in pytorch

* test import

* tests

* req

* req

* req

* torch version

* req

* req

* open dep

* reformatted

* pin strict

* pin strict extra

* req

* modify workflow, no cache

* try

* patch

* import

* fix

* dataset test

* update getattr

* pin everything to test

* remove torch preinstall from workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* print

* skip test for now

* update path join

* revert app dep version bump

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow updates

* app base req

* req

* window test failure

* add data req to assistant

* try

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing comma

* updates

* update

* typo

* requirements

* try widening req

* older torch version

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* update

* update

* update

* cleanup tests

* typo again

* update

* remove unnecessary line

* Update .github/CODEOWNERS

* Discard changes to requirements/pytorch/base.txt

* Discard changes to requirements/fabric/base.txt

* Discard changes to requirements/app/base.txt

* requirements

* requirements

* one line

* app workflow pick only app reqs

* rename package

* undo

* don't use cache

* examples CI

* pytorch and fabric CI

* try remove cache

* Apply suggestions from code review

* jirka playing

* jirka playing

* jirka playing

* blah

* flatten LightningDataset

* cleans up dataset class

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* extra

* fix dataset test

* update checkgroups

* Luca's review comments

* val error fix

* unskip test

* min

* fix precommit warning

* cpu

* docstrings

* req

* 2.0.1

* add return type

* typing errors

* req

* return types with quotations

* import for type-checking

* no botocore in cloudagnostic code

* exit args

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* backends typing

* remove oldest from data tests

* typing

* typing

* typing

* types

* type

* typing

* typing

* typing

* import fix

* Changelog

---------

Co-authored-by: Noha Alon <nohaalon@Nohas-MacBook-Air.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
2023-06-13 11:44:41 +01:00
Adrian Wälchli 9ff7d7120b
Add `rank_zero_first` utility (#17784)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-06-12 10:32:32 +00:00
Carlos Mocholí ea44aac10f
Use PL `is_overridden` in BYOT example (#17070) 2023-05-05 12:08:18 +00:00