Rooba
/
lightning
mirror of https://github.com/Lightning-AI/lightning.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,147 Commits 59 Branches 217 Tags 356 MiB
Commit Graph

56 Commits

Author SHA1 Message Date
pre-commit-ci[bot] f120c91e9f
[pre-commit.ci] pre-commit suggestions (#19229) 
* [pre-commit.ci] pre-commit suggestions

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0)
- [github.com/asottile/pyupgrade: v3.14.0 → v3.15.0](https://github.com/asottile/pyupgrade/compare/v3.14.0...v3.15.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.3 → v0.1.9](https://github.com/astral-sh/ruff-pre-commit/compare/v0.1.3...v0.1.9)
- [github.com/psf/black: 23.9.1 → 23.12.1](https://github.com/psf/black/compare/23.9.1...23.12.1)
- [github.com/pre-commit/mirrors-prettier: v3.0.3 → v4.0.0-alpha.8](https://github.com/pre-commit/mirrors-prettier/compare/v3.0.3...v4.0.0-alpha.8)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update .pre-commit-config.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* drop unused

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
 2024-01-10 13:11:48 -05:00
thomas chaton f5f4d0a264
Improve map, optimize and StreamingDataset (#18912) 
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
 2023-11-03 11:28:55 +00:00
Jirka Borovec df959aeb4f
fix `pydantic` compatibility for 2.0+ & allow new `fastAPI` (#18676) 2023-09-30 07:43:29 +02:00
Jirka Borovec 830a62a722
ruff: replace isort with ruff +TPU (#17684) 
* ruff: replace isort with ruff

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing & imports

* lines in warning test

* docs

* fix enum import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing

* import

* fix lines

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* type ClusterEnvironment

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-09-26 11:54:55 -04:00
Jirka Borovec efa7b2f9ef
docformatter: config with black (#18064) 
* docformatter: config with black

* additional_dependencies: [tomli]

* 119

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-08-09 10:44:20 -04:00
Luca Antiga a7bbbcea9a
Remove cluster commands from CLI (#18151) 
* Remove cluster commands from CLI

* Update docs

* Remove index entry

* Revert accidental conversion to static assets

* Remove reference to deleted file

* Fix highlights

* Fix highlights

* Remove file

* Update CHANGELOG

* Fix dead link

* Increase upper bound on state size to reduce flakiness
 2023-07-28 00:16:21 +02:00
Jirka Borovec 4413e98e4e
ruff: enable & fixing RET (#17540) 2023-05-05 09:34:40 +00:00
Jirka Borovec f55d10f5ee
ruff: autofix PT (#17541) 
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-05-04 11:50:39 -04:00
Jirka Borovec 156786343b
adding check for bandit vulnerabilities 1/n (#17382) 
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-04-27 09:43:12 +00:00
pre-commit-ci[bot] 91cb4b9b87
[pre-commit.ci] pre-commit suggestions (#17271) 
* [pre-commit.ci] pre-commit suggestions

updates:
- [github.com/PyCQA/docformatter: v1.4 → v1.6.0](https://github.com/PyCQA/docformatter/compare/v1.4...v1.6.0)
- [github.com/psf/black: 22.12.0 → 23.3.0](https://github.com/psf/black/compare/22.12.0...23.3.0)
- [github.com/charliermarsh/ruff-pre-commit: v0.0.237 → v0.0.260](https://github.com/charliermarsh/ruff-pre-commit/compare/v0.0.237...v0.0.260)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* apply

* fixing

* docs/lines

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
 2023-04-26 21:37:41 +02:00
Carlos Mocholí 04fb30bd97
Update fastapi dependency pins (#17173) 
* Update fastapi dependency pins

* Apply suggestions from code review

* Update test.txt

* Update requirements/app/base.txt

* Revert "Update requirements/app/base.txt"

This reverts commit 59918ffc6c.

* cloud update

* Bad merge

* fastapi 0.69.0 which pins starlette 0.15.0

* https://github.com/pydantic/pydantic/issues/1985

* Avoid CVE: https://github.com/tiangolo/fastapi/pull/3213

* Strict trio

* Skip windows test

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2023-04-12 17:33:56 +02:00
Jirka Borovec 760612fb8a
update list of fist party packages (#16859) 
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-03-03 16:55:48 +00:00
Jirka Borovec f697fff5db
docs: rename source-app (#16863) 
* docs: rename source-app

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci

* group check

* trigger

* param

* fix

* cleaning

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-02-28 10:04:43 +01:00
Jirka Borovec 8884c8970c
test/hotfix: DDPSpawnStrategy (#16889) 2023-02-27 23:16:44 +00:00
Jirka Borovec 0be025e8b7
rename docs/source-app & adjust docs links for lightning (#16676) 
* update CI

* config / import

* lightning_app imports

* source/ dir

* html

* ci: dirs

* pr

* req dir

* on push

* rename

* drop

* cleaning
 2023-02-13 10:59:02 +01:00
Jirka Borovec 377210d85d
tests: switch imports for fabric (#16592) 2023-02-01 20:34:38 +00:00
Jirka Borovec cc49e4a31f
tests: switch imports for apps (#16554) 2023-02-01 11:07:00 +00:00
Adrian Wälchli c68cfd686e
Rename LiteMultiNode to FabricMultiNode (#16505) 2023-01-26 11:36:27 +00:00
Carlos Mocholí 39b7cb80ca
Remove the FairScale integration (#16400) 
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
 2023-01-23 13:39:04 +00:00
Justus Schock e61bead324
rename flow _exit (#16378) 
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2023-01-17 11:06:50 +00:00
Jirka Borovec 0876a6412d
fix import removed `DDPShardedStrategy` (#16341) 2023-01-12 12:28:09 +01:00
Sherin Thomas d218331f85
[App] Porting fixes to autoscaler component (#16249) 
* autoscaler update

* cold-start-proxy-updates

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ethan Harris <ethanwharris@gmail.com>
 2023-01-06 12:30:59 +00:00
Carlos Mocholí 15ef52bc73
Rename LightningLite to Fabric (#16244) 
* Rename LightningLite to Fabric

* Fix introspection test

* Fix deprecated Lite tests

* Undo accidental Horovod removal

* Fixes
 2023-01-04 10:57:18 -05:00
Sherin Thomas 7cbdc68449
[App] Cold start proxy in autoscaler (#16094) 
* cold start proxy

* Update src/lightning_app/components/serve/auto_scaler.py

* changelog

* better-doc

Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2022-12-20 09:47:10 +00:00
thomas chaton 51ec949657
[App] Resolve some bugs from the Training Studio scaling (#16114) 
Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
 2022-12-19 23:12:55 +00:00
Sherin Thomas 0fd3d54205
[App] Scale out/in interval for autoscaler (#16093) 
* Adding arguments for scale out/in interval

* Tests
 2022-12-19 13:49:00 +00:00
Sherin Thomas aba5f12352
[App] Min replica=0 would break autoscaler component (#16092) 
* fixing the bug where num_replica=0 would fail

* changelog
 2022-12-19 13:16:04 +00:00
Akihiro Nitta 39d27f6370
[App] Improve the autoscaler UI (#16063) 
[App] Improve the autoscaler UI (#16063)
 2022-12-19 10:25:24 +00:00
Sherin Thomas 23013be28a
[App] Serve datatypes with better client code (#16018) 2022-12-16 16:03:08 +00:00
Justus Schock 3d509f6902
Update Multinode Warning (#16091) 2022-12-16 15:13:34 +00:00
Carlos Mocholí b1ce2639f4
Better check for programmatic lightningignore (#16080) 
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2022-12-16 09:49:17 +00:00
Carlos Mocholí 53ceb156c4
Integrate lightning_utilities==0.4.2 (#15817) 
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2022-12-13 13:13:51 +00:00
Adrian Wälchli fcd3195e68
Fix DDPStrategy import in app framework after #14952 (#16029) 2022-12-13 10:01:33 +00:00
Akihiro Nitta c1d0156e1d
[App] Fix `AutoScaler` trying to replicate multiple works in a single machine (#15991) 
* dont try to replicate new works in the existing machine

* update chglog

* Update comment

* Update src/lightning_app/components/auto_scaler.py

* add test
 2022-12-11 00:56:46 +00:00
Justus Schock d21b8992ee
Fix multinode cloud component (#15965) 
* fix multinode cloud component
* add tests
 2022-12-09 12:02:58 +00:00
Justus Schock 36aecde695
Multinode on MPS (#15748) 
* Fix restarting attribute for lr finder
* update lite executor
* update trainer executor
* update spawn executor
* add multinode component tests
* add testing helpers
* add lite tests
* add trainer tests
* update changelog
* update trainer
* update workflow
* update tests
* debug
* add reason for skipif
* Apply suggestions from code review
* switch skipif

Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
 2022-12-08 12:05:18 +00:00
Akihiro Nitta 64b19fb16f
[App] Introduce auto scaler (#15769) 
* Exlucde __pycache__ in setuptools

* Add load balancer example

* wip

* Update example

* rename

* remove prints

* _LoadBalancer -> LoadBalancer

* AutoScaler(work)

* change var name

* remove locust

* Update docs

* include autoscaler in api ref

* docs typo

* docs typo

* docs typo

* docs typo

* remove unused loadtest

* remove unused device_type

* clean up

* clean up

* clean up

* Add docstring

* type

* env vars to args

* expose an API for users to override to customise autoscaling logic

* update example

* comment

* udpate var name

* fix scale mechanism and clean up

* Update exampl

* ignore mypy

* Add test file

* .

* update impl and update tests

* Update changlog

* .

* revert docs

* update test

* update state to keep calling 'flow.run()'

Co-authored-by: Aniket Maurya <theaniketmaurya@gmail.com>

* Add aiohttp to base requirements

* Update docs

Co-authored-by: Luca Antiga <luca.antiga@gmail.com>

* Use deserializer utility

* fake trigger

* wip: protect /system/* with basic auth

* read password at runtime

* Change env var name

* import torch as optional

* Don't overcreate works

* simplify imports

* Update example

* aiohttp

* Add work_args work_kwargs

* More docs

* remove FIXME

* Apply Jirka's suggestions

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean example device

* add comment on init threshold value

* bad merge

* nit: logging format

* {in,out}put_schema -> {in,out}put_type

* lowercase

* docs on seconds

* process_time -> processing_time

* Dont modify work state from flow

* Update tests

* worker_url -> endpoint

* fix exampl

* Fix default scale logic

* Fix default scale logic

* Fix num_pending_works

* Update num_pending_works

* Fix bug creating too many works

* Remove up/downscale_threshold args

* Update example

* Add typing

* Fix example in docstring

* Fix default scale logic

* Update src/lightning_app/components/auto_scaler.py

Co-authored-by: Noha Alon <nohalon@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rename method

* rename locvar

* Add todo

* docs ci

* docs ci

* asdfafsdasdf pls docs

* Apply suggestions from code review

Co-authored-by: Ethan Harris <ethanwharris@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* .

* doc

* Update src/lightning_app/components/auto_scaler.py

Co-authored-by: Noha Alon <nohalon@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit 24983a0a5a.

* Revert "Update src/lightning_app/components/auto_scaler.py"

This reverts commit 56ea78b45f.

* Remove redefinition

* Remove load balancer run blocker

* raise RuntimeError

* remove has_sent

* lower the default timeout_batching from 10 to 1

* remove debug

* update the default timeout_batching

* .

* tighten condition

* fix endpoint

* typo in runtimeerror cond

* async lock update severs

* add a test

* {in,out}put_type typing

* Update examples/app_server_with_auto_scaler/app.py

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* Update .actions/setup_tools.py

Co-authored-by: Aniket Maurya <theaniketmaurya@gmail.com>
Co-authored-by: Luca Antiga <luca.antiga@gmail.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Noha Alon <nohalon@gmail.com>
Co-authored-by: Ethan Harris <ethanwharris@gmail.com>
Co-authored-by: Akihiro Nitta <aki@pop-os.localdomain>
Co-authored-by: thomas chaton <thomas@grid.ai>
 2022-12-07 14:27:44 +01:00
Adrian Wälchli a970f090a0
Ignore `num_nodes` when running MultiNode components locally (#15806) 2022-11-24 17:21:32 +00:00
Luca Antiga 0a1273187a
Add code_dir argument to tracer run (#15771) 2022-11-24 14:49:06 +00:00
Jirka Borovec 952b64b358
tests: split examples and pytests (#15774) 
split examples and pytests
 2022-11-23 20:15:16 +09:00
Aniket Maurya f9d906c340
add title and description to ServeGradio (#15639) 
* add title and description
* update test
* apply suggestions

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
 2022-11-12 20:48:37 +00:00
thomas chaton 757413c8e0
[App] Accelerate Multi Node Startup Time (#15650) 2022-11-11 20:08:16 +00:00
Sherin Thomas 136a090f99
Sample datatype for Serve Component (#15623) 
* introducing serve component

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up tests

* clean up tests

* doctest

* mypy

* structure-fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cleanup

* cleanup

* test fix

* addition

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* requirements

* getting future url

* url for local

* sample data typeg

* changes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* prediction

* updates

* updates

* manifest

* fix type error

* fixed test

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Rick Izzo <rick@grid.ai>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
 2022-11-10 14:39:36 -05:00
thomas chaton 10cdf0dd2c
[App] Introduce Bi-directional queues (#15582) 
* update

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update

* updatre

* update

* update

* update

* updar

* update

* update

* update

* remove print

* update

* update

* update

* update

* update

* update

* update

* update
 2022-11-10 12:21:51 +01:00
Sherin Thomas f1bd68da3b
Serve component (#15609) 
Serve component (#15609)
 2022-11-10 15:31:07 +05:30
Luca Antiga 5cf50363c3
Fix handling of script arguments in tracer (#15518) 
* Don't assume script args start with double dash

* add changelog

Co-authored-by: Luca Antiga <luca@lightning.ai>
Co-authored-by: awaelchli <aedu.waelchli@gmail.com>
 2022-11-04 16:03:46 +01:00
Luca Antiga 16960276ff
Periodically sync database to the drive (#15441) 2022-11-03 21:07:14 +00:00
Ethan Harris b340859df9
[App] Add `ServeStreamlit` work (#15400) 
* Init

* Updates

* Add test for model building

* Imports

* Fix

* Typing

* Ignore serve streamlit in mypy
 2022-10-28 16:33:53 -04:00
Ethan Harris e9a6b83437
[App] Reduce import depths and add test (#15330) 
Co-authored-by: thomas chaton <thomas@grid.ai>
 2022-10-28 13:57:35 +00:00
thomas chaton 979d728563
Add Database Component (#14995) 2022-10-19 19:52:12 +00:00