2019-08-06 20:37:58 +00:00
< div align = "center" >
2020-10-08 11:19:57 +00:00
< img src = "docs/source/_images/logos/lightning-logo.png" width = "400px" >
2019-03-31 19:32:35 +00:00
2019-08-05 20:02:48 +00:00
2020-10-08 11:27:13 +00:00
**The lightweight PyTorch wrapper for high-performance AI research.
2020-10-08 11:20:43 +00:00
Scale your models, not the boilerplate.**
2019-08-05 20:02:48 +00:00
2020-10-08 11:29:57 +00:00
```diff
2020-10-11 02:09:01 +00:00
- NEWS: Lightning 1.0 is currently in rc and is being released for GA Oct 11!
2020-10-08 11:29:57 +00:00
```
2020-10-08 11:28:32 +00:00
---
2020-08-20 16:45:40 +00:00
< p align = "center" >
2020-10-08 13:17:51 +00:00
< a href = "https://www.pytorchlightning.ai/" > Website< / a > •
2020-08-20 16:45:40 +00:00
< a href = "#key-features" > Key Features< / a > •
< a href = "#how-to-use" > How To Use< / a > •
2020-09-21 20:31:54 +00:00
< a href = "https://pytorch-lightning.readthedocs.io/en/stable/" > Docs< / a > •
2020-09-21 20:34:55 +00:00
< a href = "#examples" > Examples< / a > •
2020-08-20 16:45:40 +00:00
< a href = "#community" > Community< / a > •
< a href = "#licence" > Licence< / a >
< / p >
2020-09-15 18:32:27 +00:00
<!-- DO NOT ADD CONDA DOWNLOADS... README CHANGES MUST BE APPROVED BY EDEN OR WILL -->
2020-09-14 08:35:14 +00:00
[](https://pypi.org/project/pytorch-lightning/)
2019-08-05 20:02:48 +00:00
[](https://badge.fury.io/py/pytorch-lightning)
2019-08-18 23:17:25 +00:00
[](https://pepy.tech/project/pytorch-lightning)
2020-09-02 15:50:02 +00:00
[](https://anaconda.org/conda-forge/pytorch-lightning)
2020-08-14 20:05:53 +00:00
[](https://hub.docker.com/r/pytorchlightning/pytorch_lightning)
2020-03-14 17:01:57 +00:00
[](https://codecov.io/gh/PyTorchLightning/pytorch-lightning)
2020-04-27 21:41:46 +00:00
2020-05-15 12:36:40 +00:00
[](https://pytorch-lightning.readthedocs.io/en/stable/)
2020-06-17 19:56:19 +00:00
[](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-f6bl2l0l-JYMK3tbAgAmGRrlNr00f1A)
2020-09-02 15:50:02 +00:00
[](https://forums.pytorchlightning.ai/)
2020-01-14 12:05:26 +00:00
[](https://github.com/PytorchLightning/pytorch-lightning/blob/master/LICENSE)
2020-09-02 15:50:02 +00:00
[](https://shields.io/)
2019-10-06 16:20:13 +00:00
2020-03-27 12:36:50 +00:00
<!--
2020-06-03 16:23:14 +00:00
[](https://www.codefactor.io/repository/github/pytorchlightning/pytorch-lightning)
2019-08-15 13:54:29 +00:00
-->
2019-08-06 20:37:58 +00:00
< / div >
2019-08-05 20:02:48 +00:00
2020-08-20 15:45:28 +00:00
###### *Codecov is > 90%+ but build delays may show less
2020-09-21 21:04:05 +00:00
---
2020-10-08 13:19:14 +00:00
## Just launched GridAI
Our native platform for training models at scale on the cloud!
2020-10-09 08:43:31 +00:00
**Sign up for [early access here ](https://www.grid.ai/ )**
2020-10-08 13:17:51 +00:00
To use grid, take your regular command:
```
2020-10-08 13:20:32 +00:00
python my_model.py --learning_rate 1e-6 --layers 2 --gpus 4
2020-10-08 13:17:51 +00:00
```
And change it to use the grid train command:
```
2020-10-08 13:20:32 +00:00
grid train --grid_gpus 4 my_model.py --learning_rate 'uniform(1e-6, 1e-1, 20)' --layers '[2, 4, 8, 16]'
2020-10-08 13:17:51 +00:00
```
The above command will launch (20 * 4) experiments each running on 4 GPUs (320 GPUs!) - by making ZERO changes to
your code.
---
2020-08-20 16:45:40 +00:00
## PyTorch Lightning is just organized PyTorch
2020-09-21 20:42:38 +00:00
Lightning disentangles PyTorch code to decouple the science from the engineering.
2020-08-20 16:45:40 +00:00
2020-09-21 20:42:38 +00:00
---
## Lightning Philosophy
Lightning is designed with these principles in mind:
2020-09-21 20:45:25 +00:00
Principle 1: Enable maximal flexibility.
Principle 2: Abstract away unecessary boilerplate, but make it accessible when needed.
Principle 3: Systems should be self-contained (ie: optimizers, computation code, etc).
Principle 4: Deep learning code should be organized into 4 distinct categories.
2020-09-21 20:44:44 +00:00
- Research code (the LightningModule).
- Engineering code (you delete, and is handled by the Trainer).
- Non-essential research code (logging, etc... this goes in Callbacks).
- Data (use PyTorch Dataloaders or organize them into a LightningDataModule).
2020-08-20 16:45:40 +00:00
Once you do this, you can train on multiple-GPUs, TPUs, CPUs and even in 16-bit precision without changing your code!
2020-10-09 08:43:31 +00:00
Get started with our [2 step guide ](https://pytorch-lightning.readthedocs.io/en/stable/new-project.html )
2020-08-20 16:45:40 +00:00
2020-07-25 15:22:08 +00:00
---
2020-10-08 11:27:13 +00:00
## Inference
2020-10-09 08:43:31 +00:00
Lightning is also designed for the fast inference AI researchers and production teams need to scale up things like BERT and self-supervised learning.
Lightning can automatically export to ONNX or TorchScript for those cases.
2020-10-08 11:27:13 +00:00
---
2020-06-18 21:55:35 +00:00
## Trending contributors
2020-06-29 10:34:19 +00:00
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/0)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/1)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/2)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/3)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/4)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/5)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/6)
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/7)
2020-06-18 21:55:35 +00:00
2020-02-14 11:49:32 +00:00
---
2020-06-18 21:54:29 +00:00
2020-02-14 11:49:32 +00:00
## Continuous Integration
< center >
2020-09-25 14:20:15 +00:00
| System / PyTorch ver. | 1.3 (min. req.)* | 1.4 | 1.5 | 1.6 (latest) | 1.7 (nightly) |
| :---: | :---: | :---: | :---: | :---: | :---: |
2020-10-06 21:30:16 +00:00
| Conda py3.7 [linux] | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) |
2020-09-30 12:33:22 +00:00
| Linux py3.7 [GPUs**] | - | - |[](http://104.154.220.231/PyTorchLightning/pytorch-lightning) | - | - |
2020-09-25 14:20:15 +00:00
| Linux py3.7 [TPUs***] | - | - | - | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22TPU+tests%22+branch%3Amaster) | - |
2020-10-06 21:30:16 +00:00
| Linux py3.6 / py3.7 / py3.8 | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
| OSX py3.6 / py3.7 | - | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
| Windows py3.6 / py3.7 / py3.8 | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
2020-02-14 11:49:32 +00:00
2020-08-02 10:34:36 +00:00
- _\* `torch>=1.4` is the minimal pytorch version for Python 3.8_
- _\** tests run on two NVIDIA K80_
- _\*** tests run on Google GKE TPUv2/3_
2020-09-30 12:36:02 +00:00
- _TPU w/ py3.6/py3.7 means we support Colab and Kaggle env._
2020-08-02 10:34:36 +00:00
2020-02-14 11:49:32 +00:00
< / center >
2020-08-08 00:21:51 +00:00
---
2020-08-20 15:45:28 +00:00
2020-08-20 16:45:40 +00:00
## How To Use
2020-09-21 21:06:40 +00:00
#### Step 0: Install
2020-08-20 16:45:40 +00:00
Simple installation from PyPI
```bash
pip install pytorch-lightning
```
From Conda
```bash
conda install pytorch-lightning -c conda-forge
```
2020-08-22 13:02:27 +00:00
Install bleeding-edge (no guarantees)
```bash
pip install git+https://github.com/PytorchLightning/pytorch-lightning.git@master --upgrade
```
2020-09-21 21:06:40 +00:00
#### Step 0: Add these imports
2020-05-12 12:46:22 +00:00
```python
2020-08-20 15:45:28 +00:00
import os
import torch
2020-09-21 15:17:59 +00:00
from torch import nn
2020-08-20 15:45:28 +00:00
import torch.nn.functional as F
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader, random_split
from torchvision import transforms
import pytorch_lightning as pl
```
2020-05-12 12:46:22 +00:00
2020-09-23 11:39:46 +00:00
#### Step 1: Define a LightningModule (nn.Module subclass)
2020-09-21 15:17:59 +00:00
A LightningModule defines a full *system* (ie: a GAN, autoencoder, BERT or a simple Image Classifier).
2020-08-20 15:45:28 +00:00
```python
2020-09-21 15:17:59 +00:00
class LitAutoEncoder(pl.LightningModule):
2020-05-12 12:46:22 +00:00
def __init__ (self):
2020-05-12 12:59:23 +00:00
super().__init__()
2020-09-21 15:17:59 +00:00
self.encoder = nn.Sequential(nn.Linear(28 * 28, 128), nn.ReLU(), nn.Linear(128, 3))
self.decoder = nn.Sequential(nn.Linear(3, 128), nn.ReLU(), nn.Linear(128, 28 * 28))
2020-09-22 18:00:02 +00:00
def forward(self, x):
# in lightning, forward defines the prediction/inference actions
embedding = self.encoder(x)
return embedding
2020-05-12 12:46:22 +00:00
2020-08-20 15:45:28 +00:00
def training_step(self, batch, batch_idx):
2020-09-23 11:36:51 +00:00
# training_step defined the train loop. It is independent of forward
2020-08-20 15:45:28 +00:00
x, y = batch
2020-09-21 15:17:59 +00:00
x = x.view(x.size(0), -1)
z = self.encoder(x)
x_hat = self.decoder(z)
loss = F.mse_loss(x_hat, x)
2020-10-09 23:11:54 +00:00
self.log('train_loss', loss)
2020-09-21 15:17:59 +00:00
return loss
2020-05-12 12:46:22 +00:00
def configure_optimizers(self):
2020-09-21 15:17:59 +00:00
optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
return optimizer
```
2020-05-12 12:46:22 +00:00
2020-09-22 18:00:02 +00:00
###### Note: Training_step defines the training loop. Forward defines how the LightningModule behaves during inference/prediction.
2020-09-21 15:17:59 +00:00
#### Step 2: Train!
```python
2020-08-20 15:45:28 +00:00
dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
train, val = random_split(dataset, [55000, 5000])
2020-05-12 12:46:22 +00:00
2020-09-21 15:17:59 +00:00
autoencoder = LitAutoEncoder()
trainer = pl.Trainer()
trainer.fit(autoencoder, DataLoader(train), DataLoader(val))
2020-05-12 12:46:22 +00:00
```
2020-08-20 16:45:40 +00:00
#### And without changing a single line of code, you could run on GPUs
```python
# 8 GPUs
trainer = Trainer(max_epochs=1, gpus=8)
# 256 GPUs
trainer = Trainer(max_epochs=1, gpus=8, num_nodes=32)
```
Or TPUs
```python
# Distributes TPU core training
trainer = Trainer(tpu_cores=8)
# Single TPU core training
trainer = Trainer(tpu_cores=[1])
```
2020-10-11 02:48:50 +00:00
#### For advanced users, you can still own complex training loops
```python
class LitAutoEncoder(pl.LightningModule):
def training_step(self, batch, batch_idx, opt_idx):
(opt_a, opt_b) = self.optimizers()
loss_a = ...
self.manual_backward(loss_a, opt_a)
opt_a.step()
opt_a.zero_grad()
loss_b = ...
self.manual_backward(loss_b, opt_b, retain_graph=True)
self.manual_backward(loss_b, opt_b)
opt_b.step()
opt_b.zero_grad()
```
2020-09-21 15:19:29 +00:00
---
## Key Features
* Scale your models to run on any hardware (CPU, GPUs, TPUs) without changing your model
* Making code more readable by decoupling the research code from the engineering
* Easier to reproduce
* Less error prone by automating most of the training loop and tricky engineering
* Keeps all the flexibility (LightningModules are still PyTorch modules), but removes a ton of boilerplate
* Lightning has out-of-the-box integration with the popular logging/visualizing frameworks ([Tensorboard](https://pytorch.org/docs/stable/tensorboard.html), [MLFlow ](https://mlflow.org/ ), [Neptune.ai ](https://neptune.ai/ ), [Comet.ml ](https://www.comet.ml/site/ ), [Wandb ](https://www.wandb.com/ )).
* [Tested rigorously with every new PR ](https://github.com/PyTorchLightning/pytorch-lightning/tree/master/tests ). We test every combination of PyTorch and Python supported versions, every OS, multi GPUs and even TPUs.
* Minimal running speed overhead (about 300 ms per epoch compared with pure PyTorch).
### Lightning automates 40+ parts of DL/ML research
- GPU training
- Distributed GPU (cluster) training
- TPU training
- EarlyStopping
- Logging/Visualizing
- Checkpointing
- Experiment management
- [Full list here ](https://pytorch-lightning.readthedocs.io/en/latest/#common-use-cases )
2020-08-20 15:45:28 +00:00
---
2020-09-21 20:34:55 +00:00
## Examples
2020-08-20 16:45:40 +00:00
2020-08-20 15:45:28 +00:00
###### Hello world
2020-09-22 18:27:52 +00:00
[MNIST hello world ](https://colab.research.google.com/github/PytorchLightning/pytorch-lightning/blob/master/notebooks/01-mnist-hello-world.ipynb )
2020-08-20 16:45:40 +00:00
[MNIST on TPUs ](https://colab.research.google.com/drive/1-_LKx4HwAxl5M6xPJmqAAu444LTDQoa3 )
2020-08-20 15:45:28 +00:00
###### Contrastive Learning
[BYOL ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/self_supervised_models.html#byol )
[CPC v2 ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/self_supervised_models.html#cpc-v2 )
[Moco v2 ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/self_supervised_models.html#moco-v2 )
2020-08-20 16:45:40 +00:00
[SIMCLR ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/self_supervised_models.html#simclr )
2020-08-20 15:45:28 +00:00
###### NLP
2020-09-22 18:27:52 +00:00
[BERT ](https://colab.research.google.com/github/PytorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb )
2020-08-20 16:45:40 +00:00
[GPT-2 ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/convolutional.html#gpt-2 )
2020-08-20 15:45:28 +00:00
###### Reinforcement Learning
2020-07-28 15:43:47 +00:00
[DQN ](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=NWvMLBDySQI5 )
2020-08-20 15:45:28 +00:00
[Dueling-DQN ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/reinforce_learn.html#dueling-dqn )
[Reinforce ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/reinforce_learn.html#reinforce )
###### Vision
2020-09-22 18:27:52 +00:00
[GAN ](https://colab.research.google.com/github/PytorchLightning/pytorch-lightning/blob/master/notebooks/03-basic-gan.ipynb )
2020-08-20 15:45:28 +00:00
###### Classic ML
[Logistic Regression ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/classic_ml.html#logistic-regression )
2020-08-20 16:45:40 +00:00
[Linear Regression ](https://pytorch-lightning-bolts.readthedocs.io/en/latest/classic_ml.html#linear-regression )
2020-08-20 15:45:28 +00:00
---
2019-12-12 19:06:20 +00:00
2020-08-20 16:45:40 +00:00
## Community
2020-04-03 14:04:54 +00:00
2020-09-21 20:37:26 +00:00
The lightning community is maintained by
2020-08-27 15:27:54 +00:00
- [16 core contributors ](https://pytorch-lightning.readthedocs.io/en/latest/governance.html ) who are all a mix of professional engineers, Research Scientists, Ph.D. students from top AI labs.
2020-09-21 20:37:26 +00:00
- 280+ community contributors.
2020-08-20 16:45:40 +00:00
Lightning is also part of the [PyTorch ecosystem ](https://pytorch.org/ecosystem/ ) which requires projects to have solid testing, documentation and support.
### Asking for help
If you have any questions please:
2020-08-31 14:52:45 +00:00
1. [Read the docs ](https://pytorch-lightning.rtfd.io/en/latest/ ).
2. [Look it up in our forum (or add a new question) ](https://forums.pytorchlightning.ai/ )
2020-08-20 16:45:40 +00:00
2. [Search through the issues ](https://github.com/PytorchLightning/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question ).
3. [Join our slack ](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-f6bl2l0l-JYMK3tbAgAmGRrlNr00f1A ).
4. [Ask on stackoverflow ](https://stackoverflow.com/questions/ask?guided=false ) with the tag pytorch-lightning.
### Funding
Building open-source software with only a few part-time people is hard! We've secured funding to make sure we can
hire a full-time staff, attend conferences, and move faster through implementing features you request.
Our goal is to build an incredible research platform and a big supportive community. Many open-source projects
have gone on to fund operations through things like support and special help for big corporations!
If you are one of these corporations, please feel free to reach out to will@pytorchlightning.ai!
2020-06-09 11:43:33 +00:00
2020-08-20 15:45:28 +00:00
---
2020-08-20 02:03:22 +00:00
## Licence
2020-08-20 16:45:40 +00:00
2020-08-20 02:03:22 +00:00
Please observe the Apache 2.0 license that is listed in this repository. In addition
the Lightning framework is Patent Pending.
2020-06-08 11:22:54 +00:00
## BibTeX
2019-11-05 16:52:50 +00:00
If you want to cite the framework feel free to use this (but only if you loved it 😊):
2020-04-28 03:54:20 +00:00
2020-04-03 13:52:41 +00:00
```bibtex
2020-04-28 03:54:20 +00:00
@article {falcon2019pytorch,
title={PyTorch Lightning},
author={Falcon, WA},
2020-09-14 04:19:09 +00:00
journal={GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning},
2020-04-28 03:54:20 +00:00
volume={3},
year={2019}
2019-11-05 16:53:12 +00:00
}
```
