Build and train PyTorch models and connect them to the ML lifecycle using Lightning App templates, without handling DIY infrastructure, cost management, scaling, and other headaches.
Go to file
William Falcon 5b2351cbb9
added slurm doc (#1418)
* added slurm doc

* added slurm doc
2020-04-08 14:43:50 -04:00
.circleci faster CI testing (#1323) 2020-04-02 12:28:44 -04:00
.github fix missing images on pypi (#1407) 2020-04-07 15:42:19 -04:00
benchmarks Add parity test for simple RNN (#1351) 2020-04-03 09:43:14 -04:00
docs added slurm doc (#1418) 2020-04-08 14:43:50 -04:00
pl_examples Set precision=16 when use_amp is passed as True (#1145) 2020-04-06 08:13:24 -04:00
pytorch_lightning Update __init__.py 2020-04-08 11:53:23 -04:00
tests Added accumulation of loggers' metrics for the same steps (#1278) 2020-04-08 08:35:47 -04:00
.codecov.yml enable Codecov (#1133) 2020-03-14 13:01:57 -04:00
.drone.yml faster CI testing (#1323) 2020-04-02 12:28:44 -04:00
.gitignore Add MNIST dataset & drop torchvision dep. from tests (#986) 2020-03-30 18:25:37 -04:00
.mergify.yml add python 3.8 testing (#915) 2020-04-02 12:20:45 -04:00
.pep8speaks.yml improve partial Codecov (#1172) 2020-03-19 09:14:29 -04:00
.readthedocs.yml update org paths & convert logos (#685) 2020-01-20 14:50:31 -05:00
.run_local_tests.sh CI: split tests-examples (#990) 2020-03-25 07:46:27 -04:00
CHANGELOG.md Added accumulation of loggers' metrics for the same steps (#1278) 2020-04-08 08:35:47 -04:00
LICENSE update license (#809) 2020-02-09 14:18:50 -05:00
MANIFEST.in release update (#1405) 2020-04-07 14:04:32 -04:00
README.md added slurm doc (#1418) 2020-04-08 14:43:50 -04:00
environment.yml Add MNIST dataset & drop torchvision dep. from tests (#986) 2020-03-30 18:25:37 -04:00
pyproject.toml Split callbacks (#849) 2020-02-22 21:45:34 -05:00
requirements-extra.txt Fix requirements-extra.txt Trains package to release version (#1229) 2020-03-27 08:32:19 +01:00
requirements.txt CI: split tests-examples (#990) 2020-03-25 07:46:27 -04:00
setup.cfg system info (#1234) 2020-03-27 08:45:52 -04:00
setup.py Temporary Fix for docs build failure (#1413) 2020-04-08 08:18:36 -04:00
update.sh add CircleCI 2019-08-06 22:45:46 +02:00

README.md

Logo

PyTorch Lightning

The lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate.

PyPI Status PyPI Status codecov CodeFactor

ReadTheDocs Slack license Next Release


Continuous Integration

System / PyTorch ver. 1.1 1.2 1.3 1.4
Linux py3.6 [CPU] CircleCI CircleCI CircleCI CircleCI
Linux py3.7 [GPU] - - - Build Status
Linux py3.6 / py3.7 / py3.8 CI testing - - CI testing
OSX py3.6 / py3.7 / py3.8 CI testing - - CI testing
Windows py3.6 / py3.7 / py3.8 CI testing - - CI testing

Simple installation from PyPI

pip install pytorch-lightning

Docs

Demo

MNIST, GAN, BERT, DQN on COLAB!
MNIST on TPUs

What is it?

Lightning is a way to organize your PyTorch code to decouple the science code from the engineering. It's more of a style-guide than a framework.

In Lightning, you organize your code into 3 distinct categories:

  1. Research code (goes in the LightningModule).
  2. Engineering code (you delete, and is handled by the Trainer).
  3. Non-essential research code (logging, etc... this goes in Callbacks).

Here's an example of how to refactor your research code into a LightningModule.

PT to PL

The rest of the code is automated by the Trainer! PT to PL

Testing Rigour

All the automated code by the Trainer is tested rigorously with every new PR.

In fact, we also train a few models using a vanilla PyTorch loop and compare with the same model trained using the Trainer to make sure we achieve the EXACT same results. Check out the parity tests here.

Overall, Lightning guarantees rigorously tested, correct, modern best practices for the automated parts.

How flexible is it?

As you see, you're just organizing your PyTorch code - there's no abstraction.

And for the stuff that the Trainer abstracts out you can override any part you want to do things like implement your own distributed training, 16-bit precision, or even a custom backwards pass.

For example, here you could do your own backward pass

class LitModel(LightningModule):
  def optimizer_step(self, current_epoch, batch_idx, optimizer, optimizer_idx,
                     second_order_closure=None):
      optimizer.step()
      optimizer.zero_grad()

For anything else you might need, we have an extensive callback system you can use to add arbitrary functionality not implemented by our team in the Trainer.

Who is Lightning for?

  • Professional researchers
  • PhD students
  • Corporate production teams

If you're just getting into deep learning, we recommend you learn PyTorch first! Once you've implemented a few models, come back and use all the advanced features of Lightning :)

What does lightning control for me?

Everything in Blue! This is how lightning separates the science (red) from the engineering (blue).

Overview

How much effort is it to convert?

If your code is not a huge mess you should be able to organize it into a LightningModule in less than 1 hour. If your code IS a mess, then you needed to clean up anyhow ;)

Check out this step-by-step guide.

Starting a new project?

Use our seed-project aimed at reproducibility!

Why do I want to use lightning?

Although your research/production project might start simple, once you add things like GPU AND TPU training, 16-bit precision, etc, you end up spending more time engineering than researching. Lightning automates AND rigorously tests those parts for you.

Support

  • 8 core contributors who are all a mix of professional engineers, Research Scientists, PhD students from top AI labs.
  • 100+ community contributors.

Lightning is also part of the PyTorch ecosystem which requires projects to have solid testing, documentation and support.


README Table of Contents


Realistic example

Here's how you would organize a realistic PyTorch project into Lightning.

PT to PL

The LightningModule defines a system such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier.

In summary, you:

  1. Define a LightningModule
    class LitSystem(pl.LightningModule):

        def __init__(self):
            super().__init__()
            # not the best model...
            self.l1 = torch.nn.Linear(28 * 28, 10)

        def forward(self, x):
            return torch.relu(self.l1(x.view(x.size(0), -1)))

        def training_step(self, batch, batch_idx):
            ...
  1. Fit it with a Trainer
from pytorch_lightning import Trainer

model = LitSystem()

# most basic trainer, uses good defaults
trainer = Trainer()
trainer.fit(model)

Check out the COLAB demo here

What types of research works?

Anything! Remember, that this is just organized PyTorch code. The Training step defines the core complexity found in the training loop.

Could be as complex as a seq2seq

# define what happens for training here
def training_step(self, batch, batch_idx):
    x, y = batch

    # define your own forward and loss calculation
    hidden_states = self.encoder(x)

    # even as complex as a seq-2-seq + attn model
    # (this is just a toy, non-working example to illustrate)
    start_token = '<SOS>'
    last_hidden = torch.zeros(...)
    loss = 0
    for step in range(max_seq_len):
        attn_context = self.attention_nn(hidden_states, start_token)
        pred = self.decoder(start_token, attn_context, last_hidden)
        last_hidden = pred
        pred = self.predict_nn(pred)
        loss += self.loss(last_hidden, y[step])

    #toy example as well
    loss = loss / max_seq_len
    return {'loss': loss}

Or as basic as CNN image classification

# define what happens for validation here
def validation_step(self, batch, batch_idx):
    x, y = batch

    # or as basic as a CNN classification
    out = self(x)
    loss = my_loss(out, y)
    return {'loss': loss}

And without changing a single line of code, you could run on CPUs

trainer = Trainer(max_epochs=1)

Or GPUs

# 8 GPUs
trainer = Trainer(max_epochs=1, gpus=8)

# 256 GPUs
trainer = Trainer(max_epochs=1, gpus=8, num_nodes=32)

Or TPUs

trainer = Trainer(num_tpu_cores=8)

When you're done training, run the test accuracy

trainer.test()

Visualization

Lightning has out-of-the-box integration with the popular logging/visualizing frameworks

tensorboard-support

Lightning automates 40+ parts of DL/ML research

  • GPU training
  • Distributed GPU (cluster) training
  • TPU training
  • EarlyStopping
  • Logging/Visualizing
  • Checkpointing
  • Experiment management
  • Full list here

Examples

Check out this awesome list of research papers and implementations done with Lightning.

Tutorials

Check out our introduction guide to get started. Or jump straight into our tutorials.


Asking for help

Welcome to the Lightning community!

If you have any questions, feel free to:

  1. read the docs.
  2. Search through the issues.
  3. Ask on stackoverflow with the tag pytorch-lightning.
  4. Join our slack.

FAQ

How do I use Lightning for rapid research? Here's a walk-through

Why was Lightning created? Lightning has 3 goals in mind:

  1. Maximal flexibility while abstracting out the common boilerplate across research projects.
  2. Reproducibility. If all projects use the LightningModule template, it will be much much easier to understand what's going on and where to look! It will also mean every implementation follows a standard format.
  3. Democratizing PyTorch power user features. Distributed training? 16-bit? know you need them but don't want to take the time to implement? All good... these come built into Lightning.

How does Lightning compare with Ignite and fast.ai? Here's a thorough comparison.

Is this another library I have to learn? Nope! We use pure Pytorch everywhere and don't add unnecessary abstractions!

Are there plans to support Python 2? Nope.

Are there plans to support virtualenv? Nope. Please use anaconda or miniconda.

Which PyTorch versions do you support?

  • PyTorch 1.1.0
    # install pytorch 1.1.0 using the official instructions
    
    # install test-tube 0.6.7.6 which supports 1.1.0
    pip install test-tube==0.6.7.6
    
    # install latest Lightning version without upgrading deps
    pip install -U --no-deps pytorch-lightning
    
  • PyTorch 1.2.0, 1.3.0, Install via pip as normal

Custom installation

Bleeding edge

If you can't wait for the next release, install the most up to date code with:

  • using GIT (locally clone whole repo with full history)
    pip install git+https://github.com/PytorchLightning/pytorch-lightning.git@master --upgrade
    
  • using instant zip (last state of the repo without git history)
    pip install https://github.com/PytorchLightning/pytorch-lightning/archive/master.zip --upgrade
    

Any release installation

You can also install any past release 0.X.Y from this repository:

pip install https://github.com/PytorchLightning/pytorch-lightning/archive/0.X.Y.zip --upgrade

Lightning team

Leads

Core Maintainers

Bibtex

If you want to cite the framework feel free to use this (but only if you loved it 😊):

@misc{Falcon2019,
  author = {Falcon, W.A. et al.},
  title = {PyTorch Lightning},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/PytorchLightning/pytorch-lightning}}
}