Build and train PyTorch models and connect them to the ML lifecycle using Lightning App templates, without handling DIY infrastructure, cost management, scaling, and other headaches.
Go to file
Jiri BOROVEC 79f0856bc8 update setup 2019-08-06 22:45:27 +02:00
docs update README 2019-08-06 22:45:27 +02:00
examples pkg relative imports 2019-08-05 10:52:09 +02:00
pytorch_lightning update setup 2019-08-06 22:45:27 +02:00
tests fix tests req. 2019-08-06 22:44:13 +02:00
.codecov.yml add Codecov 2019-08-06 22:44:13 +02:00
.gitignore added coverage badge 2019-07-24 17:39:20 -07:00
.readthedocs.yml pkg relative imports 2019-08-05 10:52:09 +02:00
.travis.yml use tox 2019-08-06 22:44:13 +02:00
MANIFEST.in Fix pip install too 2019-04-03 22:47:55 +05:30
README.md update README 2019-08-06 22:45:27 +02:00
appveyor.yml use tox 2019-08-06 22:44:13 +02:00
mkdocs.yml updated docs 2019-08-01 10:11:26 -04:00
pyproject.toml Fix pip install too 2019-04-03 22:47:55 +05:30
requirements.txt pkg relative imports 2019-08-05 10:52:09 +02:00
setup.cfg pt dpp some ignores 2019-07-24 19:36:35 -04:00
setup.py update setup 2019-08-06 22:45:27 +02:00
tox.ini use tox 2019-08-06 22:44:13 +02:00
update.sh beta release to pypi 2019-03-31 15:26:23 -04:00

README.md

Logo

PyTorch Lightning

The PyTorch Keras for ML researchers. More control. Less boilerplate.

PyPI Status PyPI Status Build Status Build status codecov CodeFactor ReadTheDocs license

Simple installation from PyPI

pip install pytorch-lightning  

Docs

View the docs here

What is it?

Lightning is a very lightweight wrapper on PyTorch. It defers core training and validation logic to you and automates the rest. It guarantees correct, modern best practices for the core training logic.

Why do I want to use lightning?

When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.

With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic.

Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!

README Table of Contents

How do I do use it?

To use lightning do 2 things:

  1. Define a LightningModel
import os
import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
import torchvision.transforms as transforms

import pytorch_lightning as ptl

class CoolModel(ptl.LightningModule):

    def __init__(self):
        super(CoolModel, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def my_loss(self, y_hat, y):
        return F.cross_entropy(y_hat, y)

    def training_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
        return {'loss': self.my_loss(y_hat, y)}

    def validation_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
        return {'val_loss': self.my_loss(y_hat, y)}

    def validation_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

    def configure_optimizers(self):
        return [torch.optim.Adam(self.parameters(), lr=0.02)]

    @ptl.data_loader
    def tng_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @ptl.data_loader
    def val_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @ptl.data_loader
    def test_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
  1. Fit with a trainer
from pytorch_lightning import Trainer
from test_tube import Experiment

model = CoolModel()
exp = Experiment(save_dir=os.getcwd())

# train on cpu using only 10% of the data (for demo purposes)
trainer = Trainer(experiment=exp, max_nb_epochs=1, train_percent_check=0.1)

# train on 4 gpus
# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3])

# train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3, 4, 5, 6, 7], nb_gpu_nodes=4)

# train (1 epoch only here for demo)
trainer.fit(model)

# view tensorflow logs 
print(f'View tensorboard logs by running\ntensorboard --logdir {os.getcwd()}')
print('and going to http://localhost:6006 on your browser')

What does lightning control for me?

Everything in gray!
You define the blue parts using the LightningModule interface:

Ouverview

# what to do in the training loop
def training_step(self, data_batch, batch_nb):

# what to do in the validation loop
def validation_step(self, data_batch, batch_nb):

# how to aggregate validation_step outputs
def validation_end(self, outputs):

# and your dataloaders
def tng_dataloader():
def val_dataloader():
def test_dataloader():

Could be as complex as seq-2-seq + attention

# define what happens for training here
def training_step(self, data_batch, batch_nb):
    x, y = data_batch
    
    # define your own forward and loss calculation
    hidden_states = self.encoder(x)
     
    # even as complex as a seq-2seq + attn model
    # (this is just a toy, non-working example to illustrate)
    start_token = '<SOS>'
    last_hidden = torch.zeros(...)
    loss = 0
    for step in range(max_seq_len):
        attn_context = self.attention_nn(hidden_states, start_token)
        pred = self.decoder(start_token, attn_context, last_hidden) 
        last_hidden = pred
        pred = self.predict_nn(pred)
        loss += self.loss(last_hidden, y[step])
        
    #toy example as well
    loss = loss / max_seq_len
    return {'loss': loss} 

Or as basic as CNN image classification

# define what happens for validation here
def validation_step(self, data_batch, batch_nb):    
    x, y = data_batch
    
    # or as basic as a CNN classification
    out = self.forward(x)
    loss = my_loss(out, y)
    return {'loss': loss} 

And you also decide how to collate the output of all validation steps

def validation_end(self, outputs):
    """
    Called at the end of validation to aggregate outputs
    :param outputs: list of individual outputs of each validation step
    :return:
    """
    val_loss_mean = 0
    val_acc_mean = 0
    for output in outputs:
        val_loss_mean += output['val_loss']
        val_acc_mean += output['val_acc']

    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
    tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
    return tqdm_dic

Tensorboard

Lightning is fully integrated with tensorboard.

tensorboard-support

Lightning also adds a text column with all the hyperparameters for this experiment.

tensorboard-support

Simply note the path you set for the Experiment

from test_tube import Experiment
from pytorch-lightning import  Trainer

exp = Experiment(save_dir='/some/path')
trainer = Trainer(experiment=exp)
...

And run tensorboard from that dir

tensorboard --logdir /some/path     

Lightning automates all of the following (each is also configurable):

Checkpointing
Computing cluster (SLURM)
Debugging
Distributed training
Experiment Logging
Training loop
Validation loop

Demo

# install lightning
pip install pytorch-lightning

# clone lightning for the demo
git clone https://github.com/williamFalcon/pytorch-lightning.git
cd pytorch-lightning
cd examples/new_project_templates/

# all of the following demos use the SAME model to show no modification needs to be made to your code

# train on cpu 
python single_cpu_template.py

# train on multiple-gpus 
python single_gpu_node_template.py --gpus "0,1"

# train on 32 gpus on a cluster (run on a SLURM managed cluster)
python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'

Tutorials

FAQ

Why was Lightning created?
Lightning has 3 goals in mind:

  1. Maximal flexibility while abstracting out the common boilerplate across research projects.
  2. Reproducibility. If all projects use the LightningModule template, it will be much much easier to understand what's going on and where to look! It will also mean every implementation follows a standard format.
  3. Democratizing PyTorch power user features. Distributed training? 16-bit? know you need them but don't want to take the time to implement? All good... these come built into Lightning.

How does Lightning compare with Ignite and fast.ai?
Here's a thorough comparison.

Is this another library I have to learn?
Nope! We use pure Pytorch everywhere and don't add unecessary abstractions!

Are there plans to support Python 2?
Nope.

Contributing

Welcome to the PTL community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!

Bug fixes:

  1. Submit a github issue.
  2. Fix it.
  3. Submit a PR!

New Features:

  1. Submit a github issue.
  2. We'll agree on the feature scope.
  3. Submit a PR! (with updated docs and tests 🙃).

Bleeding edge

If you can't wait for the next release, install the most up to date code with:

pip install git+https://github.com/williamFalcon/pytorch-lightning.git@master --upgrade