2019-08-06 20:37:58 +00:00
< div align = "center" >
2023-03-15 20:02:10 +00:00
2024-07-19 15:29:17 +00:00
< img alt = "Lightning" src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/ptl_banner.png" width = "800px" style = "max-width: 100%;" >
2023-03-15 20:02:10 +00:00
< br / >
< br / >
2019-08-05 20:02:48 +00:00
2023-12-09 20:53:54 +00:00
**The deep learning framework to pretrain, finetune and deploy AI models.**
2021-02-13 17:27:44 +00:00
2024-08-22 13:59:19 +00:00
**NEW- Deploying models? Check out [LitServe ](https://github.com/Lightning-AI/litserve ), the PyTorch Lightning for model serving**
2023-03-15 01:53:53 +00:00
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2022-06-16 12:50:24 +00:00
2022-06-16 20:21:14 +00:00
< p align = "center" >
2024-08-15 22:16:43 +00:00
< a href = "#quick-start" style = "margin: 0 10px;" > Quick start< / a > •
2024-06-11 14:04:51 +00:00
< a href = "#examples" > Examples< / a > •
2024-08-15 22:16:43 +00:00
< a href = "#why-pytorch-lightning" > PyTorch Lightning< / a > •
2024-06-11 15:01:22 +00:00
< a href = "#lightning-fabric-expert-control" > Fabric< / a > •
2024-08-15 22:16:43 +00:00
< a href = "https://lightning.ai/" > Lightning AI< / a > •
2022-06-16 12:50:24 +00:00
< a href = "#community" > Community< / a > •
2024-08-15 22:16:43 +00:00
< a href = "https://pytorch-lightning.readthedocs.io/en/stable/" > Docs< / a >
2022-06-16 12:50:24 +00:00
< / p >
2021-02-13 17:27:44 +00:00
2022-06-16 20:21:14 +00:00
<!-- DO NOT ADD CONDA DOWNLOADS... README CHANGES MUST BE APPROVED BY EDEN OR WILL -->
[![PyPI - Python Version ](https://img.shields.io/pypi/pyversions/pytorch-lightning )](https://pypi.org/project/pytorch-lightning/)
[![PyPI Status ](https://badge.fury.io/py/pytorch-lightning.svg )](https://badge.fury.io/py/pytorch-lightning)
2023-08-21 16:47:21 +00:00
[![PyPI - Downloads ](https://img.shields.io/pypi/dm/pytorch-lightning )](https://pepy.tech/project/pytorch-lightning)
2023-04-27 11:54:52 +00:00
[![Conda ](https://img.shields.io/conda/v/conda-forge/lightning?label=conda&color=success )](https://anaconda.org/conda-forge/lightning)
2023-12-07 20:21:25 +00:00
[![codecov ](https://codecov.io/gh/Lightning-AI/pytorch-lightning/graph/badge.svg?token=SmzX8mnKlA )](https://codecov.io/gh/Lightning-AI/pytorch-lightning)
2022-06-16 20:21:14 +00:00
2023-03-15 12:33:40 +00:00
[![Discord ](https://img.shields.io/discord/1077906959069626439?style=plastic )](https://discord.gg/VptPCZkGNa)
2023-03-24 08:53:20 +00:00
![GitHub commit activity ](https://img.shields.io/github/commit-activity/w/lightning-ai/lightning )
2022-07-28 20:08:07 +00:00
[![license ](https://img.shields.io/badge/License-Apache%202.0-blue.svg )](https://github.com/Lightning-AI/lightning/blob/master/LICENSE)
2022-06-16 20:21:14 +00:00
<!--
2022-07-28 20:08:07 +00:00
[![CodeFactor ](https://www.codefactor.io/repository/github/Lightning-AI/lightning/badge )](https://www.codefactor.io/repository/github/Lightning-AI/lightning)
2022-06-16 20:21:14 +00:00
-->
2021-02-18 19:51:56 +00:00
2022-06-16 12:50:24 +00:00
< / div >
2023-03-16 11:38:33 +00:00
2024-06-11 15:01:22 +00:00
< div align = "center" >
< p align = "center" >
< a target = "_blank" href = "https://lightning.ai/docs/pytorch/latest/starter/introduction.html#define-a-lightningmodule" >
< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/get-started-badge.svg" height = "36px" alt = "Get started" / >
< / a >
< / p >
< / div >
2024-08-15 22:16:43 +00:00
# Lightning has 2 core packages
[PyTorch Lightning: Train and deploy PyTorch at scale ](#why-pytorch-lightning ).
< br / >
[Lightning Fabric: Expert control ](#lightning-fabric-expert-control ).
Lightning gives you granular control over how much abstraction you want to add over PyTorch.
< div align = "center" >
< img src = "https://pl-public-data.s3.amazonaws.com/assets_lightning/continuum.png" width = "80%" >
< / div >
2023-03-16 11:38:33 +00:00
2024-08-15 22:16:43 +00:00
# Quick start
Install Lightning:
2023-03-16 12:12:17 +00:00
2023-03-16 11:38:33 +00:00
```bash
pip install lightning
```
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
<!-- following section will be skipped from PyPI description -->
< details >
2024-08-15 22:16:43 +00:00
< summary > Advanced install options< / summary >
2023-03-16 12:12:17 +00:00
<!-- following section will be skipped from PyPI description -->
#### Install with optional dependencies
```bash
pip install lightning['extra']
```
#### Conda
```bash
conda install lightning -c conda-forge
```
#### Install stable version
Install future release from the source
```bash
pip install https://github.com/Lightning-AI/lightning/archive/refs/heads/release/stable.zip -U
```
#### Install bleeding-edge
Install nightly from the source (no guarantees)
```bash
pip install https://github.com/Lightning-AI/lightning/archive/refs/heads/master.zip -U
```
or from testing PyPI
```bash
pip install -iU https://test.pypi.org/simple/ pytorch-lightning
```
< / details >
<!-- end skipping PyPI description -->
2024-08-15 22:16:43 +00:00
### PyTorch Lightning example
Define the training workflow. Here's a toy example ([explore real examples](https://lightning.ai/lightning-ai/studios?view=public& section=featured& query=pytorch+lightning)):
2022-06-16 20:21:14 +00:00
```python
2023-03-16 11:38:33 +00:00
# main.py
# ! pip install torchvision
2023-04-11 12:50:59 +00:00
import torch, torch.nn as nn, torch.utils.data as data, torchvision as tv, torch.nn.functional as F
2023-03-15 01:53:53 +00:00
import lightning as L
2023-03-16 11:38:33 +00:00
# --------------------------------
2023-03-16 20:56:47 +00:00
# Step 1: Define a LightningModule
2023-03-16 11:38:33 +00:00
# --------------------------------
2023-03-16 20:56:47 +00:00
# A LightningModule (nn.Module subclass) defines a full *system*
2023-04-11 12:50:59 +00:00
# (ie: an LLM, diffusion model, autoencoder, or simple image classifier).
2022-06-16 20:21:14 +00:00
2023-03-16 20:56:47 +00:00
2023-03-15 01:53:53 +00:00
class LitAutoEncoder(L.LightningModule):
2022-06-16 20:21:14 +00:00
def __init__ (self):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(28 * 28, 128), nn.ReLU(), nn.Linear(128, 3))
self.decoder = nn.Sequential(nn.Linear(3, 128), nn.ReLU(), nn.Linear(128, 28 * 28))
def forward(self, x):
# in lightning, forward defines the prediction/inference actions
embedding = self.encoder(x)
return embedding
def training_step(self, batch, batch_idx):
# training_step defines the train loop. It is independent of forward
2024-03-15 21:56:28 +00:00
x, _ = batch
2022-06-16 20:21:14 +00:00
x = x.view(x.size(0), -1)
z = self.encoder(x)
x_hat = self.decoder(z)
loss = F.mse_loss(x_hat, x)
self.log("train_loss", loss)
return loss
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
return optimizer
2023-03-16 20:56:47 +00:00
2023-03-16 11:38:33 +00:00
# -------------------
# Step 2: Define data
# -------------------
2023-04-11 12:50:59 +00:00
dataset = tv.datasets.MNIST(".", download=True, transform=tv.transforms.ToTensor())
2023-03-16 11:38:33 +00:00
train, val = data.random_split(dataset, [55000, 5000])
2022-06-16 20:21:14 +00:00
2023-03-16 11:38:33 +00:00
# -------------------
# Step 3: Train
# -------------------
2022-06-16 20:21:14 +00:00
autoencoder = LitAutoEncoder()
2023-03-15 01:53:53 +00:00
trainer = L.Trainer()
2023-03-16 11:38:33 +00:00
trainer.fit(autoencoder, data.DataLoader(train), data.DataLoader(val))
```
Run the model on your terminal
2023-03-16 20:56:47 +00:00
```bash
2023-03-16 11:38:33 +00:00
pip install torchvision
python main.py
2022-06-16 20:21:14 +00:00
```
2023-03-16 20:56:47 +00:00
2024-08-15 22:16:43 +00:00
# Why PyTorch Lightning?
PyTorch Lightning is just organized PyTorch - Lightning disentangles PyTorch code to decouple the science from the engineering.
![PT to PL ](docs/source-pytorch/_static/images/general/pl_quick_start_full_compressed.gif )
----
### Examples
Explore various types of training possible with PyTorch Lightning. Pretrain and finetune ANY kind of model to perform ANY task like classification, segmentation, summarization and more:
| Task | Description | Run |
|-------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|---|
| [Hello world ](#hello-simple-model ) | Pretrain - Hello world example | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/pytorch-lightning-hello-world" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Image classification ](https://lightning.ai/lightning-ai/studios/image-classification-with-pytorch-lightning ) | Finetune - ResNet-34 model to classify images of cars | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/image-classification-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Image segmentation ](https://lightning.ai/lightning-ai/studios/image-segmentation-with-pytorch-lightning ) | Finetune - ResNet-50 model to segment images | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/image-segmentation-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Object detection ](https://lightning.ai/lightning-ai/studios/object-detection-with-pytorch-lightning ) | Finetune - Faster R-CNN model to detect objects | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/object-detection-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Text classification ](https://lightning.ai/lightning-ai/studios/text-classification-with-pytorch-lightning ) | Finetune - text classifier (BERT model) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/text-classification-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Text summarization ](https://lightning.ai/lightning-ai/studios/text-summarization-with-pytorch-lightning ) | Finetune - text summarization (Hugging Face transformer model) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/text-summarization-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Audio generation ](https://lightning.ai/lightning-ai/studios/finetune-a-personal-ai-music-generator ) | Finetune - audio generator (transformer model) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/finetune-a-personal-ai-music-generator" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [LLM finetuning ](https://lightning.ai/lightning-ai/studios/finetune-an-llm-with-pytorch-lightning ) | Finetune - LLM (Meta Llama 3.1 8B) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/finetune-an-llm-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
| [Image generation ](https://lightning.ai/lightning-ai/studios/train-a-diffusion-model-with-pytorch-lightning ) | Pretrain - Image generator (diffusion model) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/train-a-diffusion-model-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
2024-08-20 14:51:14 +00:00
| [Recommendation system ](https://lightning.ai/lightning-ai/studios/recommendation-system-with-pytorch-lightning ) | Train - recommendation system (factorization and embedding) | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/recommendation-system-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
2024-08-20 14:18:27 +00:00
| [Time-series forecasting ](https://lightning.ai/lightning-ai/studios/time-series-forecasting-with-pytorch-lightning ) | Train - Time-series forecasting with LSTM | < a target = "_blank" href = "https://lightning.ai/lightning-ai/studios/time-series-forecasting-with-pytorch-lightning" >< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt = "Open In Studio" /></ a > |
2024-08-15 22:16:43 +00:00
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2022-06-16 20:21:14 +00:00
## Advanced features
2023-03-15 19:19:41 +00:00
Lightning has over [40+ advanced features ](https://lightning.ai/docs/pytorch/stable/common/trainer.html#trainer-flags ) designed for professional AI research at scale.
2022-06-16 20:21:14 +00:00
Here are some examples:
< div align = "center" >
2023-03-15 01:53:53 +00:00
< img src = "https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/features_2.jpg" max-height = "600px" >
< / div >
2022-06-16 20:21:14 +00:00
< details >
2023-03-16 12:12:17 +00:00
< summary > Train on 1000s of GPUs without code changes< / summary >
2022-06-16 20:21:14 +00:00
```python
# 8 GPUs
# no code changes needed
2023-03-16 12:12:17 +00:00
trainer = Trainer(accelerator="gpu", devices=8)
2022-06-16 20:21:14 +00:00
# 256 GPUs
2023-03-16 12:12:17 +00:00
trainer = Trainer(accelerator="gpu", devices=8, num_nodes=32)
2022-06-16 20:21:14 +00:00
```
2023-03-16 20:56:47 +00:00
2023-03-16 12:12:17 +00:00
< / details >
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
< details >
< summary > Train on other accelerators like TPUs without code changes< / summary >
2022-06-16 20:21:14 +00:00
```python
# no code changes needed
trainer = Trainer(accelerator="tpu", devices=8)
```
2023-03-16 20:56:47 +00:00
2023-03-16 12:12:17 +00:00
< / details >
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
< details >
< summary > 16-bit precision< / summary >
2022-06-16 20:21:14 +00:00
```python
# no code changes needed
trainer = Trainer(precision=16)
```
2023-03-16 12:12:17 +00:00
< / details >
< details >
< summary > Experiment managers< / summary >
2022-06-16 20:21:14 +00:00
```python
2023-03-15 01:53:53 +00:00
from lightning import loggers
2022-06-16 20:21:14 +00:00
# tensorboard
trainer = Trainer(logger=TensorBoardLogger("logs/"))
# weights and biases
trainer = Trainer(logger=loggers.WandbLogger())
# comet
trainer = Trainer(logger=loggers.CometLogger())
# mlflow
trainer = Trainer(logger=loggers.MLFlowLogger())
# neptune
trainer = Trainer(logger=loggers.NeptuneLogger())
# ... and dozens more
```
2023-03-16 12:12:17 +00:00
< / details >
< details >
2023-03-16 20:56:47 +00:00
< summary > Early Stopping< / summary >
2022-06-16 20:21:14 +00:00
```python
es = EarlyStopping(monitor="val_loss")
trainer = Trainer(callbacks=[es])
```
2023-03-16 20:56:47 +00:00
2023-03-16 12:12:17 +00:00
< / details >
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
< details >
< summary > Checkpointing< / summary >
2022-06-16 20:21:14 +00:00
```python
checkpointing = ModelCheckpoint(monitor="val_loss")
trainer = Trainer(callbacks=[checkpointing])
```
2023-03-16 20:56:47 +00:00
2023-03-16 12:12:17 +00:00
< / details >
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
< details >
< summary > Export to torchscript (JIT) (production use)< / summary >
2022-06-16 20:21:14 +00:00
```python
# torchscript
autoencoder = LitAutoEncoder()
torch.jit.save(autoencoder.to_torchscript(), "model.pt")
```
2023-03-16 20:56:47 +00:00
2023-03-16 12:12:17 +00:00
< / details >
2022-06-16 20:21:14 +00:00
2023-03-16 12:12:17 +00:00
< details >
< summary > Export to ONNX (production use)< / summary >
2022-06-16 20:21:14 +00:00
```python
2023-03-15 01:53:53 +00:00
# onnx
2022-06-16 20:21:14 +00:00
with tempfile.NamedTemporaryFile(suffix=".onnx", delete=False) as tmpfile:
2023-03-15 01:53:53 +00:00
autoencoder = LitAutoEncoder()
input_sample = torch.randn((1, 64))
2022-06-16 20:21:14 +00:00
autoencoder.to_onnx(tmpfile.name, input_sample, export_params=True)
2023-03-15 01:53:53 +00:00
os.path.isfile(tmpfile.name)
2022-06-16 20:21:14 +00:00
```
< / details >
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2022-06-16 20:21:14 +00:00
## Advantages over unstructured PyTorch
- Models become hardware agnostic
- Code is clear to read because engineering code is abstracted away
- Easier to reproduce
- Make fewer mistakes because lightning handles the tricky engineering
- Keeps all the flexibility (LightningModules are still PyTorch modules), but removes a ton of boilerplate
- Lightning has dozens of integrations with popular machine learning tools.
2022-07-28 20:08:07 +00:00
- [Tested rigorously with every new PR ](https://github.com/Lightning-AI/lightning/tree/master/tests ). We test every combination of PyTorch and Python supported versions, every OS, multi GPUs and even TPUs.
2022-06-16 20:21:14 +00:00
- Minimal running speed overhead (about 300 ms per epoch compared with pure PyTorch).
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2023-03-15 01:53:53 +00:00
< div align = "center" >
2023-03-16 12:12:17 +00:00
< a href = "https://lightning.ai/docs/pytorch/stable/" > Read the PyTorch Lightning docs< / a >
2023-03-15 01:53:53 +00:00
< / div >
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2023-03-15 01:53:53 +00:00
2024-06-11 14:04:51 +00:00
2024-06-11 15:01:22 +00:00
# Lightning Fabric: Expert control
2023-03-16 12:12:17 +00:00
2023-03-16 12:39:23 +00:00
Run on any device at any scale with expert-level control over PyTorch training loop and scaling strategy. You can even write your own Trainer.
2023-03-16 12:12:17 +00:00
2023-03-26 00:17:10 +00:00
Fabric is designed for the most complex models like foundation model scaling, LLMs, diffusion, transformers, reinforcement learning, active learning. Of any size.
< table >
< tr >
< th > What to change< / th >
< th > Resulting Fabric Code (copy me!)< / th >
< / tr >
< tr >
< td >
< sub >
2023-03-15 01:53:53 +00:00
```diff
+ import lightning as L
2023-03-26 00:17:10 +00:00
import torch; import torchvision as tv
2023-06-02 10:35:44 +00:00
dataset = tv.datasets.CIFAR10("data", download=True,
train=True,
transform=tv.transforms.ToTensor())
2023-03-26 00:17:10 +00:00
+ fabric = L.Fabric()
2023-03-15 01:53:53 +00:00
+ fabric.launch()
2023-03-26 00:17:10 +00:00
model = tv.models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
- device = "cuda" if torch.cuda.is_available() else "cpu"
- model.to(device)
2023-03-15 01:53:53 +00:00
+ model, optimizer = fabric.setup(model, optimizer)
2023-03-26 00:17:10 +00:00
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
2023-03-15 01:53:53 +00:00
+ dataloader = fabric.setup_dataloaders(dataloader)
2023-03-26 00:17:10 +00:00
2023-03-15 01:53:53 +00:00
model.train()
2023-03-26 00:17:10 +00:00
num_epochs = 10
2023-03-15 01:53:53 +00:00
for epoch in range(num_epochs):
for batch in dataloader:
2023-03-26 00:17:10 +00:00
inputs, labels = batch
- inputs, labels = inputs.to(device), labels.to(device)
2023-03-15 01:53:53 +00:00
optimizer.zero_grad()
2023-03-26 00:17:10 +00:00
outputs = model(inputs)
loss = torch.nn.functional.cross_entropy(outputs, labels)
2023-03-15 01:53:53 +00:00
- loss.backward()
+ fabric.backward(loss)
optimizer.step()
2023-05-27 03:06:31 +00:00
print(loss.data)
2023-03-15 01:53:53 +00:00
```
2023-03-26 00:17:10 +00:00
< / sub >
< td >
< sub >
```Python
import lightning as L
import torch; import torchvision as tv
2023-06-02 10:35:44 +00:00
dataset = tv.datasets.CIFAR10("data", download=True,
train=True,
transform=tv.transforms.ToTensor())
2023-03-26 00:17:10 +00:00
fabric = L.Fabric()
fabric.launch()
model = tv.models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
model, optimizer = fabric.setup(model, optimizer)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
dataloader = fabric.setup_dataloaders(dataloader)
model.train()
num_epochs = 10
for epoch in range(num_epochs):
for batch in dataloader:
inputs, labels = batch
optimizer.zero_grad()
outputs = model(inputs)
loss = torch.nn.functional.cross_entropy(outputs, labels)
fabric.backward(loss)
optimizer.step()
2023-05-27 03:06:31 +00:00
print(loss.data)
2023-03-26 00:17:10 +00:00
```
< / sub >
< / td >
< / tr >
< / table >
2023-03-16 12:12:17 +00:00
## Key features
2023-03-26 00:17:10 +00:00
< details >
< summary > Easily switch from running on CPU to GPU (Apple Silicon, CUDA, …), TPU, multi-GPU or even multi-node training< / summary >
```python
# Use your available hardware
# no code changes needed
fabric = Fabric()
# Run on GPUs (CUDA or MPS)
fabric = Fabric(accelerator="gpu")
# 8 GPUs
fabric = Fabric(accelerator="gpu", devices=8)
# 256 GPUs, multi-node
fabric = Fabric(accelerator="gpu", devices=8, num_nodes=32)
# Run on TPUs
fabric = Fabric(accelerator="tpu")
```
< / details >
< details >
< summary > Use state-of-the-art distributed training strategies (DDP, FSDP, DeepSpeed) and mixed precision out of the box< / summary >
```python
# Use state-of-the-art distributed training techniques
fabric = Fabric(strategy="ddp")
fabric = Fabric(strategy="deepspeed")
fabric = Fabric(strategy="fsdp")
# Switch the precision
fabric = Fabric(precision="16-mixed")
fabric = Fabric(precision="64")
```
< / details >
< details >
< summary > All the device logic boilerplate is handled for you< / summary >
```diff
# no more of this!
- model.to(device)
- batch.to(device)
```
< / details >
< details >
< summary > Build your own custom Trainer using Fabric primitives for training checkpointing, logging, and more< / summary >
```python
import lightning as L
class MyCustomTrainer:
def __init__ (self, accelerator="auto", strategy="auto", devices="auto", precision="32-true"):
self.fabric = L.Fabric(accelerator=accelerator, strategy=strategy, devices=devices, precision=precision)
def fit(self, model, optimizer, dataloader, max_epochs):
self.fabric.launch()
model, optimizer = self.fabric.setup(model, optimizer)
dataloader = self.fabric.setup_dataloaders(dataloader)
model.train()
for epoch in range(max_epochs):
for batch in dataloader:
input, target = batch
optimizer.zero_grad()
output = model(input)
loss = loss_fn(output, target)
self.fabric.backward(loss)
optimizer.step()
```
You can find a more extensive example in our [examples ](examples/fabric/build_your_own_trainer )
< / details >
2023-03-15 01:53:53 +00:00
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2023-03-15 01:53:53 +00:00
2023-03-16 12:27:01 +00:00
< div align = "center" >
< a href = "https://lightning.ai/docs/fabric/stable/" > Read the Lightning Fabric docs< / a >
< / div >
2023-03-16 12:12:17 +00:00
2023-03-16 20:56:47 +00:00
______________________________________________________________________
2023-03-16 12:12:17 +00:00
2024-06-11 14:04:51 +00:00
2023-03-16 12:12:17 +00:00
## Examples
###### Self-supervised Learning
- [CPC transforms ](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#cpc-transforms )
2023-09-26 15:54:44 +00:00
- [Moco v2 transforms ](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#moco-v2-transforms )
2023-03-16 12:12:17 +00:00
- [SimCLR transforms ](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#simclr-transforms )
###### Convolutional Architectures
- [GPT-2 ](https://lightning-bolts.readthedocs.io/en/stable/models/convolutional.html#gpt-2 )
- [UNet ](https://lightning-bolts.readthedocs.io/en/stable/models/convolutional.html#unet )
###### Reinforcement Learning
- [DQN Loss ](https://lightning-bolts.readthedocs.io/en/stable/losses.html#dqn-loss )
- [Double DQN Loss ](https://lightning-bolts.readthedocs.io/en/stable/losses.html#double-dqn-loss )
- [Per DQN Loss ](https://lightning-bolts.readthedocs.io/en/stable/losses.html#per-dqn-loss )
###### GANs
- [Basic GAN ](https://lightning-bolts.readthedocs.io/en/stable/models/gans.html#basic-gan )
- [DCGAN ](https://lightning-bolts.readthedocs.io/en/stable/models/gans.html#dcgan )
###### Classic ML
- [Logistic Regression ](https://lightning-bolts.readthedocs.io/en/stable/models/classic_ml.html#logistic-regression )
- [Linear Regression ](https://lightning-bolts.readthedocs.io/en/stable/models/classic_ml.html#linear-regression )
2024-06-11 14:04:51 +00:00
2023-03-15 01:53:53 +00:00
## Continuous Integration
2023-10-13 13:58:31 +00:00
Lightning is rigorously tested across multiple CPUs, GPUs and TPUs and against major Python and PyTorch versions.
2023-03-15 01:53:53 +00:00
###### \*Codecov is > 90%+ but build delays may show less
< details >
< summary > Current build statuses< / summary >
< center >
2024-01-26 16:44:24 +00:00
| System / PyTorch ver. | 1.13 | 2.0 | 2.1 |
| :--------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| Linux py3.9 \[GPUs\] | | | [![Build Status ](https://dev.azure.com/Lightning-AI/lightning/_apis/build/status%2Fpytorch-lightning%20%28GPUs%29?branchName=master )](https://dev.azure.com/Lightning-AI/lightning/_build/latest?definitionId=24& branchName=master) |
| Linux (multiple Python versions) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) |
| OSX (multiple Python versions) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) |
| Windows (multiple Python versions) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) | [![Test PyTorch ](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml/badge.svg )](https://github.com/Lightning-AI/lightning/actions/workflows/ci-tests-pytorch.yml) |
2023-03-15 01:53:53 +00:00
< / center >
< / details >
2024-06-11 14:04:51 +00:00
2023-03-15 01:53:53 +00:00
2022-06-16 20:21:14 +00:00
## Community
The lightning community is maintained by
2023-04-06 17:50:24 +00:00
- [10+ core contributors ](https://lightning.ai/docs/pytorch/latest/community/governance.html ) who are all a mix of professional engineers, Research Scientists, and Ph.D. students from top AI labs.
2023-03-26 00:17:10 +00:00
- 800+ community contributors.
2022-06-16 20:21:14 +00:00
2023-03-15 19:19:41 +00:00
Want to help us build Lightning and reduce boilerplate for thousands of researchers? [Learn how to make your first contribution here ](https://lightning.ai/docs/pytorch/stable/generated/CONTRIBUTING.html )
2022-06-16 20:21:14 +00:00
Lightning is also part of the [PyTorch ecosystem ](https://pytorch.org/ecosystem/ ) which requires projects to have solid testing, documentation and support.
### Asking for help
If you have any questions please:
2023-03-15 01:53:53 +00:00
1. [Read the docs ](https://lightning.ai/docs ).
2022-07-28 20:08:07 +00:00
1. [Search through existing Discussions ](https://github.com/Lightning-AI/lightning/discussions ), or [add a new question ](https://github.com/Lightning-AI/lightning/discussions/new )
2023-03-15 01:53:53 +00:00
1. [Join our discord ](https://discord.com/invite/tfXFetEZxv ).