Docs new section (#2236)
* chlog * docs * ver++ * docs * url * docs * readme * docs ---
This commit is contained in:
Jirka Borovec
GitHub
parent
b4044f0b90
commit
596a5d771f
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
## [unreleased] - YYYY-MM-DD
|
||||
|
||||
### Added
|
||||
|
||||
### Changed
|
||||
|
||||
### Deprecated
|
||||
|
||||
### Removed
|
||||
|
||||
### Fixed
|
||||
|
||||
## [0.8.0] - 2020-06-18
|
||||
|
||||
|
|
@ -25,11 +36,11 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
|
|||
- Added a model hook `transfer_batch_to_device` that enables moving custom data structures to the target device ([1756](https://github.com/PyTorchLightning/pytorch-lightning/pull/1756))
|
||||
- Added [black](https://black.readthedocs.io/en/stable/) formatter for the code with code-checker on pull ([1610](https://github.com/PyTorchLightning/pytorch-lightning/pull/1610))
|
||||
- Added back the slow spawn ddp implementation as `ddp_spawn` ([#2115](https://github.com/PyTorchLightning/pytorch-lightning/pull/2115))
|
||||
- Added loading checkpoints from URLs ([#1667](https://github.com/PyTorchLightning/pytorch-lightning/issues/1667))
|
||||
- Added loading checkpoints from URLs ([#1667](https://github.com/PyTorchLightning/pytorch-lightning/pull/1667))
|
||||
- Added a callback method `on_keyboard_interrupt` for handling KeyboardInterrupt events during training ([#2134](https://github.com/PyTorchLightning/pytorch-lightning/pull/2134))
|
||||
- Added a decorator `auto_move_data` that moves data to the correct device when using the LightningModule for inference ([#1905](https://github.com/PyTorchLightning/pytorch-lightning/pull/1905))
|
||||
- Added `ckpt_path` option to `LightningModule.test(...)` to load particular checkpoint ([#2190](https://github.com/PyTorchLightning/pytorch-lightning/issues/2190))
|
||||
- Added `setup` and `teardown` hooks for model ([#2229](https://github.com/PyTorchLightning/pytorch-lightning/issues/2229))
|
||||
- Added `ckpt_path` option to `LightningModule.test(...)` to load particular checkpoint ([#2190](https://github.com/PyTorchLightning/pytorch-lightning/pull/2190))
|
||||
- Added `setup` and `teardown` hooks for model ([#2229](https://github.com/PyTorchLightning/pytorch-lightning/pull/2229))
|
||||
|
||||
### Changed
|
||||
|
||||
|
|
@ -67,7 +78,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
|
|||
|
||||
- Run graceful training teardown on interpreter exit ([#1631](https://github.com/PyTorchLightning/pytorch-lightning/pull/1631))
|
||||
- Fixed user warning when apex was used together with learning rate schedulers ([#1873](https://github.com/PyTorchLightning/pytorch-lightning/pull/1873))
|
||||
- Fixed multiple calls of `EarlyStopping` callback ([#1751](https://github.com/PyTorchLightning/pytorch-lightning/issues/1751))
|
||||
- Fixed multiple calls of `EarlyStopping` callback ([#1863](https://github.com/PyTorchLightning/pytorch-lightning/pull/1863))
|
||||
- Fixed an issue with `Trainer.from_argparse_args` when passing in unknown Trainer args ([#1932](https://github.com/PyTorchLightning/pytorch-lightning/pull/1932))
|
||||
- Fixed bug related to logger not being reset correctly for model after tuner algorithms ([#1933](https://github.com/PyTorchLightning/pytorch-lightning/pull/1933))
|
||||
- Fixed root node resolution for SLURM cluster with dash in host name ([#1954](https://github.com/PyTorchLightning/pytorch-lightning/pull/1954))
|
||||
|
|
|
|||
17
README.md
17
README.md
|
|
@ -21,12 +21,8 @@
|
|||
-->
|
||||
</div>
|
||||
|
||||
---
|
||||
## Trending contributors
|
||||
|
||||
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/0)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/1)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/2)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/3)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/4)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/5)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/6)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/7)
|
||||
|
||||
---
|
||||
|
||||
## Continuous Integration
|
||||
<center>
|
||||
|
||||
|
|
@ -381,6 +377,7 @@ If you have any questions, feel free to:
|
|||
4. [Join our slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-f6bl2l0l-JYMK3tbAgAmGRrlNr00f1A).
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
**How do I use Lightning for rapid research?**
|
||||
[Here's a walk-through](https://pytorch-lightning.readthedocs.io/en/latest/introduction_guide.html)
|
||||
|
|
@ -447,6 +444,14 @@ pip install https://github.com/PytorchLightning/pytorch-lightning/archive/0.X.Y.
|
|||
- Adrian Wälchli [(awaelchli)](https://github.com/awaelchli)
|
||||
- Nicki Skafte [(skaftenicki)](https://github.com/SkafteNicki)
|
||||
|
||||
---
|
||||
|
||||
### Trending contributors
|
||||
|
||||
[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/0)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/1)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/2)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/3)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/4)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/5)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/6)[](https://sourcerer.io/fame/williamFalcon/pytorchlightning/pytorch-lightning/links/7)
|
||||
|
||||
---
|
||||
|
||||
#### Funding
|
||||
Building open-source software with only a few part-time people is hard! We've secured funding to make sure we can
|
||||
hire a full-time staff, attend conferences, and move faster through implementing features you request.
|
||||
|
|
@ -463,7 +468,7 @@ If you want to cite the framework feel free to use this (but only if you loved i
|
|||
@article{falcon2019pytorch,
|
||||
title={PyTorch Lightning},
|
||||
author={Falcon, WA},
|
||||
journal={GitHub. Note: https://github. com/williamFalcon/pytorch-lightning Cited by},
|
||||
journal={GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning Cited by},
|
||||
volume={3},
|
||||
year={2019}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
Lightning offers 16-bit training for CPUs, GPUs and TPUs.
|
||||
|
||||
GPU 16-bit
|
||||
-----------
|
||||
----------
|
||||
16 bit precision can cut your memory footprint by half.
|
||||
If using volta architecture GPUs it can give a dramatic training speed-up as well.
|
||||
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ Example:
|
|||
We successfully extended functionality without polluting our super clean
|
||||
:class:`~pytorch_lightning.core.LightningModule` research code.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.base
|
||||
:noindex:
|
||||
|
|
@ -56,7 +56,7 @@ We successfully extended functionality without polluting our super clean
|
|||
_abc_impl,
|
||||
check_monitor_top_k,
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.early_stopping
|
||||
:noindex:
|
||||
|
|
@ -66,7 +66,7 @@ We successfully extended functionality without polluting our super clean
|
|||
_abc_impl,
|
||||
check_monitor_top_k,
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.gradient_accumulation_scheduler
|
||||
:noindex:
|
||||
|
|
@ -76,7 +76,7 @@ We successfully extended functionality without polluting our super clean
|
|||
_abc_impl,
|
||||
check_monitor_top_k,
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.lr_logger
|
||||
:noindex:
|
||||
|
|
@ -84,7 +84,7 @@ We successfully extended functionality without polluting our super clean
|
|||
_extract_lr,
|
||||
_find_names
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.model_checkpoint
|
||||
:noindex:
|
||||
|
|
@ -94,7 +94,7 @@ We successfully extended functionality without polluting our super clean
|
|||
_abc_impl,
|
||||
check_monitor_top_k,
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. automodule:: pytorch_lightning.callbacks.progress
|
||||
:noindex:
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ Debugging
|
|||
=========
|
||||
The following are flags that make debugging much easier.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
fast_dev_run
|
||||
------------
|
||||
|
|
@ -21,7 +21,7 @@ argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
|
|||
|
||||
trainer = Trainer(fast_dev_run=True)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Inspect gradient norms
|
||||
----------------------
|
||||
|
|
@ -35,7 +35,7 @@ argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
|
|||
# the 2-norm
|
||||
trainer = Trainer(track_grad_norm=2)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Log GPU usage
|
||||
-------------
|
||||
|
|
@ -48,7 +48,7 @@ argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
|
|||
|
||||
trainer = Trainer(log_gpu_memory=True)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Make model overfit on subset of data
|
||||
------------------------------------
|
||||
|
|
@ -70,7 +70,7 @@ argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
|
|||
With this flag, the train, val, and test sets will all be the same train set. We will also replace the sampler
|
||||
in the training set to turn off shuffle for you.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Print a summary of your LightningModule
|
||||
---------------------------------------
|
||||
|
|
@ -99,7 +99,7 @@ See Also:
|
|||
- :paramref:`~pytorch_lightning.trainer.trainer.Trainer.weights_summary` Trainer argument
|
||||
- :class:`~pytorch_lightning.core.memory.ModelSummary`
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Shorten epochs
|
||||
--------------
|
||||
|
|
@ -116,7 +116,7 @@ On larger datasets like Imagenet, this can help you debug or test a few things f
|
|||
# use 10 batches of train and 5 batches of val
|
||||
trainer = Trainer(limit_train_batches=10, limit_val_batches=5)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Set the number of validation sanity steps
|
||||
-----------------------------------------
|
||||
|
|
|
|||
|
|
@ -7,8 +7,6 @@
|
|||
Experiment Logging
|
||||
==================
|
||||
|
||||
---
|
||||
|
||||
Comet.ml
|
||||
^^^^^^^^
|
||||
|
||||
|
|
@ -49,7 +47,7 @@ The :class:`~pytorch_lightning.loggers.CometLogger` is available anywhere except
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.CometLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
MLflow
|
||||
^^^^^^
|
||||
|
|
@ -76,7 +74,7 @@ Then configure the logger and pass it to the :class:`~pytorch_lightning.trainer.
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.MLFlowLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Neptune.ai
|
||||
^^^^^^^^^^
|
||||
|
|
@ -116,7 +114,7 @@ The :class:`~pytorch_lightning.loggers.NeptuneLogger` is available anywhere exce
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.NeptuneLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
allegro.ai TRAINS
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -160,7 +158,7 @@ The :class:`~pytorch_lightning.loggers.TrainsLogger` is available anywhere in yo
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.TrainsLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Tensorboard
|
||||
^^^^^^^^^^^
|
||||
|
|
@ -186,7 +184,7 @@ The :class:`~pytorch_lightning.loggers.TensorBoardLogger` is available anywhere
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.TensorBoardLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Test Tube
|
||||
^^^^^^^^^
|
||||
|
|
@ -221,7 +219,7 @@ The :class:`~pytorch_lightning.loggers.TestTubeLogger` is available anywhere exc
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.TestTubeLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Weights and Biases
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -257,7 +255,7 @@ The :class:`~pytorch_lightning.loggers.WandbLogger` is available anywhere except
|
|||
.. seealso::
|
||||
:class:`~pytorch_lightning.loggers.WandbLogger` docs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Multiple Loggers
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ Fast Training
|
|||
There are multiple options to speed up different parts of the training by choosing to train
|
||||
on a subset of data. This could be done for speed or debugging purposes.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Check validation every n epochs
|
||||
-------------------------------
|
||||
|
|
@ -19,7 +19,7 @@ If you have a small dataset you might want to check validation every n epochs
|
|||
# DEFAULT
|
||||
trainer = Trainer(check_val_every_n_epoch=1)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Force training for min or max epochs
|
||||
------------------------------------
|
||||
|
|
@ -33,7 +33,7 @@ It can be useful to force training for a minimum number of epochs or limit to a
|
|||
# DEFAULT
|
||||
trainer = Trainer(min_epochs=1, max_epochs=1000)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Set validation check frequency within 1 training epoch
|
||||
------------------------------------------------------
|
||||
|
|
@ -52,7 +52,7 @@ Must use an int if using an IterableDataset.
|
|||
# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
|
||||
trainer = Trainer(val_check_interval=100)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Use data subset for training, validation and test
|
||||
-------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ To enable a hook, simply override the method in your LightningModule and the tra
|
|||
|
||||
3. Add it in the correct place in :mod:`pytorch_lightning.trainer` where it should be called.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Hooks lifecycle
|
||||
---------------
|
||||
|
|
@ -72,7 +72,7 @@ Test loop
|
|||
- ``torch.set_grad_enabled(True)``
|
||||
- :meth:`~pytorch_lightning.core.hooks.ModelHooks.on_post_performance_check`
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
General hooks
|
||||
-------------
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ To illustrate, here's the typical PyTorch project structure organized in a Light
|
|||
As your project grows in complexity with things like 16-bit precision, distributed training, etc... the part in blue
|
||||
quickly becomes onerous and starts distracting from the core research code.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Goal of this guide
|
||||
------------------
|
||||
|
|
@ -32,7 +32,7 @@ to use inheritance to very quickly create an AutoEncoder.
|
|||
.. note:: Any DL/ML PyTorch project fits into the Lightning structure. Here we just focus on 3 types
|
||||
of research to illustrate.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Installing Lightning
|
||||
--------------------
|
||||
|
|
@ -55,7 +55,7 @@ Or with conda
|
|||
|
||||
conda install pytorch-lightning -c conda-forge
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Lightning Philosophy
|
||||
--------------------
|
||||
|
|
@ -117,7 +117,7 @@ In Lightning this code is abstracted out by `Callbacks`.
|
|||
generated = decoder(z)
|
||||
self.experiment.log('images', generated)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Elements of a research project
|
||||
------------------------------
|
||||
|
|
@ -383,7 +383,7 @@ in the LightningModule
|
|||
Again, this is the same PyTorch code except that it has been organized by the LightningModule.
|
||||
This code is not restricted which means it can be as complicated as a full seq-2-seq, RL loop, GAN, etc...
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Training
|
||||
--------
|
||||
|
|
@ -594,11 +594,11 @@ Notice the epoch is MUCH faster!
|
|||
.. figure:: /_images/mnist_imgs/tpu_fast.png
|
||||
:alt: TPU speed
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. include:: hyperparameters.rst
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Validating
|
||||
----------
|
||||
|
|
@ -677,7 +677,7 @@ in the validation loop, you won't need to potentially wait a full epoch to find
|
|||
|
||||
.. note:: Lightning disables gradients, puts model in eval mode and does everything needed for validation.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
|
@ -748,7 +748,7 @@ You can also run the test from a saved lightning model
|
|||
|
||||
.. warning:: .test() is not stable yet on TPUs. We're working on getting around the multiprocessing challenges.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Predicting
|
||||
----------
|
||||
|
|
@ -849,7 +849,7 @@ Or maybe we have a model that we use to do generation
|
|||
How you split up what goes in `forward` vs `training_step` depends on how you want to use this model for
|
||||
prediction.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Extensibility
|
||||
-------------
|
||||
|
|
@ -910,7 +910,7 @@ you could do your own:
|
|||
Every single part of training is configurable this way.
|
||||
For a full list look at `LightningModule <lightning-module.rst>`_.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Callbacks
|
||||
---------
|
||||
|
|
@ -947,10 +947,10 @@ And pass the callbacks into the trainer
|
|||
.. note::
|
||||
See full list of 12+ hooks in the :ref:`callbacks`.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. include:: child_modules.rst
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
.. include:: transfer_learning.rst
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ Example::
|
|||
to a few metrics. Please feel free to create an issue/PR if you have a proposed
|
||||
metric or have found a bug.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Implement a metric
|
||||
------------------
|
||||
|
|
@ -48,6 +48,8 @@ handles automated DDP syncing and converts all inputs and outputs to tensors.
|
|||
Numpy metrics might slow down your training substantially,
|
||||
since every metric computation requires a GPU sync to convert tensors to numpy.
|
||||
|
||||
----------------
|
||||
|
||||
TensorMetric
|
||||
^^^^^^^^^^^^
|
||||
Here's an example showing how to implement a TensorMetric
|
||||
|
|
@ -61,6 +63,8 @@ Here's an example showing how to implement a TensorMetric
|
|||
.. autoclass:: pytorch_lightning.metrics.metric.TensorMetric
|
||||
:noindex:
|
||||
|
||||
----------------
|
||||
|
||||
NumpyMetric
|
||||
^^^^^^^^^^^
|
||||
Here's an example showing how to implement a NumpyMetric
|
||||
|
|
@ -75,7 +79,7 @@ Here's an example showing how to implement a NumpyMetric
|
|||
.. autoclass:: pytorch_lightning.metrics.metric.NumpyMetric
|
||||
:noindex:
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Class Metrics
|
||||
-------------
|
||||
|
|
@ -225,7 +229,7 @@ RMSLE
|
|||
.. autoclass:: pytorch_lightning.metrics.regression.RMSLE
|
||||
:noindex:
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Functional Metrics
|
||||
------------------
|
||||
|
|
@ -364,13 +368,11 @@ stat_scores_multiple_classes (F)
|
|||
.. autofunction:: pytorch_lightning.metrics.functional.stat_scores_multiple_classes
|
||||
:noindex:
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Metric pre-processing
|
||||
---------------------
|
||||
|
||||
Metric
|
||||
|
||||
to_categorical (F)
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
|
@ -383,7 +385,7 @@ to_onehot (F)
|
|||
.. autofunction:: pytorch_lightning.metrics.functional.to_onehot
|
||||
:noindex:
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Sklearn interface
|
||||
-----------------
|
||||
|
|
|
|||
|
|
@ -5,13 +5,13 @@ Lightning supports running on TPUs. At this moment, TPUs are available
|
|||
on Google Cloud (GCP), Google Colab and Kaggle Environments. For more information on TPUs
|
||||
`watch this video <https://www.youtube.com/watch?v=kPMpmcl_Pyw>`_.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Live demo
|
||||
----------
|
||||
Check out this `Google Colab <https://colab.research.google.com/drive/1-_LKx4HwAxl5M6xPJmqAAu444LTDQoa3>`_ to see how to train MNIST on TPUs.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
TPU Terminology
|
||||
---------------
|
||||
|
|
@ -23,7 +23,7 @@ A TPU pod hosts many TPUs on it. Currently, TPU pod v2 has 2048 cores!
|
|||
You can request a full pod from Google cloud or a "slice" which gives you
|
||||
some subset of those 2048 cores.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
How to access TPUs
|
||||
------------------
|
||||
|
|
@ -33,7 +33,7 @@ To access TPUs there are two main ways.
|
|||
2. Using Google Cloud (GCP).
|
||||
3. Using Kaggle.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Colab TPUs
|
||||
-----------
|
||||
|
|
@ -65,7 +65,7 @@ To get a TPU on colab, follow these steps:
|
|||
|
||||
6. Then set up your LightningModule as normal.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
DistributedSamplers
|
||||
-------------------
|
||||
|
|
@ -122,7 +122,7 @@ To use a full TPU pod skip to the TPU pod section.
|
|||
|
||||
That's it! Your model will train on all 8 TPU cores.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Single TPU core training
|
||||
------------------------
|
||||
|
|
@ -132,14 +132,14 @@ Lightning supports training on a single TPU core. Just pass the TPU core ID [1-8
|
|||
|
||||
trainer = pl.Trainer(tpu_cores=[1])
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
Distributed Backend with TPU
|
||||
----------------------------
|
||||
The ```distributed_backend``` option used for GPUs does not apply to TPUs.
|
||||
TPUs work in DDP mode by default (distributing over each core)
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
TPU Pod
|
||||
-------
|
||||
|
|
@ -153,7 +153,7 @@ All you need to do is submit the following command:
|
|||
--conda-env=torch-xla-nightly
|
||||
-- python /usr/share/torch-xla-0.5/pytorch/xla/test/test_train_imagenet.py --fake_data
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
16 bit precision
|
||||
-----------------
|
||||
|
|
@ -171,7 +171,7 @@ set the 16-bit flag.
|
|||
|
||||
Under the hood the xla library will use the `bfloat16 type <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>`_.
|
||||
|
||||
---
|
||||
----------------
|
||||
|
||||
About XLA
|
||||
----------
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
"""Root package info."""
|
||||
|
||||
__version__ = '0.8.0'
|
||||
__version__ = '0.8.1-dev'
|
||||
__author__ = 'William Falcon et al.'
|
||||
__author_email__ = 'waf2107@columbia.edu'
|
||||
__license__ = 'Apache-2.0'
|
||||
|
|
|
|||
Loading…
Reference in New Issue