Rooba
/
lightning
mirror of https://github.com/Lightning-AI/lightning.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update 8-bit optimizer docs (#15155)

This commit is contained in:
Adrian Wälchli 2022-10-17 16:23:56 -04:00 committed by GitHub
parent 5dd7b6a478
commit 7b185c7fc9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/source-pytorch/common/precision_intermediate.rst
View File

@ -143,5 +143,13 @@ It is also possible to use BFloat16 mixed precision on the CPU, relying on MKLDN
8-bit Optimizer
***************
It is possible to further reduce the precision using third-party libraries like `bitsandbytes <https://github.com/facebookresearch/bitsandbytes>`_. Although,
It is possible to further reduce the precision using third-party libraries like `bitsandbytes <https://github.com/TimDettmers/bitsandbytes>`_. Although,
Lightning doesn't support it out of the box yet but you can still use it by configuring it in your LightningModule and setting ``Trainer(precision=32)``.
.. code-block:: python
import bitsandbytes as bnb
# in your LightningModule, return the 8-bit optimizer
def configure_optimizers(self):
return bnb.optim.Adam8bit(model.parameters(), lr=0.001, betas=(0.9, 0.995))