Rooba
/
lightning
mirror of https://github.com/Lightning-AI/lightning.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs: add link to lightning-colossalai (#16945)

This commit is contained in:
Jirka Borovec 2023-03-05 11:43:17 +01:00 committed by GitHub
parent 4e3273a81f
commit ac4180fc2f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source-pytorch/advanced/third_party/colossalai.rst vendored
View File

@ -5,7 +5,7 @@ Colossal-AI
###########
The Colossal-AI strategy implements ZeRO-DP with chunk-based memory management.
The `Colossal-AI strategy <https://github.com/Lightning-AI/lightning-colossalai>`_ implements ZeRO-DP with chunk-based memory management.
With this chunk mechanism, really large models can be trained with a small number of GPUs.
It supports larger trainable model size and batch size than usual heterogeneous training by reducing CUDA memory fragments and CPU memory consumption.
Also, it speeds up this kind of heterogeneous training by fully utilizing all kinds of resources.