tqdm/.meta/.readme.rst

1231 lines
42 KiB
ReStructuredText
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

|Logo|
tqdm
====
|Py-Versions| |Versions| |Conda-Forge-Status| |Docker| |Snapcraft|
|Build-Status| |Coverage-Status| |Branch-Coverage-Status| |Codacy-Grade| |Libraries-Rank| |PyPI-Downloads|
|LICENCE| |OpenHub-Status| |binder-demo| |awesome-python|
``tqdm`` derives from the Arabic word *taqaddum* (تقدّم) which can mean "progress,"
and is an abbreviation for "I love you so much" in Spanish (*te quiero demasiado*).
Instantly make your loops show a smart progress meter - just wrap any
iterable with ``tqdm(iterable)``, and you're done!
.. code:: python
from tqdm import tqdm
for i in tqdm(range(10000)):
...
``76%|████████████████████████        | 7568/10000 [00:33<00:10, 229.00it/s]``
``trange(N)`` can be also used as a convenient shortcut for
``tqdm(range(N))``.
|Screenshot|
|Video| |Slides|
It can also be executed as a module with pipes:
.. code:: sh
$ seq 9999999 | tqdm --bytes | wc -l
75.2MB [00:00, 217MB/s]
9999999
$ tar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` \
> backup.tgz
32%|██████████▍ | 8.89G/27.9G [00:42<01:31, 223MB/s]
Overhead is low -- about 60ns per iteration (80ns with ``tqdm.gui``), and is
unit tested against performance regression.
By comparison, the well-established
`ProgressBar <https://github.com/niltonvolpato/python-progressbar>`__ has
an 800ns/iter overhead.
In addition to its low overhead, ``tqdm`` uses smart algorithms to predict
the remaining time and to skip unnecessary iteration displays, which allows
for a negligible overhead in most cases.
``tqdm`` works on any platform
(Linux, Windows, Mac, FreeBSD, NetBSD, Solaris/SunOS),
in any console or in a GUI, and is also friendly with IPython/Jupyter notebooks.
``tqdm`` does not require any dependencies (not even ``curses``!), just
Python and an environment supporting ``carriage return \r`` and
``line feed \n`` control characters.
------------------------------------------
.. contents:: Table of contents
:backlinks: top
:local:
Installation
------------
Latest PyPI stable release
~~~~~~~~~~~~~~~~~~~~~~~~~~
|Versions| |PyPI-Downloads| |Libraries-Dependents|
.. code:: sh
pip install tqdm
Latest development release on GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|GitHub-Status| |GitHub-Stars| |GitHub-Commits| |GitHub-Forks| |GitHub-Updated|
Pull and install pre-release ``devel`` branch:
.. code:: sh
pip install "git+https://github.com/tqdm/tqdm.git@devel#egg=tqdm"
Latest Conda release
~~~~~~~~~~~~~~~~~~~~
|Conda-Forge-Status|
.. code:: sh
conda install -c conda-forge tqdm
Latest Snapcraft release
~~~~~~~~~~~~~~~~~~~~~~~~
|Snapcraft|
There are 3 channels to choose from:
.. code:: sh
snap install tqdm # implies --stable, i.e. latest tagged release
snap install tqdm --candidate # master branch
snap install tqdm --edge # devel branch
Note that ``snap`` binaries are purely for CLI use (not ``import``-able), and
automatically set up ``bash`` tab-completion.
Latest Docker release
~~~~~~~~~~~~~~~~~~~~~
|Docker|
.. code:: sh
docker pull tqdm/tqdm
docker run -i --rm tqdm/tqdm --help
Other
~~~~~
There are other (unofficial) places where ``tqdm`` may be downloaded, particularly for CLI use:
|Repology|
.. |Repology| image:: https://repology.org/badge/tiny-repos/python:tqdm.svg
:target: https://repology.org/project/python:tqdm/versions
Changelog
---------
The list of all changes is available either on GitHub's Releases:
|GitHub-Status|, on the
`wiki <https://github.com/tqdm/tqdm/wiki/Releases>`__, or on the
`website <https://tqdm.github.io/releases/>`__.
Usage
-----
``tqdm`` is very versatile and can be used in a number of ways.
The three main ones are given below.
Iterable-based
~~~~~~~~~~~~~~
Wrap ``tqdm()`` around any iterable:
.. code:: python
from tqdm import tqdm
from time import sleep
text = ""
for char in tqdm(["a", "b", "c", "d"]):
sleep(0.25)
text = text + char
``trange(i)`` is a special optimised instance of ``tqdm(range(i))``:
.. code:: python
from tqdm import trange
for i in trange(100):
sleep(0.01)
Instantiation outside of the loop allows for manual control over ``tqdm()``:
.. code:: python
pbar = tqdm(["a", "b", "c", "d"])
for char in pbar:
sleep(0.25)
pbar.set_description("Processing %s" % char)
Manual
~~~~~~
Manual control of ``tqdm()`` updates using a ``with`` statement:
.. code:: python
with tqdm(total=100) as pbar:
for i in range(10):
sleep(0.1)
pbar.update(10)
If the optional variable ``total`` (or an iterable with ``len()``) is
provided, predictive stats are displayed.
``with`` is also optional (you can just assign ``tqdm()`` to a variable,
but in this case don't forget to ``del`` or ``close()`` at the end:
.. code:: python
pbar = tqdm(total=100)
for i in range(10):
sleep(0.1)
pbar.update(10)
pbar.close()
Module
~~~~~~
Perhaps the most wonderful use of ``tqdm`` is in a script or on the command
line. Simply inserting ``tqdm`` (or ``python -m tqdm``) between pipes will pass
through all ``stdin`` to ``stdout`` while printing progress to ``stderr``.
The example below demonstrate counting the number of lines in all Python files
in the current directory, with timing information included.
.. code:: sh
$ time find . -name '*.py' -type f -exec cat \{} \; | wc -l
857365
real 0m3.458s
user 0m0.274s
sys 0m3.325s
$ time find . -name '*.py' -type f -exec cat \{} \; | tqdm | wc -l
857366it [00:03, 246471.31it/s]
857365
real 0m3.585s
user 0m0.862s
sys 0m3.358s
Note that the usual arguments for ``tqdm`` can also be specified.
.. code:: sh
$ find . -name '*.py' -type f -exec cat \{} \; |
tqdm --unit loc --unit_scale --total 857366 >> /dev/null
100%|█████████████████████████████████| 857K/857K [00:04<00:00, 246Kloc/s]
Backing up a large directory?
.. code:: sh
$ tar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` \
> backup.tgz
44%|██████████████▊ | 153M/352M [00:14<00:18, 11.0MB/s]
This can be beautified further:
.. code:: sh
$ BYTES="$(du -sb docs/ | cut -f1)"
$ tar -cf - docs/ \
| tqdm --bytes --total "$BYTES" --desc Processing | gzip \
| tqdm --bytes --total "$BYTES" --desc Compressed --position 1 \
> ~/backup.tgz
Processing: 100%|██████████████████████| 352M/352M [00:14<00:00, 30.2MB/s]
Compressed: 42%|█████████▎ | 148M/352M [00:14<00:19, 10.9MB/s]
Or done on a file level using 7-zip:
.. code:: sh
$ 7z a -bd -r backup.7z docs/ | grep Compressing \
| tqdm --total $(find docs/ -type f | wc -l) --unit files \
| grep -v Compressing
100%|██████████████████████████▉| 15327/15327 [01:00<00:00, 712.96files/s]
Pre-existing CLI programs already outputting basic progress information will
benefit from ``tqdm``'s ``--update`` and ``--update_to`` flags:
.. code:: sh
$ seq 3 0.1 5 | tqdm --total 5 --update_to --null
100%|████████████████████████████████████| 5.0/5 [00:00<00:00, 9673.21it/s]
$ seq 10 | tqdm --update --null # 1 + 2 + ... + 10 = 55 iterations
55it [00:00, 90006.52it/s]
FAQ and Known Issues
--------------------
|GitHub-Issues|
The most common issues relate to excessive output on multiple lines, instead
of a neat one-line progress bar.
- Consoles in general: require support for carriage return (``CR``, ``\r``).
- Nested progress bars:
* Consoles in general: require support for moving cursors up to the
previous line. For example,
`IDLE <https://github.com/tqdm/tqdm/issues/191#issuecomment-230168030>`__,
`ConEmu <https://github.com/tqdm/tqdm/issues/254>`__ and
`PyCharm <https://github.com/tqdm/tqdm/issues/203>`__ (also
`here <https://github.com/tqdm/tqdm/issues/208>`__,
`here <https://github.com/tqdm/tqdm/issues/307>`__, and
`here <https://github.com/tqdm/tqdm/issues/454#issuecomment-335416815>`__)
lack full support.
* Windows: additionally may require the Python module ``colorama``
to ensure nested bars stay within their respective lines.
- Unicode:
* Environments which report that they support unicode will have solid smooth
progressbars. The fallback is an ``ascii``-only bar.
* Windows consoles often only partially support unicode and thus
`often require explicit ascii=True <https://github.com/tqdm/tqdm/issues/454#issuecomment-335416815>`__
(also `here <https://github.com/tqdm/tqdm/issues/499>`__). This is due to
either normal-width unicode characters being incorrectly displayed as
"wide", or some unicode characters not rendering.
- Wrapping generators:
* Generator wrapper functions tend to hide the length of iterables.
``tqdm`` does not.
* Replace ``tqdm(enumerate(...))`` with ``enumerate(tqdm(...))`` or
``tqdm(enumerate(x), total=len(x), ...)``.
The same applies to ``numpy.ndenumerate``.
* Replace ``tqdm(zip(a, b))`` with ``zip(tqdm(a), b)`` or even
``zip(tqdm(a), tqdm(b))``.
* The same applies to ``itertools``.
* Some useful convenience functions can be found under ``tqdm.contrib``.
- `Hanging pipes in python2 <https://github.com/tqdm/tqdm/issues/359>`__:
when using ``tqdm`` on the CLI, you may need to use Python 3.5+ for correct
buffering.
- `No intermediate output in docker-compose <https://github.com/tqdm/tqdm/issues/771>`__:
use ``docker-compose run`` instead of ``docker-compose up`` and ``tty: true``.
If you come across any other difficulties, browse and file |GitHub-Issues|.
Documentation
-------------
|Py-Versions| |README-Hits| (Since 19 May 2016)
.. code:: python
class tqdm():
"""{DOC_tqdm}"""
def __init__(self, iterable=None, desc=None, total=None, leave=True,
file=None, ncols=None, mininterval=0.1,
maxinterval=10.0, miniters=None, ascii=None, disable=False,
unit='it', unit_scale=False, dynamic_ncols=False,
smoothing=0.3, bar_format=None, initial=0, position=None,
postfix=None, unit_divisor=1000):
Parameters
~~~~~~~~~~
{DOC_tqdm.tqdm.__init__.Parameters}
Extra CLI Options
~~~~~~~~~~~~~~~~~
{DOC_tqdm.cli.CLI_EXTRA_DOC}
Returns
~~~~~~~
{DOC_tqdm.tqdm.__init__.Returns}
.. code:: python
class tqdm():
def update(self, n=1):
"""{DOC_tqdm.tqdm.update}"""
def close(self):
"""{DOC_tqdm.tqdm.close}"""
def clear(self, nomove=False):
"""{DOC_tqdm.tqdm.clear}"""
def refresh(self):
"""{DOC_tqdm.tqdm.refresh}"""
def unpause(self):
"""{DOC_tqdm.tqdm.unpause}"""
def reset(self, total=None):
"""{DOC_tqdm.tqdm.reset}"""
def set_description(self, desc=None, refresh=True):
"""{DOC_tqdm.tqdm.set_description}"""
def set_postfix(self, ordered_dict=None, refresh=True, **tqdm_kwargs):
"""{DOC_tqdm.tqdm.set_postfix}"""
@classmethod
def write(cls, s, file=sys.stdout, end="\n"):
"""{DOC_tqdm.tqdm.write}"""
@property
def format_dict(self):
"""{DOC_tqdm.tqdm.format_dict}"""
def display(self, msg=None, pos=None):
"""{DOC_tqdm.tqdm.display}"""
@classmethod
@contextmanager
def wrapattr(cls, stream, method, total=None, bytes=True, **tqdm_kwargs):
"""{DOC_tqdm.tqdm.wrapattr}"""
@classmethod
def pandas(cls, *targs, **tqdm_kwargs):
"""Registers the current `tqdm` class with `pandas`."""
def trange(*args, **tqdm_kwargs):
"""
A shortcut for `tqdm(xrange(*args), **tqdm_kwargs)`.
On Python3+, `range` is used instead of `xrange`.
"""
Convenience Functions
~~~~~~~~~~~~~~~~~~~~~
.. code:: python
def tqdm.contrib.tenumerate(iterable, start=0, total=None,
tqdm_class=tqdm.auto.tqdm, **tqdm_kwargs):
"""Equivalent of `numpy.ndenumerate` or builtin `enumerate`."""
def tqdm.contrib.tzip(iter1, *iter2plus, **tqdm_kwargs):
"""Equivalent of builtin `zip`."""
def tqdm.contrib.tmap(function, *sequences, **tqdm_kwargs):
"""Equivalent of builtin `map`."""
Submodules
~~~~~~~~~~
.. code:: python
class tqdm.notebook.tqdm(tqdm.tqdm):
"""IPython/Jupyter Notebook widget."""
class tqdm.auto.tqdm(tqdm.tqdm):
"""Automatically chooses beween `tqdm.notebook` and `tqdm.tqdm`."""
class tqdm.asyncio.tqdm(tqdm.tqdm):
"""Asynchronous version."""
@classmethod
def as_completed(cls, fs, *, loop=None, timeout=None, total=None,
**tqdm_kwargs):
"""Wrapper for `asyncio.as_completed`."""
class tqdm.gui.tqdm(tqdm.tqdm):
"""Matplotlib GUI version."""
class tqdm.tk.tqdm(tqdm.tqdm):
"""Tkinter GUI version."""
class tqdm.rich.tqdm(tqdm.tqdm):
"""`rich.progress` version."""
class tqdm.keras.TqdmCallback(keras.callbacks.Callback):
"""`keras` callback for epoch and batch progress."""
``contrib``
+++++++++++
The ``tqdm.contrib`` package also contains experimental modules:
- ``tqdm.contrib.itertools``: Thin wrappers around ``itertools``
- ``tqdm.contrib.concurrent``: Thin wrappers around ``concurrent.futures``
- ``tqdm.contrib.discord``: Posts to `Discord <https://discord.com/>`__ bots
- ``tqdm.contrib.telegram``: Posts to `Telegram <https://telegram.org/>`__ bots
- ``tqdm.contrib.bells``: Automagically enables all optional features
* ``auto``, ``pandas``, ``discord``, ``telegram``
Examples and Advanced Usage
---------------------------
- See the `examples <https://github.com/tqdm/tqdm/tree/master/examples>`__
folder;
- import the module and run ``help()``;
- consult the `wiki <https://github.com/tqdm/tqdm/wiki>`__;
* this has an
`excellent article <https://github.com/tqdm/tqdm/wiki/How-to-make-a-great-Progress-Bar>`__
on how to make a **great** progressbar;
- check out the `slides from PyData London <https://tqdm.github.io/PyData2019/slides.html>`__, or
- run the |binder-demo|.
Description and additional stats
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Custom information can be displayed and updated dynamically on ``tqdm`` bars
with the ``desc`` and ``postfix`` arguments:
.. code:: python
from tqdm import tqdm, trange
from random import random, randint
from time import sleep
with trange(10) as t:
for i in t:
# Description will be displayed on the left
t.set_description('GEN %i' % i)
# Postfix will be displayed on the right,
# formatted automatically based on argument's datatype
t.set_postfix(loss=random(), gen=randint(1,999), str='h',
lst=[1, 2])
sleep(0.1)
with tqdm(total=10, bar_format="{postfix[0]} {postfix[1][value]:>8.2g}",
postfix=["Batch", dict(value=0)]) as t:
for i in range(10):
sleep(0.1)
t.postfix[1]["value"] = i / 2
t.update()
Points to remember when using ``{postfix[...]}`` in the ``bar_format`` string:
- ``postfix`` also needs to be passed as an initial argument in a compatible
format, and
- ``postfix`` will be auto-converted to a string if it is a ``dict``-like
object. To prevent this behaviour, insert an extra item into the dictionary
where the key is not a string.
Additional ``bar_format`` parameters may also be defined by overriding
``format_dict``, and the bar itself may be modified using ``ascii``:
.. code:: python
from tqdm import tqdm
class TqdmExtraFormat(tqdm):
"""Provides a `total_time` format parameter"""
@property
def format_dict(self):
d = super(TqdmExtraFormat, self).format_dict
total_time = d["elapsed"] * (d["total"] or 0) / max(d["n"], 1)
d.update(total_time=self.format_interval(total_time) + " in total")
return d
for i in TqdmExtraFormat(
range(9), ascii=" .oO0",
bar_format="{total_time}: {percentage:.0f}%|{bar}{r_bar}"):
if i == 4:
break
.. code::
00:00 in total: 44%|0000. | 4/9 [00:00<00:00, 962.93it/s]
Note that ``{bar}`` also supports a format specifier ``[width][type]``.
- ``width``
* unspecified (default): automatic to fill ``ncols``
* ``int >= 0``: fixed width overriding ``ncols`` logic
* ``int < 0``: subtract from the automatic default
- ``type``
* ``a``: ascii (``ascii=True`` override)
* ``u``: unicode (``ascii=False`` override)
* ``b``: blank (``ascii=" "`` override)
This means a fixed bar with right-justified text may be created by using:
``bar_format="{l_bar}{bar:10}|{bar:-10b}right-justified"``
Nested progress bars
~~~~~~~~~~~~~~~~~~~~
``tqdm`` supports nested progress bars. Here's an example:
.. code:: python
from tqdm.auto import trange
from time import sleep
for i in trange(4, desc='1st loop'):
for j in trange(5, desc='2nd loop'):
for k in trange(50, desc='3rd loop', leave=False):
sleep(0.01)
On Windows `colorama <https://github.com/tartley/colorama>`__ will be used if
available to keep nested bars on their respective lines.
For manual control over positioning (e.g. for multi-processing use),
you may specify ``position=n`` where ``n=0`` for the outermost bar,
``n=1`` for the next, and so on.
However, it's best to check if ``tqdm`` can work without manual ``position``
first.
.. code:: python
from time import sleep
from tqdm import trange, tqdm
from multiprocessing import Pool, RLock, freeze_support
L = list(range(9))
def progresser(n):
interval = 0.001 / (n + 2)
total = 5000
text = "#{}, est. {:<04.2}s".format(n, interval * total)
for _ in trange(total, desc=text, position=n):
sleep(interval)
if __name__ == '__main__':
freeze_support() # for Windows support
tqdm.set_lock(RLock()) # for managing output contention
p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p.map(progresser, L)
Note that in Python 3, ``tqdm.write`` is thread-safe:
.. code:: python
from time import sleep
from tqdm import tqdm, trange
from concurrent.futures import ThreadPoolExecutor
L = list(range(9))
def progresser(n):
interval = 0.001 / (n + 2)
total = 5000
text = "#{}, est. {:<04.2}s".format(n, interval * total)
for _ in trange(total, desc=text):
sleep(interval)
if n == 6:
tqdm.write("n == 6 completed.")
tqdm.write("`tqdm.write()` is thread-safe in py3!")
if __name__ == '__main__':
with ThreadPoolExecutor() as p:
p.map(progresser, L)
Hooks and callbacks
~~~~~~~~~~~~~~~~~~~
``tqdm`` can easily support callbacks/hooks and manual updates.
Here's an example with ``urllib``:
**``urllib.urlretrieve`` documentation**
| [...]
| If present, the hook function will be called once
| on establishment of the network connection and once after each block read
| thereafter. The hook will be passed three arguments; a count of blocks
| transferred so far, a block size in bytes, and the total size of the file.
| [...]
.. code:: python
import urllib, os
from tqdm import tqdm
class TqdmUpTo(tqdm):
"""Provides `update_to(n)` which uses `tqdm.update(delta_n)`."""
def update_to(self, b=1, bsize=1, tsize=None):
"""
b : int, optional
Number of blocks transferred so far [default: 1].
bsize : int, optional
Size of each block (in tqdm units) [default: 1].
tsize : int, optional
Total size (in tqdm units). If [default: None] remains unchanged.
"""
if tsize is not None:
self.total = tsize
return self.update(b * bsize - self.n) # also sets self.n = b * bsize
eg_link = "https://caspersci.uk.to/matryoshka.zip"
with TqdmUpTo(unit='B', unit_scale=True, unit_divisor=1024, miniters=1,
desc=eg_link.split('/')[-1]) as t: # all optional kwargs
urllib.urlretrieve(eg_link, filename=os.devnull,
reporthook=t.update_to, data=None)
t.total = t.n
Inspired by `twine#242 <https://github.com/pypa/twine/pull/242>`__.
Functional alternative in
`examples/tqdm_wget.py <https://github.com/tqdm/tqdm/blob/master/examples/tqdm_wget.py>`__.
It is recommend to use ``miniters=1`` whenever there is potentially
large differences in iteration speed (e.g. downloading a file over
a patchy connection).
**Wrapping read/write methods**
To measure throughput through a file-like object's ``read`` or ``write``
methods, use ``CallbackIOWrapper``:
.. code:: python
from tqdm.auto import tqdm
from tqdm.utils import CallbackIOWrapper
with tqdm(total=file_obj.size,
unit='B', unit_scale=True, unit_divisor=1024) as t:
fobj = CallbackIOWrapper(t.update, file_obj, "read")
while True:
chunk = fobj.read(chunk_size)
if not chunk:
break
t.reset()
# ... continue to use `t` for something else
Alternatively, use the even simpler ``wrapattr`` convenience function,
which would condense both the ``urllib`` and ``CallbackIOWrapper`` examples
down to:
.. code:: python
import urllib, os
from tqdm import tqdm
eg_link = "https://caspersci.uk.to/matryoshka.zip"
with tqdm.wrapattr(open(os.devnull, "wb"), "write",
miniters=1, desc=eg_link.split('/')[-1]) as fout:
for chunk in urllib.urlopen(eg_link):
fout.write(chunk)
The ``requests`` equivalent is nearly identical, albeit with a ``total``:
.. code:: python
import requests, os
from tqdm import tqdm
eg_link = "https://caspersci.uk.to/matryoshka.zip"
response = requests.get(eg_link, stream=True)
with tqdm.wrapattr(open(os.devnull, "wb"), "write",
miniters=1, desc=eg_link.split('/')[-1],
total=int(response.headers.get('content-length', 0))) as fout:
for chunk in response.iter_content(chunk_size=4096):
fout.write(chunk)
**Custom callback**
``tqdm`` is known for intelligently skipping unnecessary displays. To make a
custom callback take advantage of this, simply use the return value of
``update()``. This is set to ``True`` if a ``display()`` was triggered.
.. code:: python
from tqdm.auto import tqdm as std_tqdm
def external_callback(*args, **kwargs):
...
class TqdmExt(std_tqdm):
def update(self, n=1):
displayed = super(TqdmExt, self).update(n):
if displayed:
external_callback(**self.format_dict)
return displayed
``asyncio``
~~~~~~~~~~~
Note that ``break`` isn't currently caught by asynchronous iterators.
This means that ``tqdm`` cannot clean up after itself in this case:
.. code:: python
from tqdm.asyncio import tqdm
async for i in tqdm(range(9)):
if i == 2:
break
Instead, either call ``pbar.close()`` manually or use the context manager syntax:
.. code:: python
from tqdm.asyncio import tqdm
with tqdm(range(9)) as pbar:
async for i in pbar:
if i == 2:
break
Pandas Integration
~~~~~~~~~~~~~~~~~~
Due to popular demand we've added support for ``pandas`` -- here's an example
for ``DataFrame.progress_apply`` and ``DataFrameGroupBy.progress_apply``:
.. code:: python
import pandas as pd
import numpy as np
from tqdm import tqdm
df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))
# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm.gui.tqdm`, `tqdm.notebook.tqdm`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")
# Now you can use `progress_apply` instead of `apply`
# and `progress_map` instead of `map`
df.progress_apply(lambda x: x**2)
# can also groupby:
# df.groupby(0).progress_apply(lambda x: x**2)
In case you're interested in how this works (and how to modify it for your
own callbacks), see the
`examples <https://github.com/tqdm/tqdm/tree/master/examples>`__
folder or import the module and run ``help()``.
Keras Integration
~~~~~~~~~~~~~~~~~
A ``keras`` callback is also available:
.. code:: python
from tqdm.keras import TqdmCallback
...
model.fit(..., verbose=0, callbacks=[TqdmCallback()])
IPython/Jupyter Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IPython/Jupyter is supported via the ``tqdm.notebook`` submodule:
.. code:: python
from tqdm.notebook import trange, tqdm
from time import sleep
for i in trange(3, desc='1st loop'):
for j in tqdm(range(100), desc='2nd loop'):
sleep(0.01)
In addition to ``tqdm`` features, the submodule provides a native Jupyter
widget (compatible with IPython v1-v4 and Jupyter), fully working nested bars
and colour hints (blue: normal, green: completed, red: error/interrupt,
light blue: no ETA); as demonstrated below.
|Screenshot-Jupyter1|
|Screenshot-Jupyter2|
|Screenshot-Jupyter3|
The ``notebook`` version supports percentage or pixels for overall width
(e.g.: ``ncols='100%'`` or ``ncols='480px'``).
It is also possible to let ``tqdm`` automatically choose between
console or notebook versions by using the ``autonotebook`` submodule:
.. code:: python
from tqdm.autonotebook import tqdm
tqdm.pandas()
Note that this will issue a ``TqdmExperimentalWarning`` if run in a notebook
since it is not meant to be possible to distinguish between ``jupyter notebook``
and ``jupyter console``. Use ``auto`` instead of ``autonotebook`` to suppress
this warning.
Note that notebooks will display the bar in the cell where it was created.
This may be a different cell from the one where it is used.
If this is not desired, either
- delay the creation of the bar to the cell where it must be displayed, or
- create the bar with ``display=False``, and in a later cell call
``display(bar.container)``:
.. code:: python
from tqdm.notebook import tqdm
pbar = tqdm(..., display=False)
.. code:: python
# different cell
display(pbar.container)
The ``keras`` callback has a ``display()`` method which can be used likewise:
.. code:: python
from tqdm.keras import TqdmCallback
cbk = TqdmCallback(display=False)
.. code:: python
# different cell
cbk.display()
model.fit(..., verbose=0, callbacks=[cbk])
Another possibility is to have a single bar (near the top of the notebook)
which is constantly re-used (using ``reset()`` rather than ``close()``).
For this reason, the notebook version (unlike the CLI version) does not
automatically call ``close()`` upon ``Exception``.
.. code:: python
from tqdm.notebook import tqdm
pbar = tqdm()
.. code:: python
# different cell
iterable = range(100)
pbar.reset(total=len(iterable)) # initialise with new `total`
for i in iterable:
pbar.update()
pbar.refresh() # force print final status but don't `close()`
Custom Integration
~~~~~~~~~~~~~~~~~~
To change the default arguments (such as making ``dynamic_ncols=True``),
simply use built-in Python magic:
.. code:: python
from functools import partial
from tqdm import tqdm as std_tqdm
tqdm = partial(std_tqdm, dynamic_ncols=True)
For further customisation,
``tqdm`` may be inherited from to create custom callbacks (as with the
``TqdmUpTo`` example `above <#hooks-and-callbacks>`__) or for custom frontends
(e.g. GUIs such as notebook or plotting packages). In the latter case:
1. ``def __init__()`` to call ``super().__init__(..., gui=True)`` to disable
terminal ``status_printer`` creation.
2. Redefine: ``close()``, ``clear()``, ``display()``.
Consider overloading ``display()`` to use e.g.
``self.frontend(**self.format_dict)`` instead of ``self.sp(repr(self))``.
Some submodule examples of inheritance:
- `tqdm/notebook.py <https://github.com/tqdm/tqdm/blob/master/tqdm/notebook.py>`__
- `tqdm/gui.py <https://github.com/tqdm/tqdm/blob/master/tqdm/gui.py>`__
- `tqdm/tk.py <https://github.com/tqdm/tqdm/blob/master/tqdm/tk.py>`__
- `tqdm/contrib/telegram.py <https://github.com/tqdm/tqdm/blob/master/tqdm/contrib/telegram.py>`__
- `tqdm/contrib/discord.py <https://github.com/tqdm/tqdm/blob/master/tqdm/contrib/discord.py>`__
Dynamic Monitor/Meter
~~~~~~~~~~~~~~~~~~~~~
You can use a ``tqdm`` as a meter which is not monotonically increasing.
This could be because ``n`` decreases (e.g. a CPU usage monitor) or ``total``
changes.
One example would be recursively searching for files. The ``total`` is the
number of objects found so far, while ``n`` is the number of those objects which
are files (rather than folders):
.. code:: python
from tqdm import tqdm
import os.path
def find_files_recursively(path, show_progress=True):
files = []
# total=1 assumes `path` is a file
t = tqdm(total=1, unit="file", disable=not show_progress)
if not os.path.exists(path):
raise IOError("Cannot find:" + path)
def append_found_file(f):
files.append(f)
t.update()
def list_found_dir(path):
"""returns os.listdir(path) assuming os.path.isdir(path)"""
listing = os.listdir(path)
# subtract 1 since a "file" we found was actually this directory
t.total += len(listing) - 1
# fancy way to give info without forcing a refresh
t.set_postfix(dir=path[-10:], refresh=False)
t.update(0) # may trigger a refresh
return listing
def recursively_search(path):
if os.path.isdir(path):
for f in list_found_dir(path):
recursively_search(os.path.join(path, f))
else:
append_found_file(path)
recursively_search(path)
t.set_postfix(dir=path)
t.close()
return files
Using ``update(0)`` is a handy way to let ``tqdm`` decide when to trigger a
display refresh to avoid console spamming.
Writing messages
~~~~~~~~~~~~~~~~
This is a work in progress (see
`#737 <https://github.com/tqdm/tqdm/issues/737>`__).
Since ``tqdm`` uses a simple printing mechanism to display progress bars,
you should not write any message in the terminal using ``print()`` while
a progressbar is open.
To write messages in the terminal without any collision with ``tqdm`` bar
display, a ``.write()`` method is provided:
.. code:: python
from tqdm.auto import tqdm, trange
from time import sleep
bar = trange(10)
for i in bar:
# Print using tqdm class method .write()
sleep(0.1)
if not (i % 3):
tqdm.write("Done task %i" % i)
# Can also use bar.write()
By default, this will print to standard output ``sys.stdout``. but you can
specify any file-like object using the ``file`` argument. For example, this
can be used to redirect the messages writing to a log file or class.
Redirecting writing
~~~~~~~~~~~~~~~~~~~
If using a library that can print messages to the console, editing the library
by replacing ``print()`` with ``tqdm.write()`` may not be desirable.
In that case, redirecting ``sys.stdout`` to ``tqdm.write()`` is an option.
To redirect ``sys.stdout``, create a file-like class that will write
any input string to ``tqdm.write()``, and supply the arguments
``file=sys.stdout, dynamic_ncols=True``.
A reusable canonical example is given below:
.. code:: python
from time import sleep
import contextlib
import sys
from tqdm import tqdm
from tqdm.contrib import DummyTqdmFile
@contextlib.contextmanager
def std_out_err_redirect_tqdm():
orig_out_err = sys.stdout, sys.stderr
try:
sys.stdout, sys.stderr = map(DummyTqdmFile, orig_out_err)
yield orig_out_err[0]
# Relay exceptions
except Exception as exc:
raise exc
# Always restore sys.stdout/err if necessary
finally:
sys.stdout, sys.stderr = orig_out_err
def some_fun(i):
print("Fee, fi, fo,".split()[i])
# Redirect stdout to tqdm.write() (don't forget the `as save_stdout`)
with std_out_err_redirect_tqdm() as orig_stdout:
# tqdm needs the original stdout
# and dynamic_ncols=True to autodetect console width
for i in tqdm(range(3), file=orig_stdout, dynamic_ncols=True):
sleep(.5)
some_fun(i)
# After the `with`, printing is restored
print("Done!")
Monitoring thread, intervals and miniters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``tqdm`` implements a few tricks to to increase efficiency and reduce overhead.
- Avoid unnecessary frequent bar refreshing: ``mininterval`` defines how long
to wait between each refresh. ``tqdm`` always gets updated in the background,
but it will display only every ``mininterval``.
- Reduce number of calls to check system clock/time.
- ``mininterval`` is more intuitive to configure than ``miniters``.
A clever adjustment system ``dynamic_miniters`` will automatically adjust
``miniters`` to the amount of iterations that fit into time ``mininterval``.
Essentially, ``tqdm`` will check if it's time to print without actually
checking time. This behaviour can be still be bypassed by manually setting
``miniters``.
However, consider a case with a combination of fast and slow iterations.
After a few fast iterations, ``dynamic_miniters`` will set ``miniters`` to a
large number. When iteration rate subsequently slows, ``miniters`` will
remain large and thus reduce display update frequency. To address this:
- ``maxinterval`` defines the maximum time between display refreshes.
A concurrent monitoring thread checks for overdue updates and forces one
where necessary.
The monitoring thread should not have a noticeable overhead, and guarantees
updates at least every 10 seconds by default.
This value can be directly changed by setting the ``monitor_interval`` of
any ``tqdm`` instance (i.e. ``t = tqdm.tqdm(...); t.monitor_interval = 2``).
The monitor thread may be disabled application-wide by setting
``tqdm.tqdm.monitor_interval = 0`` before instantiation of any ``tqdm`` bar.
Contributions
-------------
|GitHub-Commits| |GitHub-Issues| |GitHub-PRs| |OpenHub-Status| |GitHub-Contributions| |CII Best Practices|
All source code is hosted on `GitHub <https://github.com/tqdm/tqdm>`__.
Contributions are welcome.
See the
`CONTRIBUTING <https://raw.githubusercontent.com/tqdm/tqdm/master/CONTRIBUTING.md>`__
file for more information.
Developers who have made significant contributions, ranked by *SLoC*
(surviving lines of code,
`git fame <https://github.com/casperdcl/git-fame>`__ ``-wMC --excl '\.(png|gif|jpg)$'``),
are:
==================== ======================================================== ==== ================================
Name ID SLoC Notes
==================== ======================================================== ==== ================================
Casper da Costa-Luis `casperdcl <https://github.com/casperdcl>`__ ~81% primary maintainer |Gift-Casper|
Stephen Larroque `lrq3000 <https://github.com/lrq3000>`__ ~10% team member
Martin Zugnoni `martinzugnoni <https://github.com/martinzugnoni>`__ ~3%
Richard Sheridan `richardsheridan <https://github.com/richardsheridan>`__ ~1%
Guangshuo Chen `chengs <https://github.com/chengs>`__ ~1%
Kyle Altendorf `altendky <https://github.com/altendky>`__ <1%
Matthew Stevens `mjstevens777 <https://github.com/mjstevens777>`__ <1%
Hadrien Mary `hadim <https://github.com/hadim>`__ <1% team member
Ivan Ivanov `obiwanus <https://github.com/obiwanus>`__ <1%
Daniel Panteleit `danielpanteleit <https://github.com/danielpanteleit>`__ <1%
Jonas Haag `jonashaag <https://github.com/jonashaag>`__ <1%
James E. King III `jeking3 <https://github.com/jeking3>`__ <1%
Noam Yorav-Raphael `noamraph <https://github.com/noamraph>`__ <1% original author
Mikhail Korobov `kmike <https://github.com/kmike>`__ <1% team member
==================== ======================================================== ==== ================================
Ports to Other Languages
~~~~~~~~~~~~~~~~~~~~~~~~
A list is available on
`this wiki page <https://github.com/tqdm/tqdm/wiki/tqdm-ports>`__.
LICENCE
-------
Open Source (OSI approved): |LICENCE|
Citation information: |DOI|
|README-Hits| (Since 19 May 2016)
.. |Logo| image:: https://raw.githubusercontent.com/tqdm/tqdm/master/images/logo.gif
.. |Screenshot| image:: https://raw.githubusercontent.com/tqdm/img/master/tqdm.gif
.. |Video| image:: https://raw.githubusercontent.com/tqdm/img/master/video.jpg
:target: https://tqdm.github.io/video
.. |Slides| image:: https://raw.githubusercontent.com/tqdm/img/master/slides.jpg
:target: https://tqdm.github.io/PyData2019/slides.html
.. |Build-Status| image:: https://img.shields.io/github/workflow/status/tqdm/tqdm/Test/master?logo=GitHub
:target: https://github.com/tqdm/tqdm/actions?query=workflow%3ATest
.. |Coverage-Status| image:: https://img.shields.io/coveralls/github/tqdm/tqdm/master?logo=coveralls
:target: https://coveralls.io/github/tqdm/tqdm
.. |Branch-Coverage-Status| image:: https://codecov.io/gh/tqdm/tqdm/branch/master/graph/badge.svg
:target: https://codecov.io/gh/tqdm/tqdm
.. |Codacy-Grade| image:: https://app.codacy.com/project/badge/Grade/3f965571598f44549c7818f29cdcf177
:target: https://www.codacy.com/gh/tqdm/tqdm/dashboard
.. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/3264/badge
:target: https://bestpractices.coreinfrastructure.org/projects/3264
.. |GitHub-Status| image:: https://img.shields.io/github/tag/tqdm/tqdm.svg?maxAge=86400&logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/releases
.. |GitHub-Forks| image:: https://img.shields.io/github/forks/tqdm/tqdm.svg?logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/network
.. |GitHub-Stars| image:: https://img.shields.io/github/stars/tqdm/tqdm.svg?logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/stargazers
.. |GitHub-Commits| image:: https://img.shields.io/github/commit-activity/y/tqdm/tqdm.svg?logo=git&logoColor=white
:target: https://github.com/tqdm/tqdm/graphs/commit-activity
.. |GitHub-Issues| image:: https://img.shields.io/github/issues-closed/tqdm/tqdm.svg?logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/issues?q=
.. |GitHub-PRs| image:: https://img.shields.io/github/issues-pr-closed/tqdm/tqdm.svg?logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/pulls
.. |GitHub-Contributions| image:: https://img.shields.io/github/contributors/tqdm/tqdm.svg?logo=github&logoColor=white
:target: https://github.com/tqdm/tqdm/graphs/contributors
.. |GitHub-Updated| image:: https://img.shields.io/github/last-commit/tqdm/tqdm/master.svg?logo=github&logoColor=white&label=pushed
:target: https://github.com/tqdm/tqdm/pulse
.. |Gift-Casper| image:: https://img.shields.io/badge/dynamic/json.svg?color=ff69b4&label=gifts%20received&prefix=%C2%A3&query=%24..sum&url=https%3A%2F%2Fcaspersci.uk.to%2Fgifts.json
:target: https://caspersci.uk.to/donate
.. |Versions| image:: https://img.shields.io/pypi/v/tqdm.svg
:target: https://tqdm.github.io/releases
.. |PyPI-Downloads| image:: https://img.shields.io/pypi/dm/tqdm.svg?label=pypi%20downloads&logo=PyPI&logoColor=white
:target: https://pypi.org/project/tqdm
.. |Py-Versions| image:: https://img.shields.io/pypi/pyversions/tqdm.svg?logo=python&logoColor=white
:target: https://pypi.org/project/tqdm
.. |Conda-Forge-Status| image:: https://img.shields.io/conda/v/conda-forge/tqdm.svg?label=conda-forge&logo=conda-forge
:target: https://anaconda.org/conda-forge/tqdm
.. |Snapcraft| image:: https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft
:target: https://snapcraft.io/tqdm
.. |Docker| image:: https://img.shields.io/badge/docker-pull-blue.svg?logo=docker&logoColor=white
:target: https://hub.docker.com/r/tqdm/tqdm
.. |Libraries-Rank| image:: https://img.shields.io/librariesio/sourcerank/pypi/tqdm.svg?logo=koding&logoColor=white
:target: https://libraries.io/pypi/tqdm
.. |Libraries-Dependents| image:: https://img.shields.io/librariesio/dependent-repos/pypi/tqdm.svg?logo=koding&logoColor=white
:target: https://github.com/tqdm/tqdm/network/dependents
.. |OpenHub-Status| image:: https://www.openhub.net/p/tqdm/widgets/project_thin_badge?format=gif
:target: https://www.openhub.net/p/tqdm?ref=Thin+badge
.. |awesome-python| image:: https://awesome.re/mentioned-badge.svg
:target: https://github.com/vinta/awesome-python
.. |LICENCE| image:: https://img.shields.io/pypi/l/tqdm.svg
:target: https://raw.githubusercontent.com/tqdm/tqdm/master/LICENCE
.. |DOI| image:: https://img.shields.io/badge/DOI-10.5281/zenodo.595120-blue.svg
:target: https://doi.org/10.5281/zenodo.595120
.. |binder-demo| image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gh/tqdm/tqdm/master?filepath=DEMO.ipynb
.. |Screenshot-Jupyter1| image:: https://raw.githubusercontent.com/tqdm/img/master/jupyter-1.gif
.. |Screenshot-Jupyter2| image:: https://raw.githubusercontent.com/tqdm/img/master/jupyter-2.gif
.. |Screenshot-Jupyter3| image:: https://raw.githubusercontent.com/tqdm/img/master/jupyter-3.gif
.. |README-Hits| image:: https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&style=social&r=https://github.com/tqdm/tqdm&l=https://caspersci.uk.to/images/tqdm.png&f=https://raw.githubusercontent.com/tqdm/tqdm/master/images/logo.gif
:target: https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&a=plot&r=https://github.com/tqdm/tqdm&l=https://caspersci.uk.to/images/tqdm.png&f=https://raw.githubusercontent.com/tqdm/tqdm/master/images/logo.gif&style=social