2018-12-18 20:24:33 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"<h1 align=\"center\">tqdm</h1>\n",
2021-05-03 01:06:55 +00:00
"<img src=\"https://img.tqdm.ml/logo.gif\" align=\"left\" />\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"[](https://pypi.org/project/tqdm)|[](https://tqdm.github.io/releases)|[](https://anaconda.org/conda-forge/tqdm)|[](https://hub.docker.com/r/tqdm/tqdm)|[](https://snapcraft.io/tqdm)\n",
2019-05-19 20:01:23 +00:00
"-|-|-|-|-\n",
2019-01-18 23:28:10 +00:00
"\n",
2023-03-03 18:03:21 +00:00
"[](https://github.com/tqdm/tqdm/actions/workflows/test.yml)|[](https://coveralls.io/github/tqdm/tqdm)|[](https://codecov.io/gh/tqdm/tqdm)|[](https://www.codacy.com/gh/tqdm/tqdm/dashboard)|[](https://libraries.io/pypi/tqdm)|[](https://pepy.tech/project/tqdm)\n",
2019-01-18 23:28:10 +00:00
"-|-|-|-|-|-\n",
"\n",
2021-02-10 20:24:48 +00:00
"[](https://doi.org/10.5281/zenodo.595120)|[](https://raw.githubusercontent.com/tqdm/tqdm/master/LICENCE)|[](https://www.openhub.net/p/tqdm?ref=Thin+badge)|[](https://mybinder.org/v2/gh/tqdm/tqdm/master?filepath=DEMO.ipynb)|[](https://github.com/vinta/awesome-python)\n",
2020-12-24 23:40:58 +00:00
"-|-|-|-|-\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"`tqdm` derives from the Arabic word *taqaddum* (تقدّم) which can mean\n",
"\"progress,\" and is an abbreviation for \"I love you so much\" in Spanish\n",
"(*te quiero demasiado*).\n",
2019-01-18 23:28:10 +00:00
"\n",
"Instantly make your loops show a smart progress meter - just wrap any\n",
"iterable with `tqdm(iterable)`, and you're done!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"from tqdm import tqdm\n",
"for i in tqdm(range(10000)):\n",
" pass"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"`trange(N)` can be also used as a convenient shortcut for\n",
2023-03-03 15:43:42 +00:00
"`tqdm(range(N))`."
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"from tqdm import trange\n",
"for i in trange(10000, unit_scale=True, desc=\"hello\", unit=\"epoch\"):\n",
" pass"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2021-07-04 00:04:03 +00:00
"|[](https://tqdm.github.io/video) [](https://tqdm.github.io/PyData2019/slides.html) [](https://tqdm.github.io/merch)\n",
2020-03-29 23:13:41 +00:00
"-|-\n",
2019-01-18 23:28:10 +00:00
"\n",
"It can also be executed as a module with pipes:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"! seq 9999999 | tqdm --bytes | wc -l"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"```sh\n",
2020-03-29 23:13:41 +00:00
"tar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` > backup.tgz\n",
" 44%|██████████████▊ | 153M/352M [00:14<00:18, 11.0MB/s]\n",
2019-01-18 23:28:10 +00:00
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"Overhead is low -- about 60ns per iteration (80ns with `tqdm.gui`), and\n",
2019-01-18 23:28:10 +00:00
"is unit tested against performance regression. By comparison, the\n",
"well-established\n",
"[ProgressBar](https://github.com/niltonvolpato/python-progressbar) has\n",
"an 800ns/iter overhead.\n",
"\n",
"In addition to its low overhead, `tqdm` uses smart algorithms to predict\n",
"the remaining time and to skip unnecessary iteration displays, which\n",
"allows for a negligible overhead in most cases.\n",
"\n",
"`tqdm` works on any platform (Linux, Windows, Mac, FreeBSD, NetBSD,\n",
"Solaris/SunOS), in any console or in a GUI, and is also friendly with\n",
"IPython/Jupyter notebooks.\n",
"\n",
"`tqdm` does not require any dependencies (not even `curses`!), just\n",
"Python and an environment supporting `carriage return \\r` and\n",
2019-01-18 23:55:05 +00:00
"`line feed \\n` control characters.\n",
"\n",
"---\n",
"\n",
2018-12-18 20:24:33 +00:00
"## Usage\n",
"\n",
2019-01-18 23:28:10 +00:00
"`tqdm` is very versatile and can be used in a number of ways.\n",
2019-01-18 23:55:05 +00:00
"The three main ones are given below.\n",
"\n",
2018-12-18 20:24:33 +00:00
"### Iterable-based\n",
"\n",
"Wrap `tqdm()` around any iterable:"
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
"outputs": [],
"source": [
2019-01-18 23:28:10 +00:00
"from tqdm import tqdm\n",
2020-03-29 23:13:41 +00:00
"from time import sleep"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
"text = \"\"\n",
"for char in tqdm([\"a\", \"b\", \"c\", \"d\"]):\n",
2020-03-29 23:13:41 +00:00
" sleep(0.25)\n",
2018-12-18 20:24:33 +00:00
" text = text + char"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`trange(i)` is a special optimised instance of `tqdm(range(i))`:"
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
2018-12-18 20:36:39 +00:00
"from tqdm import trange\n",
"\n",
2019-01-18 23:28:10 +00:00
"for i in trange(100):\n",
2020-03-29 23:13:41 +00:00
" sleep(0.01)"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instantiation outside of the loop allows for manual control over `tqdm()`:"
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
"pbar = tqdm([\"a\", \"b\", \"c\", \"d\"])\n",
"for char in pbar:\n",
2020-03-29 23:13:41 +00:00
" sleep(0.25)\n",
2018-12-18 20:24:33 +00:00
" pbar.set_description(\"Processing %s\" % char)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Manual\n",
"\n",
2020-03-29 23:13:41 +00:00
"Manual control of `tqdm()` updates using a `with` statement:"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
"with tqdm(total=100) as pbar:\n",
" for i in range(10):\n",
2020-03-29 23:13:41 +00:00
" sleep(0.1)\n",
2018-12-18 20:24:33 +00:00
" pbar.update(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"If the optional variable `total` (or an iterable with `len()`) is\n",
"provided, predictive stats are displayed.\n",
2018-12-18 20:24:33 +00:00
"\n",
2019-01-18 23:28:10 +00:00
"`with` is also optional (you can just assign `tqdm()` to a variable,\n",
"but in this case don't forget to `del` or `close()` at the end:"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
"pbar = tqdm(total=100)\n",
"for i in range(10):\n",
2020-03-29 23:13:41 +00:00
" sleep(0.1)\n",
2018-12-18 20:24:33 +00:00
" pbar.update(10)\n",
"pbar.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"### Module"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"Perhaps the most wonderful use of `tqdm` is in a script or on the\n",
"command line. Simply inserting `tqdm` (or `python -m tqdm`) between\n",
"pipes will pass through all `stdin` to `stdout` while printing progress\n",
"to `stderr`.\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"The example below demonstrated counting the number of lines in all\n",
"Python files in the current directory, with timing information included."
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"! time find . -name '*.py' -type f -exec cat \\{} \\; | wc -l"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"! time find . -name '*.py' -type f -exec cat \\{} \\; | tqdm | wc -l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the usual arguments for `tqdm` can also be specified."
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
2021-07-04 00:04:03 +00:00
"! find . -name '*.py' -type f -exec cat \\{} \\; | tqdm --unit loc --unit-scale --total 4104300 --null"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"Backing up a large directory?\n",
"\n",
2019-01-18 23:28:10 +00:00
"```sh\n",
2020-03-29 23:13:41 +00:00
"tar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` > backup.tgz\n",
" 44%|██████████████▊ | 153M/352M [00:14<00:18, 11.0MB/s]\n",
"```\n",
"\n",
"This can be beautified further:\n",
"\n",
"```sh\n",
"BYTES=\"$(du -sb docs/ | cut -f1)\"\n",
"tar -cf - docs/ \\\n",
" | tqdm --bytes --total \"$BYTES\" --desc Processing | gzip \\\n",
" | tqdm --bytes --total \"$BYTES\" --desc Compressed --position 1 \\\n",
" > ~/backup.tgz\n",
"Processing: 100%|██████████████████████| 352M/352M [00:14<00:00, 30.2MB/s]\n",
"Compressed: 42%|█████████▎ | 148M/352M [00:14<00:19, 10.9MB/s]\n",
"```\n",
"\n",
"Or done on a file level using 7-zip:\n",
"\n",
"```sh\n",
"7z a -bd -r backup.7z docs/ | grep Compressing \\\n",
" | tqdm --total $(find docs/ -type f | wc -l) --unit files \\\n",
" | grep -v Compressing\n",
"100%|██████████████████████████▉| 15327/15327 [01:00<00:00, 712.96files/s]\n",
2019-01-18 23:28:10 +00:00
"```"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"## Documentation"
]
},
{
"cell_type": "code",
"execution_count": null,
2019-01-18 23:55:05 +00:00
"metadata": {},
2019-01-18 23:28:10 +00:00
"outputs": [],
"source": [
2019-01-18 23:55:05 +00:00
"tqdm?"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
2019-01-18 23:55:05 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
2019-01-18 23:55:05 +00:00
"! tqdm --help"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Examples and Advance Usage\n",
"\n",
2019-01-19 00:12:19 +00:00
"- See the [examples](https://github.com/tqdm/tqdm/tree/master/examples)\n",
2019-01-18 23:28:10 +00:00
" folder;\n",
"- import the module and run `help()`;\n",
"- consult the [wiki](https://github.com/tqdm/tqdm/wiki)\n",
" - this has an\n",
" [excellent article](https://github.com/tqdm/tqdm/wiki/How-to-make-a-great-Progress-Bar)\n",
2020-03-29 23:13:41 +00:00
" on how to make a **great** progressbar;\n",
"- check out the [slides from PyData London](https://tqdm.github.io/PyData2019/slides.html), or\n",
2019-01-18 23:28:10 +00:00
"- run this file!\n",
"\n",
2018-12-18 20:24:33 +00:00
"### Description and additional stats\n",
"\n",
2019-01-18 23:28:10 +00:00
"Custom information can be displayed and updated dynamically on `tqdm` bars\n",
"with the `desc` and `postfix` arguments:"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
2020-03-29 23:13:41 +00:00
"from tqdm import tqdm, trange\n",
2018-12-18 20:24:33 +00:00
"from random import random, randint\n",
"from time import sleep\n",
"\n",
2019-01-18 23:28:10 +00:00
"with trange(10) as t:\n",
2018-12-18 20:24:33 +00:00
" for i in t:\n",
" # Description will be displayed on the left\n",
" t.set_description('GEN %i' % i)\n",
" # Postfix will be displayed on the right,\n",
" # formatted automatically based on argument's datatype\n",
" t.set_postfix(loss=random(), gen=randint(1,999), str='h',\n",
" lst=[1, 2])\n",
" sleep(0.1)\n",
"\n",
"with tqdm(total=10, bar_format=\"{postfix[0]} {postfix[1][value]:>8.2g}\",\n",
" postfix=[\"Batch\", dict(value=0)]) as t:\n",
" for i in range(10):\n",
" sleep(0.1)\n",
" t.postfix[1][\"value\"] = i / 2\n",
" t.update()"
]
},
2019-01-18 23:28:10 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"Points to remember when using `{postfix[...]}` in the `bar_format` string:\n",
"\n",
"- `postfix` also needs to be passed as an initial argument in a\n",
" compatible format, and\n",
"- `postfix` will be auto-converted to a string if it is a `dict`-like\n",
" object. To prevent this behaviour, insert an extra item into the\n",
" dictionary where the key is not a string.\n",
"\n",
"Additional `bar_format` parameters may also be defined by overriding\n",
"`format_dict`, and the bar itself may be modified using `ascii`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2020-03-29 23:13:41 +00:00
"source": [
"from tqdm import tqdm\n",
"from time import sleep\n",
"\n",
"class TqdmExtraFormat(tqdm):\n",
" \"\"\"Provides a `total_time` format parameter\"\"\"\n",
" @property\n",
" def format_dict(self):\n",
" d = super(TqdmExtraFormat, self).format_dict\n",
" total_time = d[\"elapsed\"] * (d[\"total\"] or 0) / max(d[\"n\"], 1)\n",
" d.update(total_time=self.format_interval(total_time) + \" in total\")\n",
" return d\n",
"\n",
"for i in TqdmExtraFormat(\n",
" range(9), ascii=\" .oO0\",\n",
" bar_format=\"{total_time}: {percentage:.0f}%|{bar}{r_bar}\"):\n",
" if i == 4:\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that `{bar}` also supports a format specifier `[width][type]`.\n",
"\n",
"- `width`\n",
" + unspecified (default): automatic to fill `ncols`\n",
" + `int >= 0`: fixed width overriding `ncols` logic\n",
" + `int < 0`: subtract from the automatic default\n",
"- `type`\n",
" + `a`: ascii (`ascii=True` override)\n",
" + `u`: unicode (`ascii=False` override)\n",
" + `b`: blank (`ascii=\" \"` override)\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"This means a fixed bar with right-justified text may be created by\n",
"using: `bar_format=\"{l_bar}{bar:10}|{bar:-10b}right-justified\"`"
2019-01-18 23:28:10 +00:00
]
},
2018-12-18 20:24:33 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Nested progress bars\n",
"\n",
"`tqdm` supports nested progress bars. Here's an example:"
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2019-01-18 23:28:10 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
2019-01-18 23:28:10 +00:00
"from tqdm.auto import trange\n",
2018-12-18 20:24:33 +00:00
"from time import sleep\n",
"\n",
2019-01-18 23:28:10 +00:00
"for i in trange(4, desc='1st loop'):\n",
" for j in trange(5, desc='2nd loop'):\n",
2020-03-29 23:13:41 +00:00
" for k in trange(50, desc='3rd loop', leave=False):\n",
2019-01-18 23:28:10 +00:00
" sleep(0.01)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"For manual control over positioning (e.g. for multi-processing use),\n",
"you may specify `position=n` where `n=0` for the outermost bar, `n=1`\n",
"for the next, and so on. However, it's best to check if tqdm can work\n",
"without manual position first.\n",
"\n",
"```python\n",
2019-01-18 23:28:10 +00:00
"from time import sleep\n",
"from tqdm import trange, tqdm\n",
2020-10-07 22:46:13 +00:00
"from multiprocessing import Pool, RLock, freeze_support\n",
2019-01-18 23:28:10 +00:00
"\n",
"L = list(range(9))\n",
"\n",
"def progresser(n):\n",
" interval = 0.001 / (n + 2)\n",
" total = 5000\n",
" text = \"#{}, est. {:<04.2}s\".format(n, interval * total)\n",
2020-03-29 23:13:41 +00:00
" for _ in trange(total, desc=text, position=n):\n",
2019-01-18 23:28:10 +00:00
" sleep(interval)\n",
"\n",
"if __name__ == '__main__':\n",
" freeze_support() # for Windows support\n",
2020-10-07 22:46:13 +00:00
" tqdm.set_lock(RLock()) # for managing output contention\n",
2020-03-29 23:13:41 +00:00
" p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))\n",
2019-01-18 23:28:10 +00:00
" p.map(progresser, L)\n",
2020-03-29 23:13:41 +00:00
"```\n",
"\n",
"Note that in Python 3, `tqdm.write` is thread-safe:\n",
"\n",
"```python\n",
"from time import sleep\n",
"from tqdm import tqdm, trange\n",
"from concurrent.futures import ThreadPoolExecutor\n",
"\n",
"L = list(range(9))\n",
"\n",
"def progresser(n):\n",
" interval = 0.001 / (n + 2)\n",
" total = 5000\n",
" text = \"#{}, est. {:<04.2}s\".format(n, interval * total)\n",
" for _ in trange(total, desc=text):\n",
" sleep(interval).auto\n",
" if n == 6:\n",
" tqdm.write(\"n == 6 completed.\")\n",
" tqdm.write(\"`tqdm.write()` is thread-safe in py3!\")\n",
"\n",
"if __name__ == '__main__':\n",
" with ThreadPoolExecutor() as p:\n",
" p.map(progresser, L)\n",
"```"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Hooks and callbacks\n",
"\n",
"`tqdm` can easily support callbacks/hooks and manual updates.\n",
"Here's an example with `urllib`:\n",
"\n",
2020-03-29 23:13:41 +00:00
"**`urllib.urlretrieve` documentation**\n",
2019-01-18 23:28:10 +00:00
"\n",
"> [...]\n",
"> If present, the hook function will be called once\n",
"> on establishment of the network connection and once after each block read\n",
"> thereafter. The hook will be passed three arguments; a count of blocks\n",
"> transferred so far, a block size in bytes, and the total size of the file.\n",
"> [...]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2019-01-18 23:28:10 +00:00
"source": [
"import urllib, os\n",
2019-01-19 00:12:19 +00:00
"from tqdm import tqdm\n",
2021-02-10 20:23:35 +00:00
"urllib = getattr(urllib, 'request', urllib)\n",
2019-01-18 23:28:10 +00:00
"\n",
"class TqdmUpTo(tqdm):\n",
" \"\"\"Provides `update_to(n)` which uses `tqdm.update(delta_n)`.\"\"\"\n",
" def update_to(self, b=1, bsize=1, tsize=None):\n",
" \"\"\"\n",
" b : int, optional\n",
" Number of blocks transferred so far [default: 1].\n",
" bsize : int, optional\n",
" Size of each block (in tqdm units) [default: 1].\n",
" tsize : int, optional\n",
" Total size (in tqdm units). If [default: None] remains unchanged.\n",
" \"\"\"\n",
" if tsize is not None:\n",
" self.total = tsize\n",
2020-09-07 17:13:31 +00:00
" return self.update(b * bsize - self.n) # also sets self.n = b * bsize\n",
2019-01-18 23:28:10 +00:00
"\n",
"eg_link = \"https://caspersci.uk.to/matryoshka.zip\"\n",
2020-05-02 23:04:23 +00:00
"with TqdmUpTo(unit='B', unit_scale=True, unit_divisor=1024, miniters=1,\n",
2019-01-18 23:28:10 +00:00
" desc=eg_link.split('/')[-1]) as t: # all optional kwargs\n",
" urllib.urlretrieve(eg_link, filename=os.devnull,\n",
2020-03-29 23:13:41 +00:00
" reporthook=t.update_to, data=None)\n",
" t.total = t.n"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inspired by [twine#242](https://github.com/pypa/twine/pull/242).\n",
"Functional alternative in\n",
2019-01-19 00:12:19 +00:00
"[examples/tqdm_wget.py](https://github.com/tqdm/tqdm/blob/master/examples/tqdm_wget.py).\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"It is recommend to use `miniters=1` whenever there is potentially large\n",
"differences in iteration speed (e.g. downloading a file over a patchy\n",
"connection).\n",
"\n",
"**Wrapping read/write methods**\n",
"\n",
"To measure throughput through a file-like object's `read` or `write`\n",
"methods, use `CallbackIOWrapper`:\n",
"\n",
"```python\n",
"from tqdm import tqdm\n",
"from tqdm.utils import CallbackIOWrapper\n",
"\n",
"with tqdm(total=file_obj.size,\n",
" unit='B', unit_scale=True, unit_divisor=1024) as t:\n",
" fobj = CallbackIOWrapper(t.update, file_obj, \"read\")\n",
" while True:\n",
" chunk = fobj.read(chunk_size)\n",
" if not chunk:\n",
" break\n",
" t.reset()\n",
" # ... continue to use `t` for something else\n",
"```\n",
"\n",
"Alternatively, use the even simpler `wrapattr` convenience function,\n",
"which would condense both the `urllib` and `CallbackIOWrapper` examples\n",
"down to:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2020-03-29 23:13:41 +00:00
"source": [
"import urllib, os\n",
"from tqdm import tqdm\n",
"\n",
"eg_link = \"https://caspersci.uk.to/matryoshka.zip\"\n",
2021-02-10 20:23:35 +00:00
"response = getattr(urllib, 'request', urllib).urlopen(eg_link)\n",
2020-03-29 23:13:41 +00:00
"with tqdm.wrapattr(open(os.devnull, \"wb\"), \"write\",\n",
2021-02-10 20:23:35 +00:00
" miniters=1, desc=eg_link.split('/')[-1],\n",
" total=getattr(response, 'length', None)) as fout:\n",
" for chunk in response:\n",
2020-03-29 23:13:41 +00:00
" fout.write(chunk)"
2018-12-18 20:24:33 +00:00
]
},
2020-05-02 23:04:23 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
2021-02-10 20:23:35 +00:00
"The `requests` equivalent is nearly identical:"
2020-05-02 23:04:23 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2020-05-02 23:04:23 +00:00
"source": [
"import requests, os\n",
"from tqdm import tqdm\n",
"\n",
"eg_link = \"https://caspersci.uk.to/matryoshka.zip\"\n",
"response = requests.get(eg_link, stream=True)\n",
"with tqdm.wrapattr(open(os.devnull, \"wb\"), \"write\",\n",
" miniters=1, desc=eg_link.split('/')[-1],\n",
2020-05-03 14:12:42 +00:00
" total=int(response.headers.get('content-length', 0))) as fout:\n",
2020-05-02 23:04:23 +00:00
" for chunk in response.iter_content(chunk_size=4096):\n",
" fout.write(chunk)"
]
},
2018-12-18 20:24:33 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Pandas Integration\n",
"\n",
2019-01-18 23:28:10 +00:00
"Due to popular demand we've added support for `pandas` -- here's an example\n",
"for `DataFrame.progress_apply` and `DataFrameGroupBy.progress_apply`:"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2021-05-08 12:08:14 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
"import pandas as pd\n",
"import numpy as np\n",
2019-01-18 23:55:05 +00:00
"from tqdm import tqdm\n",
2018-12-18 20:24:33 +00:00
"\n",
2019-01-18 23:28:10 +00:00
"df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))\n",
2018-12-18 20:24:33 +00:00
"\n",
"# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`\n",
2020-03-29 23:13:41 +00:00
"# (can use `tqdm.gui.tqdm`, `tqdm.notebook.tqdm`, optional kwargs, etc.)\n",
2018-12-18 20:24:33 +00:00
"tqdm.pandas(desc=\"my bar!\")\n",
"\n",
"# Now you can use `progress_apply` instead of `apply`\n",
"# and `progress_map` instead of `map`\n",
"df.progress_apply(lambda x: x**2)\n",
"# can also groupby:\n",
"# df.groupby(0).progress_apply(lambda x: x**2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"In case you're interested in how this works (and how to modify it for\n",
"your own callbacks), see the\n",
"[examples](https://github.com/tqdm/tqdm/tree/master/examples) folder or\n",
"import the module and run `help()`.\n",
"\n",
"### Keras Integration\n",
"\n",
"A `keras` callback is also available:\n",
"\n",
2021-07-04 00:04:03 +00:00
"```python\n",
2020-03-29 23:13:41 +00:00
"from tqdm.keras import TqdmCallback\n",
"\n",
"...\n",
"\n",
"model.fit(..., verbose=0, callbacks=[TqdmCallback()])\n",
"```\n",
2019-01-18 23:28:10 +00:00
"\n",
"### IPython/Jupyter Integration\n",
"\n",
2020-03-29 23:13:41 +00:00
"IPython/Jupyter is supported via the `tqdm.notebook` submodule:"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
2020-03-29 23:13:41 +00:00
"from tqdm.notebook import trange, tqdm\n",
2019-01-18 23:28:10 +00:00
"from time import sleep\n",
"\n",
2020-03-29 23:13:41 +00:00
"for i in trange(3, desc='1st loop'):\n",
" for j in tqdm(range(100), desc='2nd loop'):\n",
2019-01-18 23:28:10 +00:00
" sleep(0.01)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition to `tqdm` features, the submodule provides a native Jupyter\n",
2020-03-29 23:13:41 +00:00
"widget (compatible with IPython v1-v4 and Jupyter), fully working nested\n",
"bars and colour hints (blue: normal, green: completed, red:\n",
"error/interrupt, light blue: no ETA); as demonstrated below.\n",
2019-01-18 23:28:10 +00:00
"\n",
2021-05-03 01:03:50 +00:00
"\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"The `notebook` version supports percentage or pixels for overall width\n",
"(e.g.: `ncols='100%'` or `ncols='480px'`).\n",
"\n",
"It is also possible to let `tqdm` automatically choose between console\n",
"or notebook versions by using the `autonotebook` submodule:"
2019-01-18 23:28:10 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.autonotebook import tqdm\n",
"tqdm.pandas()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2020-03-29 23:13:41 +00:00
"Note that this will issue a `TqdmExperimentalWarning` if run in a\n",
"notebook since it is not meant to be possible to distinguish between\n",
"`jupyter notebook` and `jupyter console`. Use `auto` instead of\n",
"`autonotebook` to suppress this warning.\n",
2019-01-18 23:28:10 +00:00
"\n",
2020-03-29 23:13:41 +00:00
"Note that notebooks will display the bar in the cell where it was\n",
"created. This may be a different cell from the one where it is used. If\n",
"this is not desired, the creation of the bar must be delayed/moved to\n",
"the cell where it is desired to be displayed.\n",
"\n",
"Another possibility is to have a single bar (near the top of the\n",
"notebook) which is constantly re-used (using `reset()` rather than\n",
"`close()`). For this reason, the notebook version (unlike the CLI\n",
"version) does not automatically call `close()` upon `Exception`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.notebook import tqdm\n",
"pbar = tqdm()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# different cell\n",
"iterable = range(100)\n",
"pbar.reset(total=len(iterable)) # initialise with new `total`\n",
"for i in iterable:\n",
" pbar.update()\n",
"pbar.refresh() # force print final status but don't `close()`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-12-18 20:24:33 +00:00
"### Writing messages\n",
"\n",
2020-03-29 23:13:41 +00:00
"This is a work in progress (see\n",
"[#737](https://github.com/tqdm/tqdm/issues/737)).\n",
"\n",
2019-01-18 23:28:10 +00:00
"Since `tqdm` uses a simple printing mechanism to display progress bars,\n",
2020-03-29 23:13:41 +00:00
"you should not write any message in the terminal using `print()` while a\n",
"progressbar is open.\n",
2018-12-18 20:24:33 +00:00
"\n",
2019-01-18 23:28:10 +00:00
"To write messages in the terminal without any collision with `tqdm` bar\n",
"display, a `.write()` method is provided:"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "code",
2019-01-18 23:28:10 +00:00
"execution_count": null,
2018-12-18 20:24:33 +00:00
"metadata": {},
2019-01-18 23:28:10 +00:00
"outputs": [],
2018-12-18 20:24:33 +00:00
"source": [
2020-03-29 23:13:41 +00:00
"from tqdm.auto import tqdm, trange\n",
2018-12-18 20:24:33 +00:00
"from time import sleep\n",
"\n",
"bar = trange(10)\n",
"for i in bar:\n",
" # Print using tqdm class method .write()\n",
" sleep(0.1)\n",
" if not (i % 3):\n",
" tqdm.write(\"Done task %i\" % i)\n",
" # Can also use bar.write()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2019-01-18 23:28:10 +00:00
"By default, this will print to standard output `sys.stdout`. but you can\n",
2020-03-29 23:13:41 +00:00
"specify any file-like object using the `file` argument. For example,\n",
"this can be used to redirect the messages writing to a log file or class.\n",
"\n",
2021-05-03 01:06:55 +00:00
"[](https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&a=plot&r=https://github.com/tqdm/tqdm&l=https://img.tqdm.ml/favicon.png&f=https://img.tqdm.ml/logo.gif&style=social)|(Since 19 May 2016)\n",
2020-03-29 23:13:41 +00:00
"-|-"
2018-12-18 20:24:33 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Do your own experiments here 👇\n",
"\n",
"Try `tqdm` youself by adding your code below and running your own experiments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tqdm\n",
"\n",
"# your code here\n",
"tqdm."
]
}
],
"metadata": {
"kernelspec": {
2021-02-10 20:24:48 +00:00
"display_name": "Python 3",
2018-12-18 20:24:33 +00:00
"language": "python",
2021-02-10 20:24:48 +00:00
"name": "python3"
2018-12-18 20:24:33 +00:00
},
"language_info": {
"codemirror_mode": {
2021-03-08 23:54:13 +00:00
"name": "ipython"
2018-12-18 20:24:33 +00:00
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
2021-03-08 23:54:13 +00:00
"nbconvert_exporter": "python"
2018-12-18 20:24:33 +00:00
}
},
"nbformat": 4,
"nbformat_minor": 2
}
