Rooba
/
tqdm
mirror of https://github.com/tqdm/tqdm.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

bump version, Merge branch 'positioned-tqdm'

This commit is contained in:
Casper da Costa-Luis 2016-02-04 20:46:50 +00:00
parent 0328832e04 bf394dd504
commit 35270518e8
10 changed files with 755 additions and 436 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.travis.yml
View File

@ -1,10 +1,10 @@
sudo: false
language: python
python: 3.5
branches: # remove travis double-check on pull requests in main repo
only:
- master
- /^\d\.\d+$/
# branches: # remove travis double-check on pull requests in main repo
# only:
# - master
# - /^\d\.\d+$/
env:
- TOXENV=py26
- TOXENV=py27

58
README.rst
View File

@ -126,7 +126,7 @@ If the optional variable ``total`` (or an iterable with ``len()``) is
provided, predictive stats are displayed.
``with`` is also optional (you can just assign ``tqdm()`` to a variable,
but in this case don't forget to ``close()`` at the end:
but in this case don't forget to ``del`` or ``close()`` at the end:
.. code:: python
@ -148,12 +148,11 @@ Documentation
progressbar every time a value is requested.
"""
def __init__(self, iterable=None, desc=None, total=None, leave=False,
def __init__(self, iterable=None, desc=None, total=None, leave=True,
file=sys.stderr, ncols=None, mininterval=0.1,
maxinterval=10.0, miniters=None, ascii=None,
disable=False, unit='it', unit_scale=False,
dynamic_ncols=False, smoothing=0.3, nested=False,
bar_format=None, initial=0, gui=False):
maxinterval=10.0, miniters=None, ascii=None, disable=False,
unit='it', unit_scale=False, dynamic_ncols=False,
smoothing=0.3, bar_format=None, initial=0, position=None):
Parameters
~~~~~~~~~~
@ -170,7 +169,7 @@ Parameters
True and this parameter needs subsequent updating, specify an
initial arbitrary large positive integer, e.g. int(9e9).
* leave : bool, optional
If [default: False], removes all traces of the progressbar
If [default: True], removes all traces of the progressbar
upon termination of iteration.
* file : `io.TextIOWrapper` or `io.StringIO`, optional
Specifies where to output the progress messages
@ -210,10 +209,6 @@ Parameters
Exponential moving average smoothing factor for speed estimates
(ignored in GUI mode). Ranges from 0 (average speed) to 1
(current/instantaneous speed) [default: 0.3].
* nested : bool, optional
Whether this iterable is nested in another one also managed by
`tqdm` [default: False]. Allows display of multiple, nested
progress bars.
* bar_format : str, optional
Specify a custom bar string formatting. May impact performance.
[default: '{l_bar}{bar}{r_bar}'], where l_bar is
@ -224,7 +219,9 @@ Parameters
* initial : int, optional
The initial counter value. Useful when restarting a progress
bar [default: 0].
* position : int, optional
Specify the line offset to print this bar. Useful to manage
multiple bars at once (eg, from threads).
Returns
~~~~~~~
@ -373,23 +370,25 @@ folder or import the module and run ``help()``.
Nested progress bars
~~~~~~~~~~~~~~~~~~~~
``tqdm`` supports nested progress bars, you just need to specify the
`nested=True` argument for all tqdm instantiations except the **outermost**
bar. Here's an example:
``tqdm`` supports nested progress bars. Here's an example:
.. code:: python
from tqdm import trange
from time import sleep
for i in trange(10, desc='1st loop', leave=True):
for j in trange(5, desc='2nd loop', leave=True, nested=True):
for k in trange(100, desc='3nd loop', leave=True, nested=True):
for i in trange(10, desc='1st loop'):
for j in trange(5, desc='2nd loop', leave=False):
for k in trange(100, desc='3nd loop'):
sleep(0.01)
On Windows `colorama <https://github.com/tartley/colorama>`__ will be used if
available to produce a beautiful nested display.
For manual control over positioning (e.g. for multi-threaded use),
you may specify `position=n` where `n=0` for the outermost bar,
`n=1` for the next, and so on.
How to make a good progress bar
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -400,16 +399,14 @@ a variety of use cases with no or minimal configuration.
However, there is one thing that ``tqdm`` cannot do: choose a pertinent
progress indicator. To display a useful progress bar, it is very important that
you ensure that you supply ``tqdm`` with the most pertinent progress indicator,
which will reflect most accurately the current state of your program.
``tqdm`` is supplied with the most pertinent progress indicator.
This will reflect most accurately the current state of your program.
Usually, a good way is to preprocess quickly to first evaluate the total amount
of work to do before beginning the real processing.
To illustrate the importance of a good progress indicator, let's take the
To illustrate the importance of a good progress indicator, take the
following example: you want to walk through all files of a directory and
process their contents to do your biddings.
Here is a basic program to do that:
process their contents with some external function:
.. code:: python
@ -436,7 +433,7 @@ Here is a basic program to do that:
buf = fh.read(blocksize)
dosomething(buf)
``process_content_no_progress()`` does the job alright, but it does not show
``process_content_no_progress()`` does the job, but does not show
any information about the current progress, nor how long it will take.
To quickly fix that using ``tqdm``, we can use this naive approach:
@ -485,10 +482,10 @@ now we have predictive information:
However, the progress is not smooth: it increments in steps, 1 step being
1 file processed. The problem is that we do not just walk through files tree,
but we process the files contents. Thus, if we stumble on one big fat file,
it will take a huge deal more time to process than other smaller files, but
the progress bar cannot know that, because we only supplied the files count,
so it considers that every element is of equal processing weight.
but we process the files contents. Thus, if we stumble on one very large file
which takes a great deal more time to process than other smaller files,
the progress bar
will still considers that file is of equal processing weight.
To fix this, we should use another indicator than the files count: the total
sum of all files sizes. This would be more pertinent since the data we
@ -525,6 +522,7 @@ predicted time and statistics:
47%|██████████████████▍\ \| 152K/321K [00:03<00:03, 46.2KB/s]
Contributions
-------------
@ -551,7 +549,7 @@ file for more information.
License
-------
`MIT LICENSE <https://raw.githubusercontent.com/tqdm/tqdm/master/LICENSE>`__.
Mostly `CC, MIT licence <https://raw.githubusercontent.com/tqdm/tqdm/master/LICENSE>`__.
Authors

2
examples/simple_examples.py
View File

@ -21,7 +21,7 @@ stmts = (
' ascii=True, desc="cool", dynamic_ncols=True):\n\tpass',
# Nested bars
'from tqdm import trange\nfor i in trange(10):\n\t'
'for j in trange(int(1e7), nested=True):\n\t\tpass',
'for j in trange(int(1e7), leave=False, unit_scale=True):\n\t\tpass',
# Experimental GUI demo
'import tqdm\nfor i in tqdm.tgrange(int(1e8)):\n\tpass',
# Comparison to https://code.google.com/p/python-progressbar/

193
tqdm/_tqdm.py
View File

@ -12,7 +12,7 @@ Usage:
from __future__ import division, absolute_import
# import compatibility functions and utilities
from ._utils import _supports_unicode, _environ_cols_wrapper, _range, _unich, \
_term_move_up, _unicode
_term_move_up, _unicode, WeakSet
import sys
from time import time
@ -88,11 +88,17 @@ class tqdm(object):
if not getattr(fp, 'flush', False): # pragma: no cover
fp.flush = lambda: None
def fp_write(s):
try:
fp.write(_unicode(s))
except UnicodeEncodeError: # pragma: no cover
fp.write(repr(s))
last_printed_len = [0] # closure over mutable variable (fast)
def print_status(s):
len_s = len(s)
fp.write('\r' + s + (' ' * max(last_printed_len[0] - len_s, 0)))
fp_write('\r' + s + (' ' * max(last_printed_len[0] - len_s, 0)))
fp.flush()
last_printed_len[0] = len_s
return print_status
@ -163,7 +169,7 @@ class tqdm(object):
rate_fmt = ((format_sizeof(inv_rate if inv_rate else rate)
if unit_scale else
'{0:5.2f}'.format(inv_rate if inv_rate else rate))
if elapsed else '?') \
if rate else '?') \
+ ('s' if inv_rate else unit) + '/' + (unit if inv_rate else 's')
if unit_scale:
@ -247,12 +253,44 @@ class tqdm(object):
return (prefix if prefix else '') + '{0}{1} [{2}, {3}]'.format(
n_fmt, unit, elapsed_str, rate_fmt)
def __init__(self, iterable=None, desc=None, total=None, leave=False,
def __new__(cls, *args, **kwargs):
instance = object.__new__(cls)
if "_instances" not in cls.__dict__:
cls._instances = WeakSet()
cls._instances.add(instance)
return instance
@classmethod
def _get_free_pos(cls, instance=None):
""" Skips specified instance """
try:
return max(inst.pos for inst in cls._instances
if inst is not instance) + 1
except ValueError as e:
if "arg is an empty sequence" in str(e):
return 0
raise # pragma: no cover
@classmethod
def _decr_instances(cls, instance):
"""
Remove from list and reposition other bars
so that newer bars won't overlap previous bars
"""
try: # in case instance was explicitly positioned, it won't be in set
cls._instances.remove(instance)
for inst in cls._instances:
if inst.pos > instance.pos:
inst.pos -= 1
except KeyError:
pass
def __init__(self, iterable=None, desc=None, total=None, leave=True,
file=sys.stderr, ncols=None, mininterval=0.1,
maxinterval=10.0, miniters=None, ascii=None,
disable=False, unit='it', unit_scale=False,
dynamic_ncols=False, smoothing=0.3, nested=False,
bar_format=None, initial=0, gui=False):
maxinterval=10.0, miniters=None, ascii=None, disable=False,
unit='it', unit_scale=False, dynamic_ncols=False,
smoothing=0.3, bar_format=None, initial=0, position=None,
gui=False, **kwargs):
"""
Parameters
----------
@ -268,7 +306,7 @@ class tqdm(object):
True and this parameter needs subsequent updating, specify an
initial arbitrary large positive integer, e.g. int(9e9).
leave : bool, optional
If [default: False], removes all traces of the progressbar
If [default: True], removes all traces of the progressbar
upon termination of iteration.
file : `io.TextIOWrapper` or `io.StringIO`, optional
Specifies where to output the progress messages
@ -307,10 +345,6 @@ class tqdm(object):
Exponential moving average smoothing factor for speed estimates
(ignored in GUI mode). Ranges from 0 (average speed) to 1
(current/instantaneous speed) [default: 0.3].
nested : bool, optional
Whether this iterable is nested in another one also managed by
`tqdm` [default: False]. Allows display of multiple, nested
progress bars.
bar_format : str, optional
Specify a custom bar string formatting. May impact performance.
[default: '{l_bar}{bar}{r_bar}'], where l_bar is
@ -321,6 +355,9 @@ class tqdm(object):
initial : int, optional
The initial counter value. Useful when restarting a progress
bar [default: 0].
position : int, optional
Specify the line offset to print this bar [default: 0].
Useful to manage multiple bars at once (eg, from threads).
gui : bool, optional
WARNING: internal parameter - do not use.
Use tqdm_gui(...) instead. If set, will attempt to use
@ -330,6 +367,23 @@ class tqdm(object):
-------
out : decorated iterator.
"""
if disable:
self.iterable = iterable
self.disable = disable
self.pos = self._get_free_pos(self)
self._instances.remove(self)
return
if kwargs:
self.disable = True
self.pos = self._get_free_pos(self)
self._instances.remove(self)
raise (DeprecationWarning("nested is deprecated and"
" automated.\nUse position instead"
" for manual control")
if "nested" in kwargs else
Warning("Unknown argument(s): " + str(kwargs)))
# Preprocess the arguments
if total is None and iterable is not None:
try:
@ -342,7 +396,7 @@ class tqdm(object):
if dynamic_ncols: # pragma: no cover
dynamic_ncols = _environ_cols_wrapper()
ncols = dynamic_ncols(file)
else:
else: # pragma: no cover
ncols = _environ_cols_wrapper()(file)
if miniters is None:
@ -387,24 +441,26 @@ class tqdm(object):
self.smoothing = smoothing
self.avg_time = None
self._time = time
# if nested, at initial sp() call we replace '\r' by '\n' to
# not overwrite the outer progress bar
self.nested = nested
self.bar_format = bar_format
# Init the iterations counters
self.last_print_n = initial
self.n = initial
# if nested, at initial sp() call we replace '\r' by '\n' to
# not overwrite the outer progress bar
self.pos = self._get_free_pos(self) if position is None else position
if not gui:
# Initialize the screen printer
self.sp = self.status_printer(self.fp)
if not disable:
if self.nested:
self.fp.write('\n')
self.sp(self.format_meter(self.n, total, 0,
(dynamic_ncols(file) if dynamic_ncols else ncols),
self.desc, ascii, unit, unit_scale, None, bar_format))
if self.pos:
self.moveto(self.pos)
self.sp(self.format_meter(self.n, total, 0,
(dynamic_ncols(file) if dynamic_ncols else ncols),
self.desc, ascii, unit, unit_scale, None, bar_format))
if self.pos:
self.moveto(-self.pos)
# Init the time counter
self.start_t = self.last_print_t = self._time()
@ -419,6 +475,42 @@ class tqdm(object):
self.close()
return False
def __del__(self):
self.close()
def __repr__(self):
return self.format_meter(self.n, self.total, time() - self.last_print_t,
self.ncols, self.desc, self.ascii, self.unit,
self.unit_scale, 1 / self.avg_time
if self.avg_time else None, self.bar_format)
def __lt__(self, other):
# try:
return self.pos < other.pos
# except AttributeError:
# return self.start_t < other.start_t
def __le__(self, other):
return (self < other) or (self == other)
def __eq__(self, other):
# try:
return self.pos == other.pos
# except AttributeError:
# return self.start_t == other.start_t
def __ne__(self, other):
return not (self == other)
def __gt__(self, other):
return not (self <= other)
def __ge__(self, other):
return not (self < other)
def __hash__(self):
return id(self)
def __iter__(self):
''' Backward-compatibility to use: for x in tqdm(iterable) '''
@ -475,6 +567,10 @@ class tqdm(object):
else smoothing * delta_t / delta_it + \
(1 - smoothing) * avg_time
if self.pos:
self.moveto(self.pos)
# Printing the bar's update
sp(format_meter(
n, self.total, elapsed,
(dynamic_ncols(self.fp) if dynamic_ncols
@ -482,6 +578,9 @@ class tqdm(object):
self.desc, ascii, unit, unit_scale,
1 / avg_time if avg_time else None, bar_format))
if self.pos:
self.moveto(-self.pos)
# If no `miniters` was specified, adjust automatically
# to the maximum iteration rate seen so far.
if dynamic_miniters:
@ -552,6 +651,10 @@ class tqdm(object):
raise DeprecationWarning('Please use tqdm_gui(...)'
' instead of tqdm(..., gui=True)')
if self.pos:
self.moveto(self.pos)
# Print bar's update
self.sp(self.format_meter(
self.n, self.total, elapsed,
(self.dynamic_ncols(self.fp) if self.dynamic_ncols
@ -560,6 +663,9 @@ class tqdm(object):
1 / self.avg_time if self.avg_time else None,
self.bar_format))
if self.pos:
self.moveto(-self.pos)
# If no `miniters` was specified, adjust automatically to the
# maximum iteration rate seen so far.
# e.g.: After running `tqdm.update(5)`, subsequent
@ -588,6 +694,34 @@ class tqdm(object):
if self.disable:
return
# Prevent multiple closures
self.disable = True
# decrement instance pos and remove from internal set
pos = self.pos
self._decr_instances(self)
# GUI mode
if not hasattr(self, "sp"):
return
# annoyingly, _supports_unicode isn't good enough
def fp_write(s):
try:
self.fp.write(_unicode(s))
except UnicodeEncodeError: # pragma: no cover
self.fp.write(repr(s))
try:
fp_write('')
except ValueError as e:
if 'closed' in str(e):
return
raise # pragma: no cover
if pos:
self.moveto(pos)
if self.leave:
if self.last_print_n < self.n:
cur_t = self._time()
@ -598,10 +732,16 @@ class tqdm(object):
else self.ncols),
self.desc, self.ascii, self.unit, self.unit_scale, None,
self.bar_format))
self.fp.write('\r' + _term_move_up() if self.nested else '\n')
if pos:
self.moveto(-pos)
else:
fp_write('\n')
else:
self.sp('') # clear up last bar
self.fp.write('\r' + _term_move_up() if self.nested else '\r')
if pos:
self.moveto(-pos)
else:
fp_write('\r')
def unpause(self):
"""
@ -617,6 +757,9 @@ class tqdm(object):
"""
self.desc = desc + ': ' if desc else ''
def moveto(self, n):
self.fp.write(_unicode('\n' * n + _term_move_up() * -n))
def trange(*args, **kwargs):
"""

4
tqdm/_tqdm_gui.py
View File

@ -330,6 +330,10 @@ class tqdm_gui(tqdm): # pragma: no cover
if self.disable:
return
self.disable = True
self._instances.remove(self)
# Restore toolbars
self.mpl.rcParams['toolbar'] = self.toolbar
# Return to non-interactive mode

5
tqdm/_utils.py
View File

@ -22,6 +22,11 @@ try: # pragma: no cover
except ImportError: # pragma: no cover
colorama = None
try: # pragma: no cover
from weakref import WeakSet
except ImportError: # pragma: nocover
WeakSet = set
def _is_utf(encoding):
return ('U8' == encoding) or ('utf' in encoding) or ('UTF' in encoding)

2
tqdm/_version.py
View File

@ -1,5 +1,5 @@
# Definition of the version number
version_info = 3, 8, 0 # major, minor, patch, -extra
version_info = 4, 0, 0 # major, minor, patch, -extra
# Nice string for the version
__version__ = '.'.join(map(str, version_info)).replace('.-', '-').strip('.-')

40
tqdm/tests/tests_pandas.py
View File

@ -1,21 +1,12 @@
from nose.plugins.skip import SkipTest
from tqdm import tqdm
try:
from StringIO import StringIO
except:
from io import StringIO
# Ensure we can use `with closing(...) as ... :` syntax
if getattr(StringIO, '__exit__', False) and \
getattr(StringIO, '__enter__', False):
def closing(arg):
return arg
else:
from contextlib import closing
from tests_tqdm import with_setup, pretest, posttest, StringIO, closing
@with_setup(pretest, posttest)
def test_pandas():
""" Test pandas.DataFrame.groupby(0).progress_apply """
try:
from numpy.random import randint
from tqdm import tqdm_pandas
@ -30,15 +21,18 @@ def test_pandas():
our_file.seek(0)
try:
# don't expect final output since no `leave` and
# high dynamic `miniters`
assert '100%|##########| 101/101' not in our_file.read()
except:
raise AssertionError('Did not expect:\n\t100%|##########| 101/101')
# don't expect final output since no `leave` and
# high dynamic `miniters`
nexres = '100%|##########| 101/101'
if nexres in our_file.read():
our_file.seek(0)
raise AssertionError("\nDid not expect:\n{0}\nIn:{1}\n".format(
nexres, our_file.read()))
@with_setup(pretest, posttest)
def test_pandas_leave():
""" Test pandas with `leave=True` """
try:
from numpy.random import randint
from tqdm import tqdm_pandas
@ -53,10 +47,8 @@ def test_pandas_leave():
our_file.seek(0)
try:
assert '100%|##########| 101/101' in our_file.read()
except:
exres = '100%|##########| 101/101'
if exres not in our_file.read():
our_file.seek(0)
raise AssertionError('\n'.join(('Expected:',
'100%|##########| 101/101', 'Got:',
our_file.read())))
raise AssertionError("\nExpected:\n{0}\nIn:{1}\n".format(
exres, our_file.read()))

52
tqdm/tests/tests_perf.py
View File

@ -10,22 +10,7 @@ from time import sleep, time
from tqdm import trange
from tqdm import tqdm
try:
from StringIO import StringIO
except:
from io import StringIO
# Ensure we can use `with closing(...) as ... :` syntax
if getattr(StringIO, '__exit__', False) and \
getattr(StringIO, '__enter__', False):
def closing(arg):
return arg
else:
from contextlib import closing
try:
_range = xrange
except:
_range = range
from tests_tqdm import with_setup, pretest, posttest, StringIO, closing, _range
# Use relative/cpu timer to have reliable timings when there is a sudden load
try:
@ -64,10 +49,15 @@ def checkCpuTime(sleeptime=0.2):
@contextmanager
def relative_timer():
start = process_time()
elapser = lambda: process_time() - start
def elapser():
return process_time() - start
yield lambda: elapser()
spent = process_time() - start
elapser = lambda: spent
def elapser(): # NOQA
return spent
class MockIO(StringIO):
@ -141,6 +131,7 @@ def simple_progress(iterable=None, total=None, file=sys.stdout, desc='',
return update_and_print
@with_setup(pretest, posttest)
def test_iter_overhead():
""" Test overhead of iteration based tqdm """
try:
@ -164,13 +155,12 @@ def test_iter_overhead():
our_file.write(a)
# Compute relative overhead of tqdm against native range()
try:
assert(time_tqdm() < 3 * time_bench())
except AssertionError:
if time_tqdm() > 9 * time_bench():
raise AssertionError('trange(%g): %f, range(%g): %f' %
(total, time_tqdm(), total, time_bench()))
@with_setup(pretest, posttest)
def test_manual_overhead():
""" Test overhead of manual tqdm """
try:
@ -181,12 +171,12 @@ def test_manual_overhead():
total = int(1e6)
with closing(MockIO()) as our_file:
t = tqdm(total=total * 10, file=our_file, leave=True)
a = 0
with relative_timer() as time_tqdm:
for i in _range(total):
a += i
t.update(10)
with tqdm(total=total * 10, file=our_file, leave=True) as t:
a = 0
with relative_timer() as time_tqdm:
for i in _range(total):
a += i
t.update(10)
a = 0
with relative_timer() as time_bench:
@ -195,13 +185,12 @@ def test_manual_overhead():
our_file.write(a)
# Compute relative overhead of tqdm against native range()
try:
assert(time_tqdm() < 10 * time_bench())
except AssertionError:
if time_tqdm() > 10 * time_bench():
raise AssertionError('tqdm(%g): %f, range(%g): %f' %
(total, time_tqdm(), total, time_bench()))
@with_setup(pretest, posttest)
def test_iter_overhead_hard():
""" Test overhead of iteration based tqdm (hard) """
try:
@ -233,6 +222,7 @@ def test_iter_overhead_hard():
(total, time_tqdm(), total, time_bench()))
@with_setup(pretest, posttest)
def test_manual_overhead_hard():
""" Test overhead of manual tqdm (hard) """
try:
@ -265,6 +255,7 @@ def test_manual_overhead_hard():
(total, time_tqdm(), total, time_bench()))
@with_setup(pretest, posttest)
def test_iter_overhead_simplebar_hard():
""" Test overhead of iteration based tqdm vs simple progress bar (hard) """
try:
@ -298,6 +289,7 @@ def test_iter_overhead_simplebar_hard():
(total, time_tqdm(), total, time_bench()))
@with_setup(pretest, posttest)
def test_manual_overhead_simplebar_hard():
""" Test overhead of manual tqdm vs simple progress bar (hard) """
try:

827
tqdm/tests/tests_tqdm.py
View File

File diff suppressed because it is too large Load Diff