2017-03-19 00:25:25 +00:00
|
|
|
``ioutils`` - Input/output enhancements
|
2017-03-18 19:52:12 +00:00
|
|
|
=======================================
|
2016-10-21 21:10:40 +00:00
|
|
|
|
|
|
|
|
.. automodule:: boltons.ioutils
|
|
|
|
|
|
|
|
|
|
Spooled Temporary Files
|
|
|
|
|
-----------------------
|
|
|
|
|
Spooled Temporary Files are file-like objects that start out mapped to
|
|
|
|
|
in-memory objects, but automatically roll over to a temporary file once they
|
|
|
|
|
reach a certain (configurable) threshhold. Unfortunately the built-in
|
|
|
|
|
SpooledTemporaryFile class in Python does not implement the exact API that some
|
|
|
|
|
common classes like StringIO do. SpooledTemporaryFile also spools all of it's
|
|
|
|
|
in-memory files as cStringIO instances. cStringIO instances cannot be
|
|
|
|
|
deep-copied, and they don't work with the zip library either. This along with
|
|
|
|
|
the incompatible api makes it useless for several use-cases.
|
|
|
|
|
|
|
|
|
|
To combat this but still gain the memory savings and usefulness of a true
|
|
|
|
|
spooled file-like-object, two custom classes have been implemented which have
|
|
|
|
|
a compatible API.
|
|
|
|
|
|
|
|
|
|
.. _spooledbytesio:
|
|
|
|
|
|
|
|
|
|
SpooledBytesIO
|
|
|
|
|
^^^^^^^^^^^^^^
|
|
|
|
|
.. autoclass:: boltons.ioutils.SpooledBytesIO
|
|
|
|
|
|
2016-10-21 22:56:19 +00:00
|
|
|
.. _spooledstringio:
|
|
|
|
|
|
2016-10-21 21:10:40 +00:00
|
|
|
SpooledStringIO
|
|
|
|
|
^^^^^^^^^^^^^^^
|
|
|
|
|
.. autoclass:: boltons.ioutils.SpooledStringIO
|
2016-10-21 22:56:19 +00:00
|
|
|
|
|
|
|
|
|
2016-10-24 12:06:12 +00:00
|
|
|
Examples
|
|
|
|
|
--------
|
|
|
|
|
It's not uncommon to find excessive usage of StringIO in older Python code. A
|
|
|
|
|
SpooledTemporaryFile would be a nice replacement if one wanted to reduce memory
|
|
|
|
|
overhead, but unfortunately it's api differs too much. This is a good candidate
|
|
|
|
|
for :ref:`spooledbytesio` as it is api compatible and thus may be used as a
|
|
|
|
|
drop-in replacement.
|
|
|
|
|
|
|
|
|
|
Old Code::
|
|
|
|
|
|
|
|
|
|
flo = StringIO()
|
|
|
|
|
flo.write(gigantic_string)
|
|
|
|
|
|
|
|
|
|
Updated::
|
|
|
|
|
|
|
|
|
|
from boltions.ioutils import SpooledBytesIO
|
|
|
|
|
|
|
|
|
|
flo = SpooledBytesIO()
|
|
|
|
|
flo.write(gigantic_string)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Another good use case is downloading a file from some remote location. It's
|
|
|
|
|
nice to keep it in memory if it's small, but writing a large file into memory
|
|
|
|
|
can make servers quite grumpy. If the file being downloaded happens to be a zip
|
|
|
|
|
file then things are worse. You can't use a normal SpooledTemporaryFile because
|
|
|
|
|
it isn't compatible. A :ref:`spooledbytesio` instance is a good alternative.
|
|
|
|
|
Here is a simple example using the requests library to download a zip file::
|
2016-10-21 22:56:19 +00:00
|
|
|
|
|
|
|
|
from zipfile import ZipFile
|
|
|
|
|
|
|
|
|
|
import requests
|
|
|
|
|
from boltons import ioutils
|
|
|
|
|
|
|
|
|
|
s = requests.Session()
|
|
|
|
|
r = s.get("http://127.0.0.1/test_file.zip", stream=True)
|
|
|
|
|
if r.status_code == 200:
|
|
|
|
|
flo = SpooledBytesIO()
|
|
|
|
|
|
|
|
|
|
r = self._get(url, stream=True)
|
|
|
|
|
if r.status_code == 200:
|
|
|
|
|
with ioutils.SpooledBytesIO() as flo:
|
|
|
|
|
for chunk in r.iter_content(chunk_size=64000):
|
|
|
|
|
flo.write(chunk)
|
|
|
|
|
|
|
|
|
|
flo.seek(0)
|
|
|
|
|
|
|
|
|
|
zip_doc = ZipFile(flo)
|
|
|
|
|
|
|
|
|
|
# Print all the files in the zip
|
|
|
|
|
print zip_doc.namelist()
|
