Rooba
/
tornado
mirror of https://github.com/tornadoweb/tornado.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document all the new HTTP stuff

This commit is contained in:
Ben Darnell 2014-04-26 23:56:16 -04:00
parent 500a605534
commit 425a31d2a5
15 changed files with 328 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/conf.py
View File

@ -37,6 +37,9 @@ coverage_ignore_modules = [
]
# I wish this could go in a per-module file...
coverage_ignore_classes = [
# tornado.concurrent
"TracebackFuture",
# tornado.gen
"Multi",
"Runner",

5
docs/gen.rst
View File

@ -32,6 +32,11 @@
.. autoclass:: YieldPoint
:members:
.. autofunction:: with_timeout
.. autoexception:: TimeoutError
.. autofunction:: maybe_future
Other classes
-------------

5
docs/http1connection.rst Normal file
View File

@ -0,0 +1,5 @@
``tornado.http1connection`` -- HTTP/1.x client/server implementation
====================================================================
.. automodule:: tornado.http1connection
:members:

1
docs/iostream.rst
View File

@ -48,3 +48,4 @@
----------
.. autoexception:: StreamClosedError
.. autoexception:: UnsatisfiableReadError

1
docs/networking.rst
View File

@ -6,6 +6,7 @@ Asynchronous networking
gen
ioloop
iostream
http1connection
httpclient
netutil
tcpserver

93
docs/releases/next.rst
View File

@ -24,12 +24,6 @@ Backwards-compatibility notes
of the old ``TracebackFuture`` class. ``TracebackFuture`` is now
simply an alias for ``Future``.
`tornado.httpclient`
~~~~~~~~~~~~~~~~~~~~
* The command-line HTTP client (``python -m tornado.httpclient $URL``)
now works on Python 3.
`tornado.gen`
~~~~~~~~~~~~~
@ -39,6 +33,57 @@ Backwards-compatibility notes
* Performance of coroutines has been improved.
* Coroutines no longer generate ``StackContexts`` by default, but they
will be created on demand when needed.
* New function `.with_timeout` wraps a `.Future` and raises an exception
if it doesn't complete in a given amount of time.
`tornado.http1connection`
~~~~~~~~~~~~~~~~~~~~~~~~~
* New module contains the HTTP implementation shared by `tornado.httpserver`
and ``tornado.simple_httpclient``.
`tornado.httpclient`
~~~~~~~~~~~~~~~~~~~~
* The command-line HTTP client (``python -m tornado.httpclient $URL``)
now works on Python 3.
`tornado.httpserver`
~~~~~~~~~~~~~~~~~~~~
* ``tornado.httpserver.HTTPRequest`` has moved to
`tornado.httputil.HTTPServerRequest`.
* HTTP implementation has been unified with ``tornado.simple_httpclient``
in `tornado.http1connection`.
* Now supports ``Transfer-Encoding: chunked`` for request bodies.
* Now supports ``Content-Encoding: gzip`` for request bodies if ``gzip=True``
is passed to the `.HTTPServer` constructor.
* The ``connection`` attribute of `.HTTPServerRequest` is now documented
for public use; applications are expected to write their responses
via the `.HTTPConnection` interface.
* The `.HTTPServerRequest.write` and `.HTTPServerRequest.finish` methods
are now deprecated.
* `.HTTPServer` now supports `.HTTPServerConnectionDelegate` in addition to
the old ``request_callback`` interface. The delegate interface supports
streaming of request bodies.
* `.HTTPServer` now detects the error of an application sending a
``Content-Length`` error that is inconsistent with the actual content.
* New constructor arguments ``max_header_size`` and ``max_body_size``
allow separate limits to be set for different parts of the request.
``max_body_size`` is applied even in streaming mode.
* New constructor argument ``chunk_size`` can be used to limit the amount
of data read into memory at one time per request.
* New constructor arguments ``idle_connection_timeout`` and ``body_timeout``
allow time limits to be placed on the reading of requests.
`tornado.httputil`
~~~~~~~~~~~~~~~~~~
* `.HTTPServerRequest` was moved to this module from `tornado.httpserver`.
* New base classes `.HTTPConnection`, `.HTTPServerConnectionDelegate`,
and `.HTTPMessageDelegate` define the interaction between applications
and the HTTP implementation.
`tornado.ioloop`
~~~~~~~~~~~~~~~~
@ -48,6 +93,7 @@ Backwards-compatibility notes
(when possible) to avoid a garbage-collection-related problem in unit tests.
* New method `.IOLoop.clear_instance` makes it possible to uninstall the
singleton instance.
* `.IOLoop.add_timeout` is now a bit more efficient.
`tornado.iostream`
~~~~~~~~~~~~~~~~~~
@ -57,6 +103,17 @@ Backwards-compatibility notes
for use with coroutines.
* No longer gets confused when an ``IOError`` or ``OSError`` without
an ``errno`` attribute is raised.
* `.BaseIOStream.read_bytes` now accepts a ``partial`` keyword argument,
which can be used to return before the full amount has been read.
This is a more coroutine-friendly alternative to ``streaming_callback``.
* `.BaseIOStream.read_until` and ``read_until_regex`` now acept a
``max_bytes`` keyword argument which will cause the request to fail if
it cannot be satisfied from the given number of bytes.
* `.IOStream` no longer reads from the socket into memory if it does not
need data to satisfy a pending read. As a side effect, the close callback
will not be run immediately if the other side closes the connection
while there is unconsumed data in the buffer.
* The default ``chunk_size`` has been increased to 64KB (from 4KB)
`tornado.netutil`
~~~~~~~~~~~~~~~~~
@ -81,7 +138,13 @@ Backwards-compatibility notes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Improved default cipher suite selection (Python 2.7+).
* HTTP implementation has been unified with ``tornado.httpserver``
in `tornado.http1connection`
* Streaming request bodies are now supported via the ``body_producer``
keyword argument to `tornado.httpclient.HTTPRequest`.
* The ``expect_100_continue`` keyword argument to
`tornado.httpclient.HTTPRequest` allows the use of the HTTP ``Expect:
100-continue`` feature.
`tornado.stack_context`
~~~~~~~~~~~~~~~~~~~~~~~
@ -103,6 +166,11 @@ Backwards-compatibility notes
* When gzip support is enabled, all ``text/*`` mime types will be compressed,
not just those on a whitelist.
* `.Application` now implements the `.HTTPMessageDelegate` interface.
* It is now possible to support streaming request bodies with the
`.stream_request_body` decorator and the new `.RequestHandler.data_received`
method.
* `.RequestHandler.flush` now returns a `.Future` if no callback is given.
`tornado.websocket`
~~~~~~~~~~~~~~~~~~~
@ -116,3 +184,14 @@ Backwards-compatibility notes
messages larger than 2GB on 64-bit systems.
* The fallback mechanism for detecting a missing C compiler now
works correctly on Mac OS X.
* Arguments to `.WebSocketHandler.open` are now decoded in the same way
as arguments to `.RequestHandler.get` and similar methods.
`tornado.wsgi`
~~~~~~~~~~~~~~
* New class `.WSGIAdapter` supports running a Tornado `.Application` on
a WSGI server in a way that is more compatible with Tornado's non-WSGI
`.HTTPServer`. `.WSGIApplication` is deprecated in favor of using
`.WSGIAdapter` with a regular `.Application`.
* `.WSGIAdapter` now supports gzipped output.

2
docs/web.rst
View File

@ -70,6 +70,7 @@
.. automethod:: RequestHandler.send_error
.. automethod:: RequestHandler.write_error
.. automethod:: RequestHandler.clear
.. automethod:: RequestHandler.data_received
Cookies
@ -219,6 +220,7 @@
.. autofunction:: authenticated
.. autofunction:: addslash
.. autofunction:: removeslash
.. autofunction:: stream_request_body
Everything else
---------------

2
tornado/gen.py
View File

@ -467,6 +467,8 @@ def with_timeout(timeout, future, io_loop=None):
relative to `.IOLoop.time`)
Currently only supports Futures, not other `YieldPoint` classes.
.. versionadded:: 3.3
"""
# TODO: allow yield points in addition to futures?
# Tricky to do with stack_context semantics.

82
tornado/http1connection.py
View File

@ -14,6 +14,11 @@
# License for the specific language governing permissions and limitations
# under the License.
"""Client and server implementations of HTTP/1.x.
.. versionadded:: 3.3
"""
from __future__ import absolute_import, division, print_function, with_statement
from tornado.concurrent import Future
@ -26,9 +31,22 @@ from tornado import stack_context
from tornado.util import GzipDecompressor
class HTTP1ConnectionParameters(object):
"""Parameters for `.HTTP1Connection` and `.HTTP1ServerConnection`.
"""
def __init__(self, no_keep_alive=False, protocol=None, chunk_size=None,
max_header_size=None, header_timeout=None, max_body_size=None,
body_timeout=None, use_gzip=False):
"""
:arg bool no_keep_alive: If true, always close the connection after
one request.
:arg str protocol: "http" or "https"
:arg int chunk_size: how much data to read into memory at once
:arg int max_header_size: maximum amount of data for HTTP headers
:arg float header_timeout: how long to wait for all headers (seconds)
:arg int max_body_size: maximum amount of data for body
:arg float body_timeout: how long to wait while reading body (seconds)
:arg bool use_gzip: if true, decode incoming ``Content-Encoding: gzip``
"""
self.no_keep_alive = no_keep_alive
self.protocol = protocol
self.chunk_size = chunk_size or 65536
@ -39,12 +57,19 @@ class HTTP1ConnectionParameters(object):
self.use_gzip = use_gzip
class HTTP1Connection(object):
"""Handles a connection to an HTTP client, executing HTTP requests.
"""Implements the HTTP/1.x protocol.
We parse HTTP headers and bodies, and execute the request callback
until the HTTP conection is closed.
This class can be on its own for clients, or via `HTTP1ServerConnection`
for servers.
"""
def __init__(self, stream, is_client, params=None, context=None):
"""
:arg stream: an `.IOStream`
:arg bool is_client: client or server
:arg params: a `.HTTP1ConnectionParameters` instance or ``None``
:arg context: an opaque application-defined object that can be accessed
as ``connection.context``.
"""
self.is_client = is_client
self.stream = stream
if params is None:
@ -85,6 +110,16 @@ class HTTP1Connection(object):
self._expected_content_remaining = None
def read_response(self, delegate):
"""Read a single HTTP response.
Typical client-mode usage is to write a request using `write_headers`,
`write`, and `finish`, and then call ``read_response``.
:arg delegate: a `.HTTPMessageDelegate`
Returns a `.Future` that resolves to None after the full response has
been read.
"""
if self.params.use_gzip:
delegate = _GzipMessageDelegate(delegate, self.params.chunk_size)
return self._read_message(delegate)
@ -190,10 +225,8 @@ class HTTP1Connection(object):
def set_close_callback(self, callback):
"""Sets a callback that will be run when the connection is closed.
Use this instead of accessing
`HTTPConnection.stream.set_close_callback
<.BaseIOStream.set_close_callback>` directly (which was the
recommended approach prior to Tornado 3.0).
.. deprecated:: 3.3
Use `.HTTPMessageDelegate.on_connection_close` instead.
"""
self._close_callback = stack_context.wrap(callback)
@ -211,18 +244,34 @@ class HTTP1Connection(object):
self._clear_callbacks()
def detach(self):
"""Take control of the underlying stream.
Returns the underlying `.IOStream` object and stops all further
HTTP processing. May only be called during
`.HTTPMessageDelegate.headers_received`. Intended for implementing
protocols like websockets that tunnel over an HTTP handshake.
"""
stream = self.stream
self.stream = None
return stream
def set_body_timeout(self, timeout):
"""Sets the body timeout for a single request.
Overrides the value from `.HTTP1ConnectionParameters`.
"""
self._body_timeout = timeout
def set_max_body_size(self, max_body_size):
"""Sets the body size limit for a single request.
Overrides the value from `.HTTP1ConnectionParameters`.
"""
self._max_body_size = max_body_size
def write_headers(self, start_line, headers, chunk=None, callback=None,
has_body=True):
"""Implements `.HTTPConnection.write_headers`."""
if self.is_client:
self._request_start_line = start_line
# Client requests with a non-empty body must have either a
@ -298,7 +347,7 @@ class HTTP1Connection(object):
return chunk
def write(self, chunk, callback=None):
"""Writes a chunk of output to the stream."""
"""Implements `.HTTPConnection.write`."""
if self.stream.closed():
self._write_future = Future()
self._write_future.set_exception(iostream.StreamClosedError())
@ -312,7 +361,7 @@ class HTTP1Connection(object):
return self._write_future
def finish(self):
"""Finishes the request."""
"""Implements `.HTTPConnection.finish`."""
if (self._expected_content_remaining is not None and
self._expected_content_remaining != 0 and
not self.stream.closed()):
@ -492,7 +541,14 @@ class _GzipMessageDelegate(httputil.HTTPMessageDelegate):
class HTTP1ServerConnection(object):
"""An HTTP/1.x server."""
def __init__(self, stream, params=None, context=None):
"""
:arg stream: an `.IOStream`
:arg params: a `.HTTP1ConnectionParameters` or None
:arg context: an opaque application-defined object that is accessible
as ``connection.context``
"""
self.stream = stream
if params is None:
params = HTTP1ConnectionParameters()
@ -502,6 +558,10 @@ class HTTP1ServerConnection(object):
@gen.coroutine
def close(self):
"""Closes the connection.
Returns a `.Future` that resolves after the serving loop has exited.
"""
self.stream.close()
# Block until the serving loop is done, but ignore any exceptions
# (start_serving is already responsible for logging them).
@ -511,6 +571,10 @@ class HTTP1ServerConnection(object):
pass
def start_serving(self, delegate):
"""Starts serving requests on this connection.
:arg delegate: a `.HTTPServerConnectionDelegate`
"""
assert isinstance(delegate, httputil.HTTPServerConnectionDelegate)
self._serving_future = self._server_request_loop(delegate)
# Register the future on the IOLoop so its errors get logged.

10
tornado/httpclient.py
View File

@ -270,13 +270,16 @@ class HTTPRequest(object):
:arg body: HTTP request body as a string (byte or unicode; if unicode
the utf-8 encoding will be used)
:arg body_producer: Callable used for lazy/asynchronous request bodies.
TODO: document the interface.
It is called with one argument, a ``write`` function, and should
return a `.Future`. It should call the write function with new
data as it becomes available. The write function returns a
`.Future` which can be used for flow control.
Only one of ``body`` and ``body_producer`` may
be specified. ``body_producer`` is not supported on
``curl_httpclient``. When using ``body_producer`` it is recommended
to pass a ``Content-Length`` in the headers as otherwise chunked
encoding will be used, and many servers do not support chunked
encoding on requests.
encoding on requests. New in Tornado 3.3
:arg string auth_username: Username for HTTP authentication
:arg string auth_password: Password for HTTP authentication
:arg string auth_mode: Authentication mode; default is "basic".
@ -349,6 +352,9 @@ class HTTPRequest(object):
.. versionadded:: 3.1
The ``auth_mode`` argument.
.. versionadded:: 3.3
The ``body_producer`` and ``expect_100_continue`` arguments.
"""
# Note that some of these attributes go through property setters
# defined below.

39
tornado/httpserver.py
View File

@ -41,41 +41,33 @@ from tornado.tcpserver import TCPServer
class HTTPServer(TCPServer, httputil.HTTPServerConnectionDelegate):
r"""A non-blocking, single-threaded HTTP server.
A server is defined by a request callback that takes an HTTPRequest
instance as an argument and writes a valid HTTP response with
`.HTTPServerRequest.write`. `.HTTPServerRequest.finish` finishes the request (but does
not necessarily close the connection in the case of HTTP/1.1 keep-alive
requests). A simple example server that echoes back the URI you
requested::
A server is defined by either a request callback that takes a
`.HTTPServerRequest` as an argument or a `.HTTPServerConnectionDelegate`
instance.
A simple example server that echoes back the URI you requested::
import tornado.httpserver
import tornado.ioloop
def handle_request(request):
message = "You requested %s\n" % request.uri
request.write("HTTP/1.1 200 OK\r\nContent-Length: %d\r\n\r\n%s" % (
len(message), message))
request.finish()
request.connection.write_headers(
httputil.ResponseStartLine('HTTP/1.1', 200, 'OK'),
{"Content-Length": str(len(message))})
request.connection.write(message)
request.connection.finish()
http_server = tornado.httpserver.HTTPServer(handle_request)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()
`HTTPServer` is a very basic connection handler. It parses the request
headers and body, but the request callback is responsible for producing
the response exactly as it will appear on the wire. This affords
maximum flexibility for applications to implement whatever parts
of HTTP responses are required.
Applications should use the methods of `.HTTPConnection` to write
their response.
`HTTPServer` supports keep-alive connections by default
(automatically for HTTP/1.1, or for HTTP/1.0 when the client
requests ``Connection: keep-alive``). This means that the request
callback must generate a properly-framed response, using either
the ``Content-Length`` header or ``Transfer-Encoding: chunked``.
Applications that are unable to frame their responses properly
should instead return a ``Connection: close`` header in each
response and pass ``no_keep_alive=True`` to the `HTTPServer`
constructor.
requests ``Connection: keep-alive``).
If ``xheaders`` is ``True``, we support the
``X-Real-Ip``/``X-Forwarded-For`` and
@ -135,6 +127,11 @@ class HTTPServer(TCPServer, httputil.HTTPServerConnectionDelegate):
servers if you want to create your listening sockets in some
way other than `tornado.netutil.bind_sockets`.
.. versionchanged:: 3.3
Added ``gzip``, ``chunk_size``, ``max_header_size``,
``idle_connection_timeout``, ``body_timeout``, ``max_body_size``
arguments. Added support for `.HTTPServerConnectionDelegate`
instances as ``request_callback``.
"""
def __init__(self, request_callback, no_keep_alive=False, io_loop=None,
xheaders=False, ssl_options=None, protocol=None, gzip=False,

112
tornado/httputil.py
View File

@ -316,6 +316,9 @@ class HTTPServerRequest(object):
be accessed through the "connection" attribute. Since connections
are typically kept open in HTTP/1.1, multiple requests can be handled
sequentially on a single connection.
.. versionchanged:: 3.3
Moved from ``tornado.httpserver.HTTPRequest``.
"""
def __init__(self, method=None, uri=None, version="HTTP/1.0", headers=None,
body=None, host=None, files=None, connection=None,
@ -345,7 +348,13 @@ class HTTPServerRequest(object):
self.body_arguments = {}
def supports_http_1_1(self):
"""Returns True if this request supports HTTP/1.1 semantics"""
"""Returns True if this request supports HTTP/1.1 semantics.
.. deprecated:: 3.3
Applications are less likely to need this information with the
introduction of `.HTTPConnection`. If you still need it, access
the ``version`` attribute directly.
"""
return self.version == "HTTP/1.1"
@property
@ -362,12 +371,22 @@ class HTTPServerRequest(object):
return self._cookies
def write(self, chunk, callback=None):
"""Writes the given chunk to the response stream."""
"""Writes the given chunk to the response stream.
.. deprecated:: 3.3
Use ``request.connection`` and the `.HTTPConnection` methods
to write the response.
"""
assert isinstance(chunk, bytes_type)
self.connection.write(chunk, callback=callback)
def finish(self):
"""Finishes this HTTP request on the open connection."""
"""Finishes this HTTP request on the open connection.
.. deprecated:: 3.3
Use ``request.connection`` and the `.HTTPConnection` methods
to write the response.
"""
self.connection.finish()
self._finish_time = time.time()
@ -428,37 +447,122 @@ class HTTPServerRequest(object):
class HTTPInputException(Exception):
"""Exception class for malformed HTTP requests or responses
from remote sources.
.. versionadded:: 3.3
"""
pass
class HTTPOutputException(Exception):
"""Exception class for errors in HTTP output."""
"""Exception class for errors in HTTP output.
.. versionadded:: 3.3
"""
pass
class HTTPServerConnectionDelegate(object):
"""Implement this interface to handle requests from `.HTTPServer`.
.. versionadded:: 3.3
"""
def start_request(self, server_conn, request_conn):
"""This method is called by the server when a new request has started.
:arg server_conn: is an opaque object representing the long-lived
(e.g. tcp-level) connection.
:arg request_conn: is a `.HTTPConnection` object for a single
request/response exchange.
This method should return a `.HTTPMessageDelegate`.
"""
raise NotImplementedError()
def on_close(self, server_conn):
"""This method is called when a connection has been closed.
:arg server_conn: is a server connection that has previously been
passed to ``start_request``.
"""
pass
class HTTPMessageDelegate(object):
"""Implement this interface to handle an HTTP request or response.
.. versionadded:: 3.3
"""
def headers_received(self, start_line, headers):
"""Called when the HTTP headers have been received and parsed.
:arg start_line: a `.RequestStartLine` or `.ResponseStartLine`
depending on whether this is a client or server message.
:arg headers: a `.HTTPHeaders` instance.
Some `.HTTPConnection` methods can only be called during
``headers_received``.
May return a `.Future`; if it does the body will not be read
until it is done.
"""
pass
def data_received(self, chunk):
"""Called when a chunk of data has been received.
May return a `.Future` for flow control.
"""
pass
def finish(self):
"""Called after the last chunk of data has been received."""
pass
def on_connection_close(self):
"""Called if the connection is closed without finishing the request.
If ``headers_received`` is called, either ``finish`` or
``on_connection_close`` will be called, but not both.
"""
pass
class HTTPConnection(object):
"""Applications use this interface to write their responses.
.. versionadded:: 3.3
"""
def write_headers(self, start_line, headers, chunk=None, callback=None,
has_body=True):
"""Write an HTTP header block.
:arg start_line: a `.RequestStartLine` or `.ResponseStartLine`.
:arg headers: a `.HTTPHeaders` instance.
:arg chunk: the first (optional) chunk of data. This is an optimization
so that small responses can be written in the same call as their
headers.
:arg callback: a callback to be run when the write is complete.
:arg has_body: as an optimization, may be ``False`` to indicate
that no further writes will be coming.
Returns a `.Future` if no callback is given.
"""
raise NotImplementedError()
def write(self, chunk, callback=None):
"""Writes a chunk of body data.
The callback will be run when the write is complete. If no callback
is given, returns a Future.
"""
raise NotImplementedError()
def finish(self):
"""Indicates that the last body data has been written.
"""
raise NotImplementedError()
def url_concat(url, args):
"""Concatenate url and argument dictionary regardless of whether
url has existing query parameters.

5
tornado/ioloop.py
View File

@ -159,7 +159,10 @@ class IOLoop(Configurable):
@staticmethod
def clear_instance():
"""Clear the global `IOLoop` instance."""
"""Clear the global `IOLoop` instance.
.. versionadded:: 3.3
"""
if hasattr(IOLoop, "_instance"):
del IOLoop._instance

11
tornado/web.py
View File

@ -768,6 +768,9 @@ class RequestHandler(object):
Note that only one flush callback can be outstanding at a time;
if another flush occurs before the previous flush's callback
has been run, the previous callback will be discarded.
.. versionchanged:: 3.3
Now returns a `.Future` if no callback is given.
"""
chunk = b"".join(self._write_buffer)
self._write_buffer = []
@ -1253,6 +1256,13 @@ class RequestHandler(object):
# in a finally block to avoid GC issues prior to Python 3.4.
self._prepared_future.set_result(None)
def data_received(self, chunk):
"""Implement this method to handle streamed request data.
Requires the `.stream_request_body` decorator.
"""
raise NotImplementedError()
def _log(self):
"""Logs the current request.
@ -1395,6 +1405,7 @@ def stream_request_body(cls):
"""Apply to `RequestHandler` subclasses to enable streaming body support.
This decorator implies the following changes:
* `.HTTPServerRequest.body` is undefined, and body arguments will not
be included in `RequestHandler.get_argument`.
* `RequestHandler.prepare` is called when the request headers have been

1
tornado/wsgi.py
View File

@ -174,6 +174,7 @@ class WSGIAdapter(object):
that it is not possible to use `.AsyncHTTPClient`, or the
`tornado.auth` or `tornado.websocket` modules.
.. versionadded:: 3.3
"""
def __init__(self, application):
if isinstance(application, WSGIApplication):