From 83f8f1863e0c166933b98974f4623cb25ee97a39 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Fri, 29 Sep 2017 14:20:22 +0530 Subject: [PATCH] docs: fix pickler docs, begin relabelling master/slave->parent/child --- docs/howitworks.rst | 80 ++++++++++++++++++++++----------------------- 1 file changed, 39 insertions(+), 41 deletions(-) diff --git a/docs/howitworks.rst b/docs/howitworks.rst index 96ca8852..5d0a3c84 100644 --- a/docs/howitworks.rst +++ b/docs/howitworks.rst @@ -54,7 +54,7 @@ pipe, and the child half writes the string ``EC0\n``, then begins reading the writing the decompressed result to the write-end of the UNIX pipe. To allow recovery of ``stdin`` for reuse by the bootstrapped process for -master<->slave communication, it is necessary for the first stage to avoid +parent<->child communication, it is necessary for the first stage to avoid closing ``stdin`` or reading from it until until EOF. Therefore, the master sends the :py:mod:`zlib`-compressed payload prefixed with an integer size, allowing reading by the first stage of exactly the required bytes. @@ -100,10 +100,10 @@ Preserving The `mitogen.core` Source #################################### One final trick is implemented in the first stage: after bootstrapping the new -slave, it writes a duplicate copy of the :py:mod:`mitogen.core` source it just -used to bootstrap it back into another pipe connected to the slave. The slave's +child, it writes a duplicate copy of the :py:mod:`mitogen.core` source it just +used to bootstrap it back into another pipe connected to the child. The child's module importer cache is initialized with a copy of the source, so that -subsequent bootstraps of slave-of-slaves do not require the source to be +subsequent bootstraps of children-of-children do not require the source to be fetched from the master a second time. @@ -129,7 +129,7 @@ Generating A Synthetic `mitogen` Package Since the bootstrap consists of the :py:mod:`mitogen.core` source code, and this code is loaded by Python by way of its main script (``__main__`` module), -initially the module layout in the slave will be incorrect. +initially the module layout in the child will be incorrect. The first step taken after bootstrap is to rearrange :py:data:`sys.modules` slightly so that :py:mod:`mitogen.core` appears in the correct location, and all @@ -158,12 +158,12 @@ it. Therefore the bootstrap must call :py:func:`os.wait` soon after startup. Setup Logging ############# -The slave's :py:mod:`logging` package root logger is configured to have the +The child's :py:mod:`logging` package root logger is configured to have the same log level as the root logger in the master, and :py:class:`mitogen.core.LogHandler` is installed to forward logs to the master context's :py:data:`FORWARD_LOG ` handle. -The log level is copied into the slave to avoid generating a potentially large +The log level is copied into the child to avoid generating a potentially large amount of network IO forwarding logs that will simply be filtered away once they reach the master. @@ -185,8 +185,8 @@ end is added to the IO multiplexer, and whose write end is used to overwrite the handles inherited during process creation. Even without IO redirection, something must replace ``stdin`` and ``stdout``, -otherwise it is possible for the stream used for communication between the -master and slave to be accidentally corrupted by subprocesses run by user code. +otherwise it is possible for the stream used for communication between parent +and child to be accidentally corrupted by subprocesses run by user code. The inherited ``stdin`` is replaced by a file descriptor pointing to ``/dev/null``. @@ -198,7 +198,7 @@ active, so that ``print`` statements and suchlike promptly appear in the logs. Function Call Dispatch ###################### -After all initialization is complete, the slave's main thread sits in a loop +After all initialization is complete, the child's main thread sits in a loop reading from a :py:class:`Channel ` connected to the :py:data:`CALL_FUNCTION ` handle. This handle is written to by @@ -211,14 +211,14 @@ Shutdown When the master signals the :py:data:`CALL_FUNCTION ` :py:class:`Channel ` is -closed, the slave calls :py:meth:`shutdown() ` +closed, the child calls :py:meth:`shutdown() ` followed by :py:meth:`wait() ` on its own broker, triggering graceful shutdown. -During shutdown, the master will wait a few seconds for slaves to disconnect -gracefully before force disconnecting them, while the slaves will use that time -to call :py:meth:`socket.shutdown(SHUT_WR) ` on their -:py:class:`IoLogger ` socket's write ends before +During shutdown, the master will wait a few seconds for children to disconnect +gracefully before force disconnecting them, while the children will use that +time to call :py:meth:`socket.shutdown(SHUT_WR) ` on +their :py:class:`IoLogger ` socket's write ends before draining any remaining data buffered on the read ends. An alternative approach is to wait until the socket is completely closed, with @@ -239,7 +239,7 @@ Stream Protocol --------------- Once connected, a basic framing protocol is used to communicate between -master and slave: +parent and child: +--------------------+------+------------------------------------------------------+ | Field | Size | Description | @@ -273,13 +273,13 @@ Masters listen on the following handles: .. data:: mitogen.core.ALLOCATE_ID Replies to any message sent to it with a newly allocated unique context ID, - to allow slaves to safely start their own contexts. In future this is + to allow children to safely start their own contexts. In future this is likely to be replaced by 32-bit context IDs and random allocation, with an improved ``ADD_ROUTE`` message sent upstream rather than downstream that generates NACKs if any ancestor already knows the ID. -Slaves listen on the following handles: +Children listen on the following handles: .. data:: mitogen.core.CALL_FUNCTION @@ -290,9 +290,9 @@ Slaves listen on the following handles: `class_name.func_name(\*args, \**kwargs)`. When this channel is closed (by way of sending ``_DEAD`` to it), the - slave's main thread begins graceful shutdown of its own `Broker` and - `Router`. Each slave is responsible for sending ``_DEAD`` to each of its - directly connected slaves in response to the master sending ``_DEAD`` to + child's main thread begins graceful shutdown of its own `Broker` and + `Router`. Each child is responsible for sending ``_DEAD`` to each of its + directly connected children in response to the master sending ``_DEAD`` to it, and arranging for the connection to its parent context to be closed shortly thereafter. @@ -310,7 +310,7 @@ Slaves listen on the following handles: Given a chain `master -> ssh1 -> sudo1`, no `ADD_ROUTE` message is necessary, since :py:class:`mitogen.core.Router` in the `ssh` context can - arrange to update its routes while setting up the new slave during + arrange to update its routes while setting up the new child during `proxy_connect()`. However, given a chain like `master -> ssh1 -> sudo1 -> ssh2 -> sudo2`, @@ -319,8 +319,8 @@ Slaves listen on the following handles: establishment. -Slaves that have ever been used to create a descendent child context also -listen on the following handles: +Children that have ever been used to create a descendent child also listen on +the following handles: .. data:: mitogen.core.GET_MODULE @@ -328,7 +328,7 @@ listen on the following handles: (:py:class:`mitogen.master.ModuleForwarder`) serves responses using :py:class:`mitogen.core.Importer`'s cache before forwarding the request to its parent context. The response is cached by each context in turn before - being forwarded on to the slave context that originally made the request. + being forwarded on to the child context that originally made the request. In this way, the master need never re-send a module it has already sent to a direct descendant. @@ -352,17 +352,16 @@ still registered with :py:meth:`add_handler() Use of Pickle ############# -The current implementation uses the Python :py:mod:`cPickle` module, with -mitigations to prevent untrusted slaves from triggering code excution in the -master. The primary reason for using :py:mod:`cPickle` is that it is -computationally efficient, and avoids including a potentially large body of -serialization code in the bootstrap. +The current implementation uses the Python :py:mod:`cPickle` module, with a +restrictive class whitelist to prevent triggering undesirable code execution. +The primary reason for using :py:mod:`cPickle` is that it is computationally +efficient, and avoids including a potentially large body of serialization code +in the bootstrap. -The pickler active in slave contexts will instantiate any class, however in the -master it is initially restricted to only permitting -:py:class:`CallError ` and :py:data:`_DEAD -`. While not recommended, it is possible to register more -using :py:meth:`mitogen.master.LocalStream.allow_class`. +The pickler will instantiate only built-in types and one of 3 constructor +functions, to support unpickling :py:class:`CallError +`, :py:data:`_DEAD `, and +:py:class:`Context `. The choice of Pickle is one area to be revisited later. All accounts suggest it cannot be used securely, however few of those accounts appear to be expert, and @@ -440,12 +439,12 @@ a new child context, but that is okay for now, since child contexts cannot currently allocate new context IDs anyway. -Differences Between Master And Slave Brokers +Differences Between Master And Child Brokers ############################################ The main difference between :py:class:`mitogen.core.Broker` and :py:class:`mitogen.master.Broker` is that when the stream connection to the -parent is lost in a slave, the broker will trigger its own shutdown. +parent is lost in a child, the broker will trigger its own shutdown. The Module Importer @@ -512,7 +511,6 @@ mechanism is not portable to non-UNIX operating systems, and does not work in every case, for example when Python blocks signals during a variety of :py:mod:`threading` package operations. -At some point it is likely Mitogen will be extended to support starting slaves -running on Windows. When that happens, it would be nice if the process model on -Windows and UNIX did not differ, and in fact the code used on both were -identical. +At some point it is likely Mitogen will be extended to support children running +on Windows. When that happens, it would be nice if the process model on Windows +and UNIX did not differ, and in fact the code used on both were identical.