368 lines
14 KiB
ReStructuredText
368 lines
14 KiB
ReStructuredText
|
||
Ansible Extension
|
||
=================
|
||
|
||
.. image:: images/ansible/cell_division.png
|
||
:align: right
|
||
|
||
An experimental extension to `Ansible`_ is included that implements host
|
||
connections over Mitogen, replacing embedded shell invocations with pure-Python
|
||
equivalents invoked via highly efficient remote procedure calls tunnelled over
|
||
SSH. No changes are required to the target hosts.
|
||
|
||
The extension isn't nearly in a generally dependable state yet, however it
|
||
already works well enough for testing against real-world playbooks. `Bug
|
||
reports`_ in this area are very welcome – Ansible is a huge beast, and only
|
||
significant testing will prove the extension's soundness.
|
||
|
||
Divergence from Ansible's normal behaviour is considered a bug, so please
|
||
report anything you notice, regardless of how inconsequential it may seem.
|
||
|
||
.. _Ansible: https://www.ansible.com/
|
||
|
||
.. _Bug reports: https://goo.gl/yLKZiJ
|
||
|
||
|
||
Overview
|
||
--------
|
||
|
||
You should **expect a 1.25x - 7x speedup** and a **CPU usage reduction of at
|
||
least 2x**, depending on network conditions, the specific modules executed, and
|
||
time spent by the target host already doing useful work. Mitogen cannot speed
|
||
up a module once it is executing, it can only ensure the module executes as
|
||
quickly as possible.
|
||
|
||
* **A single SSH connection is used for each target host**, in addition to one
|
||
sudo invocation per distinct user account. Subsequent playbook steps always
|
||
reuse the same connection. This is much better than SSH multiplexing combined
|
||
with pipelining, as significant state can be maintained in RAM between steps,
|
||
and the system logs aren't filled with spam from repeat SSH and sudo
|
||
invocations.
|
||
|
||
* **A single Python interpreter is used** per host and sudo account combination
|
||
for the duration of the run, avoiding the repeat cost of invoking multiple
|
||
interpreters and recompiling imports, saving 300-800 ms for every playbook
|
||
step.
|
||
|
||
* Remote interpreters reuse Mitogen's module import mechanism, caching uploaded
|
||
dependencies between steps at the host and user account level. As a
|
||
consequence, **bandwidth usage is consistently an order of magnitude lower**
|
||
compared to SSH pipelining, and around 5x fewer frames are required to
|
||
traverse the wire for a run to complete successfully.
|
||
|
||
* **No writes to the target host's filesystem occur**, unless explicitly
|
||
triggered by a playbook step. In all typical configurations, Ansible
|
||
repeatedly rewrites and extracts ZIP files to multiple temporary directories
|
||
on the target host. Since no temporary files are used, security issues
|
||
relating to those files in cross-account scenarios are entirely avoided.
|
||
|
||
|
||
Testimonials
|
||
------------
|
||
|
||
* "With mitogen **my playbook runtime went from 45 minutes to just under 3
|
||
minutes**. Awesome work!"
|
||
|
||
* "The runtime was reduced from **1.5 hours on 4 servers to just under 3
|
||
minutes**. Thanks!"
|
||
|
||
* "Oh, performance improvement using Mitogen is *huge*. As mentioned before,
|
||
running with Mitogen enables takes 7m36 (give or take a few seconds). Without
|
||
Mitogen, the same run takes 19m49! **I'm not even deploying without Mitogen
|
||
anymore** :)"
|
||
|
||
* "**Works like a charm**, thank you for your quick response"
|
||
|
||
* "I tried it out. **He is not kidding about the speed increase**."
|
||
|
||
* "I don't know what kind of dark magic @dmw_83 has done, but his Mitogen
|
||
strategy took Clojars' Ansible runs from **14 minutes to 2 minutes**. I still
|
||
can't quite believe it."
|
||
|
||
|
||
Installation
|
||
------------
|
||
|
||
.. caution::
|
||
|
||
Thoroughly review the list of limitations before use, and **do not test the
|
||
prototype in a live environment until this notice is removed**.
|
||
|
||
1. Verify Ansible 2.4 and Python 2.7 are listed in the output of ``ansible
|
||
--version``
|
||
2. Download and extract https://github.com/dw/mitogen/archive/master.zip
|
||
3. Modify ``ansible.cfg``:
|
||
|
||
.. code-block:: dosini
|
||
|
||
[defaults]
|
||
strategy_plugins = /path/to/mitogen-master/ansible_mitogen/plugins/strategy
|
||
strategy = mitogen
|
||
|
||
The ``strategy`` key is optional. If omitted, you can set the
|
||
``ANSIBLE_STRATEGY=mitogen`` environment variable on a per-run basis.
|
||
|
||
4. Cross your fingers and try it.
|
||
|
||
|
||
Limitations
|
||
-----------
|
||
|
||
This is a proof of concept: issues below are exclusively due to code immaturity.
|
||
|
||
High Risk
|
||
~~~~~~~~~
|
||
|
||
* Transfer of large (i.e. GB-sized) files using certain Ansible-internal APIs,
|
||
such as triggered via the ``copy`` module, will cause corresponding temporary
|
||
memory and CPU spikes on both host and target machine, due to delivering the
|
||
file as a single large message. If many machines are targetted with a large
|
||
file, the host machine could easily exhaust available RAM. This will be fixed
|
||
soon as it's likely to be tickled by common playbooks.
|
||
|
||
* Local actions are single threaded. Any that execute for every target will
|
||
experience artificial serialization, causing slowdown equivalent to
|
||
`task_duration * num_targets`. This will be fixed soon.
|
||
|
||
* `Asynchronous Actions And Polling
|
||
<https://docs.ansible.com/ansible/latest/playbooks_async.html>`_ has received
|
||
minimal testing. Jobs execute in a thread of the target Python interpreter.
|
||
This will fixed shortly.
|
||
|
||
* No mechanism exists yet to bound the number of interpreters created during a
|
||
run. For some playbooks that parameterize ``become_user`` over a large number
|
||
of user accounts, resource exhaustion may be triggered on the target machine.
|
||
|
||
* Only Ansible 2.4 is being used for development, with occasional tests under
|
||
2.5, 2.3 and 2.2. It should be more than possible to fully support at least
|
||
2.3, if not also 2.2.
|
||
|
||
|
||
Low Risk
|
||
~~~~~~~~
|
||
|
||
* Only UNIX machines running Python 2.x are supported, Windows will come later.
|
||
|
||
* Only the ``sudo`` become method is available, however adding new methods is
|
||
straightforward, and eventually at least ``su`` will be included.
|
||
|
||
* The only supported strategy is ``linear``, which is Ansible's default.
|
||
|
||
* In some cases ``remote_tmp`` may not be respected.
|
||
|
||
* The extension's performance benefits do not scale perfectly linearly with the
|
||
number of targets. This is a subject of ongoing investigation and
|
||
improvements will appear in time.
|
||
|
||
* Ansible defaults to requiring pseudo TTYs for most SSH invocations, in order
|
||
to allow it to handle ``sudo`` with ``requiretty`` enabled, however it
|
||
disables pseudo TTYs for certain commands where standard input is required or
|
||
``sudo`` is not in use. Mitogen does not require this, as it can simply call
|
||
:py:func:`pty.openpty` from the SSH user account during ``sudo`` setup.
|
||
|
||
A major downside to Ansible's default is that stdout and stderr of any
|
||
resulting executed command are merged, with additional carriage return
|
||
characters synthesized in the output by the TTY layer. Neither of these
|
||
problems are apparent using the Mitogen extension, which may break some
|
||
playbooks.
|
||
|
||
A future version will emulate Ansible's behaviour, once it is clear precisely
|
||
what that behaviour is supposed to be. See `Ansible#14377`_ for related
|
||
discussion.
|
||
|
||
* "Module Replacer" style modules are not yet supported. These rarely appear in
|
||
practice, and light Github code searches failed to reveal many examples of
|
||
them.
|
||
|
||
.. _Ansible#14377: https://github.com/ansible/ansible/issues/14377
|
||
|
||
|
||
Behavioural Differences
|
||
-----------------------
|
||
|
||
* Ansible permits up to ``forks`` SSH connections to be setup simultaneously,
|
||
whereas in Mitogen this is handled by a thread pool. Eventually this pool
|
||
will become per-CPU, but meanwhile, a maximum of 16 SSH connections may be
|
||
established simultaneously by default. This can be increased or decreased
|
||
setting the ``MITOGEN_POOL_SIZE`` environment variable.
|
||
|
||
* Mitogen treats connection timeouts for the SSH and become steps of a task
|
||
invocation separately, meaning that in some circumstances the configured
|
||
timeout may appear to be doubled. This is since Mitogen internally treats the
|
||
creation of an SSH account context separately to the creation of a sudo
|
||
account context proxied via that SSH account.
|
||
|
||
A future revision may detect a sudo account context created immediately
|
||
following its parent SSH account, and try to emulate Ansible's existing
|
||
timeout semantics.
|
||
|
||
* Normally with Ansible, diagnostics and use of the :py:mod:`logging` package
|
||
output on the target machine are discarded. With Mitogen, all of this is
|
||
captured and returned to the host machine, where it can be viewed as desired
|
||
with ``-vvv``.
|
||
|
||
* Ansible with SSH multiplexing enabled causes a string like ``Shared
|
||
connection to host closed`` to appear in ``stderr`` output of every executed
|
||
command. This never manifests with the Mitogen extension.
|
||
|
||
* Local commands are executed in a reuseable Python interpreter created
|
||
identically to interpreters used on remote hosts. At present only one such
|
||
interpreter per ``become_user`` exists, and so only one action may be
|
||
executed in each context simultaneously. Ansible usually permits up to
|
||
``ansible.cfg:forks`` simultaneous local actions, which may trigger a
|
||
performance regression in some playbooks. This will be fixed in a future
|
||
release.
|
||
|
||
|
||
Demo
|
||
----
|
||
|
||
Local VM connection
|
||
~~~~~~~~~~~~~~~~~~~
|
||
|
||
This demonstrates Mitogen vs. connection pipelining to a local VM, executing
|
||
the 100 simple repeated steps of ``run_hostname_100_times.yml`` from the
|
||
examples directory. Mitogen requires **43x less bandwidth and 4.25x less
|
||
time**.
|
||
|
||
.. image:: images/ansible/run_hostname_100_times.png
|
||
|
||
|
||
Kathmandu to Paris
|
||
~~~~~~~~~~~~~~~~~~
|
||
|
||
This is a full Django application playbook over a ~180ms link between Kathmandu
|
||
and Paris. Aside from large pauses where the host performs useful work, the
|
||
high latency of this link means Mitogen only manages a 1.7x speedup.
|
||
|
||
Many early roundtrips are due to inefficiencies in Mitogen's importer that will
|
||
be fixed over time, however the majority, comprising at least 10 seconds, are
|
||
due to idling while the host's previous result and next command are in-flight
|
||
on the network.
|
||
|
||
The initial extension lays groundwork for exciting structural changes to the
|
||
execution model: a future version will tackle latency head-on by delegating
|
||
some control flow to the target host, melding the performance and scalability
|
||
benefits of pull-based operation with the management simplicity of push-based
|
||
operation.
|
||
|
||
.. image:: images/ansible/costapp.png
|
||
|
||
|
||
SSH Variables
|
||
-------------
|
||
|
||
Matching Ansible's existing model, these variables are treated on a per-task
|
||
basis, causing establishment of additional reuseable interpreters as necessary
|
||
to match the configuration of each task.
|
||
|
||
This list will grow as more missing pieces are discovered.
|
||
|
||
* ``ansible_ssh_timeout``
|
||
* ``ansible_host``, ``ansible_ssh_host``
|
||
* ``ansible_user``, ``ansible_ssh_user``
|
||
* ``ansible_port``, ``ssh_port``
|
||
* ``ansible_ssh_executable``, ``ssh_executable``
|
||
* ``ansible_ssh_private_key_file``
|
||
* ``ansible_ssh_pass``, ``ansible_password`` (default: assume passwordless)
|
||
* ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args``
|
||
* ``mitogen_ssh_discriminator``: if present, a string mixed into the key used
|
||
to deduplicate connections. This permits intentional duplicate Mitogen
|
||
connections to a single host, which is probably only useful for testing.
|
||
|
||
|
||
Sudo Variables
|
||
--------------
|
||
|
||
* ``ansible_python_interpreter``
|
||
* ``ansible_sudo_exe``, ``ansible_become_exe``
|
||
* ``ansible_sudo_user``, ``ansible_become_user`` (default: ``root``)
|
||
* ``ansible_sudo_pass``, ``ansible_become_pass`` (default: assume passwordless)
|
||
* ``sudo_flags``, ``become_flags``
|
||
* ansible.cfg: ``timeout``
|
||
|
||
|
||
Docker Variables
|
||
----------------
|
||
|
||
Note: Docker support is only intended for developer testing, it might disappear
|
||
entirely prior to a stable release.
|
||
|
||
* ansible_host
|
||
|
||
|
||
Chat on IRC
|
||
-----------
|
||
|
||
Some users and developers hang out on the
|
||
`#mitogen <https://webchat.freenode.net/?channels=mitogen>`_ channel on the
|
||
FreeNode IRC network.
|
||
|
||
|
||
Debugging
|
||
---------
|
||
|
||
Mitogen's logs are integrated into Ansible's display framework. Basic high
|
||
level debug logs are produced with ``-vvv``, with logging of all IO activity on
|
||
the controller machine when ``-vvvv`` or higher is specified.
|
||
|
||
Although any use of standard IO and the logging package on remote machines is
|
||
forwarded to the controller machine, it is not possible to receive logs of all
|
||
IO activity, as the processs of receiving those logs would would in turn
|
||
generate more IO activity. To receive a complete trace of every process on
|
||
every machine, file-based logging is required. File-based logging can be
|
||
enabled by setting ``MITOGEN_ROUTER_DEBUG=1`` in your environment.
|
||
|
||
When file-based logging is enabled, one file per context will be created on the
|
||
local machine and every target machine, as ``/tmp/mitogen.<pid>.log``.
|
||
|
||
|
||
Implementation Notes
|
||
--------------------
|
||
|
||
Interpreter Reuse
|
||
~~~~~~~~~~~~~~~~~
|
||
|
||
The extension aggressively reuses the single target Python interpreter to
|
||
execute every module. While this works well, it violates an unwritten
|
||
assumption regarding Ansible modules, and so it is possible a buggy module
|
||
could cause a run to fail, or for unrelated modules to interact with each other
|
||
due to bad hygiene. Mitigations (such as forking) will be added as necessary if
|
||
problems of this sort ever actually manfest.
|
||
|
||
|
||
Runtime Patches
|
||
~~~~~~~~~~~~~~~
|
||
|
||
Three small runtime patches are employed in ``strategy.py`` to hook into
|
||
desirable locations, in order to override uses of shell, the module executor,
|
||
and the mechanism for selecting a connection plug-in. While it is hoped the
|
||
patches can be avoided in future, for interesting versions of Ansible deployed
|
||
today this simply is not possible, and so they continue to be required.
|
||
|
||
The patches are concise and behave conservatively, including by disabling
|
||
themselves when non-Mitogen connections are in use. Additional third party
|
||
plug-ins are unlikely to attempt similar patches, so the risk to an established
|
||
configuration should be minimal.
|
||
|
||
|
||
Flag Emulation
|
||
~~~~~~~~~~~~~~
|
||
|
||
Mitogen re-parses ``sudo_flags``, ``become_flags``, and ``ssh_flags`` using
|
||
option parsers extracted from `sudo(1)` and `ssh(1)` in order to emulate their
|
||
equivalent semantics. This allows:
|
||
|
||
* robust support for common ``ansible.cfg`` tricks without reconfiguration,
|
||
such as forwarding SSH agents across ``sudo`` invocations,
|
||
* reporting on conflicting flag combinations,
|
||
* reporting on unsupported flag combinations,
|
||
* internally special-casing certain behaviour (like recursive agent forwarding)
|
||
without boring the user with the details,
|
||
* avoiding opening the extension up to untestable scenarios where users can
|
||
insert arbitrary garbage between Mitogen and the components it integrates
|
||
with,
|
||
* precise emulation by an alternative implementation, for example if Mitogen
|
||
grew support for Paramiko.
|
||
|