Rooba
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,302 Commits 16 Branches 40 Tags 38 MiB
Commit Graph

396 Commits

Author SHA1 Message Date
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 2019-02-09 22:04:19 +00:00
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 2019-02-09 22:04:19 +00:00
David Wilson a2ae4ed696 SyntaxError. 2019-02-09 22:04:19 +00:00
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 2019-02-09 22:04:19 +00:00
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux 
Portable as in does nothing for the time at least for now.
 2019-02-09 22:04:18 +00:00
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 2019-02-09 22:04:18 +00:00
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork. 
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
 2019-02-09 22:04:18 +00:00
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 2019-02-09 22:04:18 +00:00
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration 
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
 2019-02-09 22:04:18 +00:00
David Wilson 1b909e8697 ansible: pin connection multiplexer to a single core 
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.

Before:
    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 4,001,276,805,120      stalled-cycles-frontend   #   72.32% frontend cycles idle     (83.30%)
 2,024,159,442,463      stalled-cycles-backend    #   36.59% backend cycles idle      (66.65%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:
    2208247.938562      task-clock (msec)         #    6.735 CPUs utilized
         8,489,840      context-switches          #    0.004 M/sec
         1,432,967      cpu-migrations            #    0.649 K/sec
         7,508,957      page-faults               #    0.003 M/sec
 5,477,293,750,357      cycles                    #    2.480 GHz                      (83.31%)
 3,984,360,350,811      stalled-cycles-frontend   #   72.74% frontend cycles idle     (83.32%)
 1,976,646,418,711      stalled-cycles-backend    #   36.09% backend cycles idle      (66.64%)
 3,196,197,480,792      instructions              #    0.58  insn per cycle
                                                  #    1.25  stalled cycles per insn  (83.36%)
   648,247,332,967      branches                  #  293.557 M/sec                    (83.35%)
    39,004,881,070      branch-misses             #    6.02% of all branches          (83.37%)

     327.876903668 seconds time elapsed
 2019-02-09 22:04:18 +00:00
David Wilson e587396e70 ansible: hook strategy and worker processes into profiler 2019-02-09 22:04:18 +00:00
David Wilson 84944a9a61 ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk. 
This has been broken for quite some time.
 2019-02-09 22:04:18 +00:00
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 2019-02-09 19:43:35 +00:00
David Wilson a1121c5a84 issue #499: respect C.BECOME_ALLOW_SAME_USER. 2019-01-29 02:53:21 +00:00
David Wilson be6ab52fe1 issue #488: fix shutdown damage caused in 6ca2677de5 
os._exit() subverted calm shutdown, meaning unix.Listener never had a
chance to cleanup its socket.

Move unix.Listener socket cleanup into its class so it is automatic
during shutdown, rather than cutpasted for each consumer.

Disable the watcher thread in the MuxProcess, it is useless.

Add .sock extension to /tmp/mitogen_unix_*, so we can write a test.
 2019-01-27 19:21:50 +00:00
David Wilson 38a553d42d issue #490: prevent double close() destroying unrelated Connection. 2019-01-27 18:46:03 +00:00
David Wilson e7fe95af88 issue #477: fix sudo_args selection. 2019-01-27 05:07:36 +00:00
David Wilson 599da0689a issue #477 / ansible: avoid a race in async job startup. 
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.

As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
 2019-01-27 04:27:41 +00:00
David Wilson 0175052099 issue #477: fix source of become_flags on 2.3. 2019-01-27 03:49:01 +00:00
David Wilson 97f3cfe4f4 issue #477: target.file_exists() wrapper. 
os.path.exists physical module name varies across major Python versions.
 2019-01-27 03:03:47 +00:00
David Wilson 8f5b65f7ec issue #477: introduce subprocess isolation. 
Since Python 2.4 fork is so defective, we must use subprocesses for
mitogen_task_isolation=fork. This has plenty of upside, since the long
term goal is to dump forking altogether. This allows a gentle
introduction of its replacement.
 2019-01-27 03:01:42 +00:00
David Wilson b9924683ac ansible: docstring fixes. 2019-01-27 03:00:46 +00:00
David Wilson 75f53faf8c issue #477: shlex.split() in 2.4 required bytes input. 2019-01-27 02:10:01 +00:00
David Wilson dc1d4251e3 ansible: synchronize module needs '.docker_cmd' attr for Docker plugin. 2019-01-27 01:10:51 +00:00
David Wilson dd30a907ce issue #477: promote setup_gil() to mitogen.utils 
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
 2019-01-26 02:06:33 +00:00
David Wilson bba5d82fc4 issue #477: fix another str/bytes mixup. 2019-01-25 16:44:57 +00:00
David Wilson c9ff4c6506 issue #477: fix 3.x failure in new target.set_file_mode() function. 2019-01-25 16:32:00 +00:00
David Wilson aa5d37af89 issue #477: fix 3.x failure in new target.set_file_mode() function. 2019-01-25 16:25:47 +00:00
David Wilson 84a0424749 issue #477: fix 3.x test regressions. 2019-01-25 16:17:12 +00:00
David Wilson 81f15028a7 issue #477: backport ansible_mitogen.runner to 2.4. 2019-01-24 17:24:56 +00:00
David Wilson 256628c149 issue #477: backport ansible_mitogen/target.py to Python2.4 2019-01-24 17:15:13 +00:00
David Wilson a48ee3a536 issue #477: vendorize the last 2.4-compatible simplejson 
This is in part so image_prep can run against an ancient CentOS 5 image
without any upfront help, and in part simply because it's very easy to
support.
 2019-01-24 17:10:27 +00:00
David Wilson 59dd0dc814 issue #477: serve up junk ansible/__init__.py just like Ansible. 2019-01-24 03:43:09 +00:00
David Wilson f52f7f6a62 ansible: merge away unused return values / functions. 2019-01-22 09:20:40 +00:00
David Wilson b5c14cecda docs: docs and docstrings. 2019-01-22 05:25:10 +00:00
David Wilson e767de3f15 issue #412: force-verbose output for mitogen_get_stack. 2019-01-22 02:26:32 +00:00
David Wilson 23866084d7 issue #412: promote "mitogen_get_stack" to the main extension. 
This is to make it easier for users to diagnose their own problems.
 2019-01-22 01:42:15 +00:00
David Wilson 115c3c5657 issue #412: add docstrings/boilerplate to transport_config.py. 2019-01-22 01:38:58 +00:00
David Wilson 2ad05f1238 issue #251, #412, #434: fix connection configuration brainwrong 
This refactors connection.py to pull the two huge dict-building
functions out into new transport_transport_config.PlayContextSpec and
MitogenViaSpec classes, leaving a lot more room to breath in both files
to figure out exactly how connection configuration should work.

The changes made in 1f21a30 / 3d58832 are updated or completely removed,
the original change was misguided, in a bid to fix connection delegation
taking variables from the wrong place when delegate_to was active.

The Python path no longer defaults to '/usr/bin/python', this does not
appear to be Ansible's normal behaviour. This has changed several times,
so it may have to change again, and it may cause breakage after release.

Connection delegation respects the c.DEFAULT_REMOTE_USER whereas the
previous version simply tried to fetch whatever was in the
'ansible_user' hostvar. Many more connection delegation variables closer
match vanilla's handling, but this still requires more work. Some of the
variables need access to the command line, and upstream are in the
process of changing all that stuff around.
 2019-01-22 00:40:32 +00:00
David Wilson 6ca2677de5 ansible: fix test failure during process exit. 
======================================================================
ERROR: tests.connection_test (unittest2.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
ImportError: Failed to import test module: tests.connection_test
Traceback (most recent call last):
  File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
RuntimeError: not holding the import lock
 2019-01-21 21:31:37 +00:00
David Wilson 6915af502e issue #61: unused import (reported by LGTM) 2019-01-20 19:49:19 +00:00
David Wilson 907c4556c8 issue #61: unused variable (reported by LGTM) 2019-01-20 19:46:16 +00:00
David Wilson f67960848b issue #61: unused import (reported by LGTM) 2019-01-20 19:45:34 +00:00
David Wilson b8ca015b83 issue #61: unused variable (reported by LGTM) 2019-01-20 19:43:56 +00:00
David Wilson bcc7bb7128 issue #61: unused import (reported by LGTM) 2019-01-20 19:42:30 +00:00
David Wilson 4bdf60326c issue #424: ansible: make put_file() raise AnsibleFileNotFound 2019-01-20 19:01:14 +00:00
David Wilson 3f31b166f6 issue #461: Ansible 2.3-compatible _get_candidate_temp_dirs(). 2019-01-20 10:51:20 +00:00
David Wilson 9401c34f64 issue #461: Ansible 2.3 did not have _load_name. 2019-01-20 10:50:38 +00:00
David Wilson 04755c3321 issue #426: tighten up PushFileService types. 
Bytes/Unicode mixing caused a hang, so prevent bytes entirely.
 2019-01-19 05:43:39 +00:00
David Wilson 1e9f3446c8 issue #426: big hack so reset_connection has task_var access 2019-01-19 05:17:39 +00:00