Rooba
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,123 Commits 16 Branches 38 Tags 33 MiB
Commit Graph

286 Commits

Author SHA1 Message Date
David Wilson 1a8ac9f4d1 issue #155: introduce mitogen.fork / Router.fork() 2018-03-24 15:50:28 +05:45
David Wilson 27175e3126 docs: tidy up signals.rst a little. 2018-03-23 16:52:54 +05:45
David Wilson 732a610246 docs: add beginnings of section on func decorators 2018-03-23 09:31:29 +05:45
David Wilson 2abe87472c issue #162: docs: begin documenting mitogen.service 2018-03-23 09:31:11 +05:45
David Wilson 54ff1c90fa issue #155: add DEL_ROUTE, propagate ADD_ROUTE upwards 
* IDs are allocated by the parent responsible for contructing a new
  child, using ALLOCATE_ID to the master as necessary to allocate new ID
  ranges.

* ADD_ROUTE is sent up the tree rather than down. This permits
  construction of the new context to complete concurrent to parent
  contexts learning about its existence. Since all streams are strictly
  ordered, it's not possible for any parent to observe messages from the
  new context prior to arrival of an ADD_ROUTE from the parent notifying
  of its existence.

  If the new context, for example, implements an Ansible async task, its
  parent can start executing that without waiting for any synchronous
  confirmation from any parent or the master.

* Since routes propagate up, it's no longer possible for a plain
  non-parent child to ever receive ADD_ROUTE, so that code can be moved
  out of core.py and into parent.py (-0.2kb compressed).

* Add a .routes attribute to parent.Stream, and respond to disconnection
  signal on the stream by propagating DEL_ROUTE for any ADD_ROUTE ever
  received from that stream.

* Centralize route management in a new parent.RouteMonitor class
 2018-03-22 11:56:24 +05:45
David Wilson aeeeb45ccb docs: farewell, glorious iframe! 2018-03-22 07:13:42 +05:45
David Wilson 1ed86774b5 issue #156: document select exception 2018-03-21 09:23:54 +05:45
David Wilson 20f5d89dfa issue #156: fix several more races 
* Don't need to sleep if queue>sleepers, can just pop the right queue
  element and return it.

* If queue>sleeping and waking==sleeping, no mechanism existed to ensure
  a thread newly added to sleeping would ever be woken. Above change
  fixes that.

* Cannot trust select() return value, scheduler might sleep us
  indefinitely while put() writes a byte.

* Sleeping threads didn't pop FIFO, they popped in whatever order
  scheduler woke them up. Must recover index and use it to pick the pop
  index.
 2018-03-20 14:53:19 +05:45
David Wilson 526b0a514b issue #156: prevent Latch.close() triggering spurious wakeups 2018-03-20 13:14:51 +05:45
David Wilson 18e2977baf docs: annoying phrasing 2018-03-20 13:05:41 +05:45
David Wilson 2c22c41819 issue #156: don't decrement `waking` if we timed out rather than being woken. 2018-03-20 13:02:46 +05:45
David Wilson 07a8994ff5 issue #156: waking thread result dictionary with an integer. 2018-03-20 12:55:55 +05:45
David Wilson b5398bd17f issue #156: docs typo 2018-03-20 09:12:50 +05:45
David Wilson 512ff77a46 issue #156: prevent non-sleeping threads from starving sleeping threads. 
See new docs
 2018-03-20 09:12:45 +05:45
David Wilson 148ce1d703 issue #155: increase context ID width to 32 bits 
Needed to make large range allocations (1000 per ALLOCATE_ID roundtrip)
feasible.
 2018-03-19 21:58:35 +05:45
David Wilson 071d9fbfb3 docs: tidy ansible docs. 2018-03-19 21:58:35 +05:45
David Wilson f241eac5ce parent: allow Python to determine its install prefix from argv[0] 
Fixes support for virtualenv. Closes #152.
 2018-03-19 21:58:35 +05:45
David Wilson dec3af375a issue #144: ansible: increase default pool size to 16. 2018-03-19 21:58:35 +05:45
David Wilson 4f361be7e7 issue #144: teach Select() to close its latch 
Causes all threads sleeping on the select to wake.
 2018-03-19 21:58:35 +05:45
David Wilson 8aada2646c core: support throwing LatchError in every sleeping thread 
This is to allow Select() to be used as a generic queueing primitive
that supports graceful shutdown.
 2018-03-19 21:58:35 +05:45
David Wilson 7a74bb0a39 docs: update ansible risks/differences. 2018-03-19 21:58:35 +05:45
David Wilson bcc15987fc docs: extra ansible paragraph. 2018-03-19 21:58:35 +05:45
David Wilson 7a394dc73e ansible: allow establishment of duplicate SSH connections 2018-03-19 21:58:35 +05:45
David Wilson 4691ce0b95 issue #150: ansible: add basic Docker support. 2018-03-19 21:58:34 +05:45
David Wilson 67ff762ba5 issue #139: docs: remove note about bad buffering 2018-03-19 21:58:34 +05:45
David Wilson d1888f1908 docs: reorder sections 2018-03-19 21:58:34 +05:45
David Wilson 014247ce66 docs: another crazy Ansible success story 2018-03-19 21:58:34 +05:45
Alex Willmer 7063d172e9 tests: Add Tox config for Python 2.6 and 2.7 
I could not get Python 2.5 or earlier to work. Too many packages
(critically docker) don't support it.
 2018-03-19 21:58:34 +05:45
David Wilson 88c198ea05 issue #141: copy Ansible's connect_timeout for sudo too. 2018-03-19 21:58:34 +05:45
David Wilson 63c3fc623c docs: note the semantic difference in Mitogen vs. Ansible timeouts 
Related to issue #141.
 2018-03-19 21:58:34 +05:45
David Wilson 8f85943083 issue #139: mention relating buffering issue 2018-03-19 21:58:34 +05:45
David Wilson 1f1d691a28 docs: update to match @moreati's code golf birdies :) 2018-03-19 21:58:34 +05:45
Alex Willmer 9aa83ef77f docs: First round of Pickle-likes survey 2018-03-19 21:58:33 +05:45
Alex Willmer a1fc21bb06 docs: Maximum size of pencode values 2018-03-19 21:58:33 +05:45
Alex Willmer e24db89f3a docs: Disco comparison 2018-03-19 21:58:33 +05:45
Alex Willmer 04f4851138 docs: multiprocessing comparison 
Not strictly a rival, but has enough commonalities to be worth noting
 2018-03-19 21:58:33 +05:45
Alex Willmer 8c227b2bdd docs: More detail about Baker 2018-03-19 21:58:33 +05:45
Alex Willmer e06e438228 docs: More detail about execnet 2018-03-19 21:58:33 +05:45
Alex Willmer da58f8595d docs: More detail about chopsticks 2018-03-19 21:58:33 +05:45
Alex Willmer d7fbb9aef6 docs: Link compared projects to their website 
All outgoing links checked with

```bash
cd docs
make linkcheck
```
 2018-03-19 21:58:33 +05:45
Alex Willmer 4615ab1a8e docs: Enable sphinx-autobuild 
```bash
cd docs
make
```

to run a webserver that automatically rerenders whenever the rST is
modified.
 2018-03-19 21:58:33 +05:45
David Wilson f8c85b549f docs: typo 2018-03-19 21:58:33 +05:45
David Wilson dc9dda08e1 Add link to IRC; closes #116 2018-03-19 21:58:33 +05:45
David Wilson b2a433dcc4 ssh: CompressionLevel is a v1-only option. 
It's ignored by newer SSHes, which only pay attention to Compression.
 2018-03-19 21:58:33 +05:45
Alex Willmer 1c20c61605 docs: Convert all URLs that support https:// 
Excluded: graphml XML namespaces, links to e.g. Fabric homepage

Fixes #128
 2018-03-19 21:58:33 +05:45
David Wilson 089f39716e docs: beginnings of testimonials section 2018-03-19 21:58:33 +05:45
David Wilson 500c4f419c docs: update limitations list for Ansible. 2018-03-19 21:58:33 +05:45
David Wilson 537a263261 docs: So many typos 2018-03-19 21:58:33 +05:45
David Wilson 6bd4c5ead5 docs: Ansible logging update (#111) 2018-03-19 21:58:33 +05:45
David Wilson 4f352d7d4b Pin Sphinx version 2018-03-19 21:58:32 +05:45
David Wilson 53e51c4af8 docs: update Python 3 support to match reality. 2018-03-19 21:58:32 +05:45
David Wilson ce9a1fef26 docs: fix float 2018-03-19 21:58:32 +05:45
David Wilson 04bb5881b6 ansible: doc updates 2018-03-19 21:58:32 +05:45
David Wilson 9515291024 docs: small fix 2018-03-19 21:58:32 +05:45
David Wilson 3ddbf1a503 ansible: basic support for ssh_args 2018-03-19 21:58:32 +05:45
David Wilson 1b28252ad0 docs: get rid of "medium risk" category 2018-03-19 21:58:32 +05:45
David Wilson da00437f1e ansible: Support ansible_ssh_private_key_file 2018-03-19 21:58:32 +05:45
David Wilson a87b665099 ansible: limited support for become_flags, more docs. 2018-03-19 21:58:32 +05:45
David Wilson 235e1df987 sudo: support parsing sudo flags back out into parameters 2018-03-19 21:58:32 +05:45
David Wilson cd53023768 docs: Fix install docs now Ansible package layout changed. 2018-03-19 21:58:32 +05:45
David Wilson 03e51fdaa7 docs: mitogen.core.Latch docs 2018-03-19 21:58:32 +05:45
David Wilson 712e5dfca1 docs: fix image labels 2018-03-19 21:58:32 +05:45
David Wilson c2793b7102 docs: import but don't link compared.rst 2018-03-19 21:58:32 +05:45
David Wilson 2ecc6f43a4 docs: Import, but do not yet link signals.rst 2018-03-19 21:58:32 +05:45
David Wilson 7cf2edc3a8 ansible: Support many more common playbook variables. 2018-03-19 21:58:32 +05:45
David Wilson ba644e184d docs: Split up limitations list, add warning 2018-03-19 21:58:31 +05:45
David Wilson 3183dd4147 ansible: initial support for async jobs 
Running in a thread to begin with, but this must change.
 2018-03-19 21:58:31 +05:45
David Wilson e913c11e50 docs: new Ansible limitation 2018-03-19 21:58:31 +05:45
David Wilson 860a635500 docs: remove delegate_to limitation 2018-03-19 21:58:31 +05:45
David Wilson dfdb7ab128 docs: more Ansible 2018-03-19 21:58:31 +05:45
David Wilson cf7f8ff490 docs: document one more Ansible limitation 2018-03-19 21:58:31 +05:45
David Wilson 7179ca163b docs: Replace shifty hacker dude with plebey skater dude 
Farewell, shifty hacker dude! May your 0days be many and your
indictments be few
 2018-03-19 21:58:31 +05:45
David Wilson 0c056ddbe2 docs: new Ansible limitation, add new heading 
Some differences are eventually likely to become permanent, because the
existing behaviour is unforgiveable.
 2018-03-19 21:58:31 +05:45
David Wilson 14251539cd docs: Docker typo. 2018-03-19 21:58:31 +05:45
David Wilson 5036a9448f Add Docker method. 2018-03-19 21:58:31 +05:45
David Wilson 77f6b31e5c docs: more images. 2018-03-19 21:58:30 +05:45
David Wilson 56b563161b docs: single-threaded connection is the most important limitation 2018-03-19 21:58:30 +05:45
David Wilson c1bea0dd49 docs: store blog images for fear of laptop crash 
There is only so much trackpad-driven yEd work one man can handle
 2018-03-19 21:58:30 +05:45
David Wilson dd9d3494e0 docs: small ansible.rst updates 2018-03-19 21:58:30 +05:45
David Wilson ce6648da51 docs: mention CPU usage reduction 2018-03-19 21:58:30 +05:45
David Wilson 270db025c0 docs: fix up Ansible install docs. 2018-03-19 21:58:30 +05:45
David Wilson e9060b3b7f docs: more modest and accurate numbers for Ansible 2018-03-19 21:58:30 +05:45
David Wilson b221eaaa10 ansible: log call timings 2018-03-19 21:58:30 +05:45
David Wilson a817f5e9e8 docs: more marketing, add lots of drama bold. 2018-03-19 21:58:30 +05:45
David Wilson 5933723d70 docs: a little more marketing 2018-03-19 21:58:30 +05:45
David Wilson de1b3602e5 docs: tidy up ansible.rst 2018-03-19 21:58:30 +05:45
David Wilson 52da0d882e ansible: add a new limitation 2018-03-19 21:58:30 +05:45
David Wilson 14b5c4d504 docs: remove fixed Ansible limitation 
The solution was that Mitogen's loader should emulate the behaviour of
ansible.executor.module_common, which restricts dependency scanning to
the ansible.module_utils namespace.
 2018-03-19 21:58:30 +05:45
David Wilson 91116810a1 ansible: delete utils.py and promote cast() to mitogen.utils 2018-03-19 21:58:30 +05:45
David Wilson 18eaf14dca ansible: migrate logging variables into utils. 2018-03-19 21:58:30 +05:45
David Wilson 10c2de2845 examples: rename playbooks for clarity. 2018-03-19 21:58:30 +05:45
David Wilson ea126bb7e8 docs: add stats tracking 2018-03-19 21:58:30 +05:45
David Wilson 875ee29d4f docs: initial Ansible extension docs. 2018-03-19 21:58:29 +05:45
David Wilson bdd8648bae ssh: enable compression by default 
Using the same test as in 7af97c0365,
transmitted wire bytes drops from 135,531 to 133,071 (-1.81%), while
received drops from 21,073 to 14,775 (-30%).

Combined, both changes shave 13,914 bytes (-8.6%) off aggregate
bandwidth usage.

Make it configurable as compression hurts in some scenarios.
 2018-03-19 21:58:29 +05:45
David Wilson 4d940f08ae importer: drop redundant prefix from pkg_present 
For the 52 submodules of ansible.modules.system, this produced a 1602
byte pkg_present list. After stripping it becomes 406 bytes, and the
entire LOAD_MODULE size drops from 1988 bytes to 792 bytes (-60%).

For the 68 submodules of ansible.module_utils, 1902 bytes pkg_present
becomes 474 bytes (-75%), and LOAD_MODULE size drops from 2867 bytes to
1439 bytes (-49%).

In a simple test running Ansible's "setup" module followed by its "apt"
module, wire bytes sent drops from 140,357 to 135,531 (-3.4%).
 2018-03-19 21:58:29 +05:45
David Wilson 71a6b9e32a master: Select.all() sugar 2018-03-19 21:58:29 +05:45
David Wilson ac4a42a43e docs: fix Select() example for new style API. 2018-03-19 21:58:29 +05:45
Alex Willmer a69d9650ad Fix indentation of Router.route anchor 2018-03-19 21:58:29 +05:45
Alex Willmer eb1ad66469 docs: Fix out of date get_data() snippets 
As of adc8fe3aed Receiver objects do not
have a get_data() method and Receiver.get() does not unpickle the
message.
 2018-03-19 21:58:29 +05:45
David Wilson 20afa5b90c Latch v2: combined queue + one self-pipe-per-thread 
Turns out it is far too easy to burn through available file descriptors,
so try something else: self-pipes are per thread, and only temporarily
associated with a Lack that wishes to sleep.

Reduce pointless locking by giving Latch its own queue, and removing
Queue.Queue() use in some places.

Temporarily undo merging of of Waker and Latch, let's do this one step
at a time.
 2018-03-19 21:58:29 +05:45
David Wilson e63e9d299e docs: add Message documentation 2018-03-19 21:58:28 +05:45
David Wilson dd088908df select: clean up API. 2018-03-19 21:58:28 +05:45
David Wilson a39cd44bf2 core: add auth_id field. 2018-03-19 21:58:28 +05:45
David Wilson 07d4d799f1 Add mitogen.main() decorator mainly for docs and demo use. 2018-03-19 21:58:28 +05:45
David Wilson 09eb3fd9ba utils: support log_to_file(usec=True) 2018-03-19 21:58:28 +05:45
David Wilson 592ebd59c2 docs: reorder sections 2018-03-19 21:58:28 +05:45
David Wilson dee856f6f4 docs: remove obsolete warning 2018-03-19 21:58:28 +05:45
David Wilson b941bce964 docs: sphinx-autobuild default target. 2018-03-19 21:58:28 +05:45
David Wilson 4d9d21c808 docs: fix typo 2018-03-19 21:58:28 +05:45
David Wilson 4a6d55ced6 docs: vastly simplify importer concurrency docs 2018-03-19 21:58:28 +05:45
David Wilson 984b39180e importer: Beginnings of howitworks section. 2018-03-19 21:58:28 +05:45
David Wilson b580ada89f Tidy up and correct the new example 2018-03-19 21:58:28 +05:45
David Wilson eb2aef866b Add shell replacement example. 2018-03-19 21:58:28 +05:45
David Wilson 641498e7f6 docs: Fix function name. 2018-03-19 21:58:28 +05:45
David Wilson f1f36cec35 Simplify the API, make Broker optional and auto-shutdown on main thread exit. 2018-03-19 21:58:28 +05:45
Alex Willmer 3831ac360f Replace all calls to file() with open() 
Although these are synonyms in Python 2.x, when using MyPy to typecheck
code use of file() causes spurious errors.

This commit also serves as one small step to Python 3.x compatibility,
since 3.x removes the file() builtin.
 2018-03-19 21:58:27 +05:45
David Wilson 038ab04908 docs: convert paragraph to footnote. 2018-03-19 21:58:27 +05:45
David Wilson e3d2c8d649 issue #49: update howitworks.rst for command line change 2018-03-19 21:58:27 +05:45
David Wilson 58102c8179 docs: waiting on multiple calls 2018-03-19 21:58:27 +05:45
David Wilson b1ad04330b docs: move Router.route() into Sphinx. 2018-03-19 21:58:27 +05:45
David Wilson cf0668b2b1 docs: Add warning to preloading section. 2018-03-19 21:58:27 +05:45
David Wilson fb759f7c16 docs: move Broker docstrings into Sphinx. 2018-03-19 21:58:27 +05:45
David Wilson ac7cada323 docs: more getting started guide 2018-03-19 21:58:27 +05:45
David Wilson ffa063cc01 docs: annother barriage of cross-reference fixes. 2018-03-19 21:58:25 +05:45
David Wilson 9372d2c3de docs: Fix up tons of references, document trust chain 2018-03-19 21:35:38 +05:45
David Wilson 051285437f importer: module preloading docs 2018-03-19 21:35:38 +05:45
David Wilson b3bd075d37 docs: getting started 2018-03-19 21:35:38 +05:45
David Wilson 792e789908 More. 2018-03-19 21:35:37 +05:45
David Wilson c15d227da4 docs: way more appropriate image 2018-03-19 21:35:37 +05:45
David Wilson b94e7b3678 docs: more getting started 2018-03-19 21:35:37 +05:45
David Wilson 709b38c36e docs: Getting started tidyups 2018-03-19 21:35:37 +05:45
David Wilson dcc45bc7de docs: Get started on Getting Started 2018-03-19 21:35:37 +05:45
David Wilson 0299f86bc9 docs: fix reference to with: 2018-03-19 21:35:37 +05:45
David Wilson c9daa2ff30 docs: move fakessh docs into Sphinx. 2018-03-19 21:35:37 +05:45
David Wilson 6a10ab605e docs: move mitogen.utils docs to Sphinx. 2018-03-19 21:35:37 +05:45
David Wilson 270b242ec5 docs: fix decorator types. 2018-03-19 21:35:37 +05:45
David Wilson ec66152e37 docs: better io_op doc, move Side docs to Sphinx. 2018-03-19 21:35:37 +05:45
David Wilson 0767abf26f docs: move BasicStream docs into Sphinx. 2018-03-19 21:35:37 +05:45
David Wilson b7a9aa46cf core: More robust shutdown 
Now there is a separate SHUTDOWN message that relies only on being
received by the broker thread, the main thread can be hung horribly and
the process will still eventually receive a SIGTERM.
 2018-03-19 21:35:37 +05:45
David Wilson 79dd00db5a master: hack to avoid executing __main__. 2018-03-19 21:35:37 +05:45
David Wilson 416ab90e02 docs: Fix index generation everywhere. 2018-03-19 21:35:37 +05:45
David Wilson 976c643f21 docs: remove note, unpickling can no longer trigger module loads 2018-03-19 21:35:37 +05:45
David Wilson decc9a900c docs: s/random/pseudorandom/ 2018-03-19 21:35:37 +05:45
David Wilson 83f8f1863e docs: fix pickler docs, begin relabelling master/slave->parent/child 2018-03-19 21:35:37 +05:45
David Wilson 4327baabfa docs: remove final references to call_with_deadline(). 2018-03-19 21:35:37 +05:45
David Wilson dd69b8feeb docs: with_context element is replaced by a decorator. 2018-03-19 21:35:37 +05:45
David Wilson 12fe884f1a Reference Select 2018-03-19 21:35:37 +05:45
David Wilson a81f804e92 Accidentally hecked in incomplete paragraph. 2018-03-19 21:35:37 +05:45
David Wilson 83e3e1b82a Typo. 2018-03-19 21:35:37 +05:45
David Wilson e9af841e60 docs: fix :return: syntax. 2018-03-19 21:35:36 +05:45