Given:
- Broker asleep in poll()
- thread B calling Latch.put()
Previously,
- B takes lock,
- B wakes socket by dropping GIL and writing to it
- Broker wakes from poll(), acquires GIL only to find Latch._lock is held
- Broker drops GIL, sleeps on futex() for _lock
- B wakes, acquires GIL, releases _lock
- Broker wakes from futex(), acquires lock
Now,
- B takes lock, updates state, releases lock
- B wakes socket by droppping GIL and writing to it
- Broker wakes from poll(), acquires GIL and _lock
- Everyone lives happily ever after.
Given:
- thread A asleep in Latch._get_sleep()
- thread B calling Latch.put()
Previously,
- B takes lock,
- B wakes socket by dropping GIL and writing to it
- A wakes from poll(), acquires GIL only to find Latch._lock is held
- A drops GIL, sleeps on futex() for _lock
- B wakes, acquires GIL, releases _lock
- A wakes from futex(), acquires lock
Now,
- B takes lock, updates state, releases lock
- B wakes socket by droppping GIL and writing to it
- A wakes from poll(), acquires GIL and _lock
- Everyone lives happily ever after.
Move all details of broker/router setup out of connection.py, instead
deferring it to a WorkerModel class exported by process.py via
get_worker_model(). The running strategy can override the configured
worker model via _get_worker_model().
ClassicWorkerModel is installed by default, which implements the
extension's existing process model.
Add optional support for the third party setproctitle module, so
children have pretty names in ps output.
Add optional support for per-CPU multiplexers to classic runs.
Now it's possible for stream.protocol to not refer to MitogenProtocol,
move the signal handler to a MitogenProtocol subclass instead.
Fixes a crash where CTRL+C during child bootstrap would print
AttributeError.
* origin/stream-refactor:
[stream-refactor] Py3.x test fixes
[stream-refactor] mark py24 as allow-fail
[stream-refactor] Debian Docker container image initctl
[stream-refactor] replace cutpaste with Stream.accept() in mitogen.unix
[stream-refactor] fix flake8 errors
[stream-refactor] fix testlib assertion format string
[stream-refactor] make mitogen-fuse work on Linux
[stream-refactor] repair preamble_size.py again
[stream-refactor] don't abort Connection until all buffers are empty
Normalize docstring formatting
[stream-refactor] fix LogHandler.uncork() race
[stream-refactor] BufferedWriter must disconenct Stream, not Protocol
[stream-refactor] statically link doas binary using musl
[stream-refactor] stop writing to /tmp/foo in fd_check.py.
[stream-refactor] yet another 2.4 issue in create_child_test
[stream-refactor] fix Py2.4 failure by implementing missing Timer method
[stream-refactor] allow up to 30 seconds to connect in unix_test
[stream-refactor] mark setns module as requiring Python >2.4
[stream-refactor] another 2.4 fix for create_child_test
.travis.yml: Add reverse shell spawn for Travis too
core: better Side attribute docstrings
[stream-refactor] remove one more getuser() usage
[stream-refactor] allow doas_test to succeed on CentOS
Pin idna==2.7 when running on Python<2.7.
[stream-refactor] Py2.4 compat fix for iter_split_test.
[stream-refactor] add descriptive task names to _container_prep
[stream-refactor] 3.x socket.send() requires bytes
[stream-refactor] fix 2.4 syntax error.
[stream-refactor] avoid os.wait3() for Py2.4.
Allow specifying -vvv to debops_tests.
[stream-refactor] send MITO002 earlier
module_finder: pass raw file to compile()
[stream-refactor] merge stdout+stderr when reporting EofError
[stream-refactor] fix crash in detach() / during async/multiple_items_loop.yml
[stream-refactor] fix crash in runner/forking_active.yml
[stream-refactor] replace old detach_popen() reference
ansible: fixturize creation of MuxProcess
unix: ensure mitogen.context_id is reset when client disconnects
[stream-refactor] make syntax 2.4 compatible
[stream-refactor] make trusty our Travis dist.
[stream-refactor] fix su_test failure (issue #363)
[stream-refactor] more readable log string format
[stream-refactor] dont doubly log last partial line
[stream-refactor] import fd_check.py used by create_child_test
[stream-refactor] port mitogen.buildah, added to master since work began
[stream-refactor] fix unix.Listener construction
[stream-refactor] fix crash when no stderr present.
[stream-refactor] fix Process constructor invocation
Add tests/ansible/.*.pid to gitignore (for ansible_mitogen/process.py)
Add extra/ to gitignore
import release-notes script.
[stream-refactor] repaired rest of create_child_test.
[stream-refactor] rename Process attrs, fix up more create_child_test
[stream-refactor] import incomplete create_child_test
issue #482: tests: check for zombie process after test.
issue #363: add test.
tests: clean up old-style SSH exception catch
issue #271: add mitogen__permdenied user to Docker image.
ssh: fix issue #271 regression due to refactor, add test.
Refactor Stream, introduce quasi-asynchronous connect, much more
core: teach iter_split() to break on callback returning False.
issue #507: log fatal errors to syslog.
testlib: have LogCapturer.raw() return unicode on 2.x.
core/master: docstring, repr, and debug log message cleanups
parent: remove unused Timer parameter.
tests: jail_test fixes.
parent: docstring improvements, cfmakeraw() regression.
core: introduce Protocol, DelimitedProtocol and BufferedWriter.
core: introduce mitogen.core.pipe()
tests/bench: import ssh-roundtrip.py.
tests: note location of related tests.
tests: add real test for doas.
tests: install OpenBSD doas port in Debian image.
tests: add setns_test that works if password localhost sudo works.
Import minimal jail_test.
core: move message encoding to Message.pack(), add+refactor tests.
master: expect forwarded logs to be in UTF-8.
tests: add some UTF-8 to ssh_login_banner to encourage breakage.
core: bootstrap FD management improvements
core: pending timers should keep broker alive.
core: more succinct iter_split().
core: replace UTF8_CODEC with encodings.utf_8.encode() function.
docs: remove bytearray from supported types list.
core: docstring style cleanups, dead code.
testlib: disable lsof warnings due to Docker crap
parent: discard cancelled events in TimerList.get_timeout().
core: split out iter_split() for use in parent.py.
parent: various style cleanups, remove unused function.
issue #170: add TimerList docstrings.
core: eliminate some quadratric behaviour from IoLogger
issue #170: update Changelog; closes#170.
issue #170: add timers to internals.rst.
issue #170: implement timers.
During early initialization under hackbench, it is possible for Broker
to be in LogHandler._send() while the main thread has already destroyed
_buffer. So we must synchronize them, but only while the handler is
corked.
Fix a race where if Stream.on_receive() detects disconnect, it calls
Stream.on_disconnect(), which fires Stream 'disconnect' event, whereas
if BufferedWriter.on_transmit() detects disconnect, it called
Protocol.on_disconnect(), which did not fire the Stream 'disconnect'
event.
Since mitogen.parent listens on Stream's 'disconnect' event to reap
children, this was causing a very difficult to trigger test failure.
Triggered after <1000 runs on a Xeon E5530 with hyperthreading using
hackbench running at the same priority:
$ hackbench -s 1048576 -l 100000000000 -g 4
Unlike on Debian, some environment variables that tickle
getpass.getuser() are being inherited. So use getuid() instead.
Also install the doas binary on CentOS. CI was changed (I believe) to
shrink the configuration matrix, and now these tests run on CentOS too.
This relies on the previous commit resetting global variables.
Update clean_shutdown() to handle duplicate calls, due to tests
repeatedly installing it.
To ensure a test process can successfully recreate an Ansible
MuxProcess, reset fork-inherited globals during disconnection.
There is basically no good place for this. Per the comments on #91, it
would be far better if the context's identity was tied to its router,
rather than some global variable.
David Wilson