Rooba
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ci: Reduce number of Jobs by parameterizing Mitogen Docker SSH tests 

This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.

Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases

refs #1058, #1059
This commit is contained in:
Alex Willmer 2024-10-10 18:33:44 +01:00
parent 9859e44ee8
commit 28e08ef94c
13 changed files with 101 additions and 138 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
.ci/README.md
View File

@ -28,14 +28,15 @@ for doing `setup.py install` while pulling a Docker container, for example.
### Environment Variables
* `DISTRO`: the `mitogen_` tests need a target Docker container distro. This
name comes from the Docker Hub `mitogen` user, i.e. `mitogen/$DISTRO-test`
* `DISTROS`: the `ansible_` tests can run against multiple targets
simultaneously, which speeds things up. This is a space-separated list of
DISTRO names, but additionally, supports:
* `MITOGEN_TEST_DISTRO_SPECS`: a space delimited list of distro specs to run
the tests against. (e.g. `centos6 ubuntu2004-py3*4`). Each spec determines
the Linux distribution, target Python interepreter & number of instances.
Only distributions with a pre-built Linux container image can be used.
* `debian-py3`: when generating Ansible inventory file, set
`ansible_python_interpreter` to `python3`, i.e. run a test where the
target interpreter is Python 3.
* `debian*16`: generate 16 Docker containers running Debian. Also works
with -py3.
* `MITOGEN_TEST_IMAGE_TEMPLATE`: specifies the Linux container image name,
and hence the container registry used for test targets.

2
.ci/ansible_tests.py
View File

@ -35,7 +35,7 @@ ci_lib.check_stray_processes(interesting)
with ci_lib.Fold('docker_setup'):
containers = ci_lib.container_specs(ci_lib.DISTROS)
containers = ci_lib.container_specs(ci_lib.DISTRO_SPECS.split())
ci_lib.start_containers(containers)

8
.ci/ci_lib.py
View File

@ -28,6 +28,10 @@ os.chdir(
)
DISTRO_SPECS = os.environ.get(
'MITOGEN_TEST_DISTRO_SPECS',
'centos6 centos8 debian9 debian11 ubuntu1604 ubuntu2004',
)
IMAGE_TEMPLATE = os.environ.get(
'MITOGEN_TEST_IMAGE_TEMPLATE',
'public.ecr.aws/n5z0e8q9/%(distro)s-test',
@ -196,10 +200,6 @@ class Fold(object):
GIT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
# Used only when MODE=mitogen
DISTRO = os.environ.get('DISTRO', 'debian9')
# Used only when MODE=ansible
DISTROS = os.environ.get('DISTROS', 'centos6 centos8 debian9 debian11 ubuntu1604 ubuntu2004').split()
TMP = TempDir().path

2
.ci/mitogen_py24_tests.py
View File

@ -8,8 +8,6 @@ import ci_lib
os.environ.update({
'NOCOVERAGE': '1',
'UNIT2': '/usr/local/python2.4.6/bin/unit2',
'MITOGEN_TEST_DISTRO': ci_lib.DISTRO,
'MITOGEN_LOG_LEVEL': 'debug',
'SKIP_ANSIBLE': '1',
})

1
.ci/mitogen_tests.py
View File

@ -6,7 +6,6 @@ import os
import ci_lib
os.environ.update({
'MITOGEN_TEST_DISTRO': ci_lib.DISTRO,
'MITOGEN_LOG_LEVEL': 'debug',
'SKIP_ANSIBLE': '1',
})

78
.github/workflows/tests.yml vendored
View File

@ -67,80 +67,14 @@ jobs:
python_version: '3.13'
tox_env: py313-mode_ansible-ansible10-strategy_linear
- name: Mito_27_centos6
tox_env: py27-mode_mitogen-distro_centos6
- name: Mito_27_centos7
tox_env: py27-mode_mitogen-distro_centos7
- name: Mito_27_centos8
tox_env: py27-mode_mitogen-distro_centos8
- name: Mito_27_debian9
tox_env: py27-mode_mitogen-distro_debian9
- name: Mito_27_debian10
tox_env: py27-mode_mitogen-distro_debian10
- name: Mito_27_debian11
tox_env: py27-mode_mitogen-distro_debian11
- name: Mito_27_ubuntu1604
tox_env: py27-mode_mitogen-distro_ubuntu1604
- name: Mito_27_ubuntu1804
tox_env: py27-mode_mitogen-distro_ubuntu1804
- name: Mito_27_ubuntu2004
tox_env: py27-mode_mitogen-distro_ubuntu2004
- name: Mito_36_centos6
- name: Mito_27
tox_env: py27-mode_mitogen
- name: Mito_36
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_centos6
- name: Mito_36_centos7
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_centos7
- name: Mito_36_centos8
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_centos8
- name: Mito_36_debian9
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_debian9
- name: Mito_36_debian10
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_debian10
- name: Mito_36_debian11
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_debian11
- name: Mito_36_ubuntu1604
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_ubuntu1604
- name: Mito_36_ubuntu1804
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_ubuntu1804
- name: Mito_36_ubuntu2004
python_version: '3.6'
tox_env: py36-mode_mitogen-distro_ubuntu2004
- name: Mito_313_centos6
tox_env: py36-mode_mitogen
- name: Mito_313
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_centos6
- name: Mito_313_centos7
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_centos7
- name: Mito_313_centos8
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_centos8
- name: Mito_313_debian9
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_debian9
- name: Mito_313_debian10
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_debian10
- name: Mito_313_debian11
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_debian11
- name: Mito_313_ubuntu1604
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_ubuntu1604
- name: Mito_313_ubuntu1804
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_ubuntu1804
- name: Mito_313_ubuntu2004
python_version: '3.13'
tox_env: py313-mode_mitogen-distro_ubuntu2004
tox_env: py313-mode_mitogen
steps:
- uses: actions/checkout@v4

2
docs/changelog.rst
View File

@ -21,6 +21,8 @@ To avail of fixes in an unreleased version, please download a ZIP file
In progress (unreleased)
------------------------
* :gh:issue:`1159` CI: Reduce number of Jobs by parameterizing Mitogen Docker
SSH tests
v0.3.13 (2024-10-09)

16
tests/README.md
View File

@ -30,11 +30,19 @@ and run the tests there.
1. Run ``test``
# Selecting a target distribution
# Selecting target distributions
Docker target images exist for testing against CentOS and Debian, with the
default being Debian. To select CentOS, specify `MITOGEN_TEST_DISTRO=centos` in
the environment.
Linux container images for testing are available at
- https://github.com/orgs/mitogen-hq/packages
- https://public.ecr.aws/n5z0e8q9
The images used are determined by two environment variables
- `MITOGEN_TEST_DISTRO_SPECS`
- `MITOGEN_TEST_IMAGE_TEMPLATE`
Defaults for these can be found in `.ci/ci_lib.py` & `tests/testlib.py`
# User Accounts

3
tests/fakessh_test.py
View File

@ -7,8 +7,8 @@ import mitogen.fakessh
import testlib
@unittest.skip('broken')
class RsyncTest(testlib.DockerMixin, testlib.TestCase):
@unittest.skip('broken')
def test_rsync_from_master(self):
context = self.docker_ssh_any()
@ -24,7 +24,6 @@ class RsyncTest(testlib.DockerMixin, testlib.TestCase):
self.assertTrue(context.call(os.path.exists, '/tmp/data'))
self.assertTrue(context.call(os.path.exists, '/tmp/data/simple_pkg/a.py'))
@unittest.skip('broken')
def test_rsync_between_direct_children(self):
# master -> SSH -> mitogen__has_sudo_pubkey -> rsync(.ssh) -> master ->
# mitogen__has_sudo -> rsync

26
tests/ssh_test.py
View File

@ -37,7 +37,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
self.assertEqual(3, context.call(plain_old_module.add, 1, 2))
class SshTest(testlib.DockerMixin, testlib.TestCase):
class SshMixin(testlib.DockerMixin):
def test_debug_decoding(self):
# ensure filter_debug_logs() decodes the logged string.
capture = testlib.LogCapturer()
@ -176,7 +176,18 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
fp.close()
class BannerTest(testlib.DockerMixin, testlib.TestCase):
for distro_spec in testlib.DISTRO_SPECS.split():
dockerized_ssh = testlib.DockerizedSshDaemon(distro_spec)
klass_name = 'SshTest%s' % (dockerized_ssh.distro.capitalize(),)
klass = type(
klass_name,
(SshMixin, testlib.TestCase),
{'dockerized_ssh': dockerized_ssh},
)
globals()[klass_name] = klass
class BannerMixin(testlib.DockerMixin):
# Verify the ability to disambiguate random spam appearing in the SSHd's
# login banner from a legitimate password prompt.
def test_verbose_enabled(self):
@ -193,6 +204,17 @@ class BannerTest(testlib.DockerMixin, testlib.TestCase):
context.shutdown(wait=True)
for distro_spec in testlib.DISTRO_SPECS.split():
dockerized_ssh = testlib.DockerizedSshDaemon(distro_spec)
klass_name = 'BannerTest%s' % (dockerized_ssh.distro.capitalize(),)
klass = type(
klass_name,
(BannerMixin, testlib.TestCase),
{'dockerized_ssh': dockerized_ssh},
)
globals()[klass_name] = klass
class StubPermissionDeniedTest(StubSshMixin, testlib.TestCase):
def test_classic_prompt(self):
self.assertRaises(mitogen.ssh.PasswordError,

13
tests/su_test.py
View File

@ -23,7 +23,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
self.assertEqual(argv[2], '-c')
class SuTest(testlib.DockerMixin, testlib.TestCase):
class SuMixin(testlib.DockerMixin):
stub_su_path = testlib.data_path('stubs/stub-su.py')
def test_slow_auth_failure(self):
@ -64,3 +64,14 @@ class SuTest(testlib.DockerMixin, testlib.TestCase):
)
context = self.router.su(via=ssh, password='rootpassword')
self.assertEqual(0, context.call(os.getuid))
for distro_spec in testlib.DISTRO_SPECS.split():
dockerized_ssh = testlib.DockerizedSshDaemon(distro_spec)
klass_name = 'SuTest%s' % (dockerized_ssh.distro.capitalize(),)
klass = type(
klass_name,
(SuMixin, testlib.TestCase),
{'dockerized_ssh': dockerized_ssh},
)
globals()[klass_name] = klass

25
tests/testlib.py
View File

@ -51,7 +51,10 @@ except NameError:
LOG = logging.getLogger(__name__)
DISTRO = os.environ.get('MITOGEN_TEST_DISTRO', 'debian9')
DISTRO_SPECS = os.environ.get(
'MITOGEN_TEST_DISTRO_SPECS',
'centos6 centos8 debian9 debian11 ubuntu1604 ubuntu2004',
)
IMAGE_TEMPLATE = os.environ.get(
'MITOGEN_TEST_IMAGE_TEMPLATE',
'public.ecr.aws/n5z0e8q9/%(distro)s-test',
@ -555,8 +558,9 @@ class DockerizedSshDaemon(object):
self.image,
]
subprocess.check_output(args)
self.port = self.get_port(self.container_name)
def __init__(self, distro=DISTRO, image_template=IMAGE_TEMPLATE):
def __init__(self, distro_spec, image_template=IMAGE_TEMPLATE):
# Code duplicated in ci_lib.py, both should be updated together
distro_pattern = re.compile(r'''
(?P<distro>(?P<family>[a-z]+)[0-9]+)
@ -565,7 +569,10 @@ class DockerizedSshDaemon(object):
''',
re.VERBOSE,
)
d = distro_pattern.match(distro).groupdict(default=None)
d = distro_pattern.match(distro_spec).groupdict(default=None)
self.distro = d['distro']
self.family = d['family']
if d.pop('py') == 'py3':
self.python_path = '/usr/bin/python3'
@ -573,9 +580,7 @@ class DockerizedSshDaemon(object):
self.python_path = '/usr/bin/python'
self.image = image_template % d
self.start_container()
self.host = get_docker_host()
self.port = self.get_port(self.container_name)
def wait_for_sshd(self):
wait_for_port(self.host, self.port, pattern='OpenSSH')
@ -648,12 +653,10 @@ class DockerMixin(RouterMixin):
if os.environ.get('SKIP_DOCKER_TESTS'):
raise unittest.SkipTest('SKIP_DOCKER_TESTS is set')
# we want to be able to override test distro for some tests that need a different container spun up
daemon_args = {}
if hasattr(cls, 'mitogen_test_distro'):
daemon_args['mitogen_test_distro'] = cls.mitogen_test_distro
cls.dockerized_ssh = DockerizedSshDaemon(**daemon_args)
# cls.dockerized_ssh is injected by dynamically generating TestCase
# subclasses.
# TODO Bite the bullet, switch to e.g. pytest
cls.dockerized_ssh.start_container()
cls.dockerized_ssh.wait_for_sshd()
@classmethod

52
tox.ini
View File

@ -56,9 +56,7 @@ envlist =
py{27,36}-mode_ansible-ansible{2.10,3,4},
py{311}-mode_ansible-ansible{2.10,3,4,5},
py{313}-mode_ansible-ansible{6,7,8,9,10},
py{27,36,313}-mode_mitogen-distro_centos{6,7,8},
py{27,36,313}-mode_mitogen-distro_debian{9,10,11},
py{27,36,313}-mode_mitogen-distro_ubuntu{1604,1804,2004},
py{27,36,313}-mode_mitogen,
report,
[testenv]
@ -105,39 +103,27 @@ setenv =
NOCOVERAGE_ERASE = 1
NOCOVERAGE_REPORT = 1
PIP_CONSTRAINT={toxinidir}/tests/constraints.txt
# Only applicable to MODE=mitogen
distro_centos5: DISTRO=centos5
distro_centos6: DISTRO=centos6
distro_centos7: DISTRO=centos7
distro_centos8: DISTRO=centos8
distro_debian9: DISTRO=debian9
distro_debian10: DISTRO=debian10
distro_debian11: DISTRO=debian11
distro_ubuntu1604: DISTRO=ubuntu1604
distro_ubuntu1804: DISTRO=ubuntu1804
distro_ubuntu2004: DISTRO=ubuntu2004
# Note the plural, only applicable to MODE=ansible
# Ansible 6 - 8 (ansible-core 2.13 - 2.15) require Python 2.7 or >= 3.5 on targets
ansible6: DISTROS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
ansible7: DISTROS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
ansible8: DISTROS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
ansible6: MITOGEN_TEST_DISTRO_SPECS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
ansible7: MITOGEN_TEST_DISTRO_SPECS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
ansible8: MITOGEN_TEST_DISTRO_SPECS=centos7 centos8 debian9 debian10 debian11 ubuntu1604 ubuntu1804 ubuntu2004
# Ansible 9 (ansible-core 2.16) requires Python 2.7 or >= 3.6 on targets
ansible9: DISTROS=centos7 centos8 debian9 debian10 debian11 ubuntu1804 ubuntu2004
ansible9: MITOGEN_TEST_DISTRO_SPECS=centos7 centos8 debian9 debian10 debian11 ubuntu1804 ubuntu2004
# Ansible 10 (ansible-core 2.17) requires Python >= 3.7 on targets
ansible10: DISTROS=debian10-py3 debian11-py3 ubuntu2004-py3
distros_centos: DISTROS=centos6 centos7 centos8
distros_centos5: DISTROS=centos5
distros_centos6: DISTROS=centos6
distros_centos7: DISTROS=centos7
distros_centos8: DISTROS=centos8
distros_debian: DISTROS=debian9 debian10 debian11
distros_debian9: DISTROS=debian9
distros_debian10: DISTROS=debian10
distros_debian11: DISTROS=debian11
distros_ubuntu: DISTROS=ubuntu1604 ubuntu1804 ubuntu2004
distros_ubuntu1604: DISTROS=ubuntu1604
distros_ubuntu1804: DISTROS=ubuntu1804
distros_ubuntu2004: DISTROS=ubuntu2004
ansible10: MITOGEN_TEST_DISTRO_SPECS=debian10-py3 debian11-py3 ubuntu2004-py3
distros_centos: MITOGEN_TEST_DISTRO_SPECS=centos6 centos7 centos8
distros_centos5: MITOGEN_TEST_DISTRO_SPECS=centos5
distros_centos6: MITOGEN_TEST_DISTRO_SPECS=centos6
distros_centos7: MITOGEN_TEST_DISTRO_SPECS=centos7
distros_centos8: MITOGEN_TEST_DISTRO_SPECS=centos8
distros_debian: MITOGEN_TEST_DISTRO_SPECS=debian9 debian10 debian11
distros_debian9: MITOGEN_TEST_DISTRO_SPECS=debian9
distros_debian10: MITOGEN_TEST_DISTRO_SPECS=debian10
distros_debian11: MITOGEN_TEST_DISTRO_SPECS=debian11
distros_ubuntu: MITOGEN_TEST_DISTRO_SPECS=ubuntu1604 ubuntu1804 ubuntu2004
distros_ubuntu1604: MITOGEN_TEST_DISTRO_SPECS=ubuntu1604
distros_ubuntu1804: MITOGEN_TEST_DISTRO_SPECS=ubuntu1804
distros_ubuntu2004: MITOGEN_TEST_DISTRO_SPECS=ubuntu2004
mode_ansible: MODE=ansible
mode_ansible: ANSIBLE_SKIP_TAGS=resource_intensive
mode_ansible: ANSIBLE_CALLBACK_WHITELIST=profile_tasks