pyodide/docs/development/testing.md

(testing)=

Testing and benchmarking

Testing

Running the Python test suite

You can use either Selenium or Playwright to run the pytest suite of tests.

Install the following dependencies into the default Python installation:

pip install pytest pytest-instafail pytest-httpserver

There are 3 test locations that are collected by pytest,

• src/tests/: general Pyodide tests and tests running the CPython test suite
• pyodide-build/pyodide_build/tests/: tests related to Pyodide build system (do not require selenium or playwright to run)
• packages/*/test_*: package specific tests.

Selenium (default)

pip install selenium

Install geckodriver and chromedriver and check that they are in your PATH.

Running the Python test suite

To run the test suite, run pytest from the root directory of Pyodide:

pytest

You can run the tests from a specific file with:

pytest path/to/test/file.py

To speed things up you may wish to filter out two of the browsers. Node runs tests the fastest so:

pytest -k "not chrome and not fire"

is often helpful. Some browsers sometimes produce informative errors than others so if you are getting confusing errors it is worth rerunning the test on each browser. If you are still confused, try {ref}manual-testing which is more flexible.

There are 5 test locations that are collected by pytest:

• src/tests/: general Pyodide tests and tests running the CPython test suite
• pyodide-test-runner/pyodide_test_runner/tests/: Tests for the testing system.
• pyodide-build/pyodide_build/tests/: tests related to Pyodide build system (do not require selenium to run)
• packages/test_common.py: common tests for packages.
• packages/*/test_*: package specific tests.

Running tests with Playwright

By default, the tests will be run with Selenium. It is possible to run tests with playwright instead as follows.

First install playwright

pip install playwright && python -m playwright install

Then use the --runner argument to specify to run tests with playwright.

pytest --runner playwright

Running the JavaScript test suite

To run tests on the JavaScript Pyodide package using Mocha, run the following commands,

cd src/js
npm test

To check TypeScript type definitions run,

npx tsd

(manual-testing)=

Manual interactive testing

To run tests manually:

1. Build Pyodide, perhaps in the docker image

2. From outside of the docker image, cd into the dist directory and run python -m http.server.

3. Once the webserver is running, simple interactive testing can be run by visiting the URL: http://localhost:<PORT>/console.html. It's recommended to use pyodide.runPython in the browser console rather than using the repl.

Benchmarking

To run common benchmarks to understand Pyodide's performance, begin by installing the same prerequisites as for testing. Then run:

PYODIDE_PACKAGES="numpy,matplotlib" make benchmark

Linting

We lint with pre-commit.

Python is linted with flake8, black and mypy. JavaScript, markdown, yaml, and html are linted with prettier. C is linted with clang-format.

To lint the code, run:

pre-commit run -a

You can have the linter automatically run whenever you commit by running

pip install pre-commit
pre-commit install

and this can later be disabled with

pre-commit uninstall

If you don't lint your code, certain lint errors will be fixed automatically by pre-commit.ci which will push fixes to your branch. If you want to push more commits, you will either have to pull in the remote changes or force push.

Testing framework

(run-in-pyodide)=

run_in_pyodide

Many tests simply involve running a chunk of code in Pyodide and ensuring it doesn't error. In this case, one can use the run_in_pyodide decorate from pyodide_test_runner.decorator, e.g.

from pyodide_test_runner import run_in_pyodide

@run_in_pyodide
def test_add(selenium):
    assert 1 + 1 == 2

In this case, the body of the function will automatically be run in Pyodide. The decorator can also be called with a packages argument to load packages before running the test. For example:

from pyodide_test_runner import run_in_pyodide

@run_in_pyodide(packages = ["regex"])
def test_regex(selenium_standalone):
    import regex
    assert regex.search("o", "foo").end() == 2

If your run_in_pyodide test is failing, it will generate much better error messages if the packages pytest and tblib are available in Pyodide, so it is recommended that you build them:

PYODIDE_PACKAGES="pytest, tblib, whatever-else" make -C packages

If you need to xfail your test on a certain browser, you can use pytest.mark.xfail_browsers. You can also use @run_in_pyodide with pytest.mark.parametrize, with hypothesis, etc. @run_in_pyodide MUST be the innermost decorator. Any decorators inside of @run_in_pyodide will be have no effect on the behavior of the test.

from pyodide_test_runner import run_in_pyodide

@pytest.mark.parametrize("x", [1, 2, 3])
@run_in_pyodide(packages = ["regex"])
def test_type_of_int(selenium, x):
    assert type(x) is int

These arguments must be picklable. You can also use fixtures as long as the return values of the fixtures are picklable (most commonly, if they are None). As a special case, the function will see the selenium fixture as None inside the test.

It is possible to use run_in_pyodide as an inner function:

def test_inner_function(selenium):
    @run_in_pyodide
    def inner_function(selenium, x):
        assert x == 6
        return 7

    assert inner_function(selenium_mock, 6) == 7

Again both the arguments and return value must be pickleable.

Also, the function will not see closure variables at all:

def test_inner_function_closure(selenium):
    x = 6
    @run_in_pyodide
    def inner_function(selenium):
        assert x == 6
        return 7

    # Raises `NameError: 'x' is not defined`
    assert inner_function(selenium_mock) == 7

Custom test marks

We support four test marks:

@pytest.mark.skip_refcount_check and pytest.mark.skip_pyproxy_check disable respectively the check for JavaScript references and the check for PyProxies. If a test creates JavaScript references or PyProxies and does not clean them up, by default the tests will fail. If a test is known to leak objects, it is possible to disable these checks with these markers.

pytest.mark.driver_timeout(timeout): Set script timeout in WebDriver. If the test is known to take a long time, you can extend the deadline with this marker.

pytest.mark.xfail_browsers(chrome="why chrome fails"): xfail a test in specific browsers.