The command line runner in #2976 finally works, but it is a large change set so
I am planning to split it up. This is the first PR split off from there.
This PR adds a patch to Python to expose pymain_run_python. We cannot use the
public API Py_RunMain for this purpose because it finalizes the Python
interpreter with Py_FinalizeEx when it is done. If we start an async task with
Py_RunMain then it will segfault. pymain_run_python does a large amount of work
and reproducing it is undesirable.
I added an args parameter which accepts command line arguments. The private
entry point pyodide._main._run_main() executes the main Python entrypoint
without shutting down the interpreter and returns the return code.
Unvendor the standard library sqlite3 to reduce the size of the main module. It reduces the size of pyodide.asm.wasm around ~1.4MB.
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This is more WIP on create_xbuildenv. I am including some of the WASM_LIB_DIR
but not all of it to keep size in check. The current cross build environment that we
would upload/download is 20mb.
To use this, we need an extra CLI entrypoint which I am working on.
sysconfig.py uses the environment variable `_PYTHON_SYSCONFIGDATA_NAME`
to decide where to look for the sysconfig data file with info about the compile target.
We also need to separately insure that our sysconfig data file is on the path. We
don't want the rest of our target stdlib on the path, so I made an extra sysconfigdata
folder, copied the sysconfig data into it, and put it on the path.
All libffi tests pass now. The only failing ctypes test is test_callback_too_many_args which doesn't segfault anymore, it only soft fails. Planning to submit a PR to cpython that fixes test_callback_too_many_args.
See also:
bugs.python.org/issue47208
https://github.com/emscripten-core/emscripten/pull/16658
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.
Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.
I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.
I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
The goal of this commit is to add the capability of skipping specific tests by
glob pattern. Many core test sets have just one test that uses subprocess or
threads. Others have a few tests that are failing due to upstream problems in
emscripten. I went through and switched to skipping specific tests in as many
cases as I could. In modules where most or all tests use threading or
subprocess, I gave up and put a comment saying something like 35/50 tests fork
and I didn't want to bother to salvage the remaining ones.
In order to accommodate using patterns to skip specific tests (and also extra
comments) I switched the format of python_tests to a yaml file. This gives us
significantly more flexibility in the structure of the file and allows us to use
ruamel.yaml to update it. We gain a lot flexibility while reducing the length of
make_test_lists.py modestly from 108 to 78 loc.
In the future, hopefully we should be able to make fixes by deleting one skip
pattern and looking into why that particular test fails.
* Patch ceval to add interrupt handler callback
* Don't use SharedArrayBuffer in test, Firefox doesn't expose it
* Fix test
* Retest
Co-authored-by: Jan Max Meyer <jmm@phorward.de>
This is needed for the upstream backend, and I believe is a noop under
the fastcomp backend
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
Co-authored-by: Joe Marshall <joe.marshall@nottingham.ac.uk>
Hood Chatham