Buster isn't available for anymore for Python 312 so we have to update.
This also updates to Python 3.11.7, which turns out to be relatively invasive. I broadly
xfailed failing tests in `python_tests.yaml` since we'll upgrade to Python 3.12 soon anyways.
One of the cpython tests (test_plistlib.TestBinaryPlistlib.test_deep_nesting)
seems to be extremely sensitive to stack utilization and stack overflows in our
CI when even a tiny amount of extra stack is used. I doubt this is a problem in
real life -- I certainly cannot reproduce it outside of the CI service. But this
rearranges the code to avoid any additional stack usage so that CI will pass.
This change switches to my external implementation of hiwire. This is the
minimal change set to do this, it uses some hacks to avoid changing any files
outside of `hiwire.{c,h,js}`. In followups, I will gradually switch to using
the new APIs rather than compatibility shims.
One of the cpython tests (test_plistlib.TestBinaryPlistlib.test_deep_nesting)
seems to be extremely sensitive to stack utilization and stack overflows in our
CI when even a tiny amount of extra stack is used. I doubt this is a problem in
real life -- I certainly cannot reproduce it outside of the CI service. But this
rearranges the code to avoid any additional stack usage so that CI will pass.
Removes / unvendors some python modules:
- Remove `_aix_support.py`, which is for supporting IBM AIX OS.
- Unvendor `_pydecimal.py`.
- _pydecimal is a pure Python implementation of `decimal` module. [Importing `decimal` fallbacks](https://github.com/python/cpython/blob/main/Lib/decimal.py) to `_pydecimal` if the C-implementation `_decimal` is not available. In our case, _decimal is available, so _pydecimal will not be normally used.
- Unvendor `pydoc_data`.
- pydoc_data contains [a large (~700KB) dictionary](https://github.com/python/cpython/blob/main/Lib/pydoc_data/topics.py) for explaining python builtins. This is mostly used when `help("...")` is called.
@ryanking13 added these very nice error messages to the `ModuleNotFound`
errors. However they introduce a few problems:
1. `find_spec` is supposed to return `None` or a spec and not to raise an error.
If it raises errors, it can cause trouble in code that tries to check if a module is
installed or not.
2. Other code that tries to add new import hooks has to know to insert them
before these error-raising import hooks.
See the discussion in #3262.
This instead patches `importlib._bootstrap` to create a function called
`_get_module_not_found_error`. We then can monkey patch this to modify
the error messages that `importlib` raises.
See Python issue: https://github.com/python/cpython/issues/100208
The command line runner in #2976 finally works, but it is a large change set so
I am planning to split it up. This is the first PR split off from there.
This PR adds a patch to Python to expose pymain_run_python. We cannot use the
public API Py_RunMain for this purpose because it finalizes the Python
interpreter with Py_FinalizeEx when it is done. If we start an async task with
Py_RunMain then it will segfault. pymain_run_python does a large amount of work
and reproducing it is undesirable.
I added an args parameter which accepts command line arguments. The private
entry point pyodide._main._run_main() executes the main Python entrypoint
without shutting down the interpreter and returns the return code.
Unvendor the standard library sqlite3 to reduce the size of the main module. It reduces the size of pyodide.asm.wasm around ~1.4MB.
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This is more WIP on create_xbuildenv. I am including some of the WASM_LIB_DIR
but not all of it to keep size in check. The current cross build environment that we
would upload/download is 20mb.
To use this, we need an extra CLI entrypoint which I am working on.
sysconfig.py uses the environment variable `_PYTHON_SYSCONFIGDATA_NAME`
to decide where to look for the sysconfig data file with info about the compile target.
We also need to separately insure that our sysconfig data file is on the path. We
don't want the rest of our target stdlib on the path, so I made an extra sysconfigdata
folder, copied the sysconfig data into it, and put it on the path.
Joe Marshall