Commit Graph

115 Commits

Author SHA1 Message Date
Hood Chatham b67660ff9d
BLD Add CPYTHON_DEBUG environment variable to request a debug Python (#3966) 2023-06-28 17:01:15 -07:00
Hood Chatham 71cc8bc925 Revert "MAINT Use sigstore to validate Python source signature (#3916)"
This reverts commit 59bb655957.
2023-06-12 15:16:55 -07:00
Hood Chatham 59bb655957
MAINT Use sigstore to validate Python source signature (#3916)
This should make Python updates a little easier and more secure.

See: python.org/download/sigstore
2023-06-12 12:28:44 -07:00
Gyeongjae Choi 73c497bfbe
Remove in-tree dependency when building unvendored python modules (#3864) 2023-06-05 19:48:48 +09:00
Christian Clauss b2201c19b1
Upgrade to Python v3.11.3 (#3741) 2023-04-12 17:38:05 +02:00
Hood Chatham 32ccad8f6e
Use upstream libffi (#3629) 2023-04-01 17:32:38 -07:00
Christian Clauss 154fbed9fa
Upgrade to Python v3.11.2 (#3614) 2023-03-01 23:57:17 +01:00
Gyeongjae Choi ed3ae1ba49
Download standard libraries in a zipfile (#3582) 2023-03-01 23:11:30 +01:00
Gyeongjae Choi 9e1e575dfe
Deepfreeze essential parts of the encodings module (#3585) 2023-02-22 18:40:05 +09:00
Gyeongjae Choi dcdc1e6941
Unvendor pydoc_data and _pydecimal (#3525)
Removes / unvendors some python modules:

- Remove `_aix_support.py`, which is for supporting IBM AIX OS.

- Unvendor `_pydecimal.py`.
  - _pydecimal is a pure Python implementation of `decimal` module. [Importing `decimal` fallbacks](https://github.com/python/cpython/blob/main/Lib/decimal.py) to `_pydecimal` if the C-implementation `_decimal` is not available. In our case, _decimal is available, so _pydecimal will not be normally used. 

- Unvendor `pydoc_data`.
  - pydoc_data contains [a large (~700KB) dictionary](https://github.com/python/cpython/blob/main/Lib/pydoc_data/topics.py) for explaining python builtins. This is mostly used when `help("...")` is called.
2023-02-01 11:00:17 +09:00
Gyeongjae Choi f50c749c9a
Apply upstream cpython commits to relocate test directories (#3507) 2023-01-30 13:34:33 +09:00
Hood Chatham 3b2952fc92
Python3.11 (#3252) 2023-01-23 19:45:59 -08:00
Hood Chatham 68b3f3d067
Fix top level multiprocessing.connection import and update yt (#3371)
This updates yt. The new yt has a top level `multiprocessing.connection` import so we patch
that so that it can be imported without raising an error.
2022-12-19 16:56:31 -08:00
Gyeongjae Choi cb29aca290
Patch ctypes.util.find_library to support Emscripten platform (#3353) 2022-12-18 11:20:00 +01:00
Hood Chatham 33d1794a6e
Use a more robust method to improve our ModuleNotFound errors (#3263)
@ryanking13 added these very nice error messages to the `ModuleNotFound`
errors. However they introduce a few problems:
1. `find_spec` is supposed to return `None` or a spec and not to raise an error.
If it raises errors, it can cause trouble in code that tries to check if a module is
installed or not.
2. Other code that tries to add new import hooks has to know to insert them
before these error-raising import hooks.

See the discussion in #3262.

This instead patches `importlib._bootstrap` to create a function called
`_get_module_not_found_error`. We then can monkey patch this to modify
the error messages that `importlib` raises.

See Python issue: https://github.com/python/cpython/issues/100208
2022-12-12 18:35:51 -08:00
Hood Chatham 1335d7ee64
Add argument handling and _run_main function (#3021)
The command line runner in #2976 finally works, but it is a large change set so
I am planning to split it up. This is the first PR split off from there.

This PR adds a patch to Python to expose pymain_run_python. We cannot use the
public API Py_RunMain for this purpose because it finalizes the Python
interpreter with Py_FinalizeEx when it is done. If we start an async task with
Py_RunMain then it will segfault. pymain_run_python does a large amount of work
and reproducing it is undesirable.

I added an args parameter which accepts command line arguments. The private
entry point pyodide._main._run_main() executes the main Python entrypoint
without shutting down the interpreter and returns the return code.
2022-08-30 08:49:06 -07:00
Hood Chatham 67aa16861d
Update Emscripten to 3.1.18 (#2950) 2022-08-21 17:17:49 -07:00
Gyeongjae Choi 26535e33cd
Unvendor sqlite3 from stdlib (#2946)
Unvendor the standard library sqlite3 to reduce the size of the main module. It reduces the size of pyodide.asm.wasm around ~1.4MB.

Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
2022-08-22 08:52:50 +09:00
Gyeongjae Choi a72ce423b4
BLD Use emscripten ports of bzip and zlib when building cpython (#2912) 2022-07-28 10:17:58 +02:00
Hood Chatham 52287a179d
More work on create_xbuildenv to allow building numpy (#2811)
This is more WIP on create_xbuildenv. I am including some of the WASM_LIB_DIR
but not all of it to keep size in check. The current cross build environment that we
would upload/download is 20mb.

To use this, we need an extra CLI entrypoint which I am working on.
2022-07-01 11:20:45 -07:00
Hood Chatham 55a53df4a5
Update libffi tag (#2772)
Since the Emscripten update, libffi can support closures with up to 1000 arguments.
2022-06-23 12:05:40 -07:00
Hood Chatham 060f1f4ba6
MAINT Use the patched version of setuptools-rust (#2735) 2022-06-20 09:28:19 +09:00
Hood Chatham c834085995
Use emscripten 3.1.13 (#2679) 2022-06-10 22:26:10 +02:00
Hood Chatham 52f27f06ad
Update to emscripten 3.1.6 (#2672) 2022-06-10 08:13:36 -07:00
Hood Chatham dbbbbb67b2
Cryptography v36.0.2 w/ Rust (#2378) 2022-06-09 10:57:34 -07:00
Hood Chatham 8a8629f2c6
MAINT Switch to using a tag to version libffi port (#2673) 2022-06-08 21:37:24 +02:00
Hood Chatham 2f4f258bb5
MAINT Use target sysconfigdata rather than host sysconfigdata when cross compiling (#2516)
sysconfig.py uses the environment variable `_PYTHON_SYSCONFIGDATA_NAME`
to decide where to look for the sysconfig data file with info about the compile target.
We also need to separately insure that our sysconfig data file is on the path. We
don't want the rest of our target stdlib on the path, so I made an extra sysconfigdata
folder, copied the sysconfig data into it, and put it on the path.
2022-05-29 18:59:10 -07:00
Ryan May 91fd7dbe89
Fix building CPython on macos (Fixes #2360) (#2554)
Updates the Makefile to avoid separating the source command from the build.
2022-05-12 15:50:01 -07:00
stonebig be59fae4f4
upgrade to SQLIte-3.38.5 (#2518) 2022-05-08 09:56:59 +09:00
echorand (Amit Saha) 213288f200
ENH Update sqlite (#2477) 2022-05-04 14:18:07 -06:00
Hood Chatham f04ba9d56a
Add bitarray (#2459) 2022-05-04 09:50:44 -06:00
Hood Chatham fda600e519
Fix interrupt handler patch (#2405) 2022-04-19 14:40:58 +02:00
Hood Chatham dfbf28856e
New keyboard interrupt patch from upstream (#2332) 2022-04-08 08:35:58 -07:00
Hood Chatham eb8afb00c2
Update libffi commit (#2350)
All libffi tests pass now. The only failing ctypes test is test_callback_too_many_args which doesn't segfault anymore, it only soft fails. Planning to submit a PR to cpython that fixes test_callback_too_many_args.
See also:
bugs.python.org/issue47208
https://github.com/emscripten-core/emscripten/pull/16658
2022-04-04 21:18:20 -07:00
Hood Chatham f06eb77387
Remove bad ctypes patch (#2331) 2022-03-30 15:03:36 -07:00
Hood Chatham 37057d1704
Set PLATFORM_TRIPLET, include platform in so names, only load compatible so files (#2299)
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.

Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.

I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.

I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
2022-03-28 20:07:36 -07:00
Hood Chatham 22fab55b04
MAINT Bump zlib version (#2327)
Apparently zlib v1.2.11 had an out of bounds memory access bug so
they yanked it. This bumps the version and gets the build working
again.
2022-03-28 18:17:25 -07:00
Hood Chatham 5897646e04
MAINT Update emscripten to 2.0.24 (#2035) 2022-03-23 23:27:46 -07:00
Hood Chatham 0f2ce88edb
Rework core tests (#2265)
The goal of this commit is to add the capability of skipping specific tests by
glob pattern. Many core test sets have just one test that uses subprocess or
threads. Others have a few tests that are failing due to upstream problems in
emscripten. I went through and switched to skipping specific tests in as many
cases as I could. In modules where most or all tests use threading or
subprocess, I gave up and put a comment saying something like 35/50 tests fork
and I didn't want to bother to salvage the remaining ones.

In order to accommodate using patterns to skip specific tests (and also extra
comments) I switched the format of python_tests to a yaml file. This gives us
significantly more flexibility in the structure of the file and allows us to use
ruamel.yaml to update it. We gain a lot flexibility while reducing the length of
make_test_lists.py modestly from 108 to 78 loc.

In the future, hopefully we should be able to make fixes by deleting one skip
pattern and looking into why that particular test fails.
2022-03-22 13:41:43 -07:00
Hood Chatham 0e1e2ac01c
Python 3.10 (#2225) 2022-03-17 15:48:56 -07:00
Hood Chatham 19261f394c
Remove function pointer cast emulation (#2019) 2021-12-09 10:31:21 -08:00
Roman Yurchak 82aa271b4e
Remove disable-set-inheritable CPython patch (#2003) 2021-11-26 14:47:04 -08:00
Jan Max Meyer ebb75cd334
ENH Makefile targets for cpython partial and full clean (#1969) 2021-11-16 20:22:50 +01:00
Hood Chatham 70e9e19dde
Patches from joemarshall/fpcast (#1708)
Co-authored-by: joemarshall <joe.marshall@nottingham.ac.uk>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
2021-11-13 12:31:11 +01:00
Gyeongjae Choi 754ae621c8
Drop Python built-in pwd module support (#1883) 2021-10-12 20:45:46 +02:00
Hood Chatham 3e3d6eb8a9
ENH Support PyErr_CheckSignals interrupts (#1294) 2021-09-26 11:53:46 -07:00
Hood Chatham 6eac16825c
Update libffi to tip of tree (#1760) 2021-07-30 16:54:27 +02:00
Hood Chatham 6059fc7652
Remove fpcast calls from CALL_FUNCTION opcode (#1699) 2021-07-18 15:04:42 -07:00
Paul m. p. P d4a705bde2
Use PYMAJOR/PYMINOR according to Python devguide (#1570)
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
2021-07-06 13:28:39 +02:00
Hood Chatham 46eee26ff8
Various improvements in core (#1673) 2021-07-01 10:47:44 -07:00