If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
This begins migrating some of the ffi code towards using externrefs, which
should make it faster and shorter.
I'm planning on migrating from the existing hiwire.h functions to new `jslib.h`
functions which will operate on externrefs, but I think it will be easiest to
do this in several PRs since it's going to be a fair amount of code that
changes (in mostly boilerplate ways though).
This makes IDEs, documentation generation tools, and linters handle our
generated files better. I set the project root to be `src` instead of `src/js`
so that it is allowed to import files directly from `src/core`. This way we
don't have to copy `error_handling.ts` we can just import
`../core/error_handling`.
I made a new folder called `src/js/generated` to place generated files into and
added typescript resolution rules so that when we import a file called
"generated/blah" we first look for `blah` in `src/js/generated` and then fall
back to a file called `blah` in `src/core`.
This also allows us to move around fewer files when building the docs
and in the makefile.
If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
This is work towards unvendoring the Pyodide foreign function interface.
Prior to this point, we included a large amount of critical functionality with `--pre-js`.
So we could create an archive called `libpyodide.a` with the object files but to use it
you would have to pass `--pre-js _pyodide.out.js` at link time. This embeds all of this
stuff in an object file called `pyodide_pre.o` which goes in our archive so you get all
the needed js runtime by linking it.
Of course someone trying to use this still has to get the Python code onto the import
path, either using `--preload-file`, using Python to unpack it as a zip archive as we now
do, with zipimporter, or otherwise. They also will have to link `libpython.a` (is CPython
going to start distributing an Emscripten libpython?) and probably various other things.
We have to use a hack to inject the JavaScript code into the object files. The normal
`EM_JS` macro cannot handle arbitrary JavaScript code -- for example it fails with many
regex. Instead we manually generate write a C source file that does what we need using
`xxd`. The generated C code is similar to what `EM_JS` generates, but it uses an array
initializer rather than a string initializer for the characters avoiding the C preprocessor /
compiler's strange opinions about strings.
This adds pyodide_build command create_xbuildenv which creates a
crossbuild environment (from a copy of Pyodide where scipy has been
build) and install_xbuildenv which installs the cross build environment
into a fresh copy of Pyodide.
I successfully installed the xbuild environment into a fresh checkout of
Pyodide then built statsmodels and scikit-learn in isolation, without
building the Python interpreter, numpy, or scipy. I dumped the generated
wheels into a copy of Pyodide downloaded from CI, and was able to import
and use them as normal.
The size of the xbuild environment is 1.5 megabytes, of which 1.2 megabytes
is Python headers.
In a subsequent PR, we can update the CI to automatically upload these
to aws s3 and then install the environment from there.
The installation is very fast, and this avoids having two copies of node
around. In particular, this avoids the need to monkey-patch uglify-js to
use the system node.
This can be further streamlined when #792 is merged
Hood Chatham