The JSPI API changed. Currently v8 implements both the old and the new version.
We want to keep support for the old version for a while so we can use stack
switching in node. This supports both.
I also simplified the implementation by quite a lot somewhat by accident in the
process of trying to work out how to support the new API.
Resolves#4883.
Include type defintion files in pyodide-core.tar.gz
Remove snapshot related files.
Remove test files.
TODO in the future: Remove source maps and webworker.js?
As suggested in #4942
We've gotten a lot of complaints about how hard it is to call certain JS
functions, but it's hard to do much to improve the situation without knowing
anything about the specific function being called.
However, if we know about the API, we can choose how to convert to/from JS as
appropriate and make things a lot easier. This adds a method to JsProxy called
`bind_sig` which adjusts how __getattr__ and __call__ work on the proxy so we
can choose our converters. I added a class called JsSignature which
guides what we do when we call the function. I completely rewrote
JsProxy_Vectorcall to use this. I added handling for Promise results so we can
automatically bind a signature to the result of the Promise.
There's a lot leftover to do:
* Generator and AsyncGenerator types
* Alternatives A | B (try A converter and if it raises a TypeError fall back to B)
* TypedDict converters
* Make calling certain objects call `Reflect.construct`
* Cleaner __getattr__ handling, __setattr__ handling
* A way to raise AttributeError if the attribute is missing from the signature
even if it's present in the JS object (important for backwards compatibility)
* Signatures for JS stdlib functions
My idea is that the signatures should simultaneously function as mypy typehints
as much as this is practical.
libgl is marked .PHONY and is a dependency for the link step which forces us to always redo the link
step even though it's completely not necessary. I moved embuilder build libgl into the link step
since that's the only place where it's needed.
dist/python_stdlib.zip and dist/pyodide-lock.json both declare a dependency on pyodide_build but
there is never a file of that name so they always have to run. Instead, run make pyodide_build so
that they can reuse the logic but they don't always run if their other deps are satisfied.
Now that dist/python_stdlib.zip doesn't always run it needs to rely on all files in src/py.
Make pyodide_build only invoke pip install if pyodide_build is not already installed. Pip generates
a lot of noise and takes a while so skipping it when we don't need it helps a lot.
Make dist/webworker.js and friends not phony since they stopped needing to be when PYODIDE_BASE_URL
was removed from them.
This allows bidirectional stack switching and other arbitrary control flow, whereas
without this patch arbitrary control flow terribly messes up the Python interpreter
state.
By far the most subtle thing here is the manipulation of the cframe state. This
required careful reading of the greenlet source code and a large amount of trial
and error to get right.
Unfortunately the file pystate.c is tightly coupled to the Python version. There are
apparently plans to try to upstream some of this code from greenlet into Python itself.
In the meantime, it should be possible to continue updating it by imitating greenlet.
This is a minor rearrangment.
I renamed "continuations" stuff to "stack_switching" and made a new folder
for stack switching logic. I also added logic in esbuild.config.mjs to calculate
which functions exported from stack_switching.mjs and automatically copy them to
Module and into the Emscripten namespace. This makes unit testing a little neater
since there is no modification of the Module object.
If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
This begins migrating some of the ffi code towards using externrefs, which
should make it faster and shorter.
I'm planning on migrating from the existing hiwire.h functions to new `jslib.h`
functions which will operate on externrefs, but I think it will be easiest to
do this in several PRs since it's going to be a fair amount of code that
changes (in mostly boilerplate ways though).
This makes IDEs, documentation generation tools, and linters handle our
generated files better. I set the project root to be `src` instead of `src/js`
so that it is allowed to import files directly from `src/core`. This way we
don't have to copy `error_handling.ts` we can just import
`../core/error_handling`.
I made a new folder called `src/js/generated` to place generated files into and
added typescript resolution rules so that when we import a file called
"generated/blah" we first look for `blah` in `src/js/generated` and then fall
back to a file called `blah` in `src/core`.
This also allows us to move around fewer files when building the docs
and in the makefile.
This is a minor rearrangment.
I renamed "continuations" stuff to "stack_switching" and made a new folder
for stack switching logic. I also added logic in esbuild.config.mjs to calculate
which functions exported from stack_switching.mjs and automatically copy them to
Module and into the Emscripten namespace. This makes unit testing a little neater
since there is no modification of the Module object.
If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
Resolves pyodide#4126.
If someone builds emsdk outside of the docker image and then attempts to use it
inside, there will be an error. This detects it and tells people to clean emsdk.
This PR removes *rollup* in favor of *esbuild*, ensuring `DEBUG` flag, when used,
is available to the produced JS and the *umd* dance has been simplified, ditching
the *amd* part from it as I think *amd* is kinda dead as module system.
Set the MIME type for wheels and zip files to application/wasm to enable CDN compression. Also optionally disable compression in zip files and wheels to make the brotli compression by the CDN more efficient.
Co-authored-by: Gyeongjae Choi <def6488@gmail.com>
This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
Hood Chatham