This allows bidirectional stack switching and other arbitrary control flow, whereas
without this patch arbitrary control flow terribly messes up the Python interpreter
state.
By far the most subtle thing here is the manipulation of the cframe state. This
required careful reading of the greenlet source code and a large amount of trial
and error to get right.
Unfortunately the file pystate.c is tightly coupled to the Python version. There are
apparently plans to try to upstream some of this code from greenlet into Python itself.
In the meantime, it should be possible to continue updating it by imitating greenlet.
This is a minor rearrangment.
I renamed "continuations" stuff to "stack_switching" and made a new folder
for stack switching logic. I also added logic in esbuild.config.mjs to calculate
which functions exported from stack_switching.mjs and automatically copy them to
Module and into the Emscripten namespace. This makes unit testing a little neater
since there is no modification of the Module object.
If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
This begins migrating some of the ffi code towards using externrefs, which
should make it faster and shorter.
I'm planning on migrating from the existing hiwire.h functions to new `jslib.h`
functions which will operate on externrefs, but I think it will be easiest to
do this in several PRs since it's going to be a fair amount of code that
changes (in mostly boilerplate ways though).
This makes IDEs, documentation generation tools, and linters handle our
generated files better. I set the project root to be `src` instead of `src/js`
so that it is allowed to import files directly from `src/core`. This way we
don't have to copy `error_handling.ts` we can just import
`../core/error_handling`.
I made a new folder called `src/js/generated` to place generated files into and
added typescript resolution rules so that when we import a file called
"generated/blah" we first look for `blah` in `src/js/generated` and then fall
back to a file called `blah` in `src/core`.
This also allows us to move around fewer files when building the docs
and in the makefile.
This is a minor rearrangment.
I renamed "continuations" stuff to "stack_switching" and made a new folder
for stack switching logic. I also added logic in esbuild.config.mjs to calculate
which functions exported from stack_switching.mjs and automatically copy them to
Module and into the Emscripten namespace. This makes unit testing a little neater
since there is no modification of the Module object.
If wasm exception handling is available, use dynamically generated modules
with wasm exception handling for the invoke stubs instead of a JS trampoline.
This is useful because JS trampolines interact poorly with JS Promise Integration.
In the future, we should switch exception handling ABIs to the compiler-provided
wasm exception handling ABI and we can remove this code. Currently we are
blocked on using compiler-provided wasm eh by Rust support.
Resolves pyodide#4126.
If someone builds emsdk outside of the docker image and then attempts to use it
inside, there will be an error. This detects it and tells people to clean emsdk.
This PR removes *rollup* in favor of *esbuild*, ensuring `DEBUG` flag, when used,
is available to the produced JS and the *umd* dance has been simplified, ditching
the *amd* part from it as I think *amd* is kinda dead as module system.
Set the MIME type for wheels and zip files to application/wasm to enable CDN compression. Also optionally disable compression in zip files and wheels to make the brotli compression by the CDN more efficient.
Co-authored-by: Gyeongjae Choi <def6488@gmail.com>
This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
This is work towards unvendoring the Pyodide foreign function interface.
Prior to this point, we included a large amount of critical functionality with `--pre-js`.
So we could create an archive called `libpyodide.a` with the object files but to use it
you would have to pass `--pre-js _pyodide.out.js` at link time. This embeds all of this
stuff in an object file called `pyodide_pre.o` which goes in our archive so you get all
the needed js runtime by linking it.
Of course someone trying to use this still has to get the Python code onto the import
path, either using `--preload-file`, using Python to unpack it as a zip archive as we now
do, with zipimporter, or otherwise. They also will have to link `libpython.a` (is CPython
going to start distributing an Emscripten libpython?) and probably various other things.
We have to use a hack to inject the JavaScript code into the object files. The normal
`EM_JS` macro cannot handle arbitrary JavaScript code -- for example it fails with many
regex. Instead we manually generate write a C source file that does what we need using
`xxd`. The generated C code is similar to what `EM_JS` generates, but it uses an array
initializer rather than a string initializer for the characters avoiding the C preprocessor /
compiler's strange opinions about strings.
This adds rust toolchain into our docker image so we don't have to install it every time.
make rust command still exists, but I removed it from the docs. So now a user (who wants to build a package that requires rust) is responsible for installing rust toolchain in their build setup, which is reasonable I think.
Co-authored-by: josephrocca <1167575+josephrocca@users.noreply.github.com>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This PR relocates files inside `tools` directory so that `tools` directory doesn't
have to be vendored when creating xbuildenv.
- cmake, pyo3 config ==> moved to `pyodide-build`.
- venv python script ==> moved to templates, then copied to dist during build.
My local build fails with:
```
RollupError: Node tried to load your configuration file as CommonJS
even though it is likely an ES module. To resolve this, change the
extension of your configuration to ".mjs", set "type": "module" in
your package.json file or pass the "--bundleConfigAsCjs" flag.
Original error: Cannot use import statement outside a module
```
This fixes it.
This allows us to embed extra wasm modules into the output. This is split off of #3210.
We need this for #3210 since
1. We don't want to assume that externref is supported but we need to use it as part of
the JS Promise integration proposal. To handle this we segment off the externrefs into
separate wasm modules. If externref is not supported, we load most everything
successfully but these small modules fail to load
2. Even supposing that we know that externref is supported, there is no way to write C
code such that the compiled code uses externref. It's possible using the emscripten
assembly format, but that is less well documented and harder to use for our purposes
than wat.
This PR doesn't actually change anything by itself.
This uses sed to insert /* webpackIgnore: true */ comments into pyodide.js.
This resolves#3087. I also enabled a check that a simple webpack config
builds without warnings. In the future it would be good to add a test that it
also runs without error.
I guess there has to be some trial and error with these. Since we
can't link `pyodide.asm.js` in the deploy docker image, we do it in
a separate step after `build-packages`. Because we have hard
coded `dist` everywhere, it's a little bit awkward building into
`dist-debug`. I do some rearrangements with `cp` and `mv` to hack
around this.
Yesterday's nightly includes rust-lang/rust#98149 which allows us to remove the
PIC setting. This also means we won't ever have to put -Zbuild-std back in.
Closes#2747
* renames packages.json to repodata.json
* renames the corresponding JS and Python variables to be a bit more explicit.
Tangentially related to #795
Currently the following code fails:
```py
from js import eval
eval("Object.create(null)")
```
with:
```py
Traceback (most recent call last):
File "<console>", line 1, in <module>
JsException: TypeError: Cannot read properties of undefined (reading 'name')
```
This fixes it.
Hood Chatham