This PR removes *rollup* in favor of *esbuild*, ensuring `DEBUG` flag, when used,
is available to the produced JS and the *umd* dance has been simplified, ditching
the *amd* part from it as I think *amd* is kinda dead as module system.
Set the MIME type for wheels and zip files to application/wasm to enable CDN compression. Also optionally disable compression in zip files and wheels to make the brotli compression by the CDN more efficient.
Co-authored-by: Gyeongjae Choi <def6488@gmail.com>
This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
This is work towards unvendoring the Pyodide foreign function interface.
Prior to this point, we included a large amount of critical functionality with `--pre-js`.
So we could create an archive called `libpyodide.a` with the object files but to use it
you would have to pass `--pre-js _pyodide.out.js` at link time. This embeds all of this
stuff in an object file called `pyodide_pre.o` which goes in our archive so you get all
the needed js runtime by linking it.
Of course someone trying to use this still has to get the Python code onto the import
path, either using `--preload-file`, using Python to unpack it as a zip archive as we now
do, with zipimporter, or otherwise. They also will have to link `libpython.a` (is CPython
going to start distributing an Emscripten libpython?) and probably various other things.
We have to use a hack to inject the JavaScript code into the object files. The normal
`EM_JS` macro cannot handle arbitrary JavaScript code -- for example it fails with many
regex. Instead we manually generate write a C source file that does what we need using
`xxd`. The generated C code is similar to what `EM_JS` generates, but it uses an array
initializer rather than a string initializer for the characters avoiding the C preprocessor /
compiler's strange opinions about strings.
This adds rust toolchain into our docker image so we don't have to install it every time.
make rust command still exists, but I removed it from the docs. So now a user (who wants to build a package that requires rust) is responsible for installing rust toolchain in their build setup, which is reasonable I think.
Co-authored-by: josephrocca <1167575+josephrocca@users.noreply.github.com>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This PR relocates files inside `tools` directory so that `tools` directory doesn't
have to be vendored when creating xbuildenv.
- cmake, pyo3 config ==> moved to `pyodide-build`.
- venv python script ==> moved to templates, then copied to dist during build.
My local build fails with:
```
RollupError: Node tried to load your configuration file as CommonJS
even though it is likely an ES module. To resolve this, change the
extension of your configuration to ".mjs", set "type": "module" in
your package.json file or pass the "--bundleConfigAsCjs" flag.
Original error: Cannot use import statement outside a module
```
This fixes it.
This allows us to embed extra wasm modules into the output. This is split off of #3210.
We need this for #3210 since
1. We don't want to assume that externref is supported but we need to use it as part of
the JS Promise integration proposal. To handle this we segment off the externrefs into
separate wasm modules. If externref is not supported, we load most everything
successfully but these small modules fail to load
2. Even supposing that we know that externref is supported, there is no way to write C
code such that the compiled code uses externref. It's possible using the emscripten
assembly format, but that is less well documented and harder to use for our purposes
than wat.
This PR doesn't actually change anything by itself.
This uses sed to insert /* webpackIgnore: true */ comments into pyodide.js.
This resolves#3087. I also enabled a check that a simple webpack config
builds without warnings. In the future it would be good to add a test that it
also runs without error.
I guess there has to be some trial and error with these. Since we
can't link `pyodide.asm.js` in the deploy docker image, we do it in
a separate step after `build-packages`. Because we have hard
coded `dist` everywhere, it's a little bit awkward building into
`dist-debug`. I do some rearrangements with `cp` and `mv` to hack
around this.
Yesterday's nightly includes rust-lang/rust#98149 which allows us to remove the
PIC setting. This also means we won't ever have to put -Zbuild-std back in.
Closes#2747
* renames packages.json to repodata.json
* renames the corresponding JS and Python variables to be a bit more explicit.
Tangentially related to #795
Currently the following code fails:
```py
from js import eval
eval("Object.create(null)")
```
with:
```py
Traceback (most recent call last):
File "<console>", line 1, in <module>
JsException: TypeError: Cannot read properties of undefined (reading 'name')
```
This fixes it.
Various fixes to improve the npm package.
Switch to publishing the dist folder, fix indexURL for node,
fix node commonJS import and ES6 import only publish core
Pyodide interpreter to npm, download and cache other wheels
on first use.
dist is both more accurate (the 'build' directory is normally where you do the build,
and normally consists of intermediate build artifacts no one cares about). dist also
occurs less frequently in the code base: after this change \bbuild\b has 466 matches,
whereas \bdist\b has 101 matches. build has 1072 matches whereas dist has 362.
The goal of this commit is to add the capability of skipping specific tests by
glob pattern. Many core test sets have just one test that uses subprocess or
threads. Others have a few tests that are failing due to upstream problems in
emscripten. I went through and switched to skipping specific tests in as many
cases as I could. In modules where most or all tests use threading or
subprocess, I gave up and put a comment saying something like 35/50 tests fork
and I didn't want to bother to salvage the remaining ones.
In order to accommodate using patterns to skip specific tests (and also extra
comments) I switched the format of python_tests to a yaml file. This gives us
significantly more flexibility in the structure of the file and allows us to use
ruamel.yaml to update it. We gain a lot flexibility while reducing the length of
make_test_lists.py modestly from 108 to 78 loc.
In the future, hopefully we should be able to make fixes by deleting one skip
pattern and looking into why that particular test fails.
One can select which benchmark (pystone, numpy, matplotlib, or all) to be run through command-line arguments.
We can divide each benchmark to separate CI jobs in the future if needed.
Moved pystone benchmark to benchmark directory, preventing it from being included in Pyodide release.
Hood Chatham