Closes https://github.com/pyodide/pyodide/issues/3687
I forgot to make the changes for the dev deployment in
https://github.com/pyodide/pyodide/pull/3667
Overall those are mostly identical lines repeated several times, however
because the cost of error is pretty high I prefer not to factorize it
for now. Instead, we should rethink this system, maybe moving dev
deployment to a separate bucket and making release deployment immutable.
Set the MIME type for wheels and zip files to application/wasm to enable CDN compression. Also optionally disable compression in zip files and wheels to make the brotli compression by the CDN more efficient.
Co-authored-by: Gyeongjae Choi <def6488@gmail.com>
This adds a short helper script which shows a gzip and brotli compressed size of a file,
and uses it in CI to check compressed size of pyodide.asm.* after build in addition to
the original file size.
This is work towards unvendoring the Pyodide foreign function interface.
Prior to this point, we included a large amount of critical functionality with `--pre-js`.
So we could create an archive called `libpyodide.a` with the object files but to use it
you would have to pass `--pre-js _pyodide.out.js` at link time. This embeds all of this
stuff in an object file called `pyodide_pre.o` which goes in our archive so you get all
the needed js runtime by linking it.
Of course someone trying to use this still has to get the Python code onto the import
path, either using `--preload-file`, using Python to unpack it as a zip archive as we now
do, with zipimporter, or otherwise. They also will have to link `libpython.a` (is CPython
going to start distributing an Emscripten libpython?) and probably various other things.
We have to use a hack to inject the JavaScript code into the object files. The normal
`EM_JS` macro cannot handle arbitrary JavaScript code -- for example it fails with many
regex. Instead we manually generate write a C source file that does what we need using
`xxd`. The generated C code is similar to what `EM_JS` generates, but it uses an array
initializer rather than a string initializer for the characters avoiding the C preprocessor /
compiler's strange opinions about strings.
This adds rust toolchain into our docker image so we don't have to install it every time.
make rust command still exists, but I removed it from the docs. So now a user (who wants to build a package that requires rust) is responsible for installing rust toolchain in their build setup, which is reasonable I think.
Co-authored-by: josephrocca <1167575+josephrocca@users.noreply.github.com>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
Currently `pip` can't invoke the Pyodide build backend, but it should still be
able to build pure Python packages from source. To allow this, we need to
restore `_PYTHON_SYSCONFIGDATA_NAME` so that it isn't inherited by
subprocesses. Otherwise, they will fail to import `sysconfig` because the
subprocess doesn't have our altered `sys.path`. Annoyingly, when pip invokes
the subprocess it seems to pass `_PYTHON_SYSCONFIGDATA_NAME` down
but not `PYTHONPATH` so just setting the `PYTHONPATH` isn't good enough.
This uses sed to insert /* webpackIgnore: true */ comments into pyodide.js.
This resolves#3087. I also enabled a check that a simple webpack config
builds without warnings. In the future it would be good to add a test that it
also runs without error.
This PR splits package test in CI so that no-numpy-dependents packages can be tested earlier. In detail, pytest will now save test results into its cache directory, and if --skip-passed option is given, it will skip previously successful tests.
This adds a step that generates an xbuildenv to the circleci CI and
deploys it to github releases when we make a release. I also updated
the url in download_xbuildenv to point to github releases.
I guess there has to be some trial and error with these. Since we
can't link `pyodide.asm.js` in the deploy docker image, we do it in
a separate step after `build-packages`. Because we have hard
coded `dist` everywhere, it's a little bit awkward building into
`dist-debug`. I do some rearrangements with `cp` and `mv` to hack
around this.
Roman Yurchak