This PR splits package test in CI so that no-numpy-dependents packages can be tested earlier. In detail, pytest will now save test results into its cache directory, and if --skip-passed option is given, it will skip previously successful tests.
This adds a step that generates an xbuildenv to the circleci CI and
deploys it to github releases when we make a release. I also updated
the url in download_xbuildenv to point to github releases.
In the old instructions to build package it was written that you should
run pyodide_build buildall from the packages/<package-name> folder.
However, the command that comes next assumes you are in the root
(like the previous commands, i.e. to create the meta.yml template).
Fixed by removing the reference to the folder.
This completes the out of tree build CLI. This PR is paired up with:
numpy/numpy#21895
I have also successfully built scikit-learn, statsmodels, pandas, and
astropy with this.
The last thing we need to do after this is set up deployment of the
cross build environment. We can deploy one version to s3 for each
tagged commit. I will do that in a separate PR after this is merged.
The functions inside `hiwire.h` and `hiwire.c` were defined in some inconsistent order. This PR shuffle things around so that they are clearly divided into multiple groups, one for each "namespace" (in the C sense):
- `hiwire_*`
- `JsObject_*`
- `JsArray_*`
- `JsString_*`
- `JsMap_*`
- `JaString_*`
This makes it easier to navigate inside the file, and will make it easier to add future functions "at the right place".
We shouldn't export symbols with `__attribute__((visibility("hidden")))`. Numpy has
a test to check that symbols with this attribute aren't exported. This fixes that test.
Symlinking cc to pywasmcross.py only works if pywasmcross.py has
execute permissions. When we install the package with pip, it will not
set execute permissions on pywasmcross.py. It does set execute flags
on entrypoints. Thus, define an entrypoint called _pywasmcross which
calls pywasmcross.main. If a script called _pywasmcross exists, we are
using an out-of-tree install so symlink cc to _pywasmcross. Otherwise,
we should be in tree and pywasmcross.py should have the execute flag
set, so symlink cc to pywasmcross.py.
On the other side, if __main__.__file__ is in a folder named pyodide_build
or bin we are being invoked normally, otherwise we are being invoked via
a symlink.
This is more WIP on create_xbuildenv. I am including some of the WASM_LIB_DIR
but not all of it to keep size in check. The current cross build environment that we
would upload/download is 20mb.
To use this, we need an extra CLI entrypoint which I am working on.
This is on top of #2734. Other build environments cannot be trusted to pass our
environment variables down #2736. We store key variables into a json file in a
temporary directory with the symlinks. If these variables are not in `os.environ`
we load them from disk.
In addition to the added generality, this also has the advantage that it makes it
much easier to debug specific compiler invocations from the build script because
the environment is stored to disk (that is, if we override the deletion of the
temporary directory by manually adjusting the code).
I guess there has to be some trial and error with these. Since we
can't link `pyodide.asm.js` in the deploy docker image, we do it in
a separate step after `build-packages`. Because we have hard
coded `dist` everywhere, it's a little bit awkward building into
`dist-debug`. I do some rearrangements with `cp` and `mv` to hack
around this.
This script will run with the target environment variables and
sysconfigdata and with the pywasmcross compiler symlinks.
Any changes to the environment will persist to the main build
step but will not be seen in the post step (or anything else
done outside of the cross build environment). The working
directory for this script is the source directory.
If the wheel is an Emscripten wheel of the wrong version, give the expected
version and the wheel version. Otherwise, complain about platform mismatch.
Roman Yurchak