Adds a new key in meta.yaml spec: requirements/executable which specifies the list of executables required to build a package. Unlike conda, we don't build or install these executables. This key exists just to halt build earlier
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This allows us to embed extra wasm modules into the output. This is split off of #3210.
We need this for #3210 since
1. We don't want to assume that externref is supported but we need to use it as part of
the JS Promise integration proposal. To handle this we segment off the externrefs into
separate wasm modules. If externref is not supported, we load most everything
successfully but these small modules fail to load
2. Even supposing that we know that externref is supported, there is no way to write C
code such that the compiled code uses externref. It's possible using the emscripten
assembly format, but that is less well documented and harder to use for our purposes
than wat.
This PR doesn't actually change anything by itself.
This is needed to finish getting numpy tests working.
It won't work for packages that need shared libraries without some extra intervention.
ryanking's work in #3234 would help to fix this.
pip install scipy makes the command line runner extremely slow. Without scipy installed,
python -c 'print(1)' runs in about 1 second, but with it installed it takes more like 10 seconds
(time to load clapack_all.so and 111 different .so files in scipy, totaling 20 megabytes). We
have to load all of this despite the fact that we won't use any of it.
This PR makes building CMake based packages easier.
- Introduce a custom toolchain file for Pyodide. It inherits most of settings from the original Emscripten toolchain file with some modifications. Packages built by pyodide-build will automatically use that toolchain file.
- Proxy cmake in pywasmcross.py, in order to proxy other compiler toolchains.
This adds a step that generates an xbuildenv to the circleci CI and
deploys it to github releases when we make a release. I also updated
the url in download_xbuildenv to point to github releases.
We are pushing pyodide_build to PyPI as a Python package, but for now,
installing pyodide_build from PyPI (i.e. pip install pyodide_build`) is almost
useless because:
there are bunch of hard-coded paths (e.g. Path(__file__).parents[2]),
its dependencies are not specified in setup.cfg.
This PR is for mitigating this situation by removing hard-coded paths and
adding tests, and is also a preparation for our new CLI
(https://github.com/pyodide/pyodide/issues/1977).
Our package build process currently has a significant flaw: we first run setup.py, recording all compilation commands, then we rewrite these compilation commands to invoke emcc and replay them, and then we pray that the cross compiled executables ended up in the right place to go into the wheel. This is not a good strategy because the build script is allowed to implement arbitrary logic, and if it moves, renames, etc any of the output files then we lose track of them. This has repeatedly caused difficulty for us.
However, we also make no particularly significant use of the two pass approach. We can just do the simpler thing: capture the compiler commands as they occur, modify them as needed, and then run the fixed command.
I also added a patch to fix the numpy feature detection for wasm so that we don't have to include _npyconfig.h and config.h, numpy can generate them in the way it would for a native build. I opened a numpy PR that would fix the detection for us upstream:
numpy/numpy#21154
This clears the way for us to switch to using pypa/build (as @henryiii has suggested) by removing our dependence on specific setuptools behavior.
This is on top of #2238.
The installation is very fast, and this avoids having two copies of node
around. In particular, this avoids the need to monkey-patch uglify-js to
use the system node.
This can be further streamlined when #792 is merged
This resolves#935. I also changed the lint make recipe to use find which is more resilient to changes in directory structure and doesn't output those annoying No such file or directory messages to stdout.
I changed the name of the apply-lints recipe to apply-lint for consistency, and I updated it to only apply lints to files with staged changes.
Modified ``Makefile`` and ``packages/Makefile`` to get ``PYODIDE_PACKAGE_ABI`` from ``Makefile.envs``
Made ``checkABI`` as a public API function.
Modified ``file_packager.py`` to check for the compatible ABI number.
Defined ABI number with which pyodide is built in Makefile.envs
Accepted ABI number in ``file_packager.py`` as an argument.
Modified ``buildpkg.py`` and ``buildall.py`` to account for the new argument.
Added the corresponding ``test_different_ABI`` test.
Gyeongjae Choi