Adds a new key in meta.yaml spec: requirements/executable which specifies the list of executables required to build a package. Unlike conda, we don't build or install these executables. This key exists just to halt build earlier
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This script will run with the target environment variables and
sysconfigdata and with the pywasmcross compiler symlinks.
Any changes to the environment will persist to the main build
step but will not be seen in the post step (or anything else
done outside of the cross build environment). The working
directory for this script is the source directory.
Change shared library installation directory from /lib/python3.10/site-packages/ to /usr/lib.
#2686 and this PR improve how we load shared libraries. The future goal
of these PRs is to load shared libraries on demand, instead of pre-loading
all shared libraries, which will reduce initial package load time.
Frequently when updating Python or Emscripten, a handlful of packages
stop working. Because of the way that the CI works, these prevent
other packages from being built or tested. However, packages with
many dependents are currently annoying to disable. This adds a key
to meta.yaml "disable: true" that turns off a package.
I also fixed the "!package" and no-numpy-dependents to be transitive.
We use graphlib to sort the packages topologically, then we traverse
the packages once in build order to locate all the packages which
transitively depend on disabled packages. Then we traverse in reverse
build order to locate all packages that are dependencies of non-disabled
requested packages.
* chore: add some incomplete types
* chore: modernize pyproject.toml
Adding more incomplete types. About 2/3 of the way through being
able to turn on the strictness flag for it.
This adds pyodide_build command create_xbuildenv which creates a
crossbuild environment (from a copy of Pyodide where scipy has been
build) and install_xbuildenv which installs the cross build environment
into a fresh copy of Pyodide.
I successfully installed the xbuild environment into a fresh checkout of
Pyodide then built statsmodels and scikit-learn in isolation, without
building the Python interpreter, numpy, or scipy. I dumped the generated
wheels into a copy of Pyodide downloaded from CI, and was able to import
and use them as normal.
The size of the xbuild environment is 1.5 megabytes, of which 1.2 megabytes
is Python headers.
In a subsequent PR, we can update the CI to automatically upload these
to aws s3 and then install the environment from there.
This moves unisolation into a package key. `cross-build-env: true` means the package
is part of the cross build environment and should be unisolated. `cross-build-files`
gives a list of files that should be copied from the built copy of the package into the host
copy of the package.
This will allow us to construct a cross build environment automatically as part of building
packages. If we have these files and the Python include directory, this is sufficient for
cross-building binary packages.
Our package build process currently has a significant flaw: we first run setup.py, recording all compilation commands, then we rewrite these compilation commands to invoke emcc and replay them, and then we pray that the cross compiled executables ended up in the right place to go into the wheel. This is not a good strategy because the build script is allowed to implement arbitrary logic, and if it moves, renames, etc any of the output files then we lose track of them. This has repeatedly caused difficulty for us.
However, we also make no particularly significant use of the two pass approach. We can just do the simpler thing: capture the compiler commands as they occur, modify them as needed, and then run the fixed command.
I also added a patch to fix the numpy feature detection for wasm so that we don't have to include _npyconfig.h and config.h, numpy can generate them in the way it would for a native build. I opened a numpy PR that would fix the detection for us upstream:
numpy/numpy#21154
This clears the way for us to switch to using pypa/build (as @henryiii has suggested) by removing our dependence on specific setuptools behavior.
This is on top of #2238.
Gyeongjae Choi