This adds pyodide_build command create_xbuildenv which creates a
crossbuild environment (from a copy of Pyodide where scipy has been
build) and install_xbuildenv which installs the cross build environment
into a fresh copy of Pyodide.
I successfully installed the xbuild environment into a fresh checkout of
Pyodide then built statsmodels and scikit-learn in isolation, without
building the Python interpreter, numpy, or scipy. I dumped the generated
wheels into a copy of Pyodide downloaded from CI, and was able to import
and use them as normal.
The size of the xbuild environment is 1.5 megabytes, of which 1.2 megabytes
is Python headers.
In a subsequent PR, we can update the CI to automatically upload these
to aws s3 and then install the environment from there.
This moves unisolation into a package key. `cross-build-env: true` means the package
is part of the cross build environment and should be unisolated. `cross-build-files`
gives a list of files that should be copied from the built copy of the package into the host
copy of the package.
This will allow us to construct a cross build environment automatically as part of building
packages. If we have these files and the Python include directory, this is sufficient for
cross-building binary packages.
We are pushing pyodide_build to PyPI as a Python package, but for now,
installing pyodide_build from PyPI (i.e. pip install pyodide_build`) is almost
useless because:
there are bunch of hard-coded paths (e.g. Path(__file__).parents[2]),
its dependencies are not specified in setup.cfg.
This PR is for mitigating this situation by removing hard-coded paths and
adding tests, and is also a preparation for our new CLI
(https://github.com/pyodide/pyodide/issues/1977).
This resolves#2189.
> build isolation would be a bit difficult to use in our case, as for instance
> when building scipy we need the patched numpy on the host and not the numpy
> version specified in pyproject.toml (which would be unpatched)
This is indeed the case, certain packages cannot be isolated. My strategy is to
make a list of packages that shouldn't be isolated and add symlinks from the
isolated build environment into the `.artifacts` directory to "unisolate" them.
Then we remove the unisolated package requirements from the list of packages to
install, in case pesky constraints aren't satisfied. In particular, packages
that expect to be used with `pypa/build` often feel free to put very specific
constraints on their build dependencies (often asking them to be == to a
particular version). Specific version constraints is good for build
reproducibility and with build isolation doesn't cost anything. So we just
ignore the constraints. Hopefully nothing goes wrong.
In particular, any package that does stuff both at build time and at runtime and
requires synchronization between the build time and run time environments needs
the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy
and scipy. pycparser needs to be unisolated because it is a dependency of cffi.
Currently I have also unisolated pythran and cython, though these are build time
only tools and do not really need to be unisolated. Cython I unisolated
specifically because numcodecs needs it but it isn't in the numcodecs build
dependencies. Pythran I unisolated because of a problem with the scipy build
which I don't fully understand (some problem with long double feature
detection).
Hood Chatham