This moves unisolation into a package key. `cross-build-env: true` means the package
is part of the cross build environment and should be unisolated. `cross-build-files`
gives a list of files that should be copied from the built copy of the package into the host
copy of the package.
This will allow us to construct a cross build environment automatically as part of building
packages. If we have these files and the Python include directory, this is sufficient for
cross-building binary packages.
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.
Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.
I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.
I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
With newer versions of emscripten, linker errors surface eariler.
This makes it easier to find function pointer cast errors without
having to execute the bad code path -- the errors happen when the
wasm modules are linked (at load time in the browser)
Anyways, this fixes more linker errors. Mostly the problems have
to do with LAPACK functions that take string arguments. Most
LAPACK functions that take string arguments use them as enums and
only care about the first character of the string. Because of the
way that f2c works, we need to replace these strings with the ascii
code of the first character so we should replace:
sTRSV( 'UPPER', 'NOTRANS', 'NONUNIT', J, H, LDH, Y, 1 )
==>
CALL sTRSV( 85, 78, 78, J, H, LDH, Y, 1 )
where 85 and 78 are the ascii codes of U and N. Various character
variables are subbed into being integer variables. The two
functions `ilaenv` and `xerbla` expect actual C strings as an
argument, but it is very annoying to produce C strings so instead
I added wrapper functions ilaenvf2c and xerblaf2c to clapack and
instead of calling ilaenv and xerbla we call the f2c versions.
This resolves#2189.
> build isolation would be a bit difficult to use in our case, as for instance
> when building scipy we need the patched numpy on the host and not the numpy
> version specified in pyproject.toml (which would be unpatched)
This is indeed the case, certain packages cannot be isolated. My strategy is to
make a list of packages that shouldn't be isolated and add symlinks from the
isolated build environment into the `.artifacts` directory to "unisolate" them.
Then we remove the unisolated package requirements from the list of packages to
install, in case pesky constraints aren't satisfied. In particular, packages
that expect to be used with `pypa/build` often feel free to put very specific
constraints on their build dependencies (often asking them to be == to a
particular version). Specific version constraints is good for build
reproducibility and with build isolation doesn't cost anything. So we just
ignore the constraints. Hopefully nothing goes wrong.
In particular, any package that does stuff both at build time and at runtime and
requires synchronization between the build time and run time environments needs
the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy
and scipy. pycparser needs to be unisolated because it is a dependency of cffi.
Currently I have also unisolated pythran and cython, though these are build time
only tools and do not really need to be unisolated. Cython I unisolated
specifically because numcodecs needs it but it isn't in the numcodecs build
dependencies. Pythran I unisolated because of a problem with the scipy build
which I don't fully understand (some problem with long double feature
detection).
Split build packages into a step for everything up to and including numpy and a second step for numpy and its dependencies.
Intended to prevent timeouts.
The installation is very fast, and this avoids having two copies of node
around. In particular, this avoids the need to monkey-patch uglify-js to
use the system node.
This can be further streamlined when #792 is merged
1. Split long shell command into multiple commands
2. Make BINARYEN_VERSION a variable instead of hardcoding in Makefile
3. Set Makefile.envs and patches as dependencies
* Update to use sdk-fastcomp-tag-<version> for emscripten, as sdk-tag-<version> is no longer available upstream.
* Update code to use new path
* Replace Circle-CI CPU_CORES hack with environment variables (now supported upstream)
Co-authored-by: AndyLockhart <AndyLockhart@users.noreply.github.com>
Modified ``Makefile`` and ``packages/Makefile`` to get ``PYODIDE_PACKAGE_ABI`` from ``Makefile.envs``
Made ``checkABI`` as a public API function.
Modified ``file_packager.py`` to check for the compatible ABI number.
Defined ABI number with which pyodide is built in Makefile.envs
Accepted ABI number in ``file_packager.py`` as an argument.
Modified ``buildpkg.py`` and ``buildall.py`` to account for the new argument.
Added the corresponding ``test_different_ABI`` test.
Hood Chatham