* chore: add some incomplete types
* chore: modernize pyproject.toml
Adding more incomplete types. About 2/3 of the way through being
able to turn on the strictness flag for it.
This adds pyodide_build command create_xbuildenv which creates a
crossbuild environment (from a copy of Pyodide where scipy has been
build) and install_xbuildenv which installs the cross build environment
into a fresh copy of Pyodide.
I successfully installed the xbuild environment into a fresh checkout of
Pyodide then built statsmodels and scikit-learn in isolation, without
building the Python interpreter, numpy, or scipy. I dumped the generated
wheels into a copy of Pyodide downloaded from CI, and was able to import
and use them as normal.
The size of the xbuild environment is 1.5 megabytes, of which 1.2 megabytes
is Python headers.
In a subsequent PR, we can update the CI to automatically upload these
to aws s3 and then install the environment from there.
This PR does the following to complete #2431:
* Calculate the Subresource integrity hash from the sha256 as explained here:
https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity
As the document explains, it is a base64 encoded string of the "binary" hash,
not the data returned by hexdigest() method.
* Implement the verification of the checksum when loading packages
This ignores the verification for NodeJS environment as the node-fetch module
doesn't support the integrity option. Node 17.5.0 has added the fetch API as
experimental, it would be prudent to come back and fix this when we are ready
to use that version.
This moves unisolation into a package key. `cross-build-env: true` means the package
is part of the cross build environment and should be unisolated. `cross-build-files`
gives a list of files that should be copied from the built copy of the package into the host
copy of the package.
This will allow us to construct a cross build environment automatically as part of building
packages. If we have these files and the Python include directory, this is sufficient for
cross-building binary packages.
* Add opencv-python
* Update comment
* Add JPEG, PNG, WEBP, ZLIB support
* Add tests for image processing
* Add more core modules
* Disable opencl
* Replace lena with baboon and add more tests
* Add file system support
* Add ffmpeg
* Add more tests
* Disable pthread in ffmpeg
* Disable canonical input processing mode in node test
* Update changelog
* Remove import test
* Allow more time in the first test
* Split out libwebp
* Fix node test
* Use a seperate CI job for opencv-python
* Fix generator
* Update changelog
* Remove protobuf package
* Try to fix CI workspace conflict
* Fix CI
* Use another CI job for generating unified packages.json
* Try to fix CI
* Fix CI again
* Disable verbose build
* Prevent from building opencv-python twice
* Persist only build artifacts
* Sepearate Cmake args into a script
* Try to reuse build packages job
* Fix CI
* Fix typo
* Fix merge conflict
* Use large resource class for package build
* Do not upload unwanted artifacts
* Do not upload unwanted artifacts
dist is both more accurate (the 'build' directory is normally where you do the build,
and normally consists of intermediate build artifacts no one cares about). dist also
occurs less frequently in the code base: after this change \bbuild\b has 466 matches,
whereas \bdist\b has 101 matches. build has 1072 matches whereas dist has 362.
We are pushing pyodide_build to PyPI as a Python package, but for now,
installing pyodide_build from PyPI (i.e. pip install pyodide_build`) is almost
useless because:
there are bunch of hard-coded paths (e.g. Path(__file__).parents[2]),
its dependencies are not specified in setup.cfg.
This PR is for mitigating this situation by removing hard-coded paths and
adding tests, and is also a preparation for our new CLI
(https://github.com/pyodide/pyodide/issues/1977).
Fixing a minor issue - pyodide-build doesn't work out of the box, it's missing two requirements.
Also updating a typo in the docs, and minor pre-commit bump.
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.
Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.
I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.
I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
With newer versions of emscripten, linker errors surface eariler.
This makes it easier to find function pointer cast errors without
having to execute the bad code path -- the errors happen when the
wasm modules are linked (at load time in the browser)
Anyways, this fixes more linker errors. Mostly the problems have
to do with LAPACK functions that take string arguments. Most
LAPACK functions that take string arguments use them as enums and
only care about the first character of the string. Because of the
way that f2c works, we need to replace these strings with the ascii
code of the first character so we should replace:
sTRSV( 'UPPER', 'NOTRANS', 'NONUNIT', J, H, LDH, Y, 1 )
==>
CALL sTRSV( 85, 78, 78, J, H, LDH, Y, 1 )
where 85 and 78 are the ascii codes of U and N. Various character
variables are subbed into being integer variables. The two
functions `ilaenv` and `xerbla` expect actual C strings as an
argument, but it is very annoying to produce C strings so instead
I added wrapper functions ilaenvf2c and xerblaf2c to clapack and
instead of calling ilaenv and xerbla we call the f2c versions.
This resolves#2189.
> build isolation would be a bit difficult to use in our case, as for instance
> when building scipy we need the patched numpy on the host and not the numpy
> version specified in pyproject.toml (which would be unpatched)
This is indeed the case, certain packages cannot be isolated. My strategy is to
make a list of packages that shouldn't be isolated and add symlinks from the
isolated build environment into the `.artifacts` directory to "unisolate" them.
Then we remove the unisolated package requirements from the list of packages to
install, in case pesky constraints aren't satisfied. In particular, packages
that expect to be used with `pypa/build` often feel free to put very specific
constraints on their build dependencies (often asking them to be == to a
particular version). Specific version constraints is good for build
reproducibility and with build isolation doesn't cost anything. So we just
ignore the constraints. Hopefully nothing goes wrong.
In particular, any package that does stuff both at build time and at runtime and
requires synchronization between the build time and run time environments needs
the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy
and scipy. pycparser needs to be unisolated because it is a dependency of cffi.
Currently I have also unisolated pythran and cython, though these are build time
only tools and do not really need to be unisolated. Cython I unisolated
specifically because numcodecs needs it but it isn't in the numcodecs build
dependencies. Pythran I unisolated because of a problem with the scipy build
which I don't fully understand (some problem with long double feature
detection).
This makes it so that buildpkg generates build.log on its own
even when it isn't run from buildall. This is helpful because
when working on a specific package it is nicer to run buildpkg
directly so you can see the build on the console. However, it
is frustrating when you want to grep the build log for instance
that there isn't a build log.
This tees stdout into a build log from insidde buildpkg. It also
fixes the issue with overwriting build.log when nothing needs to
be done (though that issue I think was also fixed in December by
calling needs_build from buildall). Now buildall can just redirect
the subprocess output to /dev/null.
This adds a find_matching_wheels function that correctly determines
which wheels the Pyodide interpreter supports according to the platform
compatibility tags spec. This should fix problems with extra host platform
wheels lying around or wheels built for different versions of Python (e.g.,
cp39 vs cp310).
Moves the logic of generating zip archive of a shared library from
`buildall` to `buildpkg` so that thread-unsafe `shutil.make_archive`
won't bother us anymore.
Our package build process currently has a significant flaw: we first run setup.py, recording all compilation commands, then we rewrite these compilation commands to invoke emcc and replay them, and then we pray that the cross compiled executables ended up in the right place to go into the wheel. This is not a good strategy because the build script is allowed to implement arbitrary logic, and if it moves, renames, etc any of the output files then we lose track of them. This has repeatedly caused difficulty for us.
However, we also make no particularly significant use of the two pass approach. We can just do the simpler thing: capture the compiler commands as they occur, modify them as needed, and then run the fixed command.
I also added a patch to fix the numpy feature detection for wasm so that we don't have to include _npyconfig.h and config.h, numpy can generate them in the way it would for a native build. I opened a numpy PR that would fix the detection for us upstream:
numpy/numpy#21154
This clears the way for us to switch to using pypa/build (as @henryiii has suggested) by removing our dependence on specific setuptools behavior.
This is on top of #2238.
* fix: lock around logging and archiving portions
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* refactor: fewer locks
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
This reduces the text output when an error is hit, removing the tracebacks for known errors. Tracebacks are for unknown/unexpected errors - if you expect an error, it's better to print out an actual error message. I've also made the patch applying a bit more verbose in case it helps, as it sometimes does.
Also improvements to `BashRunner`: it no longer hangs when an error is raised and it is now a context manager
Henry Schreiner