This PR makes building CMake based packages easier.
- Introduce a custom toolchain file for Pyodide. It inherits most of settings from the original Emscripten toolchain file with some modifications. Packages built by pyodide-build will automatically use that toolchain file.
- Proxy cmake in pywasmcross.py, in order to proxy other compiler toolchains.
Unvendor the standard library sqlite3 to reduce the size of the main module. It reduces the size of pyodide.asm.wasm around ~1.4MB.
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This is more WIP on create_xbuildenv. I am including some of the WASM_LIB_DIR
but not all of it to keep size in check. The current cross build environment that we
would upload/download is 20mb.
To use this, we need an extra CLI entrypoint which I am working on.
Yesterday's nightly includes rust-lang/rust#98149 which allows us to remove the
PIC setting. This also means we won't ever have to put -Zbuild-std back in.
This PR adds the Robot Raconteur Pyodide package, which is a modified version
of Robot Raconteur designed to run in the Pyodide environment, using WebSockets
for communication. See robotraconteur/robotraconteur for the full version of the
package. Robot Raconteur is used to communicate with robots and other automation
components, and has a large library of drivers and packages: robotraconteur/robotraconteur-directory.
It is used as the bases for PyRI, a robotics programming environment:
https://github.com/pyri-project/pyri-core/blob/master/README.md. Using a bridge, this package can
also be used to communicate with ROS2 networks: robotraconteur-contrib/robotraconteur_ros2_bridge
This drops the Rust patch and "Throw away errors in minify wasm_js" which
I suspect was fixed by an emscripten update (I'm not sure how long ago). It
replaces the two remaining emscripten patches with new versions that match
the versions I am upstreaming.
This enables WASM_BIGINT while maintaining (hypothetical) Safari 14 support
by shimming BigInt64Array and BigUint64Array if they are missing. I think the
last time we tried to enable WASM_BIGINT was before #2019 so our chances
are significantly better this time.
This will fix dynamic linking bugs and yields a minor reduction in code size.
sysconfig.py uses the environment variable `_PYTHON_SYSCONFIGDATA_NAME`
to decide where to look for the sysconfig data file with info about the compile target.
We also need to separately insure that our sysconfig data file is on the path. We
don't want the rest of our target stdlib on the path, so I made an extra sysconfigdata
folder, copied the sysconfig data into it, and put it on the path.
BrowserFS can mount custom filesystems into Emscripten.
However it requires the PATH and ERRNO_CODES exports from
Emscripten in addition to FS.
This exports `PATH` and `ERRNO_CODES` from `Module` into the `pyodide`
Javascript API so they can be used with BrowserFS.
This moves unisolation into a package key. `cross-build-env: true` means the package
is part of the cross build environment and should be unisolated. `cross-build-files`
gives a list of files that should be copied from the built copy of the package into the host
copy of the package.
This will allow us to construct a cross build environment automatically as part of building
packages. If we have these files and the Python include directory, this is sufficient for
cross-building binary packages.
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.
Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.
I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.
I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
With newer versions of emscripten, linker errors surface eariler.
This makes it easier to find function pointer cast errors without
having to execute the bad code path -- the errors happen when the
wasm modules are linked (at load time in the browser)
Anyways, this fixes more linker errors. Mostly the problems have
to do with LAPACK functions that take string arguments. Most
LAPACK functions that take string arguments use them as enums and
only care about the first character of the string. Because of the
way that f2c works, we need to replace these strings with the ascii
code of the first character so we should replace:
sTRSV( 'UPPER', 'NOTRANS', 'NONUNIT', J, H, LDH, Y, 1 )
==>
CALL sTRSV( 85, 78, 78, J, H, LDH, Y, 1 )
where 85 and 78 are the ascii codes of U and N. Various character
variables are subbed into being integer variables. The two
functions `ilaenv` and `xerbla` expect actual C strings as an
argument, but it is very annoying to produce C strings so instead
I added wrapper functions ilaenvf2c and xerblaf2c to clapack and
instead of calling ilaenv and xerbla we call the f2c versions.
This resolves#2189.
> build isolation would be a bit difficult to use in our case, as for instance
> when building scipy we need the patched numpy on the host and not the numpy
> version specified in pyproject.toml (which would be unpatched)
This is indeed the case, certain packages cannot be isolated. My strategy is to
make a list of packages that shouldn't be isolated and add symlinks from the
isolated build environment into the `.artifacts` directory to "unisolate" them.
Then we remove the unisolated package requirements from the list of packages to
install, in case pesky constraints aren't satisfied. In particular, packages
that expect to be used with `pypa/build` often feel free to put very specific
constraints on their build dependencies (often asking them to be == to a
particular version). Specific version constraints is good for build
reproducibility and with build isolation doesn't cost anything. So we just
ignore the constraints. Hopefully nothing goes wrong.
In particular, any package that does stuff both at build time and at runtime and
requires synchronization between the build time and run time environments needs
the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy
and scipy. pycparser needs to be unisolated because it is a dependency of cffi.
Currently I have also unisolated pythran and cython, though these are build time
only tools and do not really need to be unisolated. Cython I unisolated
specifically because numcodecs needs it but it isn't in the numcodecs build
dependencies. Pythran I unisolated because of a problem with the scipy build
which I don't fully understand (some problem with long double feature
detection).
Hood Chatham