We have two Python console scripts: _pyodide/console.py and pyodide/console.py.
I think there is no clear reason for them to be separated, and Sphinx autodoc often
complains about duplicate symbols. This PR merges them into pyodide/console.py.
For reasons that are a bit beyond me, `--host` and `PLATFORM_TRIPLET`
seem to be independent, in particular we've had an empty
`PLATFORM_TRIPLET`. This is unfortunate because `PLATFORM_TRIPLET`
is used to generate the SOABI config variable which in turn is used
to decide whether a .so file is a good match for loading. We'd like
for linux Pythons not to try to import emscripten .so files (it
raises `ImportError: some_file.so: invalid ELF header`). Similarly,
we'd like to avoid attempting to load linux .so files in wasm. These
platform tags are our friends.
Anyways, this PR sets `PLATFORM_TRIPLET` and ensures that .so files
built by pywasmcross are tagged with our SOABI tag.
I moved the .so file renaming from pywasmcross to buildpkg just
before running the post script. That is a better place to put it in
case the package wants to look at the .so file after linking it. It
might be surprised that we moved it.
I also improved the error message if we try to `loadWebAssemblyModule`
something that is actually say a Linux .so file and updated get_dynlibs
to filter out .so files that have an incompatible abi tag.
Without this PR,
```js
let f = pyodide.globals.get("some_async_function");
setTimeout(f, 100);
```
doesn't work because `setTimeout` calls `f` which returns a coroutine
which is left unscheduled and so the actual work in `f` is never executed.
This is surprising to people, see for instance
https://github.com/pyodide/pyodide/discussions/2229.
This changes the behavior to automatically schedule all coroutines created
from async functions called from Javascript so that async functions can be
used as Javascript event handlers.
Old usage is still accepted but causes a deprecation warning, saying
we will remove it in v0.21. Similar to #2300. I am planning to add
an option to do wasm compilation in a subsequent PR.
I also did some cleanup from #2300 and added tests for the deprecation
warnings.
When I added unpack_buffer_archive, in code review people said
it was redundant with unpack_buffer and they should be merged.
I said merging was too annoying. They were right. This merges
the functions together into a function with a 38 line docstring
and a 18 line implementation.
With newer versions of emscripten, linker errors surface eariler.
This makes it easier to find function pointer cast errors without
having to execute the bad code path -- the errors happen when the
wasm modules are linked (at load time in the browser)
Anyways, this fixes more linker errors. Mostly the problems have
to do with LAPACK functions that take string arguments. Most
LAPACK functions that take string arguments use them as enums and
only care about the first character of the string. Because of the
way that f2c works, we need to replace these strings with the ascii
code of the first character so we should replace:
sTRSV( 'UPPER', 'NOTRANS', 'NONUNIT', J, H, LDH, Y, 1 )
==>
CALL sTRSV( 85, 78, 78, J, H, LDH, Y, 1 )
where 85 and 78 are the ascii codes of U and N. Various character
variables are subbed into being integer variables. The two
functions `ilaenv` and `xerbla` expect actual C strings as an
argument, but it is very annoying to produce C strings so instead
I added wrapper functions ilaenvf2c and xerblaf2c to clapack and
instead of calling ilaenv and xerbla we call the f2c versions.
This is for future-proofing in case we decide to add more options at
some point. Old usage is still accepted but causes a deprecation warning,
saying we will remove in v0.21.
The goal of this commit is to add the capability of skipping specific tests by
glob pattern. Many core test sets have just one test that uses subprocess or
threads. Others have a few tests that are failing due to upstream problems in
emscripten. I went through and switched to skipping specific tests in as many
cases as I could. In modules where most or all tests use threading or
subprocess, I gave up and put a comment saying something like 35/50 tests fork
and I didn't want to bother to salvage the remaining ones.
In order to accommodate using patterns to skip specific tests (and also extra
comments) I switched the format of python_tests to a yaml file. This gives us
significantly more flexibility in the structure of the file and allows us to use
ruamel.yaml to update it. We gain a lot flexibility while reducing the length of
make_test_lists.py modestly from 108 to 78 loc.
In the future, hopefully we should be able to make fixes by deleting one skip
pattern and looking into why that particular test fails.
On rare occasions, a Javascript object that isn't an error gets thrown.
This PR ensures that we are handling this situation without anything going
too far astray.
This is a followup to #2236.
This resolves#2189.
> build isolation would be a bit difficult to use in our case, as for instance
> when building scipy we need the patched numpy on the host and not the numpy
> version specified in pyproject.toml (which would be unpatched)
This is indeed the case, certain packages cannot be isolated. My strategy is to
make a list of packages that shouldn't be isolated and add symlinks from the
isolated build environment into the `.artifacts` directory to "unisolate" them.
Then we remove the unisolated package requirements from the list of packages to
install, in case pesky constraints aren't satisfied. In particular, packages
that expect to be used with `pypa/build` often feel free to put very specific
constraints on their build dependencies (often asking them to be == to a
particular version). Specific version constraints is good for build
reproducibility and with build isolation doesn't cost anything. So we just
ignore the constraints. Hopefully nothing goes wrong.
In particular, any package that does stuff both at build time and at runtime and
requires synchronization between the build time and run time environments needs
the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy
and scipy. pycparser needs to be unisolated because it is a dependency of cffi.
Currently I have also unisolated pythran and cython, though these are build time
only tools and do not really need to be unisolated. Cython I unisolated
specifically because numcodecs needs it but it isn't in the numcodecs build
dependencies. Pythran I unisolated because of a problem with the scipy build
which I don't fully understand (some problem with long double feature
detection).
If no indexURL is provided, we throw and catch an error and
then use ErrorStackParser to calculate where pyodide.js has
been loaded from. Resolves#2290.
Question: But getting the URL from error stack trace is well... really
hacky. Can't we use
[`document.currentScript`](https://developer.mozilla.org/en-US/docs/Web/API/Document/currentScript)
or
[`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import.meta)
instead?
Answer: `document.currentScript` works for the browser main thread.
`import.meta` works for es6 modules. In a classic webworker, I think there
is no approach that works. Also we would need some third approach for node
when loading a commonjs module using `require`. On the other hand, this
stack trace approach works for every case without any feature detection
code.
This makes it so that buildpkg generates build.log on its own
even when it isn't run from buildall. This is helpful because
when working on a specific package it is nicer to run buildpkg
directly so you can see the build on the console. However, it
is frustrating when you want to grep the build log for instance
that there isn't a build log.
This tees stdout into a build log from insidde buildpkg. It also
fixes the issue with overwriting build.log when nothing needs to
be done (though that issue I think was also fixed in December by
calling needs_build from buildall). Now buildall can just redirect
the subprocess output to /dev/null.
This adds a find_matching_wheels function that correctly determines
which wheels the Pyodide interpreter supports according to the platform
compatibility tags spec. This should fix problems with extra host platform
wheels lying around or wheels built for different versions of Python (e.g.,
cp39 vs cp310).
Moves the logic of generating zip archive of a shared library from
`buildall` to `buildpkg` so that thread-unsafe `shutil.make_archive`
won't bother us anymore.
* Fix dangling webdriver
* Embed humor sans font
* Refactor matplotlib testing codes
* Trigger CI
* Trigger CI
* Catch error when import from js fails
* Update changelog
* Apply suggestions from the review
Having to add `emsdk/upstream/emscripten` to all the emscripten patches is a
nuisance. If at some point we want to patch a file in a different folder, we can
separate the patches into multiple folders. In particular this should make it
easy to migrate patches using `git am`, `git rebase`, and `git format-patch`.
Our package build process currently has a significant flaw: we first run setup.py, recording all compilation commands, then we rewrite these compilation commands to invoke emcc and replay them, and then we pray that the cross compiled executables ended up in the right place to go into the wheel. This is not a good strategy because the build script is allowed to implement arbitrary logic, and if it moves, renames, etc any of the output files then we lose track of them. This has repeatedly caused difficulty for us.
However, we also make no particularly significant use of the two pass approach. We can just do the simpler thing: capture the compiler commands as they occur, modify them as needed, and then run the fixed command.
I also added a patch to fix the numpy feature detection for wasm so that we don't have to include _npyconfig.h and config.h, numpy can generate them in the way it would for a native build. I opened a numpy PR that would fix the detection for us upstream:
numpy/numpy#21154
This clears the way for us to switch to using pypa/build (as @henryiii has suggested) by removing our dependence on specific setuptools behavior.
This is on top of #2238.
* fix: lock around logging and archiving portions
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* refactor: fewer locks
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Hood Chatham