This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
This adds a short helper script which shows a gzip and brotli compressed size of a file,
and uses it in CI to check compressed size of pyodide.asm.* after build in addition to
the original file size.
This PR relocates files inside `tools` directory so that `tools` directory doesn't
have to be vendored when creating xbuildenv.
- cmake, pyo3 config ==> moved to `pyodide-build`.
- venv python script ==> moved to templates, then copied to dist during build.
Resolves https://github.com/pyodide/pyodide/issues/3112
This adds a carefully designed API for controlling stdin, stdout, and stderr. It changes
the default behavior to be a bit more useful, though in doing so introduces some mild
backwards incompatibility. In particular:
1. By default, stdin reads directly from `process.stdin` in node (as before) and raises an
error if in browser (not as before).
2. By default, stdout writes directly to `process.stdout` in node (before it called console.log)
and calls console.log in browser (as before).
3. By default, stderr writes directly to `process.stderr` in node (before it called console.warn)
and calls console.warn in browser (as before).
4. In all three cases, by default isatty(stdin/stdout/stderr) is true in node and false in browser
(in the browser it used to be true).
5. As before, if you pass `stdin`, `stdout`, or `stderr` as arguments to `loadPyodide`, `isatty` of
the corresponding stream is set to false.
The stdin function is now more flexible: we now correctly handle the case where it returns an
ArrayBuffer or ArrayBufferView.
I also added 3 new functions to set streams after Pyodide is loaded which offer additional
control:
* `setStdin({stdin?, error?, isatty = false})` -- Sets the stdin function. The stdin function takes no
arguments and should return null, undefined, a string, or a buffer. Sets and `isatty(stdin)` to
`isatty` (by default `false`). If error is true, set stdin to always raise an EIO error when it is read.
* `setStdout({raw?, batched?, isatty = false})` -- If neither raw nor batched is passed, restore
default stdout behavior. If rwa is passed, the raw stdout function receives a byte which it should
interpret as a utf8 character code. Sets `isatty(stdout)` to isatty (by default `false`). If batched is
passed but not raw, it sets a batched stdout function. The stdout function receives a string and
should do something with it. In this case it ignores isatty and sets isatty(stdout) to false.
* `setStderr({raw?, batched?, isatty = false})` -- same but with stderr.
Adds a new key in meta.yaml spec: requirements/executable which specifies the list of executables required to build a package. Unlike conda, we don't build or install these executables. This key exists just to halt build earlier
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This allows us to embed extra wasm modules into the output. This is split off of #3210.
We need this for #3210 since
1. We don't want to assume that externref is supported but we need to use it as part of
the JS Promise integration proposal. To handle this we segment off the externrefs into
separate wasm modules. If externref is not supported, we load most everything
successfully but these small modules fail to load
2. Even supposing that we know that externref is supported, there is no way to write C
code such that the compiled code uses externref. It's possible using the emscripten
assembly format, but that is less well documented and harder to use for our purposes
than wat.
This PR doesn't actually change anything by itself.
This is needed to finish getting numpy tests working.
It won't work for packages that need shared libraries without some extra intervention.
ryanking's work in #3234 would help to fix this.
pip install scipy makes the command line runner extremely slow. Without scipy installed,
python -c 'print(1)' runs in about 1 second, but with it installed it takes more like 10 seconds
(time to load clapack_all.so and 111 different .so files in scipy, totaling 20 megabytes). We
have to load all of this despite the fact that we won't use any of it.
This PR makes building CMake based packages easier.
- Introduce a custom toolchain file for Pyodide. It inherits most of settings from the original Emscripten toolchain file with some modifications. Packages built by pyodide-build will automatically use that toolchain file.
- Proxy cmake in pywasmcross.py, in order to proxy other compiler toolchains.
This adds a step that generates an xbuildenv to the circleci CI and
deploys it to github releases when we make a release. I also updated
the url in download_xbuildenv to point to github releases.
We are pushing pyodide_build to PyPI as a Python package, but for now,
installing pyodide_build from PyPI (i.e. pip install pyodide_build`) is almost
useless because:
there are bunch of hard-coded paths (e.g. Path(__file__).parents[2]),
its dependencies are not specified in setup.cfg.
This PR is for mitigating this situation by removing hard-coded paths and
adding tests, and is also a preparation for our new CLI
(https://github.com/pyodide/pyodide/issues/1977).
Our package build process currently has a significant flaw: we first run setup.py, recording all compilation commands, then we rewrite these compilation commands to invoke emcc and replay them, and then we pray that the cross compiled executables ended up in the right place to go into the wheel. This is not a good strategy because the build script is allowed to implement arbitrary logic, and if it moves, renames, etc any of the output files then we lose track of them. This has repeatedly caused difficulty for us.
However, we also make no particularly significant use of the two pass approach. We can just do the simpler thing: capture the compiler commands as they occur, modify them as needed, and then run the fixed command.
I also added a patch to fix the numpy feature detection for wasm so that we don't have to include _npyconfig.h and config.h, numpy can generate them in the way it would for a native build. I opened a numpy PR that would fix the detection for us upstream:
numpy/numpy#21154
This clears the way for us to switch to using pypa/build (as @henryiii has suggested) by removing our dependence on specific setuptools behavior.
This is on top of #2238.
Christian Clauss