libgl is marked .PHONY and is a dependency for the link step which forces us to always redo the link
step even though it's completely not necessary. I moved embuilder build libgl into the link step
since that's the only place where it's needed.
dist/python_stdlib.zip and dist/pyodide-lock.json both declare a dependency on pyodide_build but
there is never a file of that name so they always have to run. Instead, run make pyodide_build so
that they can reuse the logic but they don't always run if their other deps are satisfied.
Now that dist/python_stdlib.zip doesn't always run it needs to rely on all files in src/py.
Make pyodide_build only invoke pip install if pyodide_build is not already installed. Pip generates
a lot of noise and takes a while so skipping it when we don't need it helps a lot.
Make dist/webworker.js and friends not phony since they stopped needing to be when PYODIDE_BASE_URL
was removed from them.
It is weird that test_run_docker is inside pyodide-build. This PR moves the
run_docker test file to tools directory. Maybe it is not the best place to put
this test file, but it is at least better than pyodide-build.
I also reordered prettier to come last since it's the slowest. I switched from
using black to ruff-format which says its defaults are nearly the same as black.
Since the release of Cython 3 in July 2023, many packages have been updated to
either set an upper bound "Cython < 3", to support both Cython 0.29.x and 3.0, or
to require Cython 3.
Here we remove the global constraint Cython < 3 added in #4001 and update
individual packages if necessary.
Resolves pyodide#4126.
If someone builds emsdk outside of the docker image and then attempts to use it
inside, there will be an error. This detects it and tells people to clean emsdk.
This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
This adds a short helper script which shows a gzip and brotli compressed size of a file,
and uses it in CI to check compressed size of pyodide.asm.* after build in addition to
the original file size.
This PR relocates files inside `tools` directory so that `tools` directory doesn't
have to be vendored when creating xbuildenv.
- cmake, pyo3 config ==> moved to `pyodide-build`.
- venv python script ==> moved to templates, then copied to dist during build.
Resolves https://github.com/pyodide/pyodide/issues/3112
This adds a carefully designed API for controlling stdin, stdout, and stderr. It changes
the default behavior to be a bit more useful, though in doing so introduces some mild
backwards incompatibility. In particular:
1. By default, stdin reads directly from `process.stdin` in node (as before) and raises an
error if in browser (not as before).
2. By default, stdout writes directly to `process.stdout` in node (before it called console.log)
and calls console.log in browser (as before).
3. By default, stderr writes directly to `process.stderr` in node (before it called console.warn)
and calls console.warn in browser (as before).
4. In all three cases, by default isatty(stdin/stdout/stderr) is true in node and false in browser
(in the browser it used to be true).
5. As before, if you pass `stdin`, `stdout`, or `stderr` as arguments to `loadPyodide`, `isatty` of
the corresponding stream is set to false.
The stdin function is now more flexible: we now correctly handle the case where it returns an
ArrayBuffer or ArrayBufferView.
I also added 3 new functions to set streams after Pyodide is loaded which offer additional
control:
* `setStdin({stdin?, error?, isatty = false})` -- Sets the stdin function. The stdin function takes no
arguments and should return null, undefined, a string, or a buffer. Sets and `isatty(stdin)` to
`isatty` (by default `false`). If error is true, set stdin to always raise an EIO error when it is read.
* `setStdout({raw?, batched?, isatty = false})` -- If neither raw nor batched is passed, restore
default stdout behavior. If rwa is passed, the raw stdout function receives a byte which it should
interpret as a utf8 character code. Sets `isatty(stdout)` to isatty (by default `false`). If batched is
passed but not raw, it sets a batched stdout function. The stdout function receives a string and
should do something with it. In this case it ignores isatty and sets isatty(stdout) to false.
* `setStderr({raw?, batched?, isatty = false})` -- same but with stderr.
Adds a new key in meta.yaml spec: requirements/executable which specifies the list of executables required to build a package. Unlike conda, we don't build or install these executables. This key exists just to halt build earlier
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
This allows us to embed extra wasm modules into the output. This is split off of #3210.
We need this for #3210 since
1. We don't want to assume that externref is supported but we need to use it as part of
the JS Promise integration proposal. To handle this we segment off the externrefs into
separate wasm modules. If externref is not supported, we load most everything
successfully but these small modules fail to load
2. Even supposing that we know that externref is supported, there is no way to write C
code such that the compiled code uses externref. It's possible using the emscripten
assembly format, but that is less well documented and harder to use for our purposes
than wat.
This PR doesn't actually change anything by itself.
This is needed to finish getting numpy tests working.
It won't work for packages that need shared libraries without some extra intervention.
ryanking's work in #3234 would help to fix this.
pip install scipy makes the command line runner extremely slow. Without scipy installed,
python -c 'print(1)' runs in about 1 second, but with it installed it takes more like 10 seconds
(time to load clapack_all.so and 111 different .so files in scipy, totaling 20 megabytes). We
have to load all of this despite the fact that we won't use any of it.
This PR makes building CMake based packages easier.
- Introduce a custom toolchain file for Pyodide. It inherits most of settings from the original Emscripten toolchain file with some modifications. Packages built by pyodide-build will automatically use that toolchain file.
- Proxy cmake in pywasmcross.py, in order to proxy other compiler toolchains.
Gyeongjae Choi