2021-12-18 14:54:02 +00:00
|
|
|
# Python WebAssembly (WASM) build
|
|
|
|
|
2024-03-07 02:38:00 +00:00
|
|
|
**WASI support is [tier 2](https://peps.python.org/pep-0011/#tier-2).**
|
2024-11-09 02:12:55 +00:00
|
|
|
**Emscripten support is [tier 3](https://peps.python.org/pep-0011/#tier-3).**
|
2022-02-05 19:52:01 +00:00
|
|
|
|
2021-12-18 14:54:02 +00:00
|
|
|
This directory contains configuration and helpers to facilitate cross
|
2022-09-11 07:51:23 +00:00
|
|
|
compilation of CPython to WebAssembly (WASM). Python supports Emscripten
|
|
|
|
(*wasm32-emscripten*) and WASI (*wasm32-wasi*) targets. Emscripten builds
|
|
|
|
run in modern browsers and JavaScript runtimes like *Node.js*. WASI builds
|
|
|
|
use WASM runtimes such as *wasmtime*.
|
|
|
|
|
|
|
|
Users and developers are encouraged to use the script
|
|
|
|
`Tools/wasm/wasm_build.py`. The tool automates the build process and provides
|
2023-06-02 22:15:41 +00:00
|
|
|
assistance with installation of SDKs, running tests, etc.
|
2021-12-18 14:54:02 +00:00
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
**NOTE**: If you are looking for information that is not directly related to
|
|
|
|
building CPython for WebAssembly (or the resulting build), please see
|
|
|
|
https://github.com/psf/webassembly for more information.
|
|
|
|
|
|
|
|
## wasm32-emscripten
|
|
|
|
|
|
|
|
### Build
|
2021-12-18 14:54:02 +00:00
|
|
|
|
2024-11-09 02:12:55 +00:00
|
|
|
To cross compile to the ``wasm32-emscripten`` platform you need
|
|
|
|
[the Emscripten compiler toolchain](https://emscripten.org/),
|
2024-12-02 23:30:24 +00:00
|
|
|
a Python interpreter, and an installation of Node version 18 or newer.
|
|
|
|
Emscripten version 3.1.73 or newer is recommended. All commands below are
|
|
|
|
relative to a checkout of the Python repository.
|
2022-04-10 07:29:51 +00:00
|
|
|
|
2024-11-09 02:12:55 +00:00
|
|
|
#### Install [the Emscripten compiler toolchain](https://emscripten.org/docs/getting_started/downloads.html)
|
2023-06-02 22:15:41 +00:00
|
|
|
|
2024-11-09 02:12:55 +00:00
|
|
|
You can install the Emscripten toolchain as follows:
|
|
|
|
```shell
|
|
|
|
git clone https://github.com/emscripten-core/emsdk.git --depth 1
|
|
|
|
./emsdk/emsdk install latest
|
|
|
|
./emsdk/emsdk activate latest
|
2022-04-10 07:29:51 +00:00
|
|
|
```
|
2024-11-09 02:12:55 +00:00
|
|
|
To add the Emscripten compiler to your path:
|
2023-06-02 22:15:41 +00:00
|
|
|
```shell
|
2024-11-09 02:12:55 +00:00
|
|
|
source ./emsdk/emsdk_env.sh
|
2023-06-02 22:15:41 +00:00
|
|
|
```
|
2024-11-09 02:12:55 +00:00
|
|
|
This adds `emcc` and `emconfigure` to your path.
|
2023-06-02 22:15:41 +00:00
|
|
|
|
2024-11-09 02:12:55 +00:00
|
|
|
##### Optionally: enable ccache for EMSDK
|
2023-06-02 22:15:41 +00:00
|
|
|
|
|
|
|
The ``EM_COMPILER_WRAPPER`` must be set after the EMSDK environment is
|
|
|
|
sourced. Otherwise the source script removes the environment variable.
|
|
|
|
|
|
|
|
```shell
|
2024-11-09 02:12:55 +00:00
|
|
|
export EM_COMPILER_WRAPPER=ccache
|
2023-06-02 22:15:41 +00:00
|
|
|
```
|
|
|
|
|
2024-12-02 23:30:24 +00:00
|
|
|
#### Compile and build Python interpreter
|
2021-12-18 14:54:02 +00:00
|
|
|
|
2024-11-09 02:12:55 +00:00
|
|
|
You can use `python Tools/wasm/emscripten` to compile and build targetting
|
|
|
|
Emscripten. You can do everything at once with:
|
2021-12-18 14:54:02 +00:00
|
|
|
```shell
|
2024-11-09 02:12:55 +00:00
|
|
|
python Tools/wasm/emscripten build
|
2022-02-05 19:52:01 +00:00
|
|
|
```
|
2024-11-09 02:12:55 +00:00
|
|
|
or you can break it out into four separate steps:
|
2022-02-05 19:52:01 +00:00
|
|
|
```shell
|
2024-11-09 02:12:55 +00:00
|
|
|
python Tools/wasm/emscripten configure-build-python
|
|
|
|
python Tools/wasm/emscripten make-build-python
|
2024-12-10 02:32:58 +00:00
|
|
|
python Tools/wasm/emscripten make-libffi
|
2024-11-09 02:12:55 +00:00
|
|
|
python Tools/wasm/emscripten configure-host
|
|
|
|
python Tools/wasm/emscripten make-host
|
2022-01-12 15:08:19 +00:00
|
|
|
```
|
2024-11-09 02:12:55 +00:00
|
|
|
Extra arguments to the configure steps are passed along to configure. For
|
|
|
|
instance, to do a debug build, you can use:
|
2022-04-23 07:52:16 +00:00
|
|
|
```shell
|
2024-11-09 02:12:55 +00:00
|
|
|
python Tools/wasm/emscripten build --with-py-debug
|
2022-08-13 19:56:08 +00:00
|
|
|
```
|
|
|
|
|
2024-12-02 23:30:24 +00:00
|
|
|
### Running from node
|
|
|
|
|
|
|
|
If you want to run the normal Python CLI, you can use `python.sh`. It takes the
|
|
|
|
same options as the normal Python CLI entrypoint, though the REPL does not
|
|
|
|
function and will crash.
|
|
|
|
|
|
|
|
`python.sh` invokes `node_entry.mjs` which imports the Emscripten module for the
|
|
|
|
Python process and starts it up with the appropriate settings. If you wish to
|
|
|
|
make a node application that "embeds" the interpreter instead of acting like the
|
|
|
|
CLI you will need to write your own alternative to `node_entry.mjs`.
|
|
|
|
|
|
|
|
|
|
|
|
### The Web Example
|
|
|
|
|
|
|
|
When building for Emscripten, the web example will be built automatically. It is
|
|
|
|
in the ``web_example`` directory. To run the web example, ``cd`` into the
|
|
|
|
``web_example`` directory, then run ``python server.py``. This will start a web
|
|
|
|
server; you can then visit ``http://localhost:8000/python.html`` in a browser to
|
|
|
|
see a simple REPL example.
|
|
|
|
|
|
|
|
The web example relies on a bug fix in Emscripten version 3.1.73 so if you build
|
|
|
|
with earlier versions of Emscripten it may not work. The web example uses
|
|
|
|
``SharedArrayBuffer``. For security reasons browsers only provide
|
|
|
|
``SharedArrayBuffer`` in secure environments with cross-origin isolation. The
|
|
|
|
webserver must send cross-origin headers and correct MIME types for the
|
|
|
|
JavaScript and WebAssembly files. Otherwise the terminal will fail to load with
|
|
|
|
an error message like ``ReferenceError: SharedArrayBuffer is not defined``. See
|
|
|
|
more information here:
|
|
|
|
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer#security_requirements
|
|
|
|
|
|
|
|
Note that ``SharedArrayBuffer`` is _not required_ to use Python itself, only the
|
|
|
|
web example. If cross-origin isolation is not appropriate for your use case you
|
|
|
|
may make your own application embedding `python.mjs` which does not use
|
|
|
|
``SharedArrayBuffer`` and serve it without the cross-origin isolation headers.
|
|
|
|
|
|
|
|
### Embedding Python in a custom JavaScript application
|
|
|
|
|
|
|
|
You can look at `python.worker.mjs` and `node_entry.mjs` for inspiration. At a
|
|
|
|
minimum you must import ``createEmscriptenModule`` and you need to call
|
|
|
|
``createEmscriptenModule`` with an appropriate settings object. This settings
|
|
|
|
object will need a prerun hook that installs the Python standard library into
|
|
|
|
the Emscripten file system.
|
|
|
|
|
|
|
|
#### NodeJs
|
|
|
|
|
|
|
|
In Node, you can use the NodeFS to mount the standard library in your native
|
|
|
|
file system into the Emscripten file system:
|
|
|
|
```js
|
|
|
|
import createEmscriptenModule from "./python.mjs";
|
|
|
|
|
|
|
|
await createEmscriptenModule({
|
|
|
|
preRun(Module) {
|
|
|
|
Module.FS.mount(
|
|
|
|
Module.FS.filesystems.NODEFS,
|
|
|
|
{ root: "/path/to/python/stdlib" },
|
|
|
|
"/lib/",
|
|
|
|
);
|
|
|
|
},
|
|
|
|
});
|
|
|
|
```
|
|
|
|
|
|
|
|
#### Browser
|
|
|
|
|
|
|
|
In the browser, the simplest approach is to put the standard library in a zip
|
|
|
|
file it and install it. With Python 3.14 this could look like:
|
|
|
|
```js
|
|
|
|
import createEmscriptenModule from "./python.mjs";
|
|
|
|
|
|
|
|
await createEmscriptenModule({
|
|
|
|
async preRun(Module) {
|
|
|
|
Module.FS.mkdirTree("/lib/python3.14/lib-dynload/");
|
|
|
|
Module.addRunDependency("install-stdlib");
|
|
|
|
const resp = await fetch("python3.14.zip");
|
|
|
|
const stdlibBuffer = await resp.arrayBuffer();
|
|
|
|
Module.FS.writeFile(`/lib/python314.zip`, new Uint8Array(stdlibBuffer), {
|
|
|
|
canOwn: true,
|
|
|
|
});
|
|
|
|
Module.removeRunDependency("install-stdlib");
|
|
|
|
},
|
|
|
|
});
|
|
|
|
```
|
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
### Limitations and issues
|
2022-02-05 19:52:01 +00:00
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### Network stack
|
2022-04-05 09:21:11 +00:00
|
|
|
|
|
|
|
- Python's socket module does not work with Emscripten's emulated POSIX
|
|
|
|
sockets yet. Network modules like ``asyncio``, ``urllib``, ``selectors``,
|
|
|
|
etc. are not available.
|
2022-04-04 17:31:31 +00:00
|
|
|
- Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or
|
|
|
|
``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported.
|
|
|
|
- ``socketpair`` does not work.
|
2022-02-05 19:52:01 +00:00
|
|
|
- Blocking sockets are not available and non-blocking sockets don't work
|
|
|
|
correctly, e.g. ``socket.accept`` crashes the runtime. ``gethostbyname``
|
|
|
|
does not resolve to a real IP address. IPv6 is not available.
|
|
|
|
- The ``select`` module is limited. ``select.select()`` crashes the runtime
|
|
|
|
due to lack of exectfd support.
|
2022-04-05 09:21:11 +00:00
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### processes, signals
|
2022-04-05 09:21:11 +00:00
|
|
|
|
|
|
|
- Processes are not supported. System calls like fork, popen, and subprocess
|
|
|
|
fail with ``ENOSYS`` or ``ENOSUP``.
|
2022-02-05 19:52:01 +00:00
|
|
|
- Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction``
|
|
|
|
are not available or do not work correctly. ``SIGTERM`` exits the runtime.
|
2022-04-05 09:21:11 +00:00
|
|
|
- Keyboard interrupt (CTRL+C) handling is not implemented yet.
|
|
|
|
- Resource-related functions like ``os.nice`` and most functions of the
|
|
|
|
``resource`` module are not available.
|
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### threading
|
2022-04-23 12:59:33 +00:00
|
|
|
|
|
|
|
- Threading is disabled by default. The ``configure`` option
|
|
|
|
``--enable-wasm-pthreads`` adds compiler flag ``-pthread`` and
|
2022-11-03 18:27:27 +00:00
|
|
|
linker flags ``-sUSE_PTHREADS -sPROXY_TO_PTHREAD``.
|
2022-04-23 12:59:33 +00:00
|
|
|
- pthread support requires WASM threads and SharedArrayBuffer (bulk memory).
|
|
|
|
The Node.JS runtime keeps a pool of web workers around. Each web worker
|
|
|
|
uses several file descriptors (eventfd, epoll, pipe).
|
|
|
|
- It's not advised to enable threading when building for browsers or with
|
|
|
|
dynamic linking support; there are performance and stability issues.
|
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### file system
|
2022-04-05 09:21:11 +00:00
|
|
|
|
2022-02-05 19:52:01 +00:00
|
|
|
- Most user, group, and permission related function and modules are not
|
|
|
|
supported or don't work as expected, e.g.``pwd`` module, ``grp`` module,
|
2023-01-18 00:06:38 +00:00
|
|
|
``os.setgroups``, ``os.chown``, and so on. ``lchown`` and ``lchmod`` are
|
2022-04-04 17:31:31 +00:00
|
|
|
not available.
|
|
|
|
- ``umask`` is a no-op.
|
|
|
|
- hard links (``os.link``) are not supported.
|
2022-02-05 19:52:01 +00:00
|
|
|
- Offset and iovec I/O functions (e.g. ``os.pread``, ``os.preadv``) are not
|
|
|
|
available.
|
|
|
|
- ``os.mknod`` and ``os.mkfifo``
|
|
|
|
[don't work](https://github.com/emscripten-core/emscripten/issues/16158)
|
|
|
|
and are disabled.
|
|
|
|
- Large file support crashes the runtime and is disabled.
|
|
|
|
- ``mmap`` module is unstable. flush (``msync``) can crash the runtime.
|
2022-04-05 09:21:11 +00:00
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### Misc
|
2022-04-05 09:21:11 +00:00
|
|
|
|
|
|
|
- Heap memory and stack size are limited. Recursion or extensive memory
|
|
|
|
consumption can crash Python.
|
|
|
|
- Most stdlib modules with a dependency on external libraries are missing,
|
2022-07-27 06:18:34 +00:00
|
|
|
e.g. ``ctypes``, ``readline``, ``ssl``, and more.
|
2022-04-05 09:21:11 +00:00
|
|
|
- Shared extension modules are not implemented yet. All extension modules
|
2022-04-23 07:52:16 +00:00
|
|
|
are statically linked into the main binary. The experimental configure
|
|
|
|
option ``--enable-wasm-dynamic-linking`` enables dynamic extensions
|
|
|
|
supports. It's currently known to crash in combination with threading.
|
2022-04-04 17:31:31 +00:00
|
|
|
- glibc extensions for date and time formatting are not available.
|
2022-02-05 19:52:01 +00:00
|
|
|
- ``locales`` module is affected by musl libc issues,
|
2023-01-18 00:06:38 +00:00
|
|
|
[gh-90548](https://github.com/python/cpython/issues/90548).
|
2022-02-05 19:52:01 +00:00
|
|
|
- Python's object allocator ``obmalloc`` is disabled by default.
|
|
|
|
- ``ensurepip`` is not available.
|
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
#### In the browser
|
2022-02-05 19:52:01 +00:00
|
|
|
|
2022-04-05 09:21:11 +00:00
|
|
|
- The interactive shell does not handle copy 'n paste and unicode support
|
|
|
|
well.
|
2022-02-05 19:52:01 +00:00
|
|
|
- The bundled stdlib is limited. Network-related modules,
|
2022-11-03 18:27:27 +00:00
|
|
|
multiprocessing, dbm, tests and similar modules
|
2022-02-05 19:52:01 +00:00
|
|
|
are not shipped. All other modules are bundled as pre-compiled
|
|
|
|
``pyc`` files.
|
2022-04-05 09:21:11 +00:00
|
|
|
- In-memory file system (MEMFS) is not persistent and limited.
|
2022-04-23 07:52:16 +00:00
|
|
|
- Test modules are disabled by default. Use ``--enable-test-modules`` build
|
|
|
|
test modules like ``_testcapi``.
|
2022-02-05 19:52:01 +00:00
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
## WASI (wasm32-wasi)
|
2022-04-23 07:52:16 +00:00
|
|
|
|
2024-03-07 02:38:00 +00:00
|
|
|
See [the devguide on how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi).
|
2023-06-02 22:15:41 +00:00
|
|
|
|
2023-11-30 21:01:07 +00:00
|
|
|
## Detecting WebAssembly builds
|
2023-06-02 22:15:41 +00:00
|
|
|
|
|
|
|
### Python code
|
2022-04-10 07:29:51 +00:00
|
|
|
|
2022-04-23 07:52:16 +00:00
|
|
|
```python
|
2022-04-10 07:29:51 +00:00
|
|
|
import os, sys
|
|
|
|
|
|
|
|
if sys.platform == "emscripten":
|
|
|
|
# Python on Emscripten
|
2023-11-30 00:18:25 +00:00
|
|
|
...
|
2022-04-10 07:29:51 +00:00
|
|
|
if sys.platform == "wasi":
|
|
|
|
# Python on WASI
|
2023-11-30 00:18:25 +00:00
|
|
|
...
|
2022-04-10 07:29:51 +00:00
|
|
|
|
|
|
|
if os.name == "posix":
|
|
|
|
# WASM platforms identify as POSIX-like.
|
|
|
|
# Windows does not provide os.uname().
|
|
|
|
machine = os.uname().machine
|
|
|
|
if machine.startswith("wasm"):
|
2023-11-30 00:18:25 +00:00
|
|
|
# WebAssembly (wasm32, wasm64 potentially in the future)
|
2022-04-23 07:52:16 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
```python
|
|
|
|
>>> import os, sys
|
|
|
|
>>> os.uname()
|
2022-09-11 07:51:23 +00:00
|
|
|
posix.uname_result(
|
|
|
|
sysname='Emscripten',
|
|
|
|
nodename='emscripten',
|
|
|
|
release='3.1.19',
|
|
|
|
version='#1',
|
|
|
|
machine='wasm32'
|
|
|
|
)
|
2022-04-23 07:52:16 +00:00
|
|
|
>>> os.name
|
|
|
|
'posix'
|
|
|
|
>>> sys.platform
|
|
|
|
'emscripten'
|
|
|
|
>>> sys._emscripten_info
|
|
|
|
sys._emscripten_info(
|
2022-09-11 07:51:23 +00:00
|
|
|
emscripten_version=(3, 1, 10),
|
|
|
|
runtime='Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0',
|
2022-04-23 07:52:16 +00:00
|
|
|
pthreads=False,
|
|
|
|
shared_memory=False
|
|
|
|
)
|
2022-09-11 07:51:23 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
```python
|
2022-04-23 07:52:16 +00:00
|
|
|
>>> sys._emscripten_info
|
2022-09-11 07:51:23 +00:00
|
|
|
sys._emscripten_info(
|
|
|
|
emscripten_version=(3, 1, 19),
|
|
|
|
runtime='Node.js v14.18.2',
|
|
|
|
pthreads=True,
|
|
|
|
shared_memory=True
|
|
|
|
)
|
2022-04-23 07:52:16 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
```python
|
|
|
|
>>> import os, sys
|
|
|
|
>>> os.uname()
|
2022-09-11 07:51:23 +00:00
|
|
|
posix.uname_result(
|
|
|
|
sysname='wasi',
|
|
|
|
nodename='(none)',
|
|
|
|
release='0.0.0',
|
|
|
|
version='0.0.0',
|
|
|
|
machine='wasm32'
|
|
|
|
)
|
2022-04-23 07:52:16 +00:00
|
|
|
>>> os.name
|
|
|
|
'posix'
|
|
|
|
>>> sys.platform
|
|
|
|
'wasi'
|
2022-04-10 07:29:51 +00:00
|
|
|
```
|
|
|
|
|
2023-06-02 22:15:41 +00:00
|
|
|
### C code
|
2022-04-10 07:29:51 +00:00
|
|
|
|
|
|
|
Emscripten SDK and WASI SDK define several built-in macros. You can dump a
|
|
|
|
full list of built-ins with ``emcc -dM -E - < /dev/null`` and
|
|
|
|
``/path/to/wasi-sdk/bin/clang -dM -E - < /dev/null``.
|
|
|
|
|
|
|
|
* WebAssembly ``__wasm__`` (also ``__wasm``)
|
|
|
|
* wasm32 ``__wasm32__`` (also ``__wasm32``)
|
|
|
|
* wasm64 ``__wasm64__``
|
|
|
|
* Emscripten ``__EMSCRIPTEN__`` (also ``EMSCRIPTEN``)
|
|
|
|
* Emscripten version ``__EMSCRIPTEN_major__``, ``__EMSCRIPTEN_minor__``, ``__EMSCRIPTEN_tiny__``
|
|
|
|
* WASI ``__wasi__``
|