pyodide/docs/project/release-notes/v0.17.0.md

(0-17-0-release-notes)=

# Version 0.17.0 Release Notes

Pyodide 0.17.0 is a major step forward from previous versions. It includes major
maintenance improvements, a thorough redesign of the central APIs, and careful
elimination of error leaks and memory leaks.

There have been a large number of backwards incompatible changes due to these
design improvements. Our hope is that we have fixed a significant portion of the
issues and that future releases will have fewer breaking changes.

## New Features

### Asyncio Support

We added full support for asyncio, including a new Python event loop that
schedules tasks on the browser event loop, support for top level await in
{any}`pyodide.runPythonAsync`, and implementations of `await` for {any}`JsProxy`
and {any}`PyProxy`, so that it is possible to await a Python awaitable from
JavaScript and a JavaScript thenable from Python. This allows seamless
interoperability:

```pyodide
pyodide.runPython(`
    async def test():
        from js import fetch
        # Fetch the Pyodide packages list
        r = await fetch("packages.json")
        data = await r.json()
        # return all available packages
        return data.dependencies.object_keys()
`);
let test = pyodide.globals.get("test");
// test returns a coroutine, we can await the coroutine
// from JavaScript and it will schedule it on the Python event loop
result = await test();
console.log(result); // ["asciitree", "parso", "scikit-learn", ...]
```

(Added in PRs {pr}`880`, {pr}`1158`, {pr}`1170`)

### Error Handling

Errors can now be thrown in Python and caught in JavaScript or thrown in
JavaScript and caught in Python.

Support for this is integrated at the lowest level, so calls between JavaScript
and C functions behave as expected. The error conversion code is generated by C
macros which makes implementing and debugging new logic dramatically simpler.

```pyodide
function jserror(){
   throw new Error("ooops!");
}
pyodide.runPython(`
   from js import jserror
   from pyodide import JsException
   try:
       jserror()
   except JsException as e:
      print(str(args)) # prints "TypeError: ooops!"
`);
```

(Added in PRs {pr}`1051` and {pr}`1080`)

### Python "builtin" Modules implemented in JavaScript

It is now simple to add a Python module implemented in JavaScript using
{any}`pyodide.registerJsModule`:

```pyodide
let my_module = {
    foo(x){
        return x*x + 1;
    },
    bar(y, z){
        return y*z + y + z;
    }
};
pyodide.registerJsModule("my_mod", my_module);
pyodide.runPython(`
    from my_mod import foo, bar
    foo(7) # 50
    bar(9, 5) # 59
`);
```

(Added in PR {pr}`1146`)

### New Conversion APIs

We added several new conversion APIs to give more explicit control over the
foreign function interface. In particular, the goal was to make it easier for
users to avoid leaking memory.

For the basic use cases, we have {any}`PyProxy.toJs` and {any}`JsProxy.to_py`
which respectively convert Python objects to JavaScript objects and JavaScript
objects to Python objects. We also added also "wrong-way" conversion functions
{any}`pyodide.to_js` and {any}`pyodide.toPy` which are particularly helpful for
when returning values across languages or to give extra control over the
behavior of called functions.

The promise handler methods {any}`JsProxy.then`, {any}`JsProxy.catch`, and
{any}`JsProxy.finally_` were particularly hard to use without leaking memory so
they have been updated with internal magic to automatically manage the memory.

For more advanced use cases where control over the life cycle of a {any}`PyProxy` is
needed, there are {any}`create_proxy` and {any}`create_once_callable`.

(Added in PRs {pr}`1186`, {pr}`1244`, {pr}`1344`, {pr}`1436`)

## API Changes

We removed `as_nested_list` and deprecated `pyimport`. The old loading method
using `languagePluginURL` and `languagePluginLoader` is also deprecated, use
instead {any}`globalThis.loadPyodide`. Access to Python globals via
{any}`pyodide.globals` has also changed: `pyodide.globals.x` ==>
`pyodide.globals.get("x")` (`pyodide.globals.x` is still supported but is
deprecated).

### Changes to type translations

In the past we found that one of the major pain points in using Pyodide occurred
when an object makes a round trip from Python to JavaScript and back to Python
and comes back different. This violates the expectations of the user and forces
inelegant workarounds (see {issue}`780` and {issue}`892` among others).

The type conversion module has significantly reworked in v0.17 with the goal
that round trip conversions of objects between Python and JavaScript produces an
identical object. That is, Python -> JS -> Python conversion produces an object
that's now equal to the original object, and JS -> Python -> JS conversion
verifies the `===` equality operation.

We also made extensive additions to the type conversions test suite and
documentation, though gaps remain.

See issue {issue}`900` for some of the discussion.

(Mostly implemented in PRs {pr}`1152` and {pr}`1167`, see also {pr}`1186` which )

### Changes to buffer translations

The buffer translation code in previous versions was less flexible, leaked memory,
and had serious bugs including use after free ({issue}`749`) and buffer overflow errors.

We completely reworked these: buffers are now proxied like most other objects.
In simple use cases they can be converted with a copy using {any}`PyProxy.toJs`
and {any}`JsProxy.to_py`. We added new APIs {any}`PyProxy.getBuffer`,
{any}`JsProxy.assign`, and {any}`JsProxy.assign_to` which give more fine-grained
control, though they are not yet as ergonomic as they could be.

(Implemented in PRs {pr}`1215`, {pr}`1376`, and {pr}`1411`)

## Maintenance and Bug fixes

Pyodide version 0.17.0 comes with an enormous amount of maintenance work. There
is still a lot of work to do be done before Pyodide will be ready for a version
1.0, but we made very significant headway. Significant improvements were made to
every component of Pyodide, including the build system, the test suite and
continuous integration, and the core Pyodide code.

### Upstream Emscripten

We finally completed the migration to the latest version of Emscripten
(https://emscripten.org/) compiler toolchain which uses the upstream LLVM
backend. This allows us to take advantage of recent improvements to the
toolchain reducing package size and execution time.

For instance, the scipy package shrank dramatically from 92 MB to 15 MB. Scipy
is now automatically cached in browsers, greatly improving the usability of
scientific Python packages that depend on scipy, such as scikit-image and
scikit-learn. The size of the base Pyodide environment with only the CPython
standard library shrank from 8.1 MB to 6.4 MB.

On the performance side, the latest toolchain comes with a 25% to 30% run time
improvement for pure Python code.

Because we are now using a very recent Emscripten version, we were able to
upstream many of our patches to the compiler toolchain.

(Implemented in PRs {pr}`1102`, {pr}`1184` and {pr}`1193`.)

### Core C Code maintenance

The core C code base was improved with new macros that streamline the most
common tasks and help with consistency, and in addition to the extensive
additions to the test suite we performed a manual audit of the entire C codebase
to locate leaks and logic errors.

We fixed many of the long standing bugs caused by inconsistencies in the
behavior of {any}`JsProxy`. There used to be two different code paths for
producing a {any}`JsProxy` and two different code paths for calling a
{any}`JsProxy` leading to four different behaviors, all of which were presented
errors in some cases. This fixed numerous bugs, including issues {issue}`461`,
{issue}`768`, {issue}`788`, and {issue}`1123`. The number of surprises you can
expect when using the foreign function interface has gone way down.

Similarly, we consolidated the entrypoints and removed redundant APIs. Now every
public entrypoint into C code has been consolidated into the file `pyproxy.js`.
The number of places where C calls into JavaScript is much more diverse, but
these call sites have all been wrapped in a special macro that automatically
converts JavaScript functions to use the CPython calling convention. They can
even be passed as function pointers to C functions from the Python standard
library!

### Fatal error detection

Because we have wrappers on all the entrypoints to C code and all of the
exitpoints, we can detect fatal errors: during normal execution we have careful
control over how the stack unwinds and if an exception comes out of an
unexpected place, we report a fatal error. This leads to faster, better bug
reports and less confusion (using Pyodide after a fatal error occurs can lead to
very strange behavior).

(Implemented in {pr}`1151`, tuned up in {pr}`1390` and {pr}`1478`.)

### Fixed error leaks and memory leaks

Errors in C code must be manually returned to the calling code, and memory must
be manually released. We refactored all of the existing C code to apply a
consistent approach to memory management and error handling, based on the try /
finally idiom. As much as possible, references are only freed in the finally
block at the end of the function to make it easier to check correctness.

This allowed us to fix a large number of error leaks and memory leaks. We now
have pretty complete test coverage for memory leaks. The error coverage is less
complete, though we added fault injection tests for every entrypoint.

(See for instance {pr}`1340`)