pyodide/docs/project/release-notes/v0.17.0.md

:orphan:

(0-17-0-release-notes)=

Version 0.17.0 Release Notes

Pyodide 0.17.0 is a major step forward from previous versions. It includes major maintenance improvements, a thorough redesign of the central APIs, and careful elimination of error leaks and memory leaks.

There have been a large number of backwards incompatible changes due to these design improvements. Our hope is that we have fixed a significant portion of the issues and that future releases will have fewer breaking changes.

New Features

Asyncio Support

We added full support for asyncio, including a new Python event loop that schedules tasks on the browser event loop, support for top level await in {any}pyodide.runPythonAsync, and implementations of await for {any}JsProxy <pyodide.ffi.JsProxy> and {js:class}PyProxy, so that it is possible to await a Python awaitable from JavaScript and a JavaScript thenable from Python. This allows seamless interoperability:

pyodide.runPython(`
    async def test():
        from js import fetch
        # Fetch the Pyodide packages list
        r = await fetch("packages.json")
        data = await r.json()
        # return all available packages
        return data.dependencies.object_keys()
`);
let test = pyodide.globals.get("test");
// test returns a coroutine, we can await the coroutine
// from JavaScript and it will schedule it on the Python event loop
result = await test();
console.log(result); // ["asciitree", "parso", "scikit-learn", ...]

(Added in PRs {pr}880, {pr}1158, {pr}1170)

Error Handling

Errors can now be thrown in Python and caught in JavaScript or thrown in JavaScript and caught in Python.

Support for this is integrated at the lowest level, so calls between JavaScript and C functions behave as expected. The error conversion code is generated by C macros which makes implementing and debugging new logic dramatically simpler.

function jserror(){
   throw new Error("ooops!");
}
pyodide.runPython(`
   from js import jserror
   from pyodide import JsException
   try:
       jserror()
   except JsException as e:
      print(str(args)) # prints "TypeError: ooops!"
`);

(Added in PRs {pr}1051 and {pr}1080)

Python "builtin" Modules implemented in JavaScript

It is now simple to add a Python module implemented in JavaScript using {any}pyodide.registerJsModule:

let my_module = {
    foo(x){
        return x*x + 1;
    },
    bar(y, z){
        return y*z + y + z;
    }
};
pyodide.registerJsModule("my_mod", my_module);
pyodide.runPython(`
    from my_mod import foo, bar
    foo(7) # 50
    bar(9, 5) # 59
`);

(Added in PR {pr}1146)

New Conversion APIs

We added several new conversion APIs to give more explicit control over the foreign function interface. In particular, the goal was to make it easier for users to avoid leaking memory.

For the basic use cases, we have {js:meth}PyProxy.toJs and JsProxy.to_py which respectively convert Python objects to JavaScript objects and JavaScript objects to Python objects. We also added also "wrong-way" conversion functions {any}pyodide.to_js <pyodide.ffi.to_js> and {any}pyodide.toPy which are particularly helpful for when returning values across languages or to give extra control over the behavior of called functions.

The promise handler methods JsProxy.then, JsProxy.catch, and JsProxy.finally_ were particularly hard to use without leaking memory so they have been updated with internal magic to automatically manage the memory.

For more advanced use cases where control over the life cycle of a {js:class}PyProxy is needed, there are {any}create_proxy and {any}create_once_callable.

(Added in PRs {pr}1186, {pr}1244, {pr}1344, {pr}1436)

API Changes

We removed as_nested_list and deprecated pyimport. The old loading method using languagePluginURL and languagePluginLoader is also deprecated, use instead {any}globalThis.loadPyodide. Access to Python globals via {any}pyodide.globals has also changed: pyodide.globals.x ==> pyodide.globals.get("x") (pyodide.globals.x is still supported but is deprecated).

Changes to type translations

In the past we found that one of the major pain points in using Pyodide occurred when an object makes a round trip from Python to JavaScript and back to Python and comes back different. This violates the expectations of the user and forces inelegant workarounds (see {issue}780 and {issue}892 among others).

The type conversion module has significantly reworked in v0.17 with the goal that round trip conversions of objects between Python and JavaScript produces an identical object. That is, Python -> JS -> Python conversion produces an object that's now equal to the original object, and JS -> Python -> JS conversion verifies the === equality operation.

We also made extensive additions to the type conversions test suite and documentation, though gaps remain.

See issue {issue}900 for some of the discussion.

(Mostly implemented in PRs {pr}1152 and {pr}1167, see also {pr}1186 which )

Changes to buffer translations

The buffer translation code in previous versions was less flexible, leaked memory, and had serious bugs including use after free ({issue}749) and buffer overflow errors.

We completely reworked these: buffers are now proxied like most other objects. In simple use cases they can be converted with a copy using {js:meth}PyProxy.toJs and JsProxy.to_py. We added new APIs PyProxy.getBuffer, JsProxy.assign, and JsProxy.assign_to which give more fine-grained control, though they are not yet as ergonomic as they could be.

(Implemented in PRs {pr}1215, {pr}1376, and {pr}1411)

Maintenance and Bug fixes

Pyodide version 0.17.0 comes with an enormous amount of maintenance work. There is still a lot of work to do be done before Pyodide will be ready for a version 1.0, but we made very significant headway. Significant improvements were made to every component of Pyodide, including the build system, the test suite and continuous integration, and the core Pyodide code.

Upstream Emscripten

We finally completed the migration to the latest version of Emscripten (https://emscripten.org/) compiler toolchain which uses the upstream LLVM backend. This allows us to take advantage of recent improvements to the toolchain reducing package size and execution time.

For instance, the scipy package shrank dramatically from 92 MB to 15 MB. Scipy is now automatically cached in browsers, greatly improving the usability of scientific Python packages that depend on scipy, such as scikit-image and scikit-learn. The size of the base Pyodide environment with only the CPython standard library shrank from 8.1 MB to 6.4 MB.

On the performance side, the latest toolchain comes with a 25% to 30% run time improvement for pure Python code.

Because we are now using a very recent Emscripten version, we were able to upstream many of our patches to the compiler toolchain.

(Implemented in PRs {pr}1102, {pr}1184 and {pr}1193.)

Core C Code maintenance

The core C code base was improved with new macros that streamline the most common tasks and help with consistency, and in addition to the extensive additions to the test suite we performed a manual audit of the entire C codebase to locate leaks and logic errors.

We fixed many of the long standing bugs caused by inconsistencies in the behavior of {any}JsProxy <pyodide.ffi.JsProxy>. There used to be two different code paths for producing a {any}JsProxy <pyodide.ffi.JsProxy> and two different code paths for calling a {any}JsProxy <pyodide.ffi.JsProxy> leading to four different behaviors, all of which were presented errors in some cases. This fixed numerous bugs, including issues {issue}461, {issue}768, {issue}788, and {issue}1123. The number of surprises you can expect when using the foreign function interface has gone way down.

Similarly, we consolidated the entrypoints and removed redundant APIs. Now every public entrypoint into C code has been consolidated into the file pyproxy.js. The number of places where C calls into JavaScript is much more diverse, but these call sites have all been wrapped in a special macro that automatically converts JavaScript functions to use the CPython calling convention. They can even be passed as function pointers to C functions from the Python standard library!

Fatal error detection

Because we have wrappers on all the entrypoints to C code and all of the exitpoints, we can detect fatal errors: during normal execution we have careful control over how the stack unwinds and if an exception comes out of an unexpected place, we report a fatal error. This leads to faster, better bug reports and less confusion (using Pyodide after a fatal error occurs can lead to very strange behavior).

(Implemented in {pr}1151, tuned up in {pr}1390 and {pr}1478.)

Fixed error leaks and memory leaks

Errors in C code must be manually returned to the calling code, and memory must be manually released. We refactored all of the existing C code to apply a consistent approach to memory management and error handling, based on the try / finally idiom. As much as possible, references are only freed in the finally block at the end of the function to make it easier to check correctness.

This allowed us to fix a large number of error leaks and memory leaks. We now have pretty complete test coverage for memory leaks. The error coverage is less complete, though we added fault injection tests for every entrypoint.

(See for instance {pr}1340)