ENH Rework streams handling (#4035)

This fixes a number problems with the old stream handling: 1. Not possible to set a custom errno (necessary for proper interrupt handling and possibly for other things) 2. Inefficient: in a lot of cases we have data in one buffer and we need it placed into a different buffer, but we have to implement a function that gets one byte out of the source buffer and then call it repeatedly to move one byte at a time to the target buffer. 3. Ease of implementation: in many cases we already have perfectly good buffer manipulation APIs, so if we have direct access to the true source or target buffer we can just use these. See: the node IO code, which got much simpler. This is backwards compatible, so you can still use the old input mechanism or use buffered or raw output. But it adds a new method of directly implementing read/write. For simplicity, we insure that the source/destination buffers are always `Uint8Array` views that point to exactly the region that is meant to be read/written. The old mechanisms are faster than before and can correctly support keyboard interrupts. Other than that I think the original behavior is unchanged. I added a lot more test coverage to ensure backwards compatibility since there was pretty anemic coverage before. I think the read/write APIs are mostly pretty simple to use, with the exception that someone might forget to return the number of bytes read. JavaScript's ordinary behavior coerces the `undefined` to a 0, which leads to an infinite loop where the filesystem repeatedly asks to read/write the same data since it sees no progress. I added a check that writes an error message to the console and sets EIO when undefined is returned so the infinite loop is prevented and the problem is explained.
2023-08-21 08:41:44 +02:00 · 2023-08-21 08:41:44 +02:00 · e19621d483
parent ba774aa211
commit e19621d483
18 changed files with 3184 additions and 562 deletions
--- a/.prettierignore
+++ b/.prettierignore
@ -10,3 +10,4 @@ cpython
 .pytest_cache
 .clang-format
 packages/libf2c/make.inc
+src/js/generated_struct_info32.json
--- a/conftest.py
+++ b/conftest.py
@ -5,6 +5,7 @@ import os
 import pathlib
 import re
 import sys
+from collections.abc import Sequence

 import pytest

@ -251,3 +252,18 @@ def extra_checks_test_wrapper(browser, trace_hiwire_refs, trace_pyproxies):

 def package_is_built(package_name):
    return _package_is_built(package_name, pytest.pyodide_dist_dir)
+
+
+def strip_assertions_stderr(messages: Sequence[str]) -> list[str]:
+    """Strip additional messages on stderr included when ASSERTIONS=1"""
+    res = []
+    for msg in messages:
+        if msg.strip() in [
+            "sigaction: signal type not supported: this is a no-op.",
+            "Calling stub instead of siginterrupt()",
+            "warning: no blob constructor, cannot create blobs with mimetypes",
+            "warning: no BlobBuilder",
+        ]:
+            continue
+        res.append(msg)
+    return res
--- a/docs/conf.py
+++ b/docs/conf.py
@ -61,7 +61,7 @@ autosummary_generate = True
 autodoc_default_flags = ["members", "inherited-members"]

 intersphinx_mapping = {
-    "python": ("https://docs.python.org/3.10", None),
+    "python": ("https://docs.python.org/3.11", None),
    "micropip": (f"https://micropip.pyodide.org/en/v{micropip.__version__}/", None),
    "numpy": ("https://numpy.org/doc/stable/", None),
 }
--- a/docs/project/changelog.md
+++ b/docs/project/changelog.md
@ -75,6 +75,10 @@ myst:
  `FetchResponse.raise_for_status` to raise an `OSError` for error status codes.
  {pr}`3986` {pr}`4053`

+- {{ Enhancement }} The `setStdin`, `setStdout` and `setStderr` APIs have been
+  improved with extra control and better performance.
+  {pr}`4035`
+
 ### Packages

 - OpenBLAS has been added and scipy now uses OpenBLAS rather than CLAPACK
--- a/docs/sphinx_pyodide/sphinx_pyodide/lexers.py
+++ b/docs/sphinx_pyodide/sphinx_pyodide/lexers.py
@ -22,9 +22,12 @@ class PyodideLexer(JavascriptLexer):
        ],
        "python-code": [
            (
-                rf"({quotemark})((?:\\\\|\\[^\\]|[^{quotemark}\\])*)({quotemark})",
+                rf"([A-Za-z.]*)({quotemark})((?:\\\\|\\[^\\]|[^{quotemark}\\])*)({quotemark})",
                bygroups(
-                    Token.Literal.String, using(PythonLexer), Token.Literal.String
+                    using(JavascriptLexer),
+                    Token.Literal.String,
+                    using(PythonLexer),
+                    Token.Literal.String,
                ),
                "#pop",
            )
--- a/docs/usage/streams.md
+++ b/docs/usage/streams.md
@ -7,36 +7,351 @@ Pyodide has three functions {js:func}`pyodide.setStdin`,
 behavior of reading from {py:data}`~sys.stdin` and writing to {py:data}`~sys.stdout` and
 {py:data}`~sys.stderr` respectively.

-`setStdin({stdin?, isatty?, error?})` takes a function which should take zero
-arguments and return either a string or an ArrayBufferView of information read
-from stdin. The `isatty` argument signals whether {py:func}`isatty(stdin) <os.isatty>` should be true
-or false. If you pass `error: true` then reading from stdin will return an
-error. If `setStdin` is called with no arguments, the default value is restored.
-In Node the default behavior is to read from {js:data}`process.stdin` and in the browser
-it is to throw an error.
+## Standard Input

-`setStdout({batched?, raw?, isattty?})` sets the standard out handler and
-similarly `setStderr` (same arguments) sets the stdandard error handler. If a
-`raw` handler is provided then the handler is called with a `number` for each
-byte of the output to stdout. The handler is expected to deal with this in
-whatever way it prefers. `isattty` again controls whether {py:func}`isatty(stdout) <os.isatty>`
-returns `true` or `false`.
+(streams-stdin)=

-On the other hand, a `batched` handler is only called with complete lines of
-text (or when the output is flushed). A `batched` handler cannot have `isatty`
-set to `true` because it is impossible to use such a handler to make something
-behave like a tty.
+{js:func}`pyodide.setStdin` sets the standard in handler. There are several
+different ways to do this depending on the options passed to `setStdin`.

-Passing neither `raw` nor `batched` sets the default behavior. In Node the
-default behavior is to write directly to {js:data}`process.stdout` and
-{js:data}`process.stderr` (in this case `isatty` depends on whether
-{js:data}`process.stdout` and {js:data}`process.stderr` are ttys). In browser,
-the default behavior is achieved with `pyodide.setStdout({batched: console.log})`
-and `pyodide.setStderr({batched: console.warn})`.
+### Always raise IO Error

-The arguments `stdin`, `stdout`, and `stderr` to `loadPyodide` provide a
-diminished amount of control compared to `setStdin`, `setStdout`, and
-`setStderr`. They all set `isatty` to `false` and use batched processing for
-`setStdout` and `setStderr`. In most cases, nothing is written or read to any of
-these streams while Pyodide is starting, so if you need the added flexibility
-you can wait until Pyodide is loaded and then use the `pyodide.setStdxxx` apis.
+If we pass `{error: true}`, any read from stdin raises an I/O error.
+
+```js
+pyodide.setStdin({ error: true });
+pyodide.runPython(`
+    with pytest.raises(OSError, match="I/O error"):
+        input()
+`);
+```
+
+### Set the default behavior
+
+You can set the default behavior by calling `pyodide.setStdin()` with no
+arguments. In Node the default behavior is to read directly from Node's standard
+input. In the browser, the default is the same as
+`pyodide.setStdin({ stdin: () => prompt() })`.
+
+### A stdin handler
+
+We can pass the options `{stdin, isatty}`. `stdin` should be a
+zero-argument function which should return one of:
+
+1. A string which represents a full line of text (it will have a newline
+   appended if it does not already end in one).
+2. An array buffer or Uint8Array containing utf8 encoded characters
+3. A number between 0 and 255 which indicates one byte of input
+4. `undefined` which indicates EOF.
+
+`isatty` is a boolean which
+indicates whether `sys.stdin.isatty()` should return `true` or `false`.
+
+For example, the following class plays back a list of results.
+
+```js
+class StdinHandler {
+  constructor(results, options) {
+    this.results = results;
+    this.idx = 0;
+    Object.assign(this, options);
+  }
+
+  stdin() {
+    return this.results[this.idx++];
+  }
+}
+```
+
+Here it is in use:
+
+```pyodide
+pyodide.setStdin(
+  new StdinHandler(["a", "bcd", "efg"]),
+);
+pyodide.runPython(`
+    assert input() == "a"
+    assert input() == "bcd"
+    assert input() == "efg"
+    # after this, further attempts to read from stdin will return undefined which
+    # indicates end of file
+    with pytest.raises(EOFError, match="EOF when reading a line"):
+        input()
+`);
+```
+
+Note that the `input()` function automatically reads a line of text and
+removes the trailing newline. If we use `sys.stdin.read` we see that newlines
+have been appended to strings that don't end in a newline:
+
+```pyodide
+pyodide.setStdin(
+  new StdinHandler(["a", "bcd\n", "efg", undefined, "h", "i"]),
+);
+pyodide.runPython(String.raw`
+    import sys
+    assert sys.stdin.read() == "a\nbcd\nefg\n"
+    assert sys.stdin.read() == "h\ni\n"
+`);
+```
+
+Instead of strings we can return the list of utf8 bytes for the input:
+
+```pyodide
+pyodide.setStdin(
+  new StdinHandler(
+    [0x61 /* a */, 0x0a /* \n */, 0x62 /* b */, 0x63 /* c */],
+    true,
+  ),
+);
+pyodide.runPython(`
+    assert input() == "a"
+    assert input() == "bc"
+`);
+```
+
+Or we can return a `Uint8Array` with the utf8-encoded text that we wish to
+render:
+
+```pyodide
+pyodide.setStdin(
+  new StdinHandler([new Uint8Array([0x61, 0x0a, 0x62, 0x63])]),
+);
+pyodide.runPython(`
+    assert input() == "a"
+    assert input() == "bc"
+`);
+```
+
+### A read handler
+
+A read handler takes a `Uint8Array` as an argument and is supposed to place
+the data into this buffer and return the number of bytes read. This is useful in
+Node. For example, the following class can be used to read from a Node file
+descriptor:
+
+```js
+const fs = require("fs");
+const tty = require("tty");
+class NodeReader {
+  constructor(fd) {
+    this.fd = fd;
+    this.isatty = tty.isatty(fd);
+  }
+
+  read(buffer) {
+    return fs.readSync(this.fd, buffer);
+  }
+}
+```
+
+For instance to set stdin to read from a file called `input.txt`, we can do the
+following:
+
+```js
+const fd = fs.openSync("input.txt", "r");
+py.setStdin(new NodeReader(fd));
+```
+
+Or we can read from node's stdin (the default behavior) as follows:
+
+```js
+fd = fs.openSync("/dev/stdin", "r");
+py.setStdin(new NodeReader(fd));
+```
+
+### isatty
+
+It is possible to control whether or not {py:meth}`sys.stdin.isatty() <io.IOBase.isatty>`
+returns true with the `isatty` option:
+
+```pyodide
+pyodide.setStdin(new StdinHandler([], {isatty: true}));
+pyodide.runPython(`
+    import sys
+    assert sys.stdin.isatty() # returns true as we requested
+`);
+pyodide.setStdin(new StdinHandler([], {isatty: false}));
+pyodide.runPython(`
+  assert not sys.stdin.isatty() # returns false as we requested
+`);
+```
+
+This will change the behavior of cli applications that behave differently in an
+interactive terminal, for example pytest does this.
+
+### Raising IO errors
+
+To raise an IO error in either a `stdin` or `read` handler, you should throw an
+IO error as follows:
+
+```js
+throw new pyodide.FS.ErrnoError(pyodide.ERRNO_CODES.EIO);
+```
+
+for instance, saying:
+
+```js
+pyodide.setStdin({
+  read(buf) {
+    throw new pyodide.FS.ErrnoError(pyodide.ERRNO_CODES.EIO);
+  },
+});
+```
+
+is the same as `pyodide.setStdin({error: true})`.
+
+### Handling Keyboard interrupts
+
+To handle a keyboard interrupt in an input handler, you should periodically call
+{js:func}`pyodide.checkInterrupt`. For example, the following stdin handler
+always raises a keyboard interrupt:
+
+```js
+const interruptBuffer = new Int32Array(new SharedArrayBuffer(4));
+pyodide.setInterruptBuffer(interruptBuffer);
+pyodide.setStdin({
+  read(buf) {
+    // Put signal into interrupt buffer
+    interruptBuffer[0] = 2;
+    // Call checkInterrupt to raise an error
+    pyodide.checkInterrupt();
+    console.log(
+      "This code won't ever be executed because pyodide.checkInterrupt raises an error!",
+    );
+  },
+});
+```
+
+For a more realistic example that handles reading stdin in a worker and also
+keyboard interrupts, you might something like the following code:
+
+```js
+pyodide.setStdin({read(buf) {
+  const timeoutMilliseconds = 100;
+  while(true) {
+    switch(Atomics.wait(stdinSharedBuffer, 0, 0, timeoutMilliseconds) {
+      case "timed-out":
+        // 100 ms passed but got no data, check for keyboard interrupt then return to waiting on data.
+        pyodide.checkInterrupt();
+        break;
+      case "ok":
+        // ... handle the data somehow
+        break;
+    }
+  }
+}});
+```
+
+See also {ref}`interrupting_execution`.
+
+## Standard Out / Standard Error
+
+(streams-stdout)=
+
+{js:func}`pyodide.setStdout` and {js:func}`pyodide.setStderr` respectively set
+the standard output and standard error handlers. These APIs are identical except
+in their defaults, so we will only discuss the `pyodide.setStdout` except in
+cases where they differ.
+
+As with {js:func}`pyodide.setStdin`, there are quite a few different ways to set
+the standard output handlers.
+
+### Set the default behavior
+
+As with stdin, `pyodide.setStdout()` sets the default behavior. In node, this is
+to write directly to `process.stdout`. In the browser, the default is as if you
+wrote
+`setStdout({batched: (str) => console.log(str)})`
+see below.
+
+### A batched handler
+
+A batched handler is the easiest standard out handler to implement but it is
+also the coarsest. It is intended to use with APIs like `console.log` that don't
+understand partial lines of text or for quick and dirty code.
+
+The batched handler receives a string which is either:
+
+1. a complete line of text with the newline removed or
+2. a partial line of text that was flushed.
+
+For instance after:
+
+```py
+print("hello!")
+import sys
+print("partial line", end="")
+sys.stdout.flush()
+```
+
+the batched handler is called with `"hello!"` and then with `"partial line"`.
+Note that there is no indication that `"hello!"` was a complete line of text and
+`"partial line"` was not.
+
+### A raw handler
+
+A raw handler receives the output one character code at a time. This is neither
+very convenient nor very efficient. It is present primarily for backwards
+compatibility reasons.
+
+For example, the following code:
+
+```py
+print("h")
+import sys
+print("p ", end="")
+print("l", end="")
+sys.stdout.flush()
+```
+
+will call the raw handler with the sequence of bytes:
+
+```py
+0x68 - h
+0x0A - newline
+0x70 - p
+0x20 - space
+0x6c - l
+```
+
+### A write handler
+
+A write handler takes a `Uint8Array` as an argument and is supposed to write the
+data in this buffer to standard output and return the number of bytes written.
+For example, the following class can be used to write to a Node file descriptor:
+
+```js
+const fs = require("fs");
+const tty = require("tty");
+class NodeWriter {
+  constructor(fd) {
+    this.fd = fd;
+    this.isatty = tty.isatty(fd);
+  }
+
+  write(buffer) {
+    return fs.writeSync(this.fd, buffer);
+  }
+}
+```
+
+Using it as follows redirects output from Pyodide to `out.txt`:
+
+```js
+const fd = fs.openSync("out.txt", "w");
+py.setStdout(new NodeWriter(fd));
+```
+
+Or the following gives the default behavior:
+
+```js
+const fd = fs.openSync("out.txt", "w");
+py.setStdout(new NodeWriter(process.stdout.fd));
+```
+
+### isatty
+
+As with `stdin`, is possible to control whether or not
+{py:meth}`sys.stdout.isatty() <io.IOBase.isatty>` returns true with the `isatty`
+option. You cannot combine `isatty: true` with a batched handler.
--- a/emsdk/Makefile
+++ b/emsdk/Makefile
@ -10,7 +10,18 @@ emsdk/.complete: ../Makefile.envs $(wildcard patches/*.patch)
 	cd emsdk/upstream/emscripten/ && cat ../../../patches/*.patch | patch -p1 --verbose
 	cd emsdk && ./emsdk install --build=Release $(PYODIDE_EMSCRIPTEN_VERSION) ccache-git-emscripten-64bit
 	cd emsdk && ./emsdk activate --embedded --build=Release $(PYODIDE_EMSCRIPTEN_VERSION)
+
+	# Check that generated_struct_info is up to date.
+	cmp --silent emsdk/upstream/emscripten/src/generated_struct_info32.json ../src/js/generated_struct_info32.json || \
+		( \
+			echo "" && \
+			echo "The vendored copy of generated_struct_info32.json does not match the copy in Emscripten" && \
+			exit 1 \
+		)
+
 	touch emsdk/.complete

+
+
 clean:
 	rm -rf emsdk
--- a/src/js/api.ts
+++ b/src/js/api.ts
@ -560,8 +560,19 @@ export class PyodideAPI {
   * during execution of C code.
   */
  static checkInterrupt() {
-    if (Module.__PyErr_CheckSignals()) {
-      Module._pythonexc2js();
+    if (Module._PyGILState_Check()) {
+      // GIL held, so it's okay to call __PyErr_CheckSignals.
+      if (Module.__PyErr_CheckSignals()) {
+        Module._pythonexc2js();
+      }
+      return;
+    } else {
+      // GIL not held. This is very likely because we're in a IO handler. If
+      // buffer has a 2, throwing EINTR quits out from the IO handler and tells
+      // the calling context to call `PyErr_CheckSignals`.
+      if (Module.Py_EmscriptenSignalBuffer[0] === 2) {
+        throw new Module.FS.ErrnoError(cDefs.EINTR);
+      }
    }
  }

--- a/src/js/constants.ts
+++ b/src/js/constants.ts
@ -0,0 +1,9 @@
+import type { Module as OrigModule } from "./module";
+import { defines } from "./generated_struct_info32.json";
+
+declare global {
+  export const cDefs: typeof defines;
+  export const DEBUG: boolean;
+  export var Module: OrigModule;
+  export var API: any;
+}
--- a/src/js/esbuild.config.mjs
+++ b/src/js/esbuild.config.mjs
@ -27,6 +27,27 @@ const outputs = [
 ];

 const dest = (output) => join(__dirname, "..", "..", output);
+const DEFINES = { DEBUG: DEBUG.toString() };
+
+function toDefines(o, path = "") {
+  return Object.entries(o).flatMap(([x, v]) => {
+    // Drop anything that's not a valid identifier
+    if (!/^[A-Za-z_$]*$/.test(x)) {
+      return [];
+    }
+    // Flatten objects
+    if (typeof v === "object") {
+      return toDefines(v, path + x + ".");
+    }
+    // Else convert to string
+    return [[path + x, v.toString()]];
+  });
+}
+
+const cdefsFile = join(__dirname, "generated_struct_info32.json");
+const origConstants = JSON.parse(readFileSync(cdefsFile));
+const constants = { cDefs: origConstants.defines };
+Object.assign(DEFINES, Object.fromEntries(toDefines(constants)));

 const config = ({ input, output, format, name: globalName }) => ({
  entryPoints: [join(__dirname, input + ".ts")],
@ -43,7 +64,7 @@ const config = ({ input, output, format, name: globalName }) => ({
    "vm",
    "ws",
  ],
-  define: { DEBUG: DEBUG.toString() },
+  define: DEFINES,
  minify: !DEBUG,
  keepNames: true,
  sourcemap: true,
--- a/src/js/generated_struct_info32.json
+++ b/src/js/generated_struct_info32.json
--- a/src/js/generated_struct_info32.json.d.ts
+++ b/src/js/generated_struct_info32.json.d.ts
@ -0,0 +1,2 @@
+// Trick dts-bundle-generator into behaving correctly
+export declare const defines: { [k: string]: number };
--- a/src/js/load-package.ts
+++ b/src/js/load-package.ts
@ -1,7 +1,4 @@
-declare var Module: any;
-declare var Tests: any;
-declare var API: any;
-declare var DEBUG: boolean;
+import "./constants";

 import {
  IN_NODE,
--- a/src/js/module.ts
+++ b/src/js/module.ts
@ -4,20 +4,54 @@ import { ConfigType } from "./pyodide";
 import { initializeNativeFS } from "./nativefs";
 import { loadBinaryFile } from "./compat";

-type FSNode = any;
-type FSStream = any;
+export type FSNode = {
+  timestamp: number;
+  rdev: number;
+  contents: Uint8Array;
+};
+
+export type FSStream = {
+  tty?: boolean;
+  seekable?: boolean;
+  stream_ops: FSStreamOps;
+  node: FSNode;
+};
+
+export type FSStreamOps = FSStreamOpsGen<FSStream>;
+
+export type FSStreamOpsGen<T> = {
+  open: (a: T) => void;
+  close: (a: T) => void;
+  fsync: (a: T) => void;
+  read: (
+    a: T,
+    b: Uint8Array,
+    offset: number,
+    length: number,
+    pos: number,
+  ) => number;
+  write: (
+    a: T,
+    b: Uint8Array,
+    offset: number,
+    length: number,
+    pos: number,
+  ) => number;
+};

 export interface FS {
  unlink: (path: string) => void;
  mkdirTree: (path: string, mode?: number) => void;
  chdir: (path: string) => void;
  symlink: (target: string, src: string) => FSNode;
-  createDevice: (
+  createDevice: ((
    parent: string,
    name: string,
    input?: (() => number | null) | null,
    output?: ((code: number) => void) | null,
-  ) => FSNode;
+  ) => FSNode) & {
+    major: number;
+  };
  closeStream: (fd: number) => void;
  open: (path: string, flags: string | number, mode?: number) => FSStream;
  makedev: (major: number, minor: number) => number;
@ -26,7 +60,7 @@ export interface FS {
  stat: (path: string, dontFollow?: boolean) => any;
  readdir: (node: FSNode) => string[];
  isDir: (mode: number) => boolean;
-  lookupPath: (path: string) => FSNode;
+  lookupPath: (path: string) => { node: FSNode };
  isFile: (mode: number) => boolean;
  writeFile: (path: string, contents: any, o?: { canOwn?: boolean }) => void;
  chmod: (path: string, mode: number) => void;
@ -41,6 +75,8 @@ export interface FS {
    position?: number,
  ) => number;
  close: (stream: FSStream) => void;
+  ErrnoError: { new (errno: number): Error };
+  registerDevice<T>(dev: number, ops: FSStreamOpsGen<T>): void;
 }

 export interface Module {
@ -58,6 +94,8 @@ export interface Module {
  canvas?: HTMLCanvasElement;
  addRunDependency: (id: string) => void;
  removeRunDependency: (id: string) => void;
+  reportUndefinedSymbols: () => void;
+  ERRNO_CODES: { [k: string]: number };
 }

 /**
--- a/src/js/streams.ts
+++ b/src/js/streams.ts
--- a/src/js/tsconfig.json
+++ b/src/js/tsconfig.json
@ -1,5 +1,6 @@
 {
  "$schema": "http://json.schemastore.org/tsconfig",
+  "rootDir": "../../",
  "compilerOptions": {
    "outDir": "../../dist",
    "esModuleInterop": true,
@ -8,11 +9,12 @@
    "moduleResolution": "node",
    "composite": true,
    "strict": true,
-    "noUnusedLocals": true,
+    "noUnusedLocals": false,
    "types": ["node"],
    "experimentalDecorators": true,
-    "lib": ["ES2018", "DOM"]
+    "lib": ["ES2018", "DOM"],
+    "resolveJsonModule": true
  },
-  "include": ["*.ts"],
+  "include": ["*.ts", "*.json"],
  "exclude": ["test/**/*"]
 }
--- a/src/tests/test_pyodide.py
+++ b/src/tests/test_pyodide.py
@ -1,7 +1,6 @@
 import re
 import shutil
 import subprocess
-from collections.abc import Sequence
 from pathlib import Path
 from textwrap import dedent
 from typing import Any
@ -11,26 +10,11 @@ from pytest_pyodide import run_in_pyodide
 from pytest_pyodide.fixture import selenium_standalone_noload_common
 from pytest_pyodide.server import spawn_web_server

-from conftest import DIST_PATH, ROOT_PATH
+from conftest import DIST_PATH, ROOT_PATH, strip_assertions_stderr
 from pyodide.code import CodeRunner, eval_code, find_imports, should_quiet  # noqa: E402
 from pyodide_build.build_env import get_pyodide_root


-def _strip_assertions_stderr(messages: Sequence[str]) -> list[str]:
-    """Strip additional messages on stderr included when ASSERTIONS=1"""
-    res = []
-    for msg in messages:
-        if msg.strip() in [
-            "sigaction: signal type not supported: this is a no-op.",
-            "Calling stub instead of siginterrupt()",
-            "warning: no blob constructor, cannot create blobs with mimetypes",
-            "warning: no BlobBuilder",
-        ]:
-            continue
-        res.append(msg)
-    return res
-
-
 def test_find_imports():
    res = find_imports(
        """
@ -863,7 +847,7 @@ def test_fatal_error(selenium_standalone):
        return x

    err_msg = strip_stack_trace(selenium_standalone.logs)
-    err_msg = "".join(_strip_assertions_stderr(err_msg.splitlines(keepends=True)))
+    err_msg = "".join(strip_assertions_stderr(err_msg.splitlines(keepends=True)))
    assert (
        err_msg
        == dedent(
@ -1134,177 +1118,6 @@ def test_restore_error(selenium):
    )


-@pytest.mark.skip_refcount_check
-@pytest.mark.skip_pyproxy_check
-def test_custom_stdin_stdout(selenium_standalone_noload, runtime):
-    selenium = selenium_standalone_noload
-    strings = [
-        "hello world",
-        "hello world\n",
-        "This has a \x00 null byte in the middle...",
-        "several\nlines\noftext",
-        "pyodidé",
-        "碘化物",
-        "🐍",
-    ]
-    selenium.run_js(
-        """
-        function* stdinStrings(){
-            for(let x of %s){
-                yield x;
-            }
-        }
-        let stdinStringsGen = stdinStrings();
-        function stdin(){
-            return stdinStringsGen.next().value;
-        }
-        self.stdin = stdin;
-        """
-        % strings
-    )
-    selenium.run_js(
-        """
-        self.stdoutStrings = [];
-        self.stderrStrings = [];
-        function stdout(s){
-            stdoutStrings.push(s);
-        }
-        function stderr(s){
-            stderrStrings.push(s);
-        }
-        let pyodide = await loadPyodide({
-            fullStdLib: false,
-            jsglobals : self,
-            stdin,
-            stdout,
-            stderr,
-        });
-        self.pyodide = pyodide;
-        globalThis.pyodide = pyodide;
-        """
-    )
-    outstrings: list[str] = sum((s.removesuffix("\n").split("\n") for s in strings), [])
-    print(outstrings)
-    assert (
-        selenium.run_js(
-            f"""
-            return pyodide.runPython(`
-                [input() for x in range({len(outstrings)})]
-                # ... test more stuff
-            `).toJs();
-            """
-        )
-        == outstrings
-    )
-
-    [stdoutstrings, stderrstrings] = selenium.run_js(
-        """
-        pyodide.runPython(`
-            import sys
-            print("something to stdout")
-            print("something to stderr",file=sys.stderr)
-        `);
-        return [self.stdoutStrings, self.stderrStrings];
-        """
-    )
-    assert stdoutstrings[-2:] == [
-        "something to stdout",
-    ]
-    stderrstrings = _strip_assertions_stderr(stderrstrings)
-    assert stderrstrings == ["something to stderr"]
-    IN_NODE = runtime == "node"
-    selenium.run_js(
-        f"""
-        pyodide.runPython(`
-            import sys
-            assert not sys.stdin.isatty()
-            assert not sys.stdout.isatty()
-            assert not sys.stderr.isatty()
-        `);
-        pyodide.setStdin();
-        pyodide.setStdout();
-        pyodide.setStderr();
-        pyodide.runPython(`
-            import sys
-            assert sys.stdin.isatty() is {IN_NODE}
-            assert sys.stdout.isatty() is {IN_NODE}
-            assert sys.stderr.isatty() is {IN_NODE}
-        `);
-        """
-    )
-
-
-def test_custom_stdin_stdout2(selenium):
-    result = selenium.run_js(
-        """
-        function stdin(){
-            return "hello there!\\nThis is a several\\nline\\nstring";
-        }
-        pyodide.setStdin({stdin});
-        pyodide.runPython(`
-            import sys
-            assert sys.stdin.read(1) == "h"
-            assert not sys.stdin.isatty()
-        `);
-        pyodide.setStdin({stdin, isatty: false});
-        pyodide.runPython(`
-            import sys
-            assert sys.stdin.read(1) == "e"
-        `);
-        pyodide.setStdout();
-        pyodide.runPython(`
-            assert sys.stdin.read(1) == "l"
-            assert not sys.stdin.isatty()
-        `);
-        pyodide.setStdin({stdin, isatty: true});
-        pyodide.runPython(`
-            assert sys.stdin.read(1) == "l"
-            assert sys.stdin.isatty()
-        `);
-
-        let stdout_codes = [];
-        function rawstdout(code) {
-            stdout_codes.push(code);
-        }
-        pyodide.setStdout({raw: rawstdout});
-        pyodide.runPython(`
-            print("hello")
-            assert sys.stdin.read(1) == "o"
-            assert not sys.stdout.isatty()
-            assert sys.stdin.isatty()
-        `);
-        pyodide.setStdout({raw: rawstdout, isatty: false});
-        pyodide.runPython(`
-            print("2hello again")
-            assert sys.stdin.read(1) == " "
-            assert not sys.stdout.isatty()
-            assert sys.stdin.isatty()
-        `);
-        pyodide.setStdout({raw: rawstdout, isatty: true});
-        pyodide.runPython(`
-            print("3hello")
-            assert sys.stdin.read(1) == "t"
-            assert sys.stdout.isatty()
-            assert sys.stdin.isatty()
-        `);
-        pyodide.runPython(`
-            print("partial line", end="")
-        `);
-        let result1 = new TextDecoder().decode(new Uint8Array(stdout_codes));
-        pyodide.runPython(`
-            sys.stdout.flush()
-        `);
-        let result2 = new TextDecoder().decode(new Uint8Array(stdout_codes));
-        pyodide.setStdin();
-        pyodide.setStdout();
-        pyodide.setStderr();
-        return [result1, result2];
-        """
-    )
-    assert result[0] == "hello\n2hello again\n3hello\n"
-    assert result[1] == "hello\n2hello again\n3hello\npartial line"
-
-
 def test_home_directory(selenium_standalone_noload):
    selenium = selenium_standalone_noload
    selenium.run_js(
--- a/src/tests/test_streams.py
+++ b/src/tests/test_streams.py
@ -0,0 +1,578 @@
+import pytest
+from pytest_pyodide import run_in_pyodide
+
+from conftest import strip_assertions_stderr
+
+
+@pytest.mark.skip_refcount_check
+@pytest.mark.skip_pyproxy_check
+def test_custom_stdin1(selenium_standalone_noload):
+    selenium = selenium_standalone_noload
+    strings = [
+        "hello world",
+        "hello world\n",
+        "This has a \x00 null byte in the middle...",
+        "several\nlines\noftext",
+        "pyodidé",
+        "碘化物",
+        "🐍",
+    ]
+    outstrings: list[str] = sum(
+        ((s.removesuffix("\n") + "\n").splitlines(keepends=True) for s in strings), []
+    )
+    result = selenium.run_js(
+        f"const strings = {strings};"
+        f"const numOutlines = {len(outstrings)};"
+        """
+        let stdinStringsGen = strings[Symbol.iterator]();
+        function stdin(){
+            return stdinStringsGen.next().value;
+        }
+        const pyodide = await loadPyodide({
+            fullStdLib: false,
+            jsglobals : self,
+            stdin,
+        });
+        self.pyodide = pyodide;
+        globalThis.pyodide = pyodide;
+        return pyodide.runPython(`
+            import sys
+            from js import console
+            [sys.stdin.readline() for _ in range(${numOutlines})]
+        `).toJs();
+        """
+    )
+    assert result == outstrings
+
+
+@pytest.mark.skip_refcount_check
+@pytest.mark.skip_pyproxy_check
+def test_custom_stdout1(selenium_standalone_noload, runtime):
+    selenium = selenium_standalone_noload
+    [stdoutstrings, stderrstrings] = selenium.run_js(
+        """
+        self.stdoutStrings = [];
+        self.stderrStrings = [];
+        function stdout(s){
+            stdoutStrings.push(s);
+        }
+        function stderr(s){
+            stderrStrings.push(s);
+        }
+        const pyodide = await loadPyodide({
+            fullStdLib: false,
+            jsglobals : self,
+            stdout,
+            stderr,
+        });
+        self.pyodide = pyodide;
+        globalThis.pyodide = pyodide;
+        pyodide.runPython(`
+            import sys
+            print("something to stdout")
+            print("something to stderr",file=sys.stderr)
+        `);
+        return [stdoutStrings, stderrStrings];
+        """
+    )
+    assert stdoutstrings[-2:] == [
+        "something to stdout",
+    ]
+    stderrstrings = strip_assertions_stderr(stderrstrings)
+    assert stderrstrings == ["something to stderr"]
+    IN_NODE = runtime == "node"
+    selenium.run_js(
+        f"""
+        pyodide.runPython(`
+            import sys
+            assert sys.stdin.isatty() is {IN_NODE}
+            assert not sys.stdout.isatty()
+            assert not sys.stderr.isatty()
+        `);
+        pyodide.setStdin();
+        pyodide.setStdout();
+        pyodide.setStderr();
+        pyodide.runPython(`
+            import sys
+            assert sys.stdin.isatty() is {IN_NODE}
+            assert sys.stdout.isatty() is {IN_NODE}
+            assert sys.stderr.isatty() is {IN_NODE}
+        `);
+        """
+    )
+
+
+def test_custom_stdin_stdout2(selenium):
+    result = selenium.run_js(
+        """
+        const stdinStringsGen = [
+            "hello there!\\nThis is a several\\nline\\nstring"
+        ][Symbol.iterator]();
+        function stdin(){
+            return stdinStringsGen.next().value;
+        }
+        pyodide.setStdin({stdin});
+        try {
+            pyodide.runPython(`
+                import sys
+                assert sys.stdin.read(1) == "h"
+                assert not sys.stdin.isatty()
+            `);
+            pyodide.setStdin({stdin, isatty: false});
+            pyodide.runPython(`
+                import sys
+                assert sys.stdin.read(1) == "e"
+            `);
+            pyodide.setStdout();
+            pyodide.runPython(`
+                assert sys.stdin.read(1) == "l"
+                assert not sys.stdin.isatty()
+            `);
+            pyodide.setStdin({stdin, isatty: true});
+            pyodide.runPython(`
+                assert sys.stdin.read(1) == "l"
+                assert sys.stdin.isatty()
+            `);
+
+            let stdout_codes = [];
+            function rawstdout(code) {
+                stdout_codes.push(code);
+            }
+            pyodide.setStdout({raw: rawstdout});
+            pyodide.runPython(`
+                print("hello")
+                assert sys.stdin.read(1) == "o"
+                assert not sys.stdout.isatty()
+                assert sys.stdin.isatty()
+            `);
+            pyodide.setStdout({raw: rawstdout, isatty: false});
+            pyodide.runPython(`
+                print("2hello again")
+                assert sys.stdin.read(1) == " "
+                assert not sys.stdout.isatty()
+                assert sys.stdin.isatty()
+            `);
+            pyodide.setStdout({raw: rawstdout, isatty: true});
+            pyodide.runPython(`
+                print("3hello")
+                assert sys.stdin.read(1) == "t"
+                assert sys.stdout.isatty()
+                assert sys.stdin.isatty()
+            `);
+            pyodide.runPython(`
+                print("partial line", end="")
+            `);
+            let result1 = new TextDecoder().decode(new Uint8Array(stdout_codes));
+            pyodide.runPython(`
+                sys.stdout.flush()
+            `);
+            let result2 = new TextDecoder().decode(new Uint8Array(stdout_codes));
+            return [result1, result2];
+        } finally {
+            // Flush stdin so other tests don't get messed up.
+            pyodide.runPython(`sys.stdin.read()`);
+            pyodide.runPython(`sys.stdin.read()`);
+            pyodide.runPython(`sys.stdin.read()`);
+            pyodide.setStdin();
+            pyodide.setStdout();
+            pyodide.setStderr();
+        }
+        """
+    )
+    assert result[0] == "hello\n2hello again\n3hello\n"
+    assert result[1] == "hello\n2hello again\n3hello\npartial line"
+
+
+@run_in_pyodide
+def test_stdin_undefined(selenium):
+    from pyodide.code import run_js
+
+    run_js("pyodide.setStdin({stdin: () => undefined})")
+    import sys
+
+    try:
+        print(sys.stdin.read())
+    finally:
+        run_js("pyodide.setStdin()")
+
+
+@run_in_pyodide
+def test_custom_stdin_bytes(selenium):
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    run_js(
+        """
+        const sg = [
+            0x61,
+            0x62,
+            0x00,
+            null,
+            0x63,
+            0x64,
+            null,
+            0x65,
+            0x66,
+            0x67,
+        ][Symbol.iterator]();
+        function stdin() {
+            return sg.next().value;
+        }
+        pyodide.setStdin({stdin});
+        """
+    )
+    try:
+        import sys
+
+        assert sys.stdin.read(5) == "ab\x00"
+        assert sys.stdin.read(5) == "cd"
+        assert sys.stdin.read(2) == "ef"
+        assert sys.stdin.read(2) == "g"
+        assert sys.stdin.read(2) == ""
+    finally:
+        setStdin()
+
+
+@run_in_pyodide
+def test_custom_stdin_buffer_autoeof(selenium):
+    import sys
+
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    stdin = run_js(
+        """
+        const sg = [
+            new Uint8Array([0x61, 0x62, 0x00]),
+            new Uint8Array([0x63, 0x64]),
+            null,
+            new Uint8Array([0x65, 0x66, 0x67]),
+        ][Symbol.iterator]();
+        function stdin() {
+            return sg.next().value;
+        }
+        stdin
+        """
+    )
+    try:
+        setStdin(stdin=stdin)
+        assert sys.stdin.read(5) == "ab\x00"
+        assert sys.stdin.read(5) == "cd"
+        assert sys.stdin.read(2) == ""
+        assert sys.stdin.read(2) == "ef"
+        assert sys.stdin.read(2) == "g"
+        assert sys.stdin.read(2) == ""
+    finally:
+        setStdin()
+
+
+@run_in_pyodide
+def test_custom_stdin_buffer_noautoeof(selenium):
+    import sys
+
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    stdin = run_js(
+        """
+        const sg = [
+            new Uint8Array([0x61, 0x62, 0x00]),
+            new Uint8Array([0x63, 0x64]),
+            null,
+            new Uint8Array([0x65, 0x66, 0x67]),
+        ][Symbol.iterator]();
+        function stdin() {
+            return sg.next().value;
+        }
+        stdin;
+        """
+    )
+    try:
+        setStdin(stdin=stdin, autoEOF=False)
+        assert sys.stdin.read(5) == "ab\x00cd"
+        assert sys.stdin.read(2) == "ef"
+        assert sys.stdin.read(2) == "g"
+        assert sys.stdin.read(2) == ""
+    finally:
+        setStdin()
+
+
+@run_in_pyodide
+def test_custom_stdin_string_autoeof(selenium):
+    import sys
+
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    stdin = run_js(
+        r"""
+        const sg = [
+            "ab\x00",
+            "cd",
+            null,
+            "efg",
+        ][Symbol.iterator]();
+        function stdin() {
+            return sg.next().value;
+        }
+        stdin
+        """
+    )
+    try:
+        setStdin(stdin=stdin)
+        assert sys.stdin.read(5) == "ab\x00\nc"
+        assert sys.stdin.read(5) == "d\n"
+        assert sys.stdin.read(2) == "ef"
+        assert sys.stdin.read(2) == "g\n"
+        assert sys.stdin.read(2) == ""
+    finally:
+        sys.stdin.read()
+        setStdin()
+
+
+@run_in_pyodide
+def test_custom_stdin_string_noautoeof(selenium):
+    import sys
+
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    stdin = run_js(
+        """
+        const sg = [
+            "ab\x00",
+            "cd",
+            null,
+            "efg",
+        ][Symbol.iterator]();
+        function stdin() {
+            return sg.next().value;
+        }
+        stdin;
+        """
+    )
+    try:
+        setStdin(stdin=stdin, autoEOF=False)
+        assert sys.stdin.read(7) == "ab\x00\ncd\n"
+        assert sys.stdin.read(2) == "ef"
+        assert sys.stdin.read(3) == "g\n"
+        assert sys.stdin.read(2) == ""
+    finally:
+        sys.stdin.read()
+        setStdin()
+
+
+@run_in_pyodide
+def test_stdin_error(selenium):
+    import pytest
+
+    from pyodide_js import setStdin
+
+    try:
+        setStdin(error=True)
+        with pytest.raises(OSError, match=r".Errno 29. I/O error"):
+            input()
+    finally:
+        setStdin()
+
+
+def test_custom_stdio_read_buggy(selenium):
+    @run_in_pyodide
+    def main(selenium):
+        import pytest
+
+        from pyodide.code import run_js
+        from pyodide_js import setStdin
+
+        setStdin(run_js("({read(buffer) {}})"))
+        try:
+            with pytest.raises(OSError, match=r"\[Errno 29\] I/O error"):
+                input()
+        finally:
+            setStdin()
+
+    main(selenium)
+    # Test that we warned about the buggy write implementation
+    assert selenium.logs.endswith(
+        "read returned undefined; a correct implementation must return a number"
+    )
+
+
+def test_custom_stdio_write_buggy(selenium):
+    @run_in_pyodide
+    def main(selenium):
+        import pytest
+
+        from pyodide.code import run_js
+        from pyodide_js import setStdout
+
+        setStdout(run_js("({write(buffer) {}})"))
+        try:
+            with pytest.raises(OSError, match=r"\[Errno 29\] I/O error"):
+                print("hi\\nthere!!")
+        finally:
+            setStdout()
+            # flush stdout
+            print("\n")
+
+    main(selenium)
+    # Test that we warned about the buggy write implementation
+    expected = "write returned undefined; a correct implementation must return a number"
+    assert expected in selenium.logs.splitlines()
+
+
+def test_custom_stdio_write(selenium):
+    result = selenium.run_js(
+        r"""
+        class MyWriter {
+            constructor() {
+                this.writtenBuffers = [];
+            }
+            write(buffer) {
+                this.writtenBuffers.push(buffer.slice());
+                return buffer.length;
+            }
+        }
+        const o = new MyWriter();
+        pyodide.setStdout(o);
+        pyodide.runPython(String.raw`
+            print('hi\nthere!!')
+            print("This\nis a \tmessage!!\n")
+        `);
+        pyodide.setStdout();
+        return o.writtenBuffers.map((b) => Array.from(b));
+        """
+    )
+    assert [bytes(a).decode() for a in result] == [
+        "hi\nthere!!",
+        "\n",
+        "This\nis a \tmessage!!\n",
+        "\n",
+    ]
+
+
+@run_in_pyodide
+def test_custom_stdin_read1(selenium):
+    from pyodide.code import run_js
+    from pyodide_js import setStdin
+
+    Reader = run_js(
+        r"""
+        function* genFunc(){
+            const encoder = new TextEncoder();
+            let buffer = yield;
+            for(const a of [
+                "mystring",
+                "",
+                "a",
+                "b",
+                "c\n",
+                "def\nghi",
+                "jkl",
+                ""
+            ]) {
+                encoder.encodeInto(a, buffer);
+                buffer = yield a.length;
+            }
+        }
+        class Reader {
+            constructor() {
+                this.g = genFunc();
+                this.g.next();
+            }
+            read(buffer) {
+                return this.g.next(buffer).value;
+            }
+        }
+        Reader
+        """
+    )
+    setStdin(Reader.new())
+    try:
+        assert input() == "mystring"
+        assert input() == "abc"
+        assert input() == "def"
+        assert input() == "ghijkl"
+    finally:
+        setStdin()
+
+    setStdin(Reader.new())
+    import sys
+
+    try:
+        assert sys.stdin.readline() == "mystring"
+        assert sys.stdin.readline() == "abc\n"
+        assert sys.stdin.readline() == "def\n"
+        assert sys.stdin.readline() == "ghijkl"
+    finally:
+        setStdin()
+    setStdin(Reader.new())
+    try:
+        assert sys.stdin.read() == "mystring"
+        assert sys.stdin.read() == "abc\ndef\nghijkl"
+    finally:
+        setStdin()
+
+
+@pytest.mark.parametrize("method", ["read", "stdin"])
+@run_in_pyodide
+def test_custom_stdin_interrupts(selenium, method):
+    import pytest
+
+    from pyodide.code import run_js
+
+    run_js(
+        """
+        ib = new Int32Array(1);
+        pyodide.setInterruptBuffer(ib);
+        pyodide.setStdin({
+            %s () {
+                ib[0] = 2;
+                pyodide.checkInterrupt();
+            }
+        });
+        """
+        % method
+    )
+    try:
+        with pytest.raises(KeyboardInterrupt):
+            input()
+    finally:
+        run_js(
+            """
+            pyodide.setInterruptBuffer();
+            pyodide.setStdin();
+            """
+        )
+
+
+@pytest.mark.parametrize("method", ["batched", "raw", "write"])
+@run_in_pyodide
+def test_custom_stdout_interrupts(selenium, method):
+    import pytest
+
+    from pyodide.code import run_js
+
+    run_js(
+        """
+        ib = new Int32Array(1);
+        pyodide.setInterruptBuffer(ib);
+        pyodide.setStdout({
+            %s () {
+                ib[0] = 2;
+                pyodide.checkInterrupt();
+            }
+        });
+        """
+        % method
+    )
+    try:
+        with pytest.raises(KeyboardInterrupt):
+            print()
+    finally:
+        run_js(
+            """
+            pyodide.setInterruptBuffer();
+            pyodide.setStdout();
+            """
+        )