This fixes eval_code / eval_code_async / CodeRunner.run / CodeRunner.runAsync so that the defaults for globals and locals work as documented. Resolves#3578.
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolves https://github.com/pyodide/pyodide/issues/3337
In Firefox if one writes anything with spaces then tries to copy-paste the input to a standard Python REPL, one gets,
SyntaxError: invalid non-printable character U+00A0
this is because spaces are replaced by the non-breaking space character.
This patch replaces non-breaking space characters with normal space characters in the repl.
This creates a new `pyodide.ffi` submodule and adds a bunch of new subclasses of
`PyProxy` to it.
There are three stages in which we are concerned with the behavior of the
objects we define:
1. at time of static typechecks
2. at execution time
3. when generating docs
Prior to this PR, the subtypes of PyProxy only work well for static type checks,
they work acceptably at runtime (just don't let the user access them), and the
docs don't look that great. This PR is primarily intended to improve the docs
for PyProxy, but they also make execution time checks work better: you can now
say `obj instanceof pyodide.ffi.PyCallable` instead of `obj.isCallable()` which
I is easier to understand and to cross reference against the documentation. I am
marking `isCallable` as deprecated.
I also made a bunch of edits and improvements to the docs.
I have deprecated `PyProxyCallable` in favor of `pyodide.ffi.PyCallable` and
`PyProxy.isCallable` in favor of `obj instanceof pyodide.ffi.PyCallable`.
`PyBuffer` has been renamed to `pyodide.ffi.PyBufferView` and a new `PyBuffer`
has been created which is a subtype of `PyProxy`.
This leads to more consistent rendering (functions and methods get parens after
them) and reduces chances of warnings about getting the wrong link. It is also
possible to use `~fully.quallified.name` to just show `name` if we use a specific
reference type, but it doesn't work with `any` for some reason.
Removes / unvendors some python modules:
- Remove `_aix_support.py`, which is for supporting IBM AIX OS.
- Unvendor `_pydecimal.py`.
- _pydecimal is a pure Python implementation of `decimal` module. [Importing `decimal` fallbacks](https://github.com/python/cpython/blob/main/Lib/decimal.py) to `_pydecimal` if the C-implementation `_decimal` is not available. In our case, _decimal is available, so _pydecimal will not be normally used.
- Unvendor `pydoc_data`.
- pydoc_data contains [a large (~700KB) dictionary](https://github.com/python/cpython/blob/main/Lib/pydoc_data/topics.py) for explaining python builtins. This is mostly used when `help("...")` is called.
This is work towards unvendoring the Pyodide foreign function interface.
Prior to this point, we included a large amount of critical functionality with `--pre-js`.
So we could create an archive called `libpyodide.a` with the object files but to use it
you would have to pass `--pre-js _pyodide.out.js` at link time. This embeds all of this
stuff in an object file called `pyodide_pre.o` which goes in our archive so you get all
the needed js runtime by linking it.
Of course someone trying to use this still has to get the Python code onto the import
path, either using `--preload-file`, using Python to unpack it as a zip archive as we now
do, with zipimporter, or otherwise. They also will have to link `libpython.a` (is CPython
going to start distributing an Emscripten libpython?) and probably various other things.
We have to use a hack to inject the JavaScript code into the object files. The normal
`EM_JS` macro cannot handle arbitrary JavaScript code -- for example it fails with many
regex. Instead we manually generate write a C source file that does what we need using
`xxd`. The generated C code is similar to what `EM_JS` generates, but it uses an array
initializer rather than a string initializer for the characters avoiding the C preprocessor /
compiler's strange opinions about strings.
Instead of putting stuff behind `IN_SPHINX`, define functions and call them from the `setup` function.
In these functions, if we want to expose variables as part of the config we have to assign to `app.config.some_var` which is more explicit.
We still have to make the path change at top level. To improve this, in the future we should:
1. rename the sphinx_pyodide folder to sphinx-pyodide
2. add a `pyproject.toml` and `setup.py` so we can `pip install -e` it
3. instead of modifying the path, source the virtual environment
Up to this point, we've used this dynamic subclassing method for
producing JsProxies for everything but errors. For errors, we make
a wrapper which is not a JsProxy that inherits from Exception and
give the wrapper a "js_error" attribute that points to an actual
error. We also make python2js know about this wrapper so it can
unwrap it. But the raw js_error object is a bit weird. There isn't
anything terrible about this situation but it is mildly unsatisfying.
This changes it so that errors subclass both JsProxy and Exception.
To do this we need to:
1. ensure that they have compatible memory layouts, and
2. convince Python that they have compatible memory layouts
I switched to using a union for the different subtypes of JsProxy
that need extra space: JsCallable, JsBuffer, and JsError. We need
js to be at the end so it won't get in the way of the BaseException
memory layout. I added _Static_asserts to double check that the
memory layouts do in fact agree.
To convince Python that they have compatible memory layouts, we have
to temporarily tell it that JsProxy is a subtype of BaseException.
To do this, we just set JsProxy.__mro__ = (BaseException,) before
creating the type and then restore it afterward.
This prints a better set of error messages in case someone calls os._exit() or in C code exit() is used.
In the future we might like to do something better here, but for now at least we can print a clear
error message.
Also remove almost everything from `_core.py` and moves it to `ffi/__init__.py`.
Micropip imports `IN_BROWSER` from `pyodide._core` so we leave that in there.
Resolves https://github.com/pyodide/pyodide/issues/3432
I added a new decorator called `@docs_argspec` to override the argument specification
of a function used in the docs by setting `func.__wraps__` to a fake function. This only
happens when building the docs, normally it is a no-op.
`@docs_argspec` is needed when using `@overload` because mypy requires the argspec
of the main function to be at least as general as the argspecs of all the overloads, which
causes suboptimal rendering in the api docs.
In #3461 I dropped `sphinx-panels` but of course we *were* using it. It has a successor called
`sphinx-design` which works with sphinx 5.x (still not 6.x but we have several packages that cap
sphinx <6). I also updated the use of the tabbed directive to the new sphinx-design api.
The `pyodide.ffi` stuff is defined in `_pyodide._core_docs`. We don't want `_pyodide._core_docs` to appear in the documentation because this isn't where you should import things from so we override the `__name__` of `_pyodide._core_docs` to be `pyodide.ffi`. But then Sphinx fails to locate the source for the stuff defined in `_pyodide._core_docs`. This patches `ModuleAnalyzer` to tell it to look for the source of things from `pyodide.ffi` in `_pyodide._core_docs`.
Remove the property prefix from properties, add a link for ast.Module.
Previously this included more significant changes but they have been upstreamed into sphinx-autodoc-typehints.
I opened a PR for one of the Napoleon changes:
https://github.com/sphinx-doc/sphinx/pull/11131
But we use Sphinx v5.3 so even if we upstream Napoleon fixes into Sphinx v6.x,
we won't get to use them for a while.
With this we can get up to Sphinx 5.3.0 (from Sphinx 4.5.0). I dropped sphinx-panels
since it has a version pin on Spinx < 5 (we don't seem to use it anyways). I moved to
sphinx-book-theme 0.4.0rc1 since version 0.3.0 pins Sphinx < 5.
This is relevant because new versions of sphinx-autodoc-typehints require Sphinx>=5.3
so we can't pull in my fixes to sphinx-autodoc-typehints.
In the future, I would like for version caps (==, <, <=, and =~) to have comments
explaining them. I removed all the version caps since none of them seem to actively break.
In particular, we can now use jinja2 version 3 so we don't have to pin docutils 0.16.
This adds rust toolchain into our docker image so we don't have to install it every time.
make rust command still exists, but I removed it from the docs. So now a user (who wants to build a package that requires rust) is responsible for installing rust toolchain in their build setup, which is reasonable I think.
Co-authored-by: josephrocca <1167575+josephrocca@users.noreply.github.com>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
The documentation generation seems to get confused by the default export and
displays the `version` field as `default` instead. This switches to a normal export.
Fix various sphinx build warnings.
I removed **all cross-references** in the changelog document. It causes trouble
whenever we make an API change.
For example, Hood did some great work of splitting JsProxy classes into subclasses
in this release, but the changelog of older versions still contains cross-references to
`JsProxy.<method>` which doesn't exist anymore. It doesn't make sense to change it
to e.g. `JsBuffer.<method>` or `JsArray.<method>` as those classes are not available in
that version, so I think the reasonable option would be not using cross-references.
Added type parameters to `JsArray`, `JsMap`, `JsMutableMap`, etc. Based on
[the typesheds for typing](https://github.com/python/typeshed/blob/main/stdlib/typing.pyi).
I skipped `JsCallable` and a few other more complicated types, those can be handled later.
The way we were handling the "routing" in `_core.py` (where in browser things are
imported from `_pyodide_core` and otherwise things from `_pyodide._core_docs`)
made mypy type a bunch of things as `Any`.
This switches to doing the imports from `_pyodide_core` with reflection that mypy
cannot understand so it will correctly type all of the methods with their types from
`_pyodide._core_docs`.
This created some new type errors, so I also made various fixes to make things
typecheck again.
* Don't destroy roundtrip PyProxies automatically
There's a bunch of places where we want to destroy a `PyProxy` if it is being
handed back into Python. This way, if the user wants to keep the proxy around
they can return a copy() of it (whereas otherwise they would have no way to
destroy it after the JavaScript execution is ended).
```js
function new_dict(){
const dict = pyodide.globals.get("dict");
const result = dict([[1,2],[3,4]]);
return result;
// Proxy is destroyed after it is returned!
}
```
If the `PyProxy` has roundtrip set, however, it is generally a bad idea to
destroy it in these cases. In many cases, this means that the object returned
to Python is actually unusable (because we get a destroyed double proxy).
One slightly annoying question is how to deal with the case where (1) we
want the return value NOT to be destroyed and (2) it may or may not be roundtrip
```
function f(px){
// We don't want px to be destroyed so we have to return a copy (the copy
// gets destroyed instead). But if px is roundtrip then the copy is also
// roundtrip and neither gets destroyed so there is a leak. What to do???
return px.copy();
}
This adds lifetime handling to the arguments of JavaScript sync and async generator
functions and to the arguments of `send` and `asend`. The arguments and the `send`
values are kept alive until the generator finishes. This is accomplished through a
wrapper around the generator. The return value of the generator is not destroyed by
the wrapper, but it will get destroyed in the C code for `asend`/`athrow`/`aclose` (unless
the generator wrapper is sent back to JavaScript and called directly).
Resolves https://github.com/pyodide/pyodide/issues/3112
This adds a carefully designed API for controlling stdin, stdout, and stderr. It changes
the default behavior to be a bit more useful, though in doing so introduces some mild
backwards incompatibility. In particular:
1. By default, stdin reads directly from `process.stdin` in node (as before) and raises an
error if in browser (not as before).
2. By default, stdout writes directly to `process.stdout` in node (before it called console.log)
and calls console.log in browser (as before).
3. By default, stderr writes directly to `process.stderr` in node (before it called console.warn)
and calls console.warn in browser (as before).
4. In all three cases, by default isatty(stdin/stdout/stderr) is true in node and false in browser
(in the browser it used to be true).
5. As before, if you pass `stdin`, `stdout`, or `stderr` as arguments to `loadPyodide`, `isatty` of
the corresponding stream is set to false.
The stdin function is now more flexible: we now correctly handle the case where it returns an
ArrayBuffer or ArrayBufferView.
I also added 3 new functions to set streams after Pyodide is loaded which offer additional
control:
* `setStdin({stdin?, error?, isatty = false})` -- Sets the stdin function. The stdin function takes no
arguments and should return null, undefined, a string, or a buffer. Sets and `isatty(stdin)` to
`isatty` (by default `false`). If error is true, set stdin to always raise an EIO error when it is read.
* `setStdout({raw?, batched?, isatty = false})` -- If neither raw nor batched is passed, restore
default stdout behavior. If rwa is passed, the raw stdout function receives a byte which it should
interpret as a utf8 character code. Sets `isatty(stdout)` to isatty (by default `false`). If batched is
passed but not raw, it sets a batched stdout function. The stdout function receives a string and
should do something with it. In this case it ignores isatty and sets isatty(stdout) to false.
* `setStderr({raw?, batched?, isatty = false})` -- same but with stderr.
Hood Chatham