diff --git a/CHANGELOG.md b/CHANGELOG.md index b68c1b088..b4ea5797d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,65 +1,3 @@ -## Unreleased +# Release notes -- The built-in `sqlite` and `bz2` modules of Python are now enabled. - -- New package: `nltk` - -- `micropip` can now be used from web workers. - -## Version 0.13.0 - -- Tagged versions of Pyodide are now deployed to Netlify. - -## Version 0.12.0 - -**User improvements:** - -- Packages with pure Python wheels can now be loaded directly from PyPI. See - `docs/pypi.md` for more information. - -- Thanks to PEP 562, you can now `import js` from Python and use it to access - anything in the global Javascript namespace. - -- Passing a Python object to Javascript always creates the same object in - Javascript. This makes APIs like `removeEventListener` usable. - -- Calling `dir()` in Python on a JavaScript proxy now works. - -- Passing an `ArrayBuffer` from Javascript to Python now correctly creates - a `memoryview` object. - -- Pyodide now works on Safari. - -## Version 0.11.0 - -**User improvements:** - -- Support for built-in modules: - - `sqlite`, `crypt` - -- New packages: `mne` - -**Developer improvements:** - -- The `mkpkg` command will now select an appropriate archive to use, rather than - just using the first. - -- The included version of emscripten has been upgraded to 1.38.30 (plus a - bugfix). - -- New packages: `jinja2`, `MarkupSafe` - -## Version 0.10.0 - -**User improvements:** - -- New packages: `html5lib`, `pygments`, `beautifulsoup4`, `soupsieve`, - `docutils`, `bleach`, `mne` - -**Developer improvements:** - -- `console.html` provides a simple text-only interactive console to test local - changes to Pyodide. The existing notebooks based on legacy versions of Iodide - have been removed. - -- The `run_docker` script can now be configured with environment variables. +See https://pyodide.readthedocs.io/en/latest/changelog.html diff --git a/README.md b/README.md index 24cf5cfb3..d097b1f06 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,3 @@ -**[What is Pyodide?](#what-is-pyodide)** | -**[Try Pyodide](#try-pyodide-no-installation-needed)** | -**[Getting Started](#getting-started)** | -**[Contributing](#contributing)** | -**[License](#license)** - # [Pyodide](https://github.com/iodide-project/pyodide) @@ -12,8 +6,6 @@ The Python scientific stack, compiled to WebAssembly. -[**Try Pyodide and Iodide in your browser**](https://alpha.iodide.io/notebooks/300/) - ## What is Pyodide? **Pyodide** brings the Python runtime to the browser via WebAssembly, along with the Python scientific stack including NumPy, Pandas, Matplotlib, parts of SciPy, and NetworkX. The [`packages` directory](https://github.com/iodide-project/pyodide/tree/master/packages) lists over 35 packages which are currently available. @@ -27,7 +19,9 @@ browser**. ## Try Pyodide (no installation needed) -Try the [iodide demo notebook](https://alpha.iodide.io/notebooks/300/) or fire up a [Python REPL](https://pyodide.cdn.iodide.io/console.html) directly in your browser. +Try the [iodide demo notebook](https://alpha.iodide.io/notebooks/300/) or fire +up a [Python REPL](https://pyodide.cdn.iodide.io/console.html) directly in your +browser. For further information, look through the [documentation](https://pyodide.readthedocs.io/). @@ -36,161 +30,27 @@ For further information, look through the [documentation](https://pyodide.readth Pyodide offers three different ways to get started depending on your needs and technical resources. These include: -- [Download a pre-built version](#download-a-pre-built-version) (the quickest way to get started) -- [Build Pyodide from source](#building-from-source) (this method requires installing prerequistes and using `make`. Primarily for Linux users who want to experiment or contribute back to the project.) -- [Use a Docker image](#using-docker) (recommended for Windows and macOS users and for Linux users who prefer a Debian-based Docker image on Docker Hub with the dependencies -already installed) - - -### Download a pre-built version - -Pre-built versions of Pyodide may be downloaded from -this repository's [releases page](https://github.com/iodide-project/pyodide/releases/). - - -### Building from source - -Building is easiest on Linux and relatively straightforward on Mac. For -Windows, we currently recommend using the Docker image (described below) to -build Pyodide. - -Make sure the prerequisites for [emsdk](https://github.com/emscripten-core/emsdk) are -installed. Pyodide will build a custom, patched version of emsdk, so there is no -need to build it yourself prior. - -Additional build prerequisites are: - -- A working native compiler toolchain, enough to build CPython. -- A native Python 3.7 to run the build scripts. -- PyYAML -- [lessc](http://lesscss.org/) to compile less to css. -- [uglifyjs](https://github.com/mishoo/UglifyJS) to minify Javascript builds. -- gfortran (GNU Fortran 95 compiler) -- [f2c](http://www.netlib.org/f2c/) -- [ccache](https://ccache.samba.org) (optional) *highly* recommended for much faster rebuilds. - -On Mac, you will also need: - -- [Homebrew](https://brew.sh/) for installing dependencies -- System libraries in the root directory (`sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /` should do it, see https://github.com/pyenv/pyenv/issues/1219#issuecomment-428305417) -- coreutils for md5sum and other essential Unix utilities (`brew install coreutils`) -- cmake (`brew install cmake`) -- pkg-config (`brew install pkg-config`) -- openssl (`brew install openssl`) -- gfortran (`brew cask install gfortran`) -- f2c: Install wget (`brew install wget`), and then run the buildf2c script from the root directory (`sudo ./tools/buildf2c`) - -#### Build using `make` - -After installing the build prerequisites, run from the command line: - -```bash -make -``` - -### Using Docker - -We provide a Debian-based Docker image on Docker Hub with the dependencies -already installed to make it easier to build Pyodide. Note that building from -the Docker image is *very* slow on Mac, building on the host machine is -preferred if at all possible. - -1. Install Docker - -2. From a git checkout of Pyodide, run `./run_docker` - -3. Run `make` to build. - -If running ``make`` deterministically stops at one point in each subsequent try, increasing -the maximum RAM usage available to the docker container might help [This is different -from the physical RAM capacity inside the system]. Ideally, at least 3 GB of RAM -should be available to the docker container to build `pyodide` smoothly. These settings can -be changed via Docker Preferences [See [here](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container)]. - -You can edit the files in your source checkout on your host machine, and then -repeatedly run `make` inside the Docker environment to test your changes. +- Use hosted distribution of pyodide: see [using pyodide from + Javascript](https://pyodide.readthedocs.io/en/latest/using_pyodide_from_javascript.html) + documentation. +- Download a pre-built version from this + repository's [releases + page](https://github.com/iodide-project/pyodide/releases/) and serve its contents with + a web server. +- [Build Pyodide from source](https://pyodide.readthedocs.io/en/latest/building_from_sources.html) + - Build natively with `make`: primarily for Linux users who want to + experiment or contribute back to the project. + - [Use a Docker image](https://pyodide.readthedocs.io/en/latest/building_from_sources.html#using-docker): + recommended for Windows and macOS users and for Linux users who prefer a + Debian-based Docker image with the dependencies already installed. ## Contributing -Please view the [CONTRIBUTING](CONTRIBUTING.md) document for tips on filing -issues, making changes, and submitting pull requests. The following sections -describe how to run tests, run Pyodide benchmarks, and lint the source code. - - -### Testing - -Install the following dependencies into the default Python installation: - -```bash -pip install pytest selenium pytest-instafail -``` - -Install [geckodriver](https://github.com/mozilla/geckodriver/releases) and -[chromedriver](https://sites.google.com/a/chromium.org/chromedriver/downloads) -and check that they are in your `PATH`. - -### Automated testing - -To run the pytest suite of tests, type on the command line: - -```bash -pytest test/ packages/ -``` - -### Manual interactive testing - -To run manual interactive tests, a docker environment and a webserver will be -used. - -1. Bind port 8000 for testing. To automatically bind port 8000 of the docker -environment and the host system, run: `./run_docker` - -2. Now, this can be used to test the `pyodide` builds running within the -docker environment using external browser programs on the host system. To do -this, run: `./bin/pyodide serve` - -3. This serves the ``build`` directory of the ``pyodide`` project on port 8000. - * To serve a different directory, use the ``--build_dir`` argument followed - by the path of the directory. - * To serve on a different port, use the ``--port`` argument followed by the - desired port number. Make sure that the port passed in ``--port`` argument - is same as the one defined as ``DOCKER_PORT`` in the ``run_docker`` script. - - -4. Once the webserver is running, simple interactive testing can be run by - visiting this URL: - [http://localhost:8000/console.html](http://localhost:8000/console.html) - -### Benchmarking - -To run common benchmarks to understand Pyodide's performance, begin by -installing the same prerequisites as for testing. Then run: - -```bash -make benchmark -``` - -### Linting - -Python is linted with `flake8`. C and Javascript are linted with -`clang-format`. - -To lint the code, run: - -```bash -make lint -``` +Please view the +[contributing guide](https://pyodide.readthedocs.io/en/latest/rootdir.html#how-to-contribute) +for tips on filing issues, making changes, and submitting pull requests. ## License Pyodide uses the Mozilla Public License Version 2.0. See the [LICENSE file](LICENSE) for more details. - ---- - -**[What is Pyodide?](#what-is-pyodide)** | -**[Try Pyodide](#try-pyodide-no-installation-needed)** | -**[Getting Started](#getting-started)** | -**[Contributing](#contributing)** | -**[License](#license)** | -**[Back to top](#pyodide)** diff --git a/docs/about.md b/docs/about.md new file mode 100644 index 000000000..f12a363c9 --- /dev/null +++ b/docs/about.md @@ -0,0 +1,18 @@ +# About the Project + +The Python scientific stack, compiled to WebAssembly. + +Pyodide brings the Python runtime to the browser via WebAssembly, along with +the Python scientific stack including NumPy, Pandas, Matplotlib, parts of +SciPy, and NetworkX. The [packages +directory](https://github.com/iodide-project/pyodide/tree/master/packages) +lists over 35 packages which are currently available. + +Pyodide provides transparent conversion of objects between Javascript and +Python. When used inside a browser, Python has full access to the Web APIs. + +While closely related to the [iodide project](https://iodide.io), a tool for +*literate scientific computing and communication for the web*, Pyodide goes +beyond running in a notebook environment. To maximize the flexibility of the +modern web, **Pyodide** may be used standalone in any context where you want to +**run Python inside a web browser**. diff --git a/docs/building_from_sources.md b/docs/building_from_sources.md index 078fad156..7bcc0de3a 100644 --- a/docs/building_from_sources.md +++ b/docs/building_from_sources.md @@ -1,8 +1,64 @@ # Building from sources -To install Pyodide from sources follow the steps in the -[readme](./rootdir.html#building-from-source). +Building is easiest on Linux and relatively straightforward on Mac. For +Windows, we currently recommend using the Docker image (described below) to +build Pyodide. +Make sure the prerequisites for [emsdk](https://github.com/emscripten-core/emsdk) are +installed. Pyodide will build a custom, patched version of emsdk, so there is no +need to build it yourself prior. + +Additional build prerequisites are: + +- A working native compiler toolchain, enough to build CPython. +- A native Python 3.7 to run the build scripts. +- PyYAML +- [lessc](http://lesscss.org/) to compile less to css. +- [uglifyjs](https://github.com/mishoo/UglifyJS) to minify Javascript builds. +- gfortran (GNU Fortran 95 compiler) +- [f2c](http://www.netlib.org/f2c/) +- [ccache](https://ccache.samba.org) (optional) *highly* recommended for much faster rebuilds. + +On Mac, you will also need: + +- [Homebrew](https://brew.sh/) for installing dependencies +- System libraries in the root directory (`sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /` should do it, see https://github.com/pyenv/pyenv/issues/1219#issuecomment-428305417) +- coreutils for md5sum and other essential Unix utilities (`brew install coreutils`) +- cmake (`brew install cmake`) +- pkg-config (`brew install pkg-config`) +- openssl (`brew install openssl`) +- gfortran (`brew cask install gfortran`) +- f2c: Install wget (`brew install wget`), and then run the buildf2c script from the root directory (`sudo ./tools/buildf2c`) + +#### Build using `make` + +After installing the build prerequisites, run from the command line: + +```bash +make +``` + +### Using Docker + +We provide a Debian-based Docker image on Docker Hub with the dependencies +already installed to make it easier to build Pyodide. Note that building from +the Docker image is *very* slow on Mac, building on the host machine is +preferred if at all possible. + +1. Install Docker + +2. From a git checkout of Pyodide, run `./run_docker` + +3. Run `make` to build. + +If running ``make`` deterministically stops at one point in each subsequent try, increasing +the maximum RAM usage available to the docker container might help [This is different +from the physical RAM capacity inside the system]. Ideally, at least 3 GB of RAM +should be available to the docker container to build `pyodide` smoothly. These settings can +be changed via Docker Preferences [See [here](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container)]. + +You can edit the files in your source checkout on your host machine, and then +repeatedly run `make` inside the Docker environment to test your changes. ## Partial builds diff --git a/docs/changelog.md b/docs/changelog.md new file mode 100644 index 000000000..7c9ae55ba --- /dev/null +++ b/docs/changelog.md @@ -0,0 +1,67 @@ +# Release notes + +## Unreleased + +- The built-in `sqlite` and `bz2` modules of Python are now enabled. + +- New package: `nltk` + +- `micropip` can now be used from web workers. + +## Version 0.13.0 + +- Tagged versions of Pyodide are now deployed to Netlify. + +## Version 0.12.0 + +**User improvements:** + +- Packages with pure Python wheels can now be loaded directly from PyPI. See + `docs/pypi.md` for more information. + +- Thanks to PEP 562, you can now `import js` from Python and use it to access + anything in the global Javascript namespace. + +- Passing a Python object to Javascript always creates the same object in + Javascript. This makes APIs like `removeEventListener` usable. + +- Calling `dir()` in Python on a JavaScript proxy now works. + +- Passing an `ArrayBuffer` from Javascript to Python now correctly creates + a `memoryview` object. + +- Pyodide now works on Safari. + +## Version 0.11.0 + +**User improvements:** + +- Support for built-in modules: + - `sqlite`, `crypt` + +- New packages: `mne` + +**Developer improvements:** + +- The `mkpkg` command will now select an appropriate archive to use, rather than + just using the first. + +- The included version of emscripten has been upgraded to 1.38.30 (plus a + bugfix). + +- New packages: `jinja2`, `MarkupSafe` + +## Version 0.10.0 + +**User improvements:** + +- New packages: `html5lib`, `pygments`, `beautifulsoup4`, `soupsieve`, + `docutils`, `bleach`, `mne` + +**Developer improvements:** + +- `console.html` provides a simple text-only interactive console to test local + changes to Pyodide. The existing notebooks based on legacy versions of Iodide + have been removed. + +- The `run_docker` script can now be configured with environment variables. diff --git a/docs/rootdir.rst b/docs/code-of-conduct.rst similarity index 60% rename from docs/rootdir.rst rename to docs/code-of-conduct.rst index 3d398ab15..c56675024 100644 --- a/docs/rootdir.rst +++ b/docs/code-of-conduct.rst @@ -1,11 +1,4 @@ .. This rST file provides a way to easily add Markdown files which are located in the root directory of the repo. -About the Project -================= - -.. mdinclude:: ../README.md - -.. mdinclude:: ../CONTRIBUTING.md - .. mdinclude:: ../CODE-OF-CONDUCT.md diff --git a/CONTRIBUTING.md b/docs/contributing.md similarity index 86% rename from CONTRIBUTING.md rename to docs/contributing.md index 0482c9288..191ff42c9 100644 --- a/CONTRIBUTING.md +++ b/docs/contributing.md @@ -2,9 +2,13 @@ Thank you for your interest in contributing to PYODIDE! There are many ways to contribute, and we appreciate all of them. Here are some guidelines & pointers for diving into it. +## Development Workflow + +See [building from sources](./building_from_sources.html) and [testing](./testing.html) documentation. + ## Code of Conduct -PYODIDE has adopted a [Code of Conduct](CODE-OF-CONDUCT.md) that we expect all contributors and core members to adhere to. +PYODIDE has adopted a [Code of Conduct](code-of-conduct.html) that we expect all contributors and core members to adhere to. ## Development @@ -14,7 +18,7 @@ We’ll do our best to keep `master` in a non-breaking state, ideally with tests PYODIDE follows semantic versioning (http://semver.org/) - major versions for breaking changes (x.0.0), minor versions for new features (0.x.0), and patches for bug fixes (0.0.x). -We keep a file, [CHANGELOG.md](CHANGELOG.md), outlining changes to PYODIDE in each release. We like to think of the audience for changelogs as non-developers who primarily run the latest stable. So the change log will primarily outline user-visible changes such as new features and deprecations, and will exclude things that might otherwise be inconsequential to the end user experience, such as infrastructure or refactoring. +We keep a file, [doc/changelog.md](./changelog.html), outlining changes to PYODIDE in each release. We like to think of the audience for changelogs as non-developers who primarily run the latest stable. So the change log will primarily outline user-visible changes such as new features and deprecations, and will exclude things that might otherwise be inconsequential to the end user experience, such as infrastructure or refactoring. ## Bugs & Issues @@ -36,9 +40,6 @@ We use [py.test](https://pytest.org), driving [Selenium](https://www.seleniumhq. All code submissions should pass `make lint`. Python is checked with the default settings of `flake8`. C and Javascript are checked against the Mozilla style in `clang-format`. -## Development Workflow - -See `README.md`. ## License @@ -51,4 +52,4 @@ All contributions to PYODIDE will be licensed under the [Mozilla Public License -[tl;drLegal entry]:https://tldrlegal.com/license/mozilla-public-license-2.0-(mpl-2) \ No newline at end of file +[tl;drLegal entry]:https://tldrlegal.com/license/mozilla-public-license-2.0-(mpl-2) diff --git a/docs/index.rst b/docs/index.rst index de0615405..02126b35e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -60,12 +60,16 @@ information about the project's organization. building_from_sources.md new_packages.md type_conversions.md + contributing.md + testing.md .. toctree:: :titlesonly: :caption: Project - rootdir.rst + about.md + code-of-conduct + changelog.md Indices and tables ================== diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 000000000..6a45d4848 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,67 @@ +# Testing and benchmarking + +## Testing + +### Requirements + +Install the following dependencies into the default Python installation: + +```bash +pip install pytest selenium pytest-instafail +``` + +Install [geckodriver](https://github.com/mozilla/geckodriver/releases) and +[chromedriver](https://sites.google.com/a/chromium.org/chromedriver/downloads) +and check that they are in your `PATH`. + +### Running the test suite + +To run the pytest suite of tests, type on the command line: + +```bash +pytest test/ packages/ +``` + +### Manual interactive testing + +To run manual interactive tests, a docker environment and a webserver will be +used. + +1. Bind port 8000 for testing. To automatically bind port 8000 of the docker +environment and the host system, run: `./run_docker` + +2. Now, this can be used to test the `pyodide` builds running within the +docker environment using external browser programs on the host system. To do +this, run: `./bin/pyodide serve` + +3. This serves the ``build`` directory of the ``pyodide`` project on port 8000. + * To serve a different directory, use the ``--build_dir`` argument followed + by the path of the directory. + * To serve on a different port, use the ``--port`` argument followed by the + desired port number. Make sure that the port passed in ``--port`` argument + is same as the one defined as ``DOCKER_PORT`` in the ``run_docker`` script. + + +4. Once the webserver is running, simple interactive testing can be run by + visiting this URL: + [http://localhost:8000/console.html](http://localhost:8000/console.html) + +## Benchmarking + +To run common benchmarks to understand Pyodide's performance, begin by +installing the same prerequisites as for testing. Then run: + +```bash +make benchmark +``` + +## Linting + +Python is linted with `flake8`. C and Javascript are linted with +`clang-format`. + +To lint the code, run: + +```bash +make lint +```