From 1a9c0d3da02cd9cc67d97240bcaac412b60bf513 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Sat, 3 Feb 2024 15:26:00 -0800 Subject: [PATCH] Maintainers docs on releases, docker images, python updates (#4470) --- cpython/Makefile | 9 +- docs/development/maintainers.md | 336 +++++++++++++++++++++++--------- 2 files changed, 251 insertions(+), 94 deletions(-) diff --git a/cpython/Makefile b/cpython/Makefile index f9f78d571..ed5594753 100644 --- a/cpython/Makefile +++ b/cpython/Makefile @@ -65,7 +65,14 @@ clean-all: clean $(TARBALL): [ -d $(ROOT)/downloads ] || mkdir $(ROOT)/downloads wget -q -O $@ $(PYTHON_ARCHIVE_URL) - shasum --algorithm 256 $(TARBALL) | grep -q $(PYTHON_ARCHIVE_SHA256) + @GOT_SHASUM=`shasum --algorithm 256 $(TARBALL) | cut -f1 -d' '` \ + && (echo $$GOT_SHASUM | grep -q $(PYTHON_ARCHIVE_SHA256)) \ + || (\ + echo "Got unexpected shasum $$GOT_SHASUM" \ + && echo "If you are updating the Python version, set PYTHON_ARCHIVE_SHA256 in Makefile.envs to this." \ + && exit 1 \ + ) + $(BUILD)/.patched: $(TARBALL) diff --git a/docs/development/maintainers.md b/docs/development/maintainers.md index 89cd6971f..1689bf79a 100644 --- a/docs/development/maintainers.md +++ b/docs/development/maintainers.md @@ -8,13 +8,85 @@ For branch organization we use a variation of the [GitHub Flow](https://guides.github.com/introduction/flow/) with the latest release branch named `stable` (due to ReadTheDocs constraints). +### Preparation for making a major release + +Generally we make a tracking issue with a title like "0.25.0 release planning". + +Follow the steps in {ref}`updating-packages`. + +Read the changelog and tidy it up by adding subsections and proof reading it. + +Generate the list of contributors for the release at the end of the +changelog entry with + +```sh +git shortlog -s LAST_TAG.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s +``` + +where `LAST_TAG` is the tag for the last release. + +Make a pull request with these changes titled "Rearrange changelog for 0.25.0 +release" and merge it. + +### Preparation for making a minor release + +Make a branch called `backports-for-v.vv.v`: + +```sh +git checkout stable +git pull upstream +git checkout -b backports-for-0.23.1 +``` + +Locate the commits you want to backport in the main branch and cherry pick them: + +```sh +git cherry-pick +``` + +Make a pull request from `backports-for-0.23.1` targeting the stable branch. If +you're using the github cli this can be done with: + +```sh +gh pr create -w -B stable +``` + +In the pull request description add a task: + +```md +- [ ] Merge don't squash +``` + +This pull request is a good place to @mention various people to ask if they have +opinions about what should be backported. + +Add an extra commit organizing the changelog into sections and editing changelog +messages. Generate the list of contributors for the release at the end of the +changelog entry with + +```sh +git shortlog -s LAST_TAG.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s +``` + +where `LAST_TAG` is the tag for the last release. Make a branch from main called +`changelog-for-v.vv.v` and apply the same changelog rearrangements there. + +Merge `changelog-for-v.vv.v` and `backports-for-v.vv.v` and then follow the +relevant steps from {ref}`release-instructions`. + +### Preparation for making an alpha release + +Name the first alpha release `x.x.xa1` and in subsequent alphas increment the +final number. No prepration is necessary. Don't update anything in the +changelog. Follow the relevant steps from {ref}`release-instructions`. + (release-instructions)= ### Release Instructions 1. From the root directory of the repository run - ```bash + ```sh ./tools/bump_version.py --new-version # ./tools/bump_version.py --new_version --dry-run ``` @@ -23,35 +95,27 @@ the latest release branch named `stable` (due to ReadTheDocs constraints). make sure there are no extra old versions lying around e.g., `rg -F "0.18"`, `rg -F dev0`, `rg -F dev.0`. -2. Make sure the change log is up-to-date. (Skip for alpha releases.) - - - Indicate the release date in the change log. - - Generate the list of contributors for the release at the end of the - changelog entry with, - ```bash - git shortlog -s LAST_TAG.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s - ``` - where `LAST_TAG` is the tag for the last release. +2. (Skip for alpha release.) Add a heading to the changelog indicating version + and release date 3. Make a PR with the updates from steps 1 and 2. Merge the PR. -4. (Major release only.) Assuming the upstream `stable` branch exists, - rename it to a release branch for the previous major version. For instance if - last release was, `0.20.0`, the corresponding release branch would be - `0.20.X`, +4. (Major release only.) Rename the `stable` branch to a release branch for the + previous major version. For instance if last release was, `0.20.0`, the + corresponding release branch would be `0.20.X`: - ```bash - git fetch upstream - git checkout stable - git checkout -b 0.20.X - git push upstream 0.20.X - git branch -D stable # delete locally + ```sh + git fetch upstream stable:stable + git branch 0.20.X stable + git push -u upstream 0.20.X ``` 5. Create a tag `X.Y.Z` (without leading `v`) and push it to upstream, - ```bash + ```sh + git checkout main + git pull upstream git tag X.Y.Z git push upstream X.Y.Z ``` @@ -60,97 +124,183 @@ the latest release branch named `stable` (due to ReadTheDocs constraints). 6. (Major release only). Create a new `stable` branch from this tag, - ```bash - git checkout -b stable + ```sh + git checkout main + git checkout -B stable git push upstream stable --force ``` -7. Revert the release commit. If making a major release, increment the version - to the next development version specified by Semantic Versioning. - +7. (Major or alpha but not minor release.) Set the version number back to the + development version. If you just released `0.22.0`, set the version to + `0.23.0.dev0`. If you just released `0.22.0a1` then you'll set the version to + `0.22.0.dev0`. Make a new commit from this and push it to upstream. ```sh - # If you just released 0.22.0, then set the next version to 0.23.0 + git checkout main ./tools/bump_version.py --new-version 0.23.0.dev0 + git add -u + git commit -m "0.23.0.dev0" + git push upstream main ``` -8. Update these instructions with any relevant changes. - -### Making a minor release - -For a minor release, commits need to be added to the `stable` branch, ideally via a PR. -This can be done with either, - -- git cherry picking individual commits, - ```bash - git checkout stable - git pull - git checkout -b backport-branch - git cherry-pick - ``` -- or with interactive rebase, - ```bash - git fetch upstream - git checkout stable - git pull - git checkout -b backport-branch - git rebase -i upstream/main - ``` - and indicate which commits to take from `main` in the UI. - -Then follow the relevant steps from {ref}`release-instructions`. - -### Making an alpha release - -Name the first alpha release `x.x.xa1` and in subsequent alphas increment the -final number. Follow the relevant steps from {ref}`release-instructions`. - -### Fixing documentation for a released version +## Fixing documentation for a released version Cherry pick the corresponding documentation commits to the `stable` branch. Use -`[skip ci]` in the commit message. +`git commit --amend` to add `[skip ci]` to the commit message. + +## Updating the Docker image + +Anyone with an account on hub.docker.com can follow the following steps: + +1. Make whatever changes are needed to the Dockerfile. +2. Build the docker image with `docker build .` in the Pyodide root directory. + If the build succeeds, docker will give you a hash for the built image. +3. Use `python ./tools/docker_image_tag.py` to find out what the new image tag + should be. Tag the image with: + ```sh + docker image tag /pyodide-env: + ``` +4. Push the image with: + ```sh + docker image push /pyodide-env: + ``` +5. Replace the image in `.circleci/config.yml` with your newly created image. + Open a pull request with your changes to `Dockerfile` and `.circleci/config.yml`. +6. When the tests pass and the pull request is approved, a maintainer must copy + the new image into the `pyodide` dockerhub account. +7. Then replace the image tag in `.circleci/config.yml`, + `.devcontainer/devcontainer.json`, and `run_docker` with the new image under + the `pyodide` dockerhub account. + +It's also possible to update the docker image by pushing your changes to the +`Dockerfile` to a branch in the `pyodide/pyodide` repo (not on a fork) and +clicking `Run workflow` on +https://github.com/pyodide/pyodide/actions/workflows/docker_image.yml. + +(updating-packages)= + +## Updating packages + +Before updating the Python version and before making a major Pyodide release, we +try to update all packages that are not too much trouble. Run + +```sh +make -C packages update-all +``` + +to update all packages and make a pull request with these changes. There will be +build/test failures, revert the packages that fail the build or tests and make a +note to update them independently. ## Upgrading pyodide to a new version of CPython -Prerequisites -- The desired version of CPython must be available at: +### Prerequisites + +The desired version of CPython must be available at: 1. The `specific release` section of https://www.python.org/downloads 2. https://hub.docker.com/_/python 3. https://github.com/actions/python-versions/releases -For example: `v3.11.1` -> `v3.11.2` +If doing a major version update, save time by {ref}`updating-packages` first. -A project maintainer must create an up-to-date Docker image: +### Steps -1. In the pyodide/pyodide github repository (not a fork) change the Python - version at the top of `Dockerfile` to the new version. -2. Click `Run workflow` on https://github.com/pyodide/pyodide/actions/workflows/docker_image.yml - - This will build and upload a new Docker image to https://hub.docker.com/r/pyodide/pyodide-env/tags -3. Re-tag that image with the correct browser and Python versions: `20230301-chrome109-firefox109-py311` -4. Open a new issue for an interested contributor to execute the following tasks... +1. Follow the steps in "Updating the Docker image" to create a docker image for + the new Python version. -Any contributor can complete the Python upgrade: +2. Make sure you are in a Python virtual environment with the new version of + Python and with `requirements.txt` installed. (It is also possible to work in + the docker image as an alternative.) -1. Ensure that the new Docker image has been tagged at https://hub.docker.com/r/pyodide/pyodide-env/tags -2. Download the **Gzipped source tarball** at https://www.python.org/downloads/release/python-3112 into `downloads/` -3. `shasum -a 256 downloads/Python-3.11.2.tgz > cpython/checksums` - - Ensure the path in `cpython/checksums` starts with `downloads/Python-` -4. `git grep --name-only "3.11.1" ` # All of these files will need to be updated. -5. In `.circleci/config.yml` modify the image name to match the image tag on Docker Hub. - - `image: pyodide/pyodide-env:20230301-chrome109-firefox109-py311` -6. In `run_docker` modify the `PYODIDE_IMAGE_TAG` to match the image tag on Docker Hub. - - `PYODIDE_IMAGE_TAG="20230301-chrome109-firefox109-py311"` -7. Rebase any patches which do not apply cleanly. -8. Create a pull request and fix any failing tests. This is historically quite - complicated for major releases of CPython. It may be useful to look at - historical Python upgrades: - | version | pr | - |---|---| - | 3.11 | {pr}`3252` | - | 3.10 |{pr}`2225` | - | 3.9 | {pr}`1637` | - | 3.8 | {pr}`712` | - | 3.7 | {pr}`77` | -9. Apply upgrade_pythoncapi.py to the C extension in `src/code`. +3. Update the Python version in Makefile.envs + +4. Update the Python version in the following locations: + + - `.github/workflows/main.yml` + - `docs/conf.py` + - `docs/development/contributing.md` + - `docs/development/building-and-testing-packages.md` + - `environment.yml` + - `.pre-commit-config.yaml` + - `pyodide-build/pyodide_build/tools/pyo3_config.ini` (two places) + - `pyproject.toml` + + (TODO: make this list shorter.) + +5. Rebase the patches: + + - Clone cpython and cd into it. Checkout the Python version you are upgrading + from. For instance, if the old version is 3.11.3, use `git checkout v3.11.3` + (Python tags have a leading v.) Run + + ```sh + git am ~/path/to/pyodide/cpython/patches/* + ``` + + - Rebase the patches onto the new version of Python. For instance if updating + from Python v3.11.3 to Python 3.12.1: + + ```sh + git rebase v3.11.3 --onto v3.12.1 + ``` + + - Resolve conflicts / drop patches that have been upstreamed. If you have + conflicts, make sure you are using diff3: + + ```sh + git config --global merge.conflictstyle diff3 + ``` + + - Generate the new patches: + ```sh + rm ~/path/to/pyodide/cpython/patches/* + git format-patch v3.12.1 -o ~/path/to/pyodide/cpython/patches/ + ``` + +6. Try to build Python with `make -C cpython`. Fix any build errors. If you + modify the Python source in tree after a failed build it may be useful to run + `make rebuild`. + +7. Try to finish the build with a top level `make`. Fix compile errors in + `src/core` and any link errors. It may be useful to apply + `upgrade_pythoncapi.py --no-compat` to the C extension in `src/code`. https://github.com/python/pythoncapi-compat/blob/main/upgrade_pythoncapi.py - Remove the `#include pythoncapi_compat.h` headers (it injects backwards - compatibility definitions and we don't intend to be backwards compatible). + + The file most tightly coupled to the CPython version is + `src/core/stack_switching/pystate.c`. Consult the following greenlet file to + figure out how to fix it: + https://github.com/python-greenlet/greenlet/blob/master/src/greenlet/TPythonState.cpp + +8. In the virtual environment with the new Python version, run + + ```sh + python src/tests/make_test_list.py + ``` + + Then run the core tests `pytest src/tests/test_core_python.py` and either fix + the failures or update `src/tests/python_tests.yaml` to skip or xfail them. + +9. Try to build packages with: + + ```sh + pyodide build-recipes '*' + ``` + + Disable packages until the build succeeds. Then fix the build failures. In + many cases, this just requires updating to the most recent version of the + package. If you have trouble, try searching on the package's issue tracker + for "python 3.12" (or whatever the new version is). It's best to create + separate PRs for tricky package upgrades. + +10. Fix failing package tests. + +### Old major Python upgrades + +| version | pr | +| ------- | ---------- | +| 3.12 | {pr}`4435` | +| 3.11 | {pr}`3252` | +| 3.10 | {pr}`2225` | +| 3.9 | {pr}`1637` | +| 3.8 | {pr}`712` | +| 3.7 | {pr}`77` |