From db37355d313065b4d793bc9da61bb2d898838c92 Mon Sep 17 00:00:00 2001 From: Jirka Borovec <6035284+Borda@users.noreply.github.com> Date: Thu, 14 Sep 2023 23:57:47 +0200 Subject: [PATCH] ci: add description how to clean machines (#18553) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Carlos Mocholí --- .azure/README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/.azure/README.md b/.azure/README.md index 917532b083..3ae170f47f 100644 --- a/.azure/README.md +++ b/.azure/README.md @@ -44,3 +44,27 @@ done ```bash ps aux | grep start.sh ``` + +# Machine maintenance + +Since most of our jobs/checks are running in a Docker container, the OS/machine can become polluted and fail to run with errors such as: + +``` +No space left on device : '/azp/agent-litGPU-21_0,1/_diag/pages/8bb191f4-a8c2-419a-8788-66e3f0522bea_1.log' +``` + +In such cases, you need to log in to the machine and run `docker system prune`. + +## Automated ways + +Let's explore adding a cron job for periodically removing all Docker caches: + +1. Open your user's cron tab for editing: `crontab -e` +1. Schedule/add the command with the `--force` flag to force pruning without interactive confirmation: + ```bash + # every day at 2:00 AM clean docker caches + 0 2 * * * docker system prune --force + ``` +1. Verify the entry: `crontab -l` + +Note: You may need to add yourself to the Docker group by running `sudo usermod -aG docker ` to have permission to execute this command without needing `sudo` and entering the password.