oss-fuzz/.github/workflows/project_tests.yml

91 lines
2.6 KiB
YAML
Raw Normal View History

2020-06-12 21:39:16 +00:00
name: Project tests
Improve Actions hygiene (#5361) 👋 hello there! I'm a fellow Googler who works on projects that leverage GitHub Actions for CI/CD. Recently I noticed a large increase in our queue time, and I've tracked it down to the [limit of 180 concurrent jobs](https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration) for an organization. To help be better citizens, I'm proposing changes across a few repositories that will reduce GitHub Actions hours and consumption. I hope these changes are reasonable and I'm happy to talk through them in more detail. - Only run GitHub Actions for pushes and PRs against the main branch of the repository. If your team uses a forking model, this change will not affect you. If your team pushes branches to the repository directly, this changes actions to only run against the primary branches or if you open a Pull Request against a primary branch. - For long-running jobs (especially tests), I added the "Cancel previous" workflow. This is very helpful to prevent a large queue backlog when you are doing rapid development and pushing multiple commits. Without this, GitHub Actions' default behavior is to run all actions on all commits. There are other changes you could make, depending on your project (but I'm not an expert): - If you have tests that should only run when a subset of code changes, consider gating your workflow to particular file paths. For example, we have some jobs that do Terraform linting, but [they only run when Terraform files are changed](https://github.com/google/exposure-notifications-verification-server/blob/c4f59fee71042cf668747e599e7c769fca736554/.github/workflows/terraform.yml#L3-L11). Hopefully these changes are not too controversial and also hopefully you can see how this would reduce actions consumption to be good citizens to fellow Googlers. If you have any questions, feel free to respond here or ping me on chat. Thank you!
2021-03-11 02:14:23 +00:00
permissions:
contents: read
Improve Actions hygiene (#5361) 👋 hello there! I'm a fellow Googler who works on projects that leverage GitHub Actions for CI/CD. Recently I noticed a large increase in our queue time, and I've tracked it down to the [limit of 180 concurrent jobs](https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration) for an organization. To help be better citizens, I'm proposing changes across a few repositories that will reduce GitHub Actions hours and consumption. I hope these changes are reasonable and I'm happy to talk through them in more detail. - Only run GitHub Actions for pushes and PRs against the main branch of the repository. If your team uses a forking model, this change will not affect you. If your team pushes branches to the repository directly, this changes actions to only run against the primary branches or if you open a Pull Request against a primary branch. - For long-running jobs (especially tests), I added the "Cancel previous" workflow. This is very helpful to prevent a large queue backlog when you are doing rapid development and pushing multiple commits. Without this, GitHub Actions' default behavior is to run all actions on all commits. There are other changes you could make, depending on your project (but I'm not an expert): - If you have tests that should only run when a subset of code changes, consider gating your workflow to particular file paths. For example, we have some jobs that do Terraform linting, but [they only run when Terraform files are changed](https://github.com/google/exposure-notifications-verification-server/blob/c4f59fee71042cf668747e599e7c769fca736554/.github/workflows/terraform.yml#L3-L11). Hopefully these changes are not too controversial and also hopefully you can see how this would reduce actions consumption to be good citizens to fellow Googlers. If you have any questions, feel free to respond here or ping me on chat. Thank you!
2021-03-11 02:14:23 +00:00
on:
pull_request:
branches:
- master
2020-06-12 21:39:16 +00:00
jobs:
build:
runs-on: ubuntu-latest
concurrency:
group: ${{ github.workflow }}-${{ matrix.sanitizer }}-${{ matrix.engine }}-${{ matrix.architecture }}-${{ github.ref }}
cancel-in-progress: true
permissions:
actions: write
2020-06-12 21:39:16 +00:00
strategy:
fail-fast: false
matrix:
engine:
- libfuzzer
sanitizer:
- address
- memory
- undefined
- coverage
2020-06-12 21:39:16 +00:00
architecture:
- x86_64
include:
- engine: afl
sanitizer: address
architecture: x86_64
- engine: honggfuzz
sanitizer: address
architecture: x86_64
- engine: libfuzzer
sanitizer: address
architecture: i386
- engine: none
sanitizer: address
architecture: x86_64
- engine: wycheproof
sanitizer: none
architecture: x86_64
Centipede integration (#8046) * Add Centipede as a fuzzer * Specify dictionary param of Centipede * Update docs * Mark Centipede as experimental * More accurate description * Remove garbage * Simplify code * Move mkdir to dockerfile * Add the weak.c trick * Install deps with Centipede's script & uninstall new deps * Fix doc * Reuse libweak_sancov_stubs.so * Reorganise flags * format * Consistent file type * Reuse the weak references defined in Centipede * Replace the shared library of weak symbols with a static one * Correct the place to call mkdir * Allow 2G of SHM for Centipede * Create dirs in run_fuzzer * Keep Centipede up-to-date * Avoid duplicating Centipede's binary * The params of Centipede and their explanations * The engine info of centipede * Save the target binary (with san) in a subdir of the project * Set the target (with san) dir in check_build * Create the target (with san) first to avoid side-effects * Fic clone * Fix format * Add periods * Fix comments * Fix dirs * Fix parameters * Adding Centipede as a fuzzing engine for Scarecrow * Add CI support * Represent sanitizer with a variable * Remove the unnecessary definition of FUZZER_OUT * Reorganise binary directories * format * A minor note * Present issues with dirs that alread exist * Use os.path.join to join path * Make a function to get the out/ in check build * Reusing existing flags in .bazel * Avoid hardcoding sanitizer, set rss_limit_mb=4096, leave address_space_limit_mb disabled * Better ways to add bazel build options * A better way to add bazel flags * Remove redundant --bazelrc * Better Cohesion * Avoid code duplication * Simplify code * Exit on crash
2022-09-06 02:34:58 +00:00
- engine: centipede
sanitizer: address
architecture: x86_64
2023-02-06 20:47:29 +00:00
- engine: centipede
sanitizer: none
architecture: x86_64
2020-06-12 21:39:16 +00:00
env:
ENGINE: ${{ matrix.engine }}
SANITIZER: ${{ matrix.sanitizer }}
ARCHITECTURE: ${{ matrix.architecture }}
steps:
- uses: actions/checkout@v4
with: # Needed for git diff to work. (get_changed_files)
fetch-depth: 0
- run: |
2020-06-12 21:39:16 +00:00
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/master
- name: Clear unnecessary files
run: |
df -h
2020-06-12 21:39:16 +00:00
sudo swapoff -a
sudo rm -f /swapfile
sudo apt clean
docker rmi $(docker images -a -q)
df -h
echo "Remove large unused folders, inspired by https://github.com/apache/flink/blame/master/tools/azure-pipelines/free_disk_space.sh"
sudo bash -c '(ionice -c 3 nice -n 19 rm -rf /usr/share/dotnet/ /usr/local/graalvm/ /usr/local/.ghcup/ /usr/local/share/powershell /usr/local/share/chromium /usr/local/lib/android /usr/local/lib/node_modules)&'
2020-06-12 21:39:16 +00:00
- name: Setup python environment
uses: actions/setup-python@v5
2020-06-12 21:39:16 +00:00
with:
2022-12-27 17:40:55 +00:00
python-version: 3.8
cache: pip
cache-dependency-path: |
infra/ci/requirements.txt
2020-06-12 21:39:16 +00:00
- name: Install dependencies
2020-07-15 16:46:24 +00:00
run: |
python -m pip install --upgrade pip
pip install -r infra/ci/requirements.txt
2020-06-12 21:39:16 +00:00
- name: Run project tests
run: python infra/ci/build.py