1. Move most of the generic stuff into BasePlatformConfig.
2. Make a GCB platform config that sets WORKSPACE and PROJECT_SRC_PATH so users don't have to.
3. Make a skeleton prow platform_config for Mitchel to fill out.
4. Make users explicitly specify the CFL_PLATFORM so we can pick CI environments
Fix diffing on non-GitHub and improve config
1. Remove obsolete comments.
2. Remove unused function get_pr_ref
3. Correct comment on git_sha
4. Rename commit_sha to git_sha
5. Make base_commit, pr_ref, and base_ref non-github specific and move to
CiEnvironment.
6. Hoist get_diff_base to base class to allow diffing
7. Fix downloading coverage on non-Github.
8. Add TODO so that we don't assume github actions is run on github.com
(enterprise users)
9. Rename repo_url to git_url.
10. Add missing git_url method base class.
11. Clarify what git_url, git_sha and pr_ref are for and leave TODOs about how
we can eliminate them.
12. Fix typos.
Getting the LATEST_BUILD_WINDOW latest commits from HEAD^ is wrong. We
should use the diff base instead.
Also modify the Ci.get_diff_base methods to not include "..." in the
return value. This will be appended in get_changed_code_under_test
instead.
This also potentially fixes a behaviour mismatch with GitHub
PRs and diffing against the latest base branch rather than
git merge-base base-branch HEAD.
1 Fix usage of gsutil tool.
2 Get rid of run_fuzzers_mode and change to mode. Fixes: #6677
3 Install requirements before copying source code to make iterative development of cifuzz code faster.
The current way adds a lot of ordering assumptions, and doesn't fit too
well with parallel batch fuzzing either. Add a "upload-build" boolean action
input that can be added to "build_fuzzers" to upload latest builds
instead.
Builds are now uploaded by commit hash, rather than a fixed "latest" name.
ClusterFuzzLite's download_latest_build will check the last 3 commits and download the
first available build by git hash.
* [cifuzz] Create validate method on BaseConfig
Use it to validate that either OSS_FUZZ_PROJECT_NAME
or BUILD_INTEGRATION_PATH is set.
Also use it to validate that workspace is set
(rather than duplicate code).
Add tests.
* Use env var hack to bypass valdiation
* fix
* fix
* fmt
* fix
* tmp
* fix
Don't assume that path is relative to workspace if it is not absolute.
Only make this assumption on GitHub where it will always be true.
Also, add tests.
- Add storage-repo, storage-repo-branch, and
storage-repo-branch-coverage fields to the actions, to indicate that
the Git filestore should be used.
- The CI provided filestore is still used for crashes and builds.
- Replace generic Filestore.upload_directory with typed upload methods
which matches the download methods.
- Rename upload_latest_build to upload_build to make it more generic.
- Make artifact name prefixes an implementation detail of the store.
For #6052.
Also rename code uses from `project_name` to `oss_fuzz_project_name`, to
be more explicit, and use it to determine whether or not we're running
an OSS-Fuzz project or not.
For external fuzzing #6051.
Also:
1. Support coverage builds.
2. Add an integration test for coverage builds and reports.
3. Refactor docker code so that there is less duplication in getting arguments for docker run, in particular when handling container (production) vs no container (testing).
4. Rename coverage.py to get_coverage_report.
5. Add tests for untested functions in docker.py
6. Add a test for get_fuzz_target_runner.
1. Moving the _create_config and create_config functions from build_fuzzers_test.py and run_fuzzers_test.py into test_helpers.py (which is now part of cifuzz instead of infra) and share code between them.
2. Rename artifacts_dir to crashes_dir in run_fuzzers.py. "artifacts" is ambiguous.
3. Make some small changes to pytest.ini to improve debugging.
This will mainly be useful for non-OSS-Fuzz users.
Though it can also be used by OSS-Fuzz users to speed things
up (very slightly in most cases).
Fixes#4377
* [CIFuzz] Add functionality to save diskspace.
Add a LOW_DISK_SPACE env/config var. When this is specified
(always true for Github actions) run_fuzzers will delete
base-builder and the project builder image before fuzzing.
After it finishes fuzzing with a target, it will also
delete the targets, its seed corpus and its corpus.
Related: #4879
Abstract away OSS-Fuzz specific bits into the OSSFuzz implementation
of the ClusterFuzzDeployment class. This will make it easier to implement
support for other deployments of ClusterFuzz (including ClusterFuzzLite).
1. Use ssh_url.
This only affects external (non-oss-fuzz) users.
Since there are none, it doesn't affect anyone.
Even if it did, exploitation would require owning the network
Github actions runs on.
This is to prevent MITM attacks.
2. Affected fuzzers bug:
We accidentally were skipping the remove unaffected functionality.
* [CIFuzz] Fix diffing + Refactor
Make diffing work in two scenarios where it previously failed:
1. Commit fuzzing. In this case, we diff $COMMIT against
$COMMIT^1 because the intent here is to fuzz the commit.
2. Fuzzing PRs that aren't to master. In this case, we previously
were diffing against origin/master. Instead, diff against the local
version of the base repo. This also has the nice effect of handling
PRs that havent pulled from master recently enough.
Also do refactoring.
1. Move code that is different for differenct CI systems to continuous_integration.py
2. Change how configuration in build_fuzzers is handled.
Previously configuration was gotten in build_fuzzers_entrypoint
and passed as individual params. This made code ugly and hard to
read. Instead, move code dealing with config to it's own module
config_utils. This module implements a config class which can
be used to create objects that are passed around to code that needs
it. Making the code much easier to read.
TODO: Move run_fuzzers code to new config system.
jonathanmetzman