Commit Graph

26 Commits

Author SHA1 Message Date
jonathanmetzman a458e3c8ed
[cifuzz] Copy repo from image before checking out. (#6941)
* [cifuzz] Copy repo from image before checking out.

Do this instead of cloning repo anew.
Fixes: https://github.com/google/oss-fuzz/issues/6755

* fix tests

* Add test code for cifuzz-example

* fix
2021-12-01 09:36:08 -05:00
jonathanmetzman 36e5931a19
Make a better api for platform specific config. (#6708)
1. Move most of the generic stuff into BasePlatformConfig.
2. Make a GCB platform config that sets WORKSPACE and PROJECT_SRC_PATH so users don't have to.
3. Make a skeleton prow platform_config for Mitchel to fill out.
4. Make users explicitly specify the CFL_PLATFORM so we can pick CI environments
2021-11-01 12:29:38 +00:00
jonathanmetzman 4ce563f882
Fix diffing on non-GitHub and improve config (#6707)
Fix diffing on non-GitHub and improve config

1. Remove obsolete comments.
2. Remove unused function get_pr_ref
3. Correct comment on git_sha
4. Rename commit_sha to git_sha
5. Make base_commit, pr_ref, and base_ref non-github specific and move to
CiEnvironment.
6. Hoist get_diff_base to base class to allow diffing
7. Fix downloading coverage on non-Github.
8. Add TODO so that we don't assume github actions is run on github.com
(enterprise users)
9. Rename repo_url to git_url.
10. Add missing git_url method base class.
11. Clarify what git_url, git_sha and pr_ref are for and leave TODOs about how
we can eliminate them.
12. Fix typos.
2021-10-31 20:36:07 -04:00
jonathanmetzman b77a55b9b4
[ClusterFuzzLite] Support GCB and gsutil/gcs as filestore. (#6629)
* add gsutil filestore

* lint

* Fix

* Add build image script

* get gcb fuzzing working

* fmt and fix config_utils_test

* Check that crashes are uploaded

* Add no_filestore

* fix test

* fix tests

* fix

* Print crash URL

* Fix

* fix

* fmt

* lnt

* fix

* fmt
2021-10-27 10:00:04 -04:00
jonathanmetzman 1d588e62cd
clusterfuzzlite: Upload builds after bad build check. (#6531)
Fixes: #6525
Depends on: #6530
2021-10-04 15:21:28 +00:00
jonathanmetzman b30e4f4727
[cifuzz] Handle upgrade to Ubuntu 20.04 (#6249)
Handle upgrade to Ubuntu 20.04

1. Don't do any special handling for MSAN anymore. It isn't needed.
2. Don't do any special handling for msan-libs-builder it doesn't
exist anymore.

Related: https://github.com/google/oss-fuzz/issues/6180
2021-08-26 10:02:19 -07:00
Oliver Chang a4bc23909b
Don't upload builds in run_fuzzers. (#6151)
The current way adds a lot of ordering assumptions, and doesn't fit too
well with parallel batch fuzzing either. Add a "upload-build" boolean action
input that can be added to "build_fuzzers" to upload latest builds
instead.

Builds are now uploaded by commit hash, rather than a fixed "latest" name.
ClusterFuzzLite's download_latest_build will check the last 3 commits and download the
first available build by git hash.
2021-08-17 06:36:06 +00:00
jonathanmetzman 01247be731
[infra][tests][NFC] Change mocked function prefix from "mocked_" to "mock_" (#6198)
Also rename mock_ functions to have impl suffix so we can use mock_ in place of mocked_
2021-08-12 11:25:57 -07:00
jonathanmetzman d01808333d
[cifuzz] Fuzz in cifuzz-base (#6142)
Fixes: #5926
2021-08-05 13:27:24 -07:00
jonathanmetzman f0a42fbb1f
[CIFuzz] Force external to use .clusterfuzzlite for build integrations. (#6167) 2021-08-04 16:13:51 -07:00
jonathanmetzman 729a8fe695
[NFC][cifuzz] Improve env var handling. (#6137)
Also delete useless TODOs.
2021-07-29 08:31:51 -07:00
jonathanmetzman 6baa5e3e68
[CIFuzz] Fix bug in tar_directory (#6083)
We need to split the archive_path
2021-07-21 10:19:32 -07:00
jonathanmetzman 6004ddf363
[NFC][cifuzz] Delete debug code and improve naming (#6082) 2021-07-21 09:18:44 -07:00
Oliver Chang c7cab23c20
Make oss-fuzz-project-name optional. (#6071)
Also rename code uses from `project_name` to `oss_fuzz_project_name`, to
be more explicit, and use it to determine whether or not we're running
an OSS-Fuzz project or not.

For external fuzzing #6051.
2021-07-21 05:32:32 +00:00
jonathanmetzman 857e9c27d0
[CIFuzz][Coverage] Make coverage reports world readable (#6077)
This is needed to upload coverage reports.
Related: #6054
2021-07-20 11:50:27 -07:00
jonathanmetzman 86392ca2f4
[CIFuzz][ClusterFuzzLite] Prepare to support coverage reports for external users (#6074)
1. Make coverage script accept an env var that controls where the generated report is written to. Use this in generate_coverage_report.
2. Rename CoverageGetter to OSSFuzzCoverage
3. Make affected_fuzz_targets get coverage object from clusterfuzz_deployment.
4. Add stubbed FilesystemCoverage class that will be used by non-OSS-Fuzz users.
5. Make ClusterFuzzLite able to return coverage object by downloading old coverage report and returning a FilesystemCoverage object.
6. Add stub to ClusterFuzzLite for uploading coverage.

Related: #6054
2021-07-20 10:04:57 -07:00
jonathanmetzman 0672aa4e1f
[CIFuzz] Don't make everything a subdirectory of /out (#5970)
Use different subdirectories of workspace for builds, old builds, coverage reports, corpora and artifacts/testscases.
2021-06-30 07:34:42 -07:00
jonathanmetzman 343e1b2d33
[CIFuzz] Add coverage report generation. (#5937)
Also:
1. Support coverage builds.
2. Add an integration test for coverage builds and reports.
3. Refactor docker code so that there is less duplication in getting arguments for docker run, in particular when handling container (production) vs no container (testing).
4. Rename coverage.py to get_coverage_report.
5. Add tests for untested functions in docker.py
6. Add a test for get_fuzz_target_runner.
2021-06-23 07:30:11 -07:00
jonathanmetzman 8c4ad095e9
[CIFuzz] Refactor in preparation for filestore (#5841)
1. Moving the _create_config and create_config functions from build_fuzzers_test.py and run_fuzzers_test.py into test_helpers.py (which is now part of cifuzz instead of infra) and share code between them.
2. Rename artifacts_dir to crashes_dir in run_fuzzers.py. "artifacts" is ambiguous.
3. Make some small changes to pytest.ini to improve debugging.
2021-05-26 09:45:22 -07:00
jonathanmetzman d8b0c77eae
[CIFuzz] Support non-github action CI for external projects (#5824)
This is incomplete for a few reasons.
1. It probably can't diff properly.
2. It assumes the project source should be put in /src/$PROJECT_NAME.
2021-05-24 21:22:01 +00:00
jonathanmetzman ffe4f892b1
[helper] Fix build_image --pull and refactor (#5642)
Fix behavior of build_image --pull (Fixes #5640)

Also refactor helper.py:

1. Change behavior of functions so that most return True on success and False on failure.
2. Only main will return 1 on failure and 0 on success now. Previous behavior was very error prone.
3. Rename _get_output_dir to _get_out_dir.
4. Make function docstrings use descriptive tense.
5. Make helper.py print help when no argument is specified.
2021-04-21 14:28:26 +00:00
jonathanmetzman 5792e5c529
[NFC][CIFuzz] Rename test_files to test_data and delete unneeded testcases directory (#5448) 2021-03-19 11:49:38 -07:00
jonathanmetzman 28b35e36fe
[CIFuzz] Support languages non-C++ projects (e.g. Python projects) (#5222)
Allow use of non-C++ projects by specifying the language in the workflow file.
Fixes #5195
2021-02-19 11:54:15 -08:00
jonathanmetzman 9f4d0e5360
[CIFuzz][NFC] Ensure more tests don't modify the repo (#5114)
Ensure more tests dont modify the repo by using a temporary directory.
Also, use setUp to create a tempdir to save code repitition.
2021-02-04 09:07:45 -08:00
jonathanmetzman 5fc28a9d86
[cifuzz][NFC] Use a temporary directory in a test of check_fuzzer_build (#5113)
This ensures that the test doesn't modify the repo.
2021-02-04 08:23:41 -08:00
jonathanmetzman a21e218511
[cifuzz][NFC] Rename cifuzz.py to build_fuzzers.py (#5107)
Since cifuzz.py now only deals with building fuzzers, change the
name to build_fuzzers.py
2021-02-04 06:52:22 -08:00