Commit Graph

1651 Commits

Author SHA1 Message Date
René Fischer 369fbc53a2
Add missing replace param to upload_corpus() (#7069) 2021-12-29 15:09:04 -05:00
jonathanmetzman 6bfab63749
Revert "Update afl++ (#7008)" (#7022)
This reverts commit 95e437c599.
2021-12-16 00:38:25 +00:00
jonathanmetzman 5015790e72
[CIFuzz] Fix affected fuzz targets. (#7014)
Fixes affected fuzz targets by normalizing path names.
Fixes: https://github.com/google/oss-fuzz/issues/7011
2021-12-14 15:01:22 -05:00
jonathanmetzman 2c6a7dde49
[cifuzz] Fix timeout in reproduce. (#6982)
* [cifuzz] Fix timeout in reproduce.

Consider the testcase unreproducible.

Fixes: https://github.com/google/oss-fuzz/issues/6952

* tmp

* tmp

* fix

* add comment

* tmp

* fix annoying heisenbug

* fmt
2021-12-14 11:32:18 -05:00
jonathanmetzman d7c9d57b6d
[cifuzz] Don't enforce OOM limit when no OOMs aren't reported (#6997)
Fixes https://github.com/google/clusterfuzzlite/issues/66
2021-12-14 09:46:16 -05:00
van Hauser 95e437c599
Update afl++ (#7008)
* update afl++ commit id

* update afl++ commit id

* fix for afl++

* attempt fix for curl

* allow easy reproducable afl++ builds

* new commit id

* fixes

* another afl++ commit increase

* not a relevant update, but why not
2021-12-14 06:23:58 -08:00
jonathanmetzman e96a5f8f40
Allow projects to always upload crashes even if they are "unreportable". (#6998)
Fixes https://github.com/google/clusterfuzzlite/issues/65
2021-12-13 12:04:04 -05:00
jonathanmetzman 9d864ee0e3
Use llvmorg-14-init-7378-gaee49255 for LLVM (#6979)
This version is confirmed to fix the AFL++ issues and was the last version we
used prior to the breakages on December 2nd.
2021-12-07 11:47:58 -05:00
MarcoFalke 79e5dc3d77
[base-clang] Change OUR_LLVM_REVISION (#6970)
* [base-clang] Change OUR_LLVM_REVISION

* llvmorg-14-init-8033-gabb2a91b
2021-12-06 10:24:41 -05:00
jonathanmetzman e3ecb89696
[base-clang] Use llvmorg-14-init-8564-g34b903d8 (#6961)
Use clang version before this change: https://chromium-review.googlesource.com/c/chromium/src/+/3310960
Fixes: #6957
2021-12-03 14:42:20 -05:00
jonathanmetzman a458e3c8ed
[cifuzz] Copy repo from image before checking out. (#6941)
* [cifuzz] Copy repo from image before checking out.

Do this instead of cloning repo anew.
Fixes: https://github.com/google/oss-fuzz/issues/6755

* fix tests

* Add test code for cifuzz-example

* fix
2021-12-01 09:36:08 -05:00
van Hauser 574fd118ed
update afl++ commit id (#6936)
* update afl++ commit id
2021-11-30 10:31:30 -06:00
van Hauser cfa0a24958
Allow for easy reproducable builds with afl++ (#6889)
* update afl++ commit id

* update afl++ commit id

* fix for afl++

* attempt fix for curl

* allow easy reproducable afl++ builds
2021-11-30 08:13:58 -05:00
jonathanmetzman 18a5b07b6b
[java][coverage] Make sure filenames is a string and not a list. (#6928)
* single quotes

* [java][coverage] Make sure filenames is a string and not a list.

Fixes: https://github.com/google/oss-fuzz/issues/6913

* fmt
2021-11-30 08:10:08 -05:00
van Hauser 24f5d036e7
attempted fix for curl (#6882)
* update afl++ commit id

* update afl++ commit id

* fix for afl++

* attempt fix for curl
2021-11-22 10:01:08 -05:00
van Hauser 3eeb973727
fix afl++ issue (#6866)
* update afl++ commit id

* update afl++ commit id

* fix for afl++
2021-11-19 16:54:00 +00:00
van Hauser b011780e30
fix for afl++ (#6843)
* update afl++ commit id

* update afl++ commit id
2021-11-17 16:09:58 +00:00
Navidem e82e180c5f
Employ cloudbuild for build status (#6767)
This resolves #5968 by employing cloud build to run update_build_status.py
2021-11-17 13:47:39 +11:00
MarcoFalke e23200e1e4
infra: Remove redundant --cap-add SYS_PTRACE (#6808)
* infra: Remove redundant --cap-add SYS_PTRACE

* format?
2021-11-15 11:24:55 -05:00
van Hauser 77ba9c1f4d
update afl++ commit id (#6809) 2021-11-11 19:34:28 -05:00
jonathanmetzman fb856de70b
[ClusterFuzzLite] Fix fuzz target search for coverage (#6799)
Coverage uses a different mechanism for determining if a file
is a fuzz target: It considers any executables in the top level
of /out as fuzz targets.

Fixes #6768
2021-11-09 07:49:21 -05:00
jonathanmetzman 5814852f52
Make gsutil the default filestore on GCB. (#6771)
Also clean up https://github.com/google/oss-fuzz/pull/6744
2021-11-05 09:13:41 -04:00
Oliver Chang 3895accdee
Remove stack_parser. (#6763)
Fixes #6762.
2021-11-04 09:34:32 -04:00
jonathanmetzman cc49e689b9
[cfl] Temporary fix for testcase_path in reproduce. (#6760)
See: https://github.com/google/oss-fuzz/issues/6758
2021-11-04 09:31:13 -04:00
Oliver Chang 69e63c52c9
Write project.yaml for external generator. (#6764) 2021-11-04 09:30:49 -04:00
Navidem 176e32b9c0
fix Dockerfile COPY command (#6757) 2021-11-03 15:49:42 -04:00
Oliver Chang a75844def0
Don't check for novel crash in batch fuzzing. (#6749)
* Don't check for novel crash in batch fuzzing.

* format
2021-11-03 10:10:42 -04:00
Oliver Chang 2affa1e8e7
Use nargs='*' instead of REMAINDER for helper.py. (#6750)
argparse.REMAINDER is too greedy and will consume known optional
arguments as well.

Fixes #6686.
2021-11-03 10:09:55 -04:00
Oliver Chang a72ef3884f
Don't set seed for batch fuzzing. (#6736)
Fixes #6714.
2021-11-03 14:22:12 +11:00
Mitchel Herman caf9958a8f
Add DOCKER_IN_DOCKER and filestore to platform config (#6744)
Do this to reduce configuration required for GCB and Prow.
2021-11-03 02:03:18 +00:00
Mitchel Herman 6634315600
Add Prow PlatformConfig (#6732) 2021-11-02 16:14:56 -04:00
jonathanmetzman 7693e9640c
[clusterfuzzlite] Make timeout and OOM reporting optional. (#6711)
OOMs will be reported by default. Timeouts wont.
Fixes: https://github.com/google/oss-fuzz/issues/6703
Should also fix: https://github.com/google/oss-fuzz/issues/6619
Fixes: https://github.com/google/oss-fuzz/issues/3432
Related: https://github.com/google/oss-fuzz/issues/6685
2021-11-02 08:01:46 -04:00
Oliver Chang ef0e5413c9
CFLite: Logging fixes. (#6715)
Don't output stacktrace again in detected bug log message.

Disable Python buffering to avoid log interleaving issues.
2021-11-02 16:50:59 +11:00
jonathanmetzman 010865ad76
Fix typo (#6731) 2021-11-01 15:39:27 -04:00
jonathanmetzman 922daf2d81
Fix pushing clusterfuzzlite-build-fuzzers (#6730) 2021-11-01 15:33:06 -04:00
jonathanmetzman 36e5931a19
Make a better api for platform specific config. (#6708)
1. Move most of the generic stuff into BasePlatformConfig.
2. Make a GCB platform config that sets WORKSPACE and PROJECT_SRC_PATH so users don't have to.
3. Make a skeleton prow platform_config for Mitchel to fill out.
4. Make users explicitly specify the CFL_PLATFORM so we can pick CI environments
2021-11-01 12:29:38 +00:00
jonathanmetzman 1ece14aeb2
Abstract away duplicate code in prepare_for_fuzzer_build (#6706)
Share more code between the 4 implementations of `prepare_for_fuzzer_build`.

This simplifies the code of these implementations, reduces repetition and makes them easier to
understand.

Create helper functions/methods for:
1. Creating a failed `BuildPreparationResult`: `get_build_preparation_failure`
2. Building an external project docker image: `_build_external_project_docker_image` 
3. Cloning a repo and checking out the specified commit/pr: `_clone_repo_and_checkout`
4. Detecting the main repo `_detect_main_repo`
5. Creating a repo manager from an existing checkout: `_create_repo_manager_for_project_src_path`

Change `ExternalGeneric` implementation of `prepare_for_fuzzer_build` to:
1. Call `_create_repo_manager_for_project_src_path`
2. Call `_build_external_project_docker_image`

Change `InternalGeneric` implementation of `prepare_for_fuzzer_build` to:
1. Call `_detect_main_repo`
2. Call `_create_repo_manager_for_project_src_path`

Change `ExternalGithub` implementation of `prepare_for_fuzzer_build` to:
1. Call `_clone_repo_and_checkout`
2. Call `_build_external_project_docker_image`

Change `InternalGithub` implementation of `prepare_for_fuzzer_build` to:
1. Call `_detect_main_repo`
2. Call `_clone_repo_and_checkout`
2021-11-01 02:59:44 +00:00
jonathanmetzman 4ce563f882
Fix diffing on non-GitHub and improve config (#6707)
Fix diffing on non-GitHub and improve config

1. Remove obsolete comments.
2. Remove unused function get_pr_ref
3. Correct comment on git_sha
4. Rename commit_sha to git_sha
5. Make base_commit, pr_ref, and base_ref non-github specific and move to
CiEnvironment.
6. Hoist get_diff_base to base class to allow diffing
7. Fix downloading coverage on non-Github.
8. Add TODO so that we don't assume github actions is run on github.com
(enterprise users)
9. Rename repo_url to git_url.
10. Add missing git_url method base class.
11. Clarify what git_url, git_sha and pr_ref are for and leave TODOs about how
we can eliminate them.
12. Fix typos.
2021-10-31 20:36:07 -04:00
jonathanmetzman 4fd64a3792
[clusterfuzzlite] Upload builds after doing bad build check (#6712)
Builds shouldn't be uploaded if the check fails.
Fixes: https://github.com/google/oss-fuzz/issues/6667
2021-11-01 11:26:59 +11:00
jonathanmetzman 32bf394e2a
[build-images.sh] Improve testing (#6709) 2021-10-31 09:54:39 -04:00
Oliver Chang 3c564bca6c
Fix download_latest_build for pull requests. (#6688)
Getting the LATEST_BUILD_WINDOW latest commits from HEAD^ is wrong. We
should use the diff base instead.

Also modify the Ci.get_diff_base methods to not include "..." in the
return value. This will be appended in get_changed_code_under_test
instead.

This also potentially fixes a behaviour mismatch with GitHub
PRs and diffing against the latest base branch rather than
git merge-base base-branch HEAD.
2021-10-29 09:23:33 -04:00
jonathanmetzman b2fa3547b2
[clusterfuzzlite] Fixes for gsutil (#6683)
1 Fix usage of gsutil tool.
2 Get rid of run_fuzzers_mode and change to mode. Fixes: #6677
3 Install requirements before copying source code to make iterative development of cifuzz code faster.
2021-10-28 15:00:44 -04:00
jonathanmetzman b6375d601f
[clusterfuzzlite] Don't use cifuzz term in image name (#6661)
Make images that dont reference cifuzz but which reference
clusterfuzzlite instead.
2021-10-28 08:30:09 -04:00
Oliver Chang 094e40044f
Use shutil.copytree instead of dir_util.copy_tree. (#6673)
We can use shutil.copytree's dirs_exist_ok now that we have a new
Python.
2021-10-28 15:46:24 +11:00
jonathanmetzman b77a55b9b4
[ClusterFuzzLite] Support GCB and gsutil/gcs as filestore. (#6629)
* add gsutil filestore

* lint

* Fix

* Add build image script

* get gcb fuzzing working

* fmt and fix config_utils_test

* Check that crashes are uploaded

* Add no_filestore

* fix test

* fix tests

* fix

* Print crash URL

* Fix

* fix

* fmt

* lnt

* fix

* fmt
2021-10-27 10:00:04 -04:00
Oliver Chang d951635512
Upload crash artifacts per target by its name. (#6660)
Rather than always upload to the "current" artifact.
2021-10-27 04:02:18 +00:00
Oliver Chang f460c03c8a
cifuzz: Fix artifacts uploading issues. (#6646) 2021-10-27 12:24:26 +11:00
Oliver Chang 570c150a05
cifuzz: set log level to info by default. (#6604)
Also fix some logging issues:

- Don't output every single stacktrace when determining if a crash is reproducible.
  It outputs too many things into the logs and makes it hard to read. 

- Log OSS-Fuzz corpus path

- Fix a minor typo and an error log with deleting the xenial image.
2021-10-26 02:18:17 +00:00
Fabian Meumertzheim 6562f44c8a
Fix builds after Jazzer breaking change (#6622)
Follow-up to f043a72a0e, which became necessary due to
24069c3885
2021-10-20 17:24:25 +00:00
Oliver Chang 9aa85855b4
cifuzz: Rename run-fuzzers-mode to mode. (#6600)
Also change the default value of "ci" to the more consistent and
descriptive "code-change".
2021-10-20 15:02:17 +11:00