`go-118-fuzz-build` is undergoing refactoring to fix some runtime issues
affecting Rekor and other projects. Currently none of Rekors fuzzers
run. This PR has been tested on all of Rekors fuzzers that will run with
this PR.
This PR modifies the infra OSS-Fuzz build script to adapt to the
upstream changes.
Cloning the `dev` branch of `go-118-fuzz-build` is a temporary solution
until it gets merged into `main` upstream.
Cloning go-118-fuzz-build in each build script is also a temporary
measure until `dev` gets merged into `main`.
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: David Korczynski <david@adalogics.com>
Cross referencing https://github.com/google/fuzztest/pull/69
Signed-off-by: David Korczynski <david@adalogics.com>
Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
Solves the CI failure in #8690, and uses the chance to add more thorough
tests for the sanitized target binary and the unsanitized target binary
for `Centipede`, which is something we thought about but did not have
the chance to implement:
1. When building sanitized binaries with `helper.py` (i.e., local or
GitHub CI): Unsanitized ones will be built automatically into the same
docker container. Now bad_build_check tests both
* a) If main fuzz targets can run with the auxiliary sanitized binaries,
and
* b) If the auxiliaries are built with the correct sanitizers.
3. When In the Trial build and production build: Two kinds of binaries
will be in separate buckets / docker containers. Now Script
bad_build_check tests either
* a) If the unsanitized binaries can run without the sanitized ones, or
* b) If the sanitized binaries are built with the correct sanitizers.
Co-authored-by: Jonathan Metzman <metzman@chromium.org>
Update `Centipede` to fix [the SegFault when using `AddressSanitizer`
with
`address_space_limit_mb`](https://github.com/google/centipede/issues/166).
Also set `address_space_limit_mb` as the issue has been fixed.
* Add Centipede as a fuzzer
* Specify dictionary param of Centipede
* Update docs
* Mark Centipede as experimental
* More accurate description
* Remove garbage
* Simplify code
* Move mkdir to dockerfile
* Add the weak.c trick
* Install deps with Centipede's script & uninstall new deps
* Fix doc
* Reuse libweak_sancov_stubs.so
* Reorganise flags
* format
* Consistent file type
* Reuse the weak references defined in Centipede
* Replace the shared library of weak symbols with a static one
* Correct the place to call mkdir
* Allow 2G of SHM for Centipede
* Create dirs in run_fuzzer
* Keep Centipede up-to-date
* Avoid duplicating Centipede's binary
* The params of Centipede and their explanations
* The engine info of centipede
* Save the target binary (with san) in a subdir of the project
* Set the target (with san) dir in check_build
* Create the target (with san) first to avoid side-effects
* Fic clone
* Fix format
* Add periods
* Fix comments
* Fix dirs
* Fix parameters
* Adding Centipede as a fuzzing engine for Scarecrow
* Add CI support
* Represent sanitizer with a variable
* Remove the unnecessary definition of FUZZER_OUT
* Reorganise binary directories
* format
* A minor note
* Present issues with dirs that alread exist
* Use os.path.join to join path
* Make a function to get the out/ in check build
* Reusing existing flags in .bazel
* Avoid hardcoding sanitizer, set rss_limit_mb=4096, leave address_space_limit_mb disabled
* Better ways to add bazel build options
* A better way to add bazel flags
* Remove redundant --bazelrc
* Better Cohesion
* Avoid code duplication
* Simplify code
* Exit on crash
* infra: have timeout per fuzz target for coverage
As is done for other languages
* ngolo-fuzzing: remove temporary workaround
now that https://github.com/golang/go/issues/53190 is closed
* ngolo-fuzzing: use built go toolchain in its directory
without copying it to /root/.go/
in order to get coverage for std lib in the end
* infra: ability to get coverage for additional golang package
And uses it with ngolo-fuzzing :
ngolo-fuzzing fuzz targets live in a different repository than
the code being fuzzed, and we we want to get the coverage, for
both the fuzz target and the package being fuzzed
* fixup bash unbound
* fixup ngolo-fuzzing only match at beginning for std package
* stricter check for every additional go package
* infra: fix per-target coverage html generation
When generating coverage reports for single targets we should not
include all fuzz targets (in the $objects var), but only the object
(binary file) for the given target.
Ref: https://github.com/ossf/fuzz-introspector/issues/340
* coverage: remove potential race condition
to make it easier to catch issues like
https://github.com/google/honggfuzz/issues/465.
With -V when unreproducible crashes are triggered honggfuzz shows something like
```
[2022-06-10T15:21:12+0000][W][2373157] subproc_checkTimeLimit():532 pid=2374490 took too much time (limit 30 s). Killing it with SIGKILL
Sz:280 Tm:30,021,052us (i/b/h/e/p/c) New:0/0/0/0/0/3, Cur:0/0/0/0/0/591
Launching verifier for HASH: 1b859677dd (iteration: 1 out of 5)
[2022-06-10T15:21:13+0000][E][2373157] fuzz_runVerifier():280 Verifier stack mismatch: (original) 1b859677dd != (new) 0
```
It should make it easier to figure out why exactly
`./infra/helper.py check_build project fuzz-target` fails
by turning
```
INFO: performing bad build checks for /tmp/not-out/tmpa4lph9dr/fuzz-bus-message
ERROR:root:Check build failed.
```
into something like
```
INFO: performing bad build checks for /tmp/not-out/tmpa4lph9dr/fuzz-bus-message
BAD BUILD: /tmp/not-out/tmpa4lph9dr/fuzz-bus-message seems to have either startup crash or exit:
/tmp/not-out/tmpa4lph9dr/fuzz-bus-message -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null
/tmp/not-out/tmpa4lph9dr/fuzz-bus-message: error while loading shared libraries: libcap.so.2: cannot open shared object file: No such file or directory
ERROR:root:Check build failed.
```
* infra: use only valid targets for python coverage
Some python coverage reports gets clobbered because non-fuzz targets are
included in FUZZ_TARGETS. The consequence of this is that some reports
will have erroneous data. This commit ensures only valid targets are
used when running coverage, which consequently fixes the reports.
* Move executable logic into python fuzzer builder instead
The current run_fuzzer will fail when multiple libfuzzer options are
specificed in a .options file. Currently, if there are multiple options
then none of the options will be used, but rather an issue about a
"binary operator expected" will occur. One implication of this is that
check_build fails to use it and may, therefore, report erroneously. This
fixes it by enabling the use of zero, one and many libfuzzer options.
* infra: fix python coverage
The python coverage has not started working on oss-fuzz.com yet. This is
an effort to fix this.
* add logs and stats files
* log stderr too
Both the JaCoCo .exec file and the XML report are very useful for
automated analysis of coverage data and IDE integration, so serve them
just like the HTML report.
* update afl++ commit id
* update afl++ commit id
* fix for afl++
* attempt fix for curl
* allow easy reproducable afl++ builds
* new commit id
* fixes
* another afl++ commit increase
* not a relevant update, but why not
* two afl++ fixes
* add debug script
* update commit id (not relevant though)
* add wget
* update afl++ commit id
* just afl++ doc updates, but maybe this results in some attention
* ignore afl++ fuzz setup problems during building
* update afl++ commit id
* fix laf-intel implementation
* ensure fuzz tests are running without issues for afl++
* update afl++ commit id
* coverage: introduces llvm-cov-rel
Cf https://github.com/google/oss-fuzz/issues/6268
Latest clang-14 and clang-13 used by rust or swift have a slightly
different profraw file format
llvm-cov-rel is tool that will update the profraw file produced
by clang-13 to one readable by clang-14 llvm-cov tools
* Suricata as a rust project
* rust coverage: remaps every rust subdirectory in fuzz
So that projects not using default fuzz_targets subdir
get the good remap, and hence the good coverage report
AdamKorcz