Commit Graph

908 Commits

Author SHA1 Message Date
DavidKorczynski 529a009be5
infra: refine fuzztest work (#8992)
Enable Fuzztest fuzzers for Tensorflow.

This depends on https://github.com/google/fuzztest/pull/79 and
eventually a PR on the tensorflow repo with the (to be refined) diff in
this PR.

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2022-11-28 18:17:41 +11:00
Navidem 41f9210a29
Revert "[infra] Update Jazzer to latest main" (#9055)
Reverts google/oss-fuzz#9027
per @fmeum request.
2022-11-24 02:21:20 +00:00
Dongge Liu 3ba9eabc27
Update centipede (#9029)
Update `Centipede` to its latest version.
2022-11-23 10:01:38 +11:00
Fabian Meumertzheim 33c5ce4d6f
[infra] Update Jazzer to latest main (#9027)
Reuses the previous Jazzer file names so that no changes to ClusterFuzz
are required.
2022-11-22 09:23:42 -05:00
Dongge Liu 6a4ecab2fb
Disable -Werror on unused-command-line-argument (#9030)
Some projects use `-Werror` to turn all warnings into errors.
This affects `Centipede` as we do not separate build and linking flags
as it expects, which leads to `unused-command-line-argument` warnings.
This PR disables turning that specific warning into errors and keeps the
rest the same.
2022-11-22 21:29:33 +11:00
AdamKorcz 63c99caae9
Refactor infra of Go 1.18 fuzzers (#8937)
`go-118-fuzz-build` is undergoing refactoring to fix some runtime issues
affecting Rekor and other projects. Currently none of Rekors fuzzers
run. This PR has been tested on all of Rekors fuzzers that will run with
this PR.

This PR modifies the infra OSS-Fuzz build script to adapt to the
upstream changes.

Cloning the `dev` branch of `go-118-fuzz-build` is a temporary solution
until it gets merged into `main` upstream.
Cloning go-118-fuzz-build in each build script is also a temporary
measure until `dev` gets merged into `main`.

Signed-off-by: AdamKorcz <adam@adalogics.com>

Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-11-21 17:55:11 +00:00
DavidKorczynski 216ec0b8a4
infra: base-builder: refine fuzz-introspector building (#9004)
- Pass coverage_url to both c/c++ and python report generation.
- Remove a double calling of introspector report generation.

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-21 07:21:59 -08:00
jonathanmetzman 3ebdb071dc
Remove centipede's .git (#8986)
It takes up space, is uneeded and causes irrelevant data to be displayed
in revisions.
2022-11-16 07:53:04 -05:00
DavidKorczynski 45895f868a
infra: bump fuzz-introspector (#8979)
Contains fixes for some python projects that are failing to be analysed.

Ref: https://github.com/ossf/fuzz-introspector/pull/606
2022-11-14 20:42:52 +00:00
DavidKorczynski 9452275da3
fuzztest integration (#8784)
Signed-off-by: David Korczynski <david@adalogics.com>

Cross referencing https://github.com/google/fuzztest/pull/69

Signed-off-by: David Korczynski <david@adalogics.com>
Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2022-11-14 13:34:29 +11:00
DavidKorczynski 4e2f86d3ea
infra: bump fuzz introspector (#8970)
This has a fix for coverage urls for per-fuzzer coverage reports:
https://github.com/ossf/fuzz-introspector/pull/605

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-10 14:12:33 -08:00
DavidKorczynski 8b9cf2d081
infra: bump introspector and update python compile script (#8957)
This is a follow-up to https://github.com/google/oss-fuzz/pull/8952

This bump is needed to have the Python projects work properly. Scan
should be done for all projects now, so have switched it on by default
in fuzz introspector.

Bump also contains
- Fix for util-linux where per-target coverage report links were broken.
- Reduced logging.

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
Co-authored-by: Navidem <navid.emamdoost@gmail.com>
2022-11-09 17:49:17 -08:00
DavidKorczynski 50205ec9e9
infra: bump and refine introspector (#8952)
Bumps fuzz introspector and updates the way python fuzzers are build for
introspector builds.

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-08 23:13:31 -08:00
DavidKorczynski 7797279c27
infra: base-builder: compile: wrap find names in quotes (#8943)
Wrap wildcards in quotes as otherwise the script will autocomplete if
there is a e.g. .json file in the current directory. This causes the
wrong files to be moved, which means `all_cov.json` will not be moved to
the introspector folder and thus wrong (no) coverage data will be used.

This error happens in e.g. g-api-auth-library-python
2022-11-09 09:24:41 +11:00
DavidKorczynski 184b3563c2
infa: bump fuzz introspector (#8917)
Fixes: https://github.com/google/oss-fuzz/issues/8913
2022-11-03 09:47:22 -07:00
DavidKorczynski 95a8398fb8
infra: fuzz-introspector; install matplotlib from binary (#8912)
This avoids compiling certain parts of matplotlib, which speeds up
runtime of fuzz-introspector runs locally by a significant (~5-10min)
time.

Ref:
https://github.com/ossf/fuzz-introspector/pull/579#issuecomment-1300339783
Ref: https://github.com/ossf/fuzz-introspector/issues/465

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-02 12:54:18 -07:00
Navidem bc81324c01
Bump Fuzz Introspector (#8907) 2022-11-01 21:54:34 -07:00
DavidKorczynski c12585c056
infa: add python fuzz-introspector support (#8865)
Adds necessary logic for running fuzz-introspector's Python frontend.

Signed-off-by: David Korczynski <david@adalogics.com>

Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-01 11:51:50 -07:00
Navidem c57a1712eb
Upgrade Atheris (#8886)
Fixes #8682
2022-10-31 10:27:27 -07:00
Navidem 165ee30241
Set introspector env variable in compile (#8878) 2022-10-28 12:06:15 -07:00
Robert Löhning e874146ca5
[infra] Upgrade cmake to latest release (3.24.2) (#8681)
CMake 3.21.1 is known to cause issues when building Qt.
2022-10-27 10:21:10 -04:00
jonathanmetzman 998791514d
Get rid of :introspector images. Build introspector in main images (#8868)
Actually install introspector in every image.
Don't do extra introspector build steps.
Fixes https://github.com/google/oss-fuzz/issues/8625
2022-10-27 08:27:12 -04:00
jonathanmetzman ce2a5a2903
[aflplusplus] Remove 32 MB from image size by deleting .git (#8869) 2022-10-27 12:48:12 +11:00
jonathanmetzman bc02fd0c63
Don't use centipede's install script (#8870)
We can't simply autoremove the packages it installs and it adds 500 MB
to the image size.
Install the only dependency we don't already have libssl-dev since it is very small anyway.
2022-10-26 18:52:22 -04:00
jonathanmetzman 86279540ba
Reduce size of instrospector install by 70 MB (#8862) 2022-10-26 21:00:17 +00:00
jonathanmetzman 474411cbb0
[base-runner] Fix build so that go layer isn't saved. (#8861) 2022-10-26 15:59:09 -04:00
Navidem f5e748bb84
Test OSS-Fuzz with introspector images [DO NOT MERGE] (#8628) 2022-10-25 16:31:04 -07:00
DavidKorczynski a64bdf16a0
Bump fuzz introspector (#8812)
To fetch various improvements, e.g.
- https://github.com/ossf/fuzz-introspector/pull/528
- https://github.com/ossf/fuzz-introspector/pull/546
- https://github.com/ossf/fuzz-introspector/pull/548
- https://github.com/ossf/fuzz-introspector/pull/549
2022-10-18 23:57:07 +00:00
Dongge Liu 947683a594
More thorough build checks for Centipede (#8697)
Solves the CI failure in #8690, and uses the chance to add more thorough
tests for the sanitized target binary and the unsanitized target binary
for `Centipede`, which is something we thought about but did not have
the chance to implement:
1. When building sanitized binaries with `helper.py` (i.e., local or
GitHub CI): Unsanitized ones will be built automatically into the same
docker container. Now bad_build_check tests both
* a) If main fuzz targets can run with the auxiliary sanitized binaries,
and
    * b) If the auxiliaries are built with the correct sanitizers.
3. When In the Trial build and production build: Two kinds of binaries
will be in separate buckets / docker containers. Now Script
bad_build_check tests either
* a) If the unsanitized binaries can run without the sanitized ones, or
* b) If the sanitized binaries are built with the correct sanitizers.

Co-authored-by: Jonathan Metzman <metzman@chromium.org>
2022-10-17 12:48:04 +11:00
Catena cyber 16cdd32578
Runner without go (#8703)
For #8701 cc @jonathanmetzman
2022-10-12 14:55:49 -04:00
AdamKorcz 099fa2e5f5
infra: bump Go to 1.19 (#8568)
Most projects have upgraded to 1.19 which is the latest version.

This upgrades Go to 1.19 in the base-image.

Signed-off-by: AdamKorcz <adam@adalogics.com>

Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-09-22 11:59:15 -07:00
John Howard 880dba286e
Always add `testing` package usage in native_go_fuzzer (#8546)
This fixes an issue where the `testing.F` is replaced with a new
package; if this was the only usage of `testing` in the file then we end
up with `testing` being an unused import. A workaround for this is to
just always use `testing` via some placeholder variable.
2022-09-21 10:49:21 -07:00
Dongge Liu 2757ed6601
Pin Centipede to a more recent version (#8543)
A recent commit allows `Centipede` to add timestamps to its log, which
can be very helpful for debugging.
This PR intends to update `Centipede` to add that commit.
2022-09-21 15:02:44 +10:00
Fabian Meumertzheim 7765e4d548
infra: Update Jazzer to fix coverage build failures (#8495)
Fixes #8241
2022-09-20 13:37:22 -04:00
van Hauser c8a9c23666
update afl++ commit id (#8216)
this fixes a bug that affects coverage.

Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
2022-09-19 18:48:32 -04:00
Thomas Van Lenten 12c881f772
Use Swift 5.7.0. (#8501)
Swift 5.4.2 was released back in June 2021.

Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
2022-09-19 18:40:00 -04:00
jonathanmetzman c0f72989f4
[centipede][base-builder] Properly delete bazel cache (#8494)
Fixes #8468

Co-authored-by: Alan32Liu <donggeliu@google.com>
2022-09-16 13:17:26 +10:00
Dongge Liu 97a68b049b
Update Centipede to fix SegFault (#8507)
Update `Centipede` to fix [the SegFault when using `AddressSanitizer`
with
`address_space_limit_mb`](https://github.com/google/centipede/issues/166).
Also set `address_space_limit_mb` as the issue has been fixed.
2022-09-16 12:33:20 +10:00
Navidem 009a098795
Bump Introspector to pick up latest updates (#8506) 2022-09-16 10:57:10 +10:00
Dongge Liu 2fa71e3c7f
Centipede's CI build, trial build, and build tests (#8422)
Adding CI build, trial build, and build tests.

Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2022-09-16 09:25:49 +10:00
Navidem b13cf7a510
Bump introspector (#8498) 2022-09-15 01:12:21 +00:00
Dongge Liu 46cc569049
Do not remove the symlink as they are used by `Centipede`'s `Bazel` build (#8486) 2022-09-14 02:54:49 +00:00
jonathanmetzman 27b929d05b
Don't install git via ppa (#8470)
Reduce complexity and bloat
2022-09-13 08:37:38 +01:00
DavidKorczynski dc5adbf754
infra: add support for netdriver fuzzing with honggfuzz (#7351)
Adds an example with the mongoose webserver
2022-09-12 11:58:12 +10:00
jonathanmetzman d50dacbfb4
Don't install clang-repl (#8471)
It's unneeded and adds 100MB of unneeded bloat.
2022-09-12 09:39:20 +10:00
jonathanmetzman d3e4ba3bb8
[centipede] Fix centipede (#8467)
Pin centipede to a commit that builds.
Also remove 1.3GB of image bloat it adds
CC @ussuri @Alan32Liu 
Fixes #8458
2022-09-11 13:20:02 -04:00
Navidem b8ef6a216d
Bump Introspector (#8436) 2022-09-08 01:16:05 +00:00
Oliver Chang 91acda2611
Centipede fixes. (#8417)
- Replace newlines with spaces in centipede/clang-flags.txt. Build
  systems don't like them.

- Fix a missing newline in output logs.
2022-09-06 07:41:13 +00:00
Dongge Liu 22488896f6
Centipede integration (#8046)
* Add Centipede as a fuzzer

* Specify dictionary param of Centipede

* Update docs

* Mark Centipede as experimental

* More accurate description

* Remove garbage

* Simplify code

* Move mkdir to dockerfile

* Add the weak.c trick

* Install deps with Centipede's script & uninstall new deps

* Fix doc

* Reuse libweak_sancov_stubs.so

* Reorganise flags

* format

* Consistent file type

* Reuse the weak references defined in Centipede

* Replace the shared library of weak symbols with a static one

* Correct the place to call mkdir

* Allow 2G of SHM for Centipede

* Create dirs in run_fuzzer

* Keep Centipede up-to-date

* Avoid duplicating Centipede's binary

* The params of Centipede and their explanations

* The engine info of centipede

* Save the target binary (with san) in a subdir of the project

* Set the target (with san) dir in check_build

* Create the target (with san) first to avoid side-effects

* Fic clone

* Fix format

* Add periods

* Fix comments

* Fix dirs

* Fix parameters

* Adding Centipede as a fuzzing engine for Scarecrow

* Add CI support

* Represent sanitizer with a variable

* Remove the unnecessary definition of FUZZER_OUT

* Reorganise binary directories

* format

* A minor note

* Present issues with dirs that alread exist

* Use os.path.join to join path

* Make a function to get the out/ in check build

* Reusing existing flags in .bazel

* Avoid hardcoding sanitizer, set rss_limit_mb=4096, leave address_space_limit_mb disabled

* Better ways to add bazel build options

* A better way to add bazel flags

* Remove redundant --bazelrc

* Better Cohesion

* Avoid code duplication

* Simplify code

* Exit on crash
2022-09-06 02:34:58 +00:00
jonathanmetzman dabee12f18
Roll clang to llvmorg-15-init-1464-gbf7f8d6f (#8313)
* Roll clang to llvmorg-15-init-1464-gbf7f8d6f

This is incremental roll meant to break less than #8108

* Pin some projects

* pin cryptofuzz
2022-08-24 10:31:28 -04:00