* [CIFuzz] Allow run_fuzzers to use a preexisting corpus.
Also improve comment in cifuzz-base/Dockerfile.
* improve comment
* Fix run_fuzzers
Add --cache option to helper.py's build_image command which
uses the docker image cache when building and use this in
ci/build.py.
Change build_image_impl's argument no_cache to cache for clarity.
Eliminate default argument cache in build_fuzzers_impl since no users
use it.
Exposing the mount location for local sources to a command-line
argument makes it possible to reproduce findings with local source
even for umbrella projects with WORKDIR $SRC/.
* Filter non-fuzz target binaries in coverage script
* Filter non-fuzz target binaries in helper.py
* Fix fuzz target executable bit check in helper.py
Python and JVM fuzz target executables created via the docs template
set the exectuable via `chmod u+x` as the root user, which means that
os.access checks in infra/helper.py don't see the exectuable bit if
not run as root locally.
With this commit, the check now looks for any of the three exectuable
bits.
Coverage is collected by running the fuzz targets with the JaCoCo agent
in Jazzer's no instrumentation mode.
Since JaCoCo does not support llvm-cov style coverage reports, a simple
Python helper creates it using the information contained in the JaCoCo
XML report.
As the Java build process does not maintain a mapping between source
files and build artifacts and JaCoCo needs to be passed the root folder
of the package tree, we use the Maven directory layout convention to
heuristically detect these roots.
Fix behavior of build_image --pull (Fixes#5640)
Also refactor helper.py:
1. Change behavior of functions so that most return True on success and False on failure.
2. Only main will return 1 on failure and 0 on success now. Previous behavior was very error prone.
3. Rename _get_output_dir to _get_out_dir.
4. Make function docstrings use descriptive tense.
5. Make helper.py print help when no argument is specified.
* build_image with pull should pull all images.
right now, build_image with pull only pulled the build image,
and not the runner image. this led to
https://github.com/google/oss-fuzz/issues/5480
so, pull all images to match pull_images functionality.
* address review comment.
gsutil recently started asking daily for a security key jiggle for accounts
using it. This means if we don't print stderr in real time, users won't see
why a corpus isn't downloading and helper will hang until 2FA request times out.
Rewrite test_all in python.
Bash is quite annoying to write and test.
One issue with bash is it is even worse than Python for parallelism (which may be causing #4707).
Rewrite test_all in python and optimize base-runner/Dockerfile for fast development.
Also, combine some docker layers.
This is a best effort attempt to parse the relevant copy command from
the Dockerfile.
Also add a main function to build_specified_commit to make it easier to
test.
* Max comments
* Added bisection tests
* Added test cases
* Docker run command added to build_specific commit
* Infer main repo uses docker image rather than docker file
* Added mores tests for repo infer
* With verbosity
* Formatting
* Bisection fixed
* Formatting updates
* Oliver small comments.
* Script infer main repo
* Detect repo and test module finished
* Detect main repo from script
* Regex searching
* Regex searching
* Max comments + bisector test script
* Oliver comments
* Max's comments
* String concat update
* Jonathan comments
* Starting the git python API
* repo manager working with tests
* Docker repo manager working, can update image with specific commit
* Working bisection
* Formatting and linting
* Working bisection
* Specific commit checkout
* Specific commit checkout
* Removed bisector and updated helper
* remove branch bug
* Max comments
* Added tests for infer main repo
* Oliver comments
* helper.py function names change
* Oliver comments pt. 2
* Olivers comments
* Formatting updates
* Removing DockerRepoManager class
* Removing DockerRepoManager class
* Changing from copying repo to docker image to mounting repo on docker image
* Jonathan comments
* Build image from commit tests
* Oliver comments pt.3
* Oliver comments pt.4
* Linting
* Max comments pt. 2
* change check commit exists
* Max comments pt. 3
* Regex updated
* Formatting updates
* [infra] Add support for dataflow builds to the helper script and build check (#1632).
* Update travis config file.
* Address self-review comments and specify dataflow sanitizer for zstd as well.
* Fix fuzzing_engines in project.yaml
* Fix bad build check for DFSan.
* Use "hasattr" in helper.py to check the sanitizer argument.
* Address more review comments.
* Remove DataFlow config from zstd.
* fix a typo
jonathanmetzman