* [CIFuzz] Allow run_fuzzers to use a preexisting corpus.
Also improve comment in cifuzz-base/Dockerfile.
* improve comment
* Fix run_fuzzers
Add --cache option to helper.py's build_image command which
uses the docker image cache when building and use this in
ci/build.py.
Change build_image_impl's argument no_cache to cache for clarity.
Eliminate default argument cache in build_fuzzers_impl since no users
use it.
* [infra][jvm] Add Jazzer UBSan support
* [java-example] Reenable and plant UB
* [docs] Mention support for Java UBSan in docs
Also adds a link to the java-example build.sh to the docs.
1. Moving the _create_config and create_config functions from build_fuzzers_test.py and run_fuzzers_test.py into test_helpers.py (which is now part of cifuzz instead of infra) and share code between them.
2. Rename artifacts_dir to crashes_dir in run_fuzzers.py. "artifacts" is ambiguous.
3. Make some small changes to pytest.ini to improve debugging.
* Exclude Jazzer classes from coverage reports
* Generate file-level coverage data for Java
Adds per-file coverage information to llvm-cov style *.json files for
Java targets.
This provides full CI Fuzz support for "jvm" projects.
For proper support of Bazel's runfiles tree, a shell script wrapper is
also needed for C++ targets, see
https://github.com/bazelbuild/rules_fuzzing/pull/149.
This commit allows shell script fuzz targets for all languages. This
also helps to consolidate the various fuzz target checks in OSS-Fuzz
by removing a dependence on fuzzing_language.
* [infra] Make `targets_list` detect JVM/Python targets
`targets_list` should not detect `jazzer_driver` and `jazzer_driver_asan` as fuzz targets, but should consider shell scripts that contain `LLVMFuzzerTestOneInput`.
@inferno-chromium This should fix https://github.com/google/oss-fuzz/pull/5770#issuecomment-841851786.
* Fix
Builds of base-runner fail as the Chromium coverage helper script has a
transitive dependency on MarkupSafe, which as of version 2.0 requires
additional python packaging tools to be installed.
This commit installs python3-setuptools and wheels, which fixes the
build.
* Filter non-fuzz target binaries in coverage script
* Filter non-fuzz target binaries in helper.py
* Fix fuzz target executable bit check in helper.py
Python and JVM fuzz target executables created via the docs template
set the exectuable via `chmod u+x` as the root user, which means that
os.access checks in infra/helper.py don't see the exectuable bit if
not run as root locally.
With this commit, the check now looks for any of the three exectuable
bits.
Coverage is collected by running the fuzz targets with the JaCoCo agent
in Jazzer's no instrumentation mode.
Since JaCoCo does not support llvm-cov style coverage reports, a simple
Python helper creates it using the information contained in the JaCoCo
XML report.
As the Java build process does not maintain a mapping between source
files and build artifacts and JaCoCo needs to be passed the root folder
of the package tree, we use the Maven directory layout convention to
heuristically detect these roots.
* retry sequentially if multiprocessing do_bad_build_check detects failures
https://github.com/google/oss-fuzz/issues/5441
The error seen in the build log is:
Whoops, the target binary crashed suddenly, before receiving any input
from the fuzzer!
suggesting that the fuzzer crashed before it got to do anything.
Debugging locally what I tend to see is that
a) in src/afl-forkserver.c afl_fsrv_start the read_s32_timed call
returns 0 and that triggers kill(fsrv->fsrv_pid, fsrv->kill_signal);
(SIGKILL)
b) read_s32_timed returns 0 because *stop_soon_p is non-zero at
restart_read:
c) *stop_soon_p becomes non-zero in handle_stop_sig of
src/afl-fuzz-init.c due to receiving SIGINT
d) that SIGINT is sent by the timeout script used in bad_build_check so
it is that "outer" timeout process which is sending SIGINT which then
triggers afl-forkserver's internal SIGKILL to kill the process
I get improved results if I retry the killed off fuzzers sequentially
* Remove unneeded semicolons to fix presubmit
Co-authored-by: Abhishek Arya <inferno@chromium.org>
* Fixes go coverage with modules
* Golang coverage html report turning off modules
Otherwise, we get the error
working directory is not part of a module
Reduce image size by:
1. Not installing go toolchain in final image. Build go tools in
seperate image that doesn't become base-runner.
2. Download the JVM zip in the same step we remove it.
compile, bad_build_check, and presubmit.py require small tweaks to
support JVM fuzz targets, most of which are similar to those required
for Python. The following additional changes are required:
* Since the Jazzer driver binary already links in libFuzzer, it should
not be built as a static library.
* It is not clear how to do architecture checks as JVM fuzz targets can
load their native dependencies dynamically at runtime. For now, the
check is disabled.
* The Jazzer binaries are moved into $OUT and need to be skipped over in
find_fuzz_targets.
Jazzer is built from HEAD using Bazel and the clang toolchain provided
by base-clang. While it could be built with OpenJDK 8, which is
available as a package, JVM fuzz targets should not be forced to be
compatible with Java 8. For this reason, the official binary release of
OpenJDK 15 is pulled into both base-builder and base-runner and set as
JAVA_HOME. It is trimmed down in size by removing src.zip and the jmods
directory.
Jazzer consists of the following four components:
* The API (`jazzer_api_deploy.jar`), which is required for fuzz targets
that use FuzzedDataProvider or custom method hooks, is made available
in /usr/local/lib in base-builder.
* The driver (`jazzer_driver`), which links in libFuzzer and is reused
across fuzz targets. Since it is used to run fuzz targets, it is
included into base-runner.
* The ASanified driver (`jazzer_driver_asan`), which is obtained from
`jazzer_driver` by linking in ASan.
* The agent (`jazzer_agent_deploy.jar`), which bundles the runtime
instrumentation agent with the Jazzer API. It is loaded by the driver
and thus also included into base-runner.
The changes to the infra scripts required by JVM fuzz targets will be
submitted as a separate PR.
It should make the script compatible with binutils-2.36.1 (where
"callq" is no longer present in the output of objdump)
It was spotted in https://github.com/systemd/systemd/pull/18528
Otherwise get errors like these for libsass.
[+] All right - fork server is up.
[-] PROGRAM ABORT : AFL_MAP_SIZE is not set and fuzzing target reports that the required size is very large. Solution: Run the fuzzing target stand-alone with the environment variable AFL_DEBUG=1 set and set the value for __afl_final_loc in the AFL_MAP_SIZE environment variable for afl-fuzz.
Location : report_error_and_exit(), src/afl-forkserver.c:321
1. Fix problem where permissions were being changed to root by non-root test (test was doing this by invoking test_all.py within docker).
2. Mark tests as integration tests so that cifuzz_test.py can be run in a reasonable amount of time.
3. Prevent some unittests from polluting source repo.
4. Add .venv to .gitignore
5. Rename test_test_all.py to the correctly formatted name "test_all_test.py"
Abhishek Arya