While installing `libxext-dev` in `base-runner` makes it available for local fuzzing via `infra/helper.py run_fuzzer`, it is not installed in the image used by ClusterFuzz to run fuzz targets. Fuzz targets that rely on any JDK system dependency rather have to install it manually using the method described in https://google.github.io/oss-fuzz/further-reading/fuzzer-environment/#runtime-dependencies.
By removing this package from `base-runner`, a failure to do so will already be detected via a local `check_build` rather than only during fuzzer execution on ClusterFuzz.
Builds of base-runner fail as the Chromium coverage helper script has a
transitive dependency on MarkupSafe, which as of version 2.0 requires
additional python packaging tools to be installed.
This commit installs python3-setuptools and wheels, which fixes the
build.
Coverage is collected by running the fuzz targets with the JaCoCo agent
in Jazzer's no instrumentation mode.
Since JaCoCo does not support llvm-cov style coverage reports, a simple
Python helper creates it using the information contained in the JaCoCo
XML report.
As the Java build process does not maintain a mapping between source
files and build artifacts and JaCoCo needs to be passed the root folder
of the package tree, we use the Maven directory layout convention to
heuristically detect these roots.
* Fixes go coverage with modules
* Golang coverage html report turning off modules
Otherwise, we get the error
working directory is not part of a module
Reduce image size by:
1. Not installing go toolchain in final image. Build go tools in
seperate image that doesn't become base-runner.
2. Download the JVM zip in the same step we remove it.
Jazzer is built from HEAD using Bazel and the clang toolchain provided
by base-clang. While it could be built with OpenJDK 8, which is
available as a package, JVM fuzz targets should not be forced to be
compatible with Java 8. For this reason, the official binary release of
OpenJDK 15 is pulled into both base-builder and base-runner and set as
JAVA_HOME. It is trimmed down in size by removing src.zip and the jmods
directory.
Jazzer consists of the following four components:
* The API (`jazzer_api_deploy.jar`), which is required for fuzz targets
that use FuzzedDataProvider or custom method hooks, is made available
in /usr/local/lib in base-builder.
* The driver (`jazzer_driver`), which links in libFuzzer and is reused
across fuzz targets. Since it is used to run fuzz targets, it is
included into base-runner.
* The ASanified driver (`jazzer_driver_asan`), which is obtained from
`jazzer_driver` by linking in ASan.
* The agent (`jazzer_agent_deploy.jar`), which bundles the runtime
instrumentation agent with the Jazzer API. It is loaded by the driver
and thus also included into base-runner.
The changes to the infra scripts required by JVM fuzz targets will be
submitted as a separate PR.
Rewrite test_all in python.
Bash is quite annoying to write and test.
One issue with bash is it is even worse than Python for parallelism (which may be causing #4707).
Rewrite test_all in python and optimize base-runner/Dockerfile for fast development.
Also, combine some docker layers.
* [infra] Skeleton of the changes needed for collecting DFT on the builder (#1632).
* move ENGINE_INFO to the helper as well
* make collect_dft +x
* syntax fixes
* add actual dataflow tracer script
* format
* more refactoring and cleanup
* format
* address Oliver's feedback
* format
* more fixes
* format
* do not redirect stderr to stdout
* add exit at the end of main
* address feedback from Oliver
Whether string_string_checks will be turned on or not is
being discussed in https://github.com/google/oss-fuzz/issues/542.
Until it's settled, it'd probably be better to remove the option
from ASAN_OPTIONS altogether so as not to confuse anybody.
* [infra] Use coverage utils from Chromium and other fixes (follow-up #1547).
* Sort dependencies in an alpha order plus make more readable.
* Re-order arguments passed to coverage_helper script.
* Rename REPORT_DIR variable and put summary.json into platform specific dir.
* Fix -src-root-dir value.
* [infra] Add script for downloading a batch of corpus backups (for #1547).
* Add missing wget dependency and replace a tab with spaces.
* Convert more tabs to spaces.
* [infra] Enable clange code coverage reports generation for local runs.
* Use runner image and move corpus management to the helper.py
.
* Clean up, delete unnecessary stuff, add comments.
* Run fuzz targets in parallel. Do not exit in case of an error.
* Address review feedback, except of the threading thing.
* Fix a typo.
* Use ThreadPool implementatino available in standard python2.7 package.
* Add dry run support + no corpus download option.
* Fix flags handling + add log output in case of an error.
* Append arguments for fuzz target instead of replacing them.
* Remove dry run functionality as it currently errors out after two runs.
* Fix some spacing in the code.
* Update documentation regarding new code coverage script.
* [WIP] Add post build checks for catching and reporting bad builds + test projects.
* Move all type of issues into a single project. Tested all sanitizers.
* First version of the script for performing bad build checks. Still not ready to commit.\
* Added a valid example that should pass all the checks + fixed instrumentation check.
* Use SIGTERM and the handler calling _exit(0) for the startup crash case.
* Small fixes.
* Ignore startup_crash check_for fuzzing engines other than libFuzzer.
* Use "local" for local variables in the script, fix container name.
* Add COPY step for bad_build_check into base-runner Dockerfile.
* Make bad_build_check temporary noop unless EXPERIMENTAL_BAD_BUILD_CHECK is set to 1.
* Temporary remove test binaries step.
* Remove empty line.
* Remove unnecessary env variable.
Catena cyber