* retry sequentially if multiprocessing do_bad_build_check detects failures
https://github.com/google/oss-fuzz/issues/5441
The error seen in the build log is:
Whoops, the target binary crashed suddenly, before receiving any input
from the fuzzer!
suggesting that the fuzzer crashed before it got to do anything.
Debugging locally what I tend to see is that
a) in src/afl-forkserver.c afl_fsrv_start the read_s32_timed call
returns 0 and that triggers kill(fsrv->fsrv_pid, fsrv->kill_signal);
(SIGKILL)
b) read_s32_timed returns 0 because *stop_soon_p is non-zero at
restart_read:
c) *stop_soon_p becomes non-zero in handle_stop_sig of
src/afl-fuzz-init.c due to receiving SIGINT
d) that SIGINT is sent by the timeout script used in bad_build_check so
it is that "outer" timeout process which is sending SIGINT which then
triggers afl-forkserver's internal SIGKILL to kill the process
I get improved results if I retry the killed off fuzzers sequentially
* Remove unneeded semicolons to fix presubmit
Co-authored-by: Abhishek Arya <inferno@chromium.org>
This will mainly be useful for non-OSS-Fuzz users.
Though it can also be used by OSS-Fuzz users to speed things
up (very slightly in most cases).
Fixes#4377
gsutil recently started asking daily for a security key jiggle for accounts
using it. This means if we don't print stderr in real time, users won't see
why a corpus isn't downloading and helper will hang until 2FA request times out.
Repos cloned with `--branch BRANCH` will only track that branch, even
when we unshallow. If we provide a git SHA from another branch, it will
not be recognized.
To fix, this, we update the remote tracking config and fetch them.
For google/osv#88.
Generate badges also for projects with no coverage builds at all (e.g.
JVM and Python projects). For these projects, the badge only has the two
possible states "build passing" and "build failing".
Delete unneeded LLVM tools, clang libraries and testing tools.
This reduces the image size from 1.71 GB to 901 MB.
It may be possible to improve on this by deleting some LLVM
libraries though I don't know which ones we should delete
because AFL++ might use some).
Related https://github.com/google/oss-fuzz/issues/5170
* [CIFuzz] Add functionality to save diskspace.
Add a LOW_DISK_SPACE env/config var. When this is specified
(always true for Github actions) run_fuzzers will delete
base-builder and the project builder image before fuzzing.
After it finishes fuzzing with a target, it will also
delete the targets, its seed corpus and its corpus.
Related: #4879
* Fixes go coverage with modules
* Golang coverage html report turning off modules
Otherwise, we get the error
working directory is not part of a module
Reduce image size by:
1. Not installing go toolchain in final image. Build go tools in
seperate image that doesn't become base-runner.
2. Download the JVM zip in the same step we remove it.
Precompile AFL like we already do for honggfuzz.
This saves about a minute in compilation time of AFL targets by doing it in base-builder
It only adds about 30 MB to the image size.
Reduce build time by doing the following:
1. Building the second stage clang build with a clang binary we download
from chromium.
2. Changing NPROC to be half of the cores instead of assuming it's 16
cores. This still addresses the OOM when building on GCB but speeds up
local building.
3. Don't install recommended packages and use --depth 1 when possible
(very minor improvements compared to the above).
In all this reduces local build time of base-clang from 32 minutes
to 11 minutes.
Because build times are reduced, it will be easier to
iteratively develop changes needed for #5170
Instead of recompiling libFuzzer each time we do a libFuzzer
build of a project, always use Clang's builtin version of libFuzzer.
Do this by copying the builtin libFuzzer to /usr/local/lib/FuzzingEngine.a.
This means that the projects that aren't using -fsanitize=fuzzer now also
use the builtin libFuzzer. And we no longer need to compile a sanitized
libFuzzer for them.
This change improves fuzzing performance and developer experience.
1. It improves developer experience by saving time spent compiling libFuzzer
when recompiling fuzzers.
The time saved is about 25 seconds on my machine.
This will make iterating on fuzzer integration much easier.
2. It improves fuzzer performance. The builtin libFuzzer isn't sanitized so it is faster.
In some cases (see [here](https://bugs.chromium.org/p/chromium/issues/detail?id=934639))
sanitized libFuzzers can waste 37% of the time running non-performant implementations
of code that the builtin-libFuzzer can do almost instantaneously (assembly vs C code).
The consequences of improving developer experience and
fuzzer performance aren't so easy to measure (though
we will look for perf consequences on ClusterFuzz).
But some of the consequences of saving time compiling libFuzzer
are easy to figure out and quite important. They are:
1. Saving $14646 a year on build costs. Based on the following:
build time saved (on GCB): ~38 seconds
libFuzzer builds per day: 990
builds per year: >365
price per build-minute (32 core instance, https://cloud.google.com/build/pricing): 0.064
38/60*.064*990*365 = 14,646
2. Speeding up infra-tests.
Many of the integration tests build fuzzers and so building libFuzzer
was a considerable bottleneck.
On my many-core machine the savings were good and noticeable
(and are probably larger on the less performant CI machines).
| | With compiling libfuzzer | Without compiling libfuzzer |
| ---------------------- | ------------------------------- | ----------------------------------- |
| Parallel tests | 45 | 34 |
| Sequential tests | 276 | 190 |
3. Speeding up CIFuzz.
CIFuzz needs to be fast but it spends about 40 seconds compiling libFuzzer.
In a run where no bugs are discovered which is intended to take about 20 minutes
compiling libFuzzer takes about 3% of the time (40/(20*60)*100).
Now we don't need to waste that time.
See https://github.com/google/oss-fuzz/issues/5180, which this partially fixes.
This bug fixes https://github.com/google/oss-fuzz/issues/2312 and https://github.com/google/oss-fuzz/issues/4677.
* Makes vitess build local
As it uses vitess.io instead of github
* Completes minify project
* Completes quic-go
* Local build for nats project
* Completes ipfs
* run go mod tidy after adding go module
* Right bash sequence for go mod tidy
* go: right bash condition for changing directory
* go-json-iterator: uses git clone
So as to copy fuzz target in right directory
* go: uses tags when running go list
* go-redis: uses git clone and builds local fuzz target
* cascadia: uses git clone instead of go get
compile, bad_build_check, and presubmit.py require small tweaks to
support JVM fuzz targets, most of which are similar to those required
for Python. The following additional changes are required:
* Since the Jazzer driver binary already links in libFuzzer, it should
not be built as a static library.
* It is not clear how to do architecture checks as JVM fuzz targets can
load their native dependencies dynamically at runtime. For now, the
check is disabled.
* The Jazzer binaries are moved into $OUT and need to be skipped over in
find_fuzz_targets.
Jazzer is built from HEAD using Bazel and the clang toolchain provided
by base-clang. While it could be built with OpenJDK 8, which is
available as a package, JVM fuzz targets should not be forced to be
compatible with Java 8. For this reason, the official binary release of
OpenJDK 15 is pulled into both base-builder and base-runner and set as
JAVA_HOME. It is trimmed down in size by removing src.zip and the jmods
directory.
Jazzer consists of the following four components:
* The API (`jazzer_api_deploy.jar`), which is required for fuzz targets
that use FuzzedDataProvider or custom method hooks, is made available
in /usr/local/lib in base-builder.
* The driver (`jazzer_driver`), which links in libFuzzer and is reused
across fuzz targets. Since it is used to run fuzz targets, it is
included into base-runner.
* The ASanified driver (`jazzer_driver_asan`), which is obtained from
`jazzer_driver` by linking in ASan.
* The agent (`jazzer_agent_deploy.jar`), which bundles the runtime
instrumentation agent with the Jazzer API. It is loaded by the driver
and thus also included into base-runner.
The changes to the infra scripts required by JVM fuzz targets will be
submitted as a separate PR.
It should make the script compatible with binutils-2.36.1 (where
"callq" is no longer present in the output of objdump)
It was spotted in https://github.com/systemd/systemd/pull/18528
Make unittests take 20 seconds to run instead of 35.
Make integration tests take 50 seconds to run instead of 6 minutes.
Make CI take 6 minutes instead of 12 minutes.
1. Allow running tests in parallel. Locally this takes the time for running all tests (including integration tests) from 6 minutes to ~50 seconds. We don't do parallel by default since it doesn't really save any time unless running integration tests on my machine (probably due to overhead of starting ~70 processes). This also speeds up CI from about 12 minutes to 6 minutes (since github actions has 2 cores per machine).
2. Fix how we run tests. I'm not exactly sure why, but the method we used for discovering tests, recursing through every directory and passing to unittest caused the build/infra tests to execute twice. Fixing this makes running unittests take ~20 seconds instead of ~35.
This change also uses pytest for running tests since it's easy to use it to run tests in parallel.
This change was made possible by #5113
1. Use pyfakefs when possible instead of tempdir
2. Favor decorators instead of contextmanagers when mocking for less indentation and greater consistency.
Abstract away OSS-Fuzz specific bits into the OSSFuzz implementation
of the ClusterFuzzDeployment class. This will make it easier to implement
support for other deployments of ClusterFuzz (including ClusterFuzzLite).
Otherwise get errors like these for libsass.
[+] All right - fork server is up.
[-] PROGRAM ABORT : AFL_MAP_SIZE is not set and fuzzing target reports that the required size is very large. Solution: Run the fuzzing target stand-alone with the environment variable AFL_DEBUG=1 set and set the value for __afl_final_loc in the AFL_MAP_SIZE environment variable for afl-fuzz.
Location : report_error_and_exit(), src/afl-forkserver.c:321
1. Use ssh_url.
This only affects external (non-oss-fuzz) users.
Since there are none, it doesn't affect anyone.
Even if it did, exploitation would require owning the network
Github actions runs on.
This is to prevent MITM attacks.
2. Affected fuzzers bug:
We accidentally were skipping the remove unaffected functionality.
* [CIFuzz] Fix diffing + Refactor
Make diffing work in two scenarios where it previously failed:
1. Commit fuzzing. In this case, we diff $COMMIT against
$COMMIT^1 because the intent here is to fuzz the commit.
2. Fuzzing PRs that aren't to master. In this case, we previously
were diffing against origin/master. Instead, diff against the local
version of the base repo. This also has the nice effect of handling
PRs that havent pulled from master recently enough.
Also do refactoring.
1. Move code that is different for differenct CI systems to continuous_integration.py
2. Change how configuration in build_fuzzers is handled.
Previously configuration was gotten in build_fuzzers_entrypoint
and passed as individual params. This made code ugly and hard to
read. Instead, move code dealing with config to it's own module
config_utils. This module implements a config class which can
be used to create objects that are passed around to code that needs
it. Making the code much easier to read.
TODO: Move run_fuzzers code to new config system.
* Get list of changed files from branch head, instead of master.
Fixes https://github.com/google/oss-fuzz/issues/5022
* Add debug with subprocess.call.
* Try again debugginig.
* Try again
* Fix works!
The previous approach of only running tests in changed directories is broken.
Tests can fail even when files outside of their directory are modified.
Also blocklist failing tests (see https://github.com/google/oss-fuzz/issues/5025) for why build tests are blocklisted.
Previously region count was used instead of region covered.
This means that unaffected fuzzers only worked when a file wasn't linked
into a fuzzer build (i.e. it was mostly broken).
Add tests to ensure this doesn't happen again.
Fixes: #5013
Also refactor.
1. Create a coverage module and move coverage functionality there.
2. Remove some overly-defensive programming. We probably aren't going to be given an invalid repo dir in coverage module.
3. Convert integration test to unittest.
4. Add helpers for: normalizing paths, getting coverage per file, determining if file is covered (which was done incorrectly before), and getting fuzzer stats dir url to make code easier to understand.
5. Add a class for getting coverage info.
6. Create an affected_fuzz_targets module and move functionality dealing with affected_fuzz_targets there.
7. Add is_fuzz_target_affected helper and log more.
8. Refer to fuzz targets as fuzz targets instead of fuzzers.
9. Move `url_join` to `utils`.
10. Move `GSUTIL_BASE_URL` to `utils`.
11. Add a util function for converting gs:// URLs into https:// urls.
12. Add a util function for removing prefixes (instead of using `.replace` which operates on the whole string not just the prefix).
13. Use more common style of mocking in unittests.
14. Delete unnecessary tests like '' when an invalid fuzzer is already tested.
15. Make constants capitalized in tests.
16. Better variable naming and consistency also reuse variables in tests.
17. Leave TODOs around code that looks suspicious.
18. Cleanup pylint directives.
19. Use single instead of double quotes.
Move entrypoints for CIFuzz to cifuzz.
This allows us to reduce some complexity by getting rid of
an unnecessary copy in docker and a hack to making importing work.
`go list` will fail if all files in the fuzzed package use a build tag
restriction (like the common `gofuzz` tag).
Also, pass the `gofuzz` tag in the teleport build to plumb it through.
Abhishek Arya