oss-fuzz/infra/base-images/base-builder/compile_libfuzzer

23 lines
899 B
Plaintext
Raw Normal View History

#!/bin/bash -eu
# Copyright 2016 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
echo -n "Compiling libFuzzer to $LIB_FUZZING_ENGINE... "
export LIB_FUZZING_ENGINE="-fsanitize=fuzzer"
[base-builder] Use builtin libFuzzer instead of recompling. (#4682) Instead of recompiling libFuzzer each time we do a libFuzzer build of a project, always use Clang's builtin version of libFuzzer. Do this by copying the builtin libFuzzer to /usr/local/lib/FuzzingEngine.a. This means that the projects that aren't using -fsanitize=fuzzer now also use the builtin libFuzzer. And we no longer need to compile a sanitized libFuzzer for them. This change improves fuzzing performance and developer experience. 1. It improves developer experience by saving time spent compiling libFuzzer when recompiling fuzzers. The time saved is about 25 seconds on my machine. This will make iterating on fuzzer integration much easier. 2. It improves fuzzer performance. The builtin libFuzzer isn't sanitized so it is faster. In some cases (see [here](https://bugs.chromium.org/p/chromium/issues/detail?id=934639)) sanitized libFuzzers can waste 37% of the time running non-performant implementations of code that the builtin-libFuzzer can do almost instantaneously (assembly vs C code). The consequences of improving developer experience and fuzzer performance aren't so easy to measure (though we will look for perf consequences on ClusterFuzz). But some of the consequences of saving time compiling libFuzzer are easy to figure out and quite important. They are: 1. Saving $14646 a year on build costs. Based on the following: build time saved (on GCB): ~38 seconds libFuzzer builds per day: 990 builds per year: >365 price per build-minute (32 core instance, https://cloud.google.com/build/pricing): 0.064 38/60*.064*990*365 = 14,646 2. Speeding up infra-tests. Many of the integration tests build fuzzers and so building libFuzzer was a considerable bottleneck. On my many-core machine the savings were good and noticeable (and are probably larger on the less performant CI machines). | | With compiling libfuzzer | Without compiling libfuzzer | | ---------------------- | ------------------------------- | ----------------------------------- | | Parallel tests | 45 | 34 | | Sequential tests | 276 | 190 | 3. Speeding up CIFuzz. CIFuzz needs to be fast but it spends about 40 seconds compiling libFuzzer. In a run where no bugs are discovered which is intended to take about 20 minutes compiling libFuzzer takes about 3% of the time (40/(20*60)*100). Now we don't need to waste that time. See https://github.com/google/oss-fuzz/issues/5180, which this partially fixes. This bug fixes https://github.com/google/oss-fuzz/issues/2312 and https://github.com/google/oss-fuzz/issues/4677.
2021-02-24 19:36:03 +00:00
cp /usr/local/lib/clang/*/lib/linux/libclang_rt.fuzzer-$ARCHITECTURE.a \
$LIB_FUZZING_ENGINE_DEPRECATED
echo " done."