oss-fuzz/projects/proxygen/build.sh

35 lines
1.3 KiB
Bash
Raw Normal View History

Add fuzzing support for proxygen (#2856) This adds support for compiling and running the fuzzers present in the proxygen repository. Right now there's only one fuzzer committed there, but this build script is generic and will pull all of them in as we add more (if oss-fuzz integration proves fruitful). Test plan is below - following https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally I verified the base image builds: python infra/helper.py build_image proxygen I built and verified the ASAN fuzzer works: python infra/helper.py build_fuzzers --sanitizer address proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Similar thing for UBSAN: python infra/helper.py build_fuzzers --sanitizer undefined proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Note the last one seemed to run ASAN build by default so I pulled out the command it runs and ran it manually: docker run --rm -i --privileged -e FUZZING_ENGINE=libfuzzer -e SANITIZER=undefined -e ARCHITECTURE=x86_64 -v /home/mhl/oss-fuzz/build/out/proxygen:/out -t gcr.io/oss-fuzz-base/base-runner test_all I tested the coverage build: python infra/helper.py build_fuzzers --sanitizer coverage proxygen python infra/helper.py coverage proxygen ProxygenHTTP1xFuzzer Note that this "runs" but threw some warnings which I will file a separate issue for. It does generate the files though. NOTE: I didn't run the MSAN build as I would have to figure out instrumenting all dependencies. We can investigate that in a follow up. Similarly, I haven't yet tried the dataflow build. Note that I haven't tried testing this with the AFL build yet either. There were no instructions on the page (https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally) on how to do so -- if someone can mention them here I am happy to test that too before committing.
2019-09-19 23:32:39 +00:00
#!/bin/bash -eu
# Copyright 2019 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
cd proxygen
[proxygen] Re-enable AFL; change build script to put libunwind in the generated directory and make the fuzzer find it (#2872) I'm re-enabling AFL since the issue with gmock's main being present was fixed in https://github.com/facebook/proxygen/commit/e8616a31f4df1486a522dde62fd337ae0d4e87e9 This libunwind changes solve the issues we were seeing with the fuzzers not running in the clusterfuzz bot environment. What this PR does, roughly: * Copy the .so from the build image into `/out/lib` * Patch the binaries so they have an rpath which specifies looking in `/out/lib` for libraries in addition to the normal search path This will work *assuming* `/out/lib` is copied over in the bot environment and is available. I'm relying on code reviewers to let me know if this is true or not. If not, it should be an easy path update. Test plan: Verifying the AFL build was easy: python infra/helper.py build_fuzzers --sanitizer address --engine afl proxygen python infra/helper.py check_build --engine afl proxygen python infra/helper.py run_fuzzer --engine afl proxygen ProxygenHTTP1xFuzzer I verified the libunwind changes by using the shell command (thanks for the tip, didn't know that was there!). I first built the binary using this build script. I then used `python infra/helper.py shell --sanitizer address proxygen` In the shell, I: * Ran `/out/ProxygenHTTP1xFuzzer` and verified it worked * Ran `ldd` on it and showed it pointed to `/out/lib` for `libunwind.so.8` * Uninstalled libunwind * Verified it still worked * Used `patchelf --print-rpath ProxygenHTTP1xFuzzer` to verify that the rpath was set as I expected (inside `/out/lib`) * Removed the patch using `patchelf --remove-rpath to_patch` * Verified that the fuzzer no longer runs (crashes on startup, complaining about missing `libunwind.so.8`) * I verified that the binary still finds the system one if rpath isn't set, by reinstalling it, using `patchelf --print-rpath` again, verifying that it prints the path to the system `libunwind` when I run `ldd`, and that the fuzzer runs fine. This implies it can find other system libraries fine too (and I saw that in the `ldd` output) I don't think I can do any further testing, so we will just have to hope that this works in the bot environment.
2019-09-22 03:08:51 +00:00
# Link to, and copy over, libunwind
Add fuzzing support for proxygen (#2856) This adds support for compiling and running the fuzzers present in the proxygen repository. Right now there's only one fuzzer committed there, but this build script is generic and will pull all of them in as we add more (if oss-fuzz integration proves fruitful). Test plan is below - following https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally I verified the base image builds: python infra/helper.py build_image proxygen I built and verified the ASAN fuzzer works: python infra/helper.py build_fuzzers --sanitizer address proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Similar thing for UBSAN: python infra/helper.py build_fuzzers --sanitizer undefined proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Note the last one seemed to run ASAN build by default so I pulled out the command it runs and ran it manually: docker run --rm -i --privileged -e FUZZING_ENGINE=libfuzzer -e SANITIZER=undefined -e ARCHITECTURE=x86_64 -v /home/mhl/oss-fuzz/build/out/proxygen:/out -t gcr.io/oss-fuzz-base/base-runner test_all I tested the coverage build: python infra/helper.py build_fuzzers --sanitizer coverage proxygen python infra/helper.py coverage proxygen ProxygenHTTP1xFuzzer Note that this "runs" but threw some warnings which I will file a separate issue for. It does generate the files though. NOTE: I didn't run the MSAN build as I would have to figure out instrumenting all dependencies. We can investigate that in a follow up. Similarly, I haven't yet tried the dataflow build. Note that I haven't tried testing this with the AFL build yet either. There were no instructions on the page (https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally) on how to do so -- if someone can mention them here I am happy to test that too before committing.
2019-09-19 23:32:39 +00:00
# We need to link to libunwind to ensure exception handling works
# See https://clang.llvm.org/docs/Toolchain.html#unwind-library
export LDFLAGS="-lunwind"
[proxygen] Re-enable AFL; change build script to put libunwind in the generated directory and make the fuzzer find it (#2872) I'm re-enabling AFL since the issue with gmock's main being present was fixed in https://github.com/facebook/proxygen/commit/e8616a31f4df1486a522dde62fd337ae0d4e87e9 This libunwind changes solve the issues we were seeing with the fuzzers not running in the clusterfuzz bot environment. What this PR does, roughly: * Copy the .so from the build image into `/out/lib` * Patch the binaries so they have an rpath which specifies looking in `/out/lib` for libraries in addition to the normal search path This will work *assuming* `/out/lib` is copied over in the bot environment and is available. I'm relying on code reviewers to let me know if this is true or not. If not, it should be an easy path update. Test plan: Verifying the AFL build was easy: python infra/helper.py build_fuzzers --sanitizer address --engine afl proxygen python infra/helper.py check_build --engine afl proxygen python infra/helper.py run_fuzzer --engine afl proxygen ProxygenHTTP1xFuzzer I verified the libunwind changes by using the shell command (thanks for the tip, didn't know that was there!). I first built the binary using this build script. I then used `python infra/helper.py shell --sanitizer address proxygen` In the shell, I: * Ran `/out/ProxygenHTTP1xFuzzer` and verified it worked * Ran `ldd` on it and showed it pointed to `/out/lib` for `libunwind.so.8` * Uninstalled libunwind * Verified it still worked * Used `patchelf --print-rpath ProxygenHTTP1xFuzzer` to verify that the rpath was set as I expected (inside `/out/lib`) * Removed the patch using `patchelf --remove-rpath to_patch` * Verified that the fuzzer no longer runs (crashes on startup, complaining about missing `libunwind.so.8`) * I verified that the binary still finds the system one if rpath isn't set, by reinstalling it, using `patchelf --print-rpath` again, verifying that it prints the path to the system `libunwind` when I run `ldd`, and that the fuzzer runs fine. This implies it can find other system libraries fine too (and I saw that in the `ldd` output) I don't think I can do any further testing, so we will just have to hope that this works in the bot environment.
2019-09-22 03:08:51 +00:00
mkdir -p $OUT/lib
cp /usr/lib/x86_64-linux-gnu/libunwind.so.8 $OUT/lib/
# Build everything
Add fuzzing support for proxygen (#2856) This adds support for compiling and running the fuzzers present in the proxygen repository. Right now there's only one fuzzer committed there, but this build script is generic and will pull all of them in as we add more (if oss-fuzz integration proves fruitful). Test plan is below - following https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally I verified the base image builds: python infra/helper.py build_image proxygen I built and verified the ASAN fuzzer works: python infra/helper.py build_fuzzers --sanitizer address proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Similar thing for UBSAN: python infra/helper.py build_fuzzers --sanitizer undefined proxygen python infra/helper.py check_build proxygen python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer Note the last one seemed to run ASAN build by default so I pulled out the command it runs and ran it manually: docker run --rm -i --privileged -e FUZZING_ENGINE=libfuzzer -e SANITIZER=undefined -e ARCHITECTURE=x86_64 -v /home/mhl/oss-fuzz/build/out/proxygen:/out -t gcr.io/oss-fuzz-base/base-runner test_all I tested the coverage build: python infra/helper.py build_fuzzers --sanitizer coverage proxygen python infra/helper.py coverage proxygen ProxygenHTTP1xFuzzer Note that this "runs" but threw some warnings which I will file a separate issue for. It does generate the files though. NOTE: I didn't run the MSAN build as I would have to figure out instrumenting all dependencies. We can investigate that in a follow up. Similarly, I haven't yet tried the dataflow build. Note that I haven't tried testing this with the AFL build yet either. There were no instructions on the page (https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally) on how to do so -- if someone can mention them here I am happy to test that too before committing.
2019-09-19 23:32:39 +00:00
./build.sh -m --no-install-dependencies --build-for-fuzzing
[proxygen] Re-enable AFL; change build script to put libunwind in the generated directory and make the fuzzer find it (#2872) I'm re-enabling AFL since the issue with gmock's main being present was fixed in https://github.com/facebook/proxygen/commit/e8616a31f4df1486a522dde62fd337ae0d4e87e9 This libunwind changes solve the issues we were seeing with the fuzzers not running in the clusterfuzz bot environment. What this PR does, roughly: * Copy the .so from the build image into `/out/lib` * Patch the binaries so they have an rpath which specifies looking in `/out/lib` for libraries in addition to the normal search path This will work *assuming* `/out/lib` is copied over in the bot environment and is available. I'm relying on code reviewers to let me know if this is true or not. If not, it should be an easy path update. Test plan: Verifying the AFL build was easy: python infra/helper.py build_fuzzers --sanitizer address --engine afl proxygen python infra/helper.py check_build --engine afl proxygen python infra/helper.py run_fuzzer --engine afl proxygen ProxygenHTTP1xFuzzer I verified the libunwind changes by using the shell command (thanks for the tip, didn't know that was there!). I first built the binary using this build script. I then used `python infra/helper.py shell --sanitizer address proxygen` In the shell, I: * Ran `/out/ProxygenHTTP1xFuzzer` and verified it worked * Ran `ldd` on it and showed it pointed to `/out/lib` for `libunwind.so.8` * Uninstalled libunwind * Verified it still worked * Used `patchelf --print-rpath ProxygenHTTP1xFuzzer` to verify that the rpath was set as I expected (inside `/out/lib`) * Removed the patch using `patchelf --remove-rpath to_patch` * Verified that the fuzzer no longer runs (crashes on startup, complaining about missing `libunwind.so.8`) * I verified that the binary still finds the system one if rpath isn't set, by reinstalling it, using `patchelf --print-rpath` again, verifying that it prints the path to the system `libunwind` when I run `ldd`, and that the fuzzer runs fine. This implies it can find other system libraries fine too (and I saw that in the `ldd` output) I don't think I can do any further testing, so we will just have to hope that this works in the bot environment.
2019-09-22 03:08:51 +00:00
# Patch rpath so fuzzers can find libunwind
2019-09-23 17:43:47 +00:00
find ./_build/proxygen/fuzzers -type f -executable -exec patchelf --set-rpath '$ORIGIN/lib' {} \;
[proxygen] Re-enable AFL; change build script to put libunwind in the generated directory and make the fuzzer find it (#2872) I'm re-enabling AFL since the issue with gmock's main being present was fixed in https://github.com/facebook/proxygen/commit/e8616a31f4df1486a522dde62fd337ae0d4e87e9 This libunwind changes solve the issues we were seeing with the fuzzers not running in the clusterfuzz bot environment. What this PR does, roughly: * Copy the .so from the build image into `/out/lib` * Patch the binaries so they have an rpath which specifies looking in `/out/lib` for libraries in addition to the normal search path This will work *assuming* `/out/lib` is copied over in the bot environment and is available. I'm relying on code reviewers to let me know if this is true or not. If not, it should be an easy path update. Test plan: Verifying the AFL build was easy: python infra/helper.py build_fuzzers --sanitizer address --engine afl proxygen python infra/helper.py check_build --engine afl proxygen python infra/helper.py run_fuzzer --engine afl proxygen ProxygenHTTP1xFuzzer I verified the libunwind changes by using the shell command (thanks for the tip, didn't know that was there!). I first built the binary using this build script. I then used `python infra/helper.py shell --sanitizer address proxygen` In the shell, I: * Ran `/out/ProxygenHTTP1xFuzzer` and verified it worked * Ran `ldd` on it and showed it pointed to `/out/lib` for `libunwind.so.8` * Uninstalled libunwind * Verified it still worked * Used `patchelf --print-rpath ProxygenHTTP1xFuzzer` to verify that the rpath was set as I expected (inside `/out/lib`) * Removed the patch using `patchelf --remove-rpath to_patch` * Verified that the fuzzer no longer runs (crashes on startup, complaining about missing `libunwind.so.8`) * I verified that the binary still finds the system one if rpath isn't set, by reinstalling it, using `patchelf --print-rpath` again, verifying that it prints the path to the system `libunwind` when I run `ldd`, and that the fuzzer runs fine. This implies it can find other system libraries fine too (and I saw that in the `ldd` output) I don't think I can do any further testing, so we will just have to hope that this works in the bot environment.
2019-09-22 03:08:51 +00:00
# Copy fuzzers over to the destination
2019-09-23 17:43:47 +00:00
find ./_build/proxygen/fuzzers -type f -executable -exec cp {} $OUT/ \;