mirror of https://github.com/google/oss-fuzz.git
270 lines
9.6 KiB
Markdown
270 lines
9.6 KiB
Markdown
# Setting up New Target
|
|
|
|
Fuzzer build configurations are placed into [`targets/`](https://github.com/google/oss-fuzz/tree/master/targets)
|
|
subdirectories of the [oss-fuzz repo] on GitHub.
|
|
For example, fuzzers for the expat library are located in
|
|
[`targets/expat/`](https://github.com/google/oss-fuzz/tree/master/targets/expat).
|
|
|
|
## Prerequisites
|
|
[Install Docker](https://docs.docker.com/engine/installation). Googlers: [go/installdocker](https://goto.google.com/installdocker).
|
|
([Why Docker?](faq.md#why-do-you-use-docker))
|
|
|
|
*NOTE: if you want to run `docker` without `sudo` also follow the optional [Create a docker group](https://docs.docker.com/engine/installation/linux/ubuntulinux/#/create-a-docker-group) section.*
|
|
|
|
*NOTE: Docker images can consume significant disk space. Run*
|
|
*[docker-cleanup](https://gist.github.com/mikea/d23a839cba68778d94e0302e8a2c200f)*
|
|
*periodically to garbage collect unused images.*
|
|
|
|
## Overview
|
|
|
|
To add a new OSS target to oss-fuzz, 3 supporting files have to be added to oss-fuzz source code repository:
|
|
|
|
* `targets/<target_name>/Dockerfile` - defines an container environment with all the dependencies
|
|
needed to build the project and the fuzzer.
|
|
* `targets/<target_name>/build.sh` - build script that will be executed inside the container.
|
|
* `targets/<target_name>/Jenkinsfile` - will be needed to integrate fuzzers with ClusterFuzz build and distributed execution system.
|
|
Specify your target VCS location in it.
|
|
|
|
To create a new directory for the target and *automatically generate* these 3 files a python script can be used:
|
|
|
|
```bash
|
|
$ cd /path/to/oss-fuzz
|
|
$ export TARGET_NAME=target_name
|
|
$ python infra/helper.py generate $TARGET_NAME
|
|
```
|
|
|
|
Create a fuzzer and add it to the *target_name/* directory as well.
|
|
|
|
## Dockerfile
|
|
|
|
This is the Docker image definition that build.sh will be executed in.
|
|
It is very simple for most libraries:
|
|
```docker
|
|
FROM ossfuzz/base-libfuzzer # base image with clang toolchain
|
|
MAINTAINER YOUR_EMAIL # each file should have a maintainer
|
|
RUN apt-get install -y ... # install required packages to build a project
|
|
RUN git checkout <git_url> # checkout all sources needed to build your target
|
|
COPY build.sh fuzzer.cc /src/ # install build script and other source files.
|
|
```
|
|
Expat example: [expat/Dockerfile](../expat/Dockerfile)
|
|
|
|
## Create Fuzzer Source File
|
|
|
|
Create a new .cc file, define a `LLVMFuzzerTestOneInput` function and call
|
|
your target:
|
|
|
|
```c++
|
|
#include <stddef.h>
|
|
#include <stdint.h>
|
|
|
|
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
|
|
// put your fuzzing code here and use data+size as input.
|
|
return 0;
|
|
}
|
|
```
|
|
|
|
Make sure you add the file to your Docker image:
|
|
```docker
|
|
COPY build.sh my_fuzzer.cc /src/ # install build script & fuzzer.
|
|
```
|
|
|
|
There are [lots](../libxml2/libxml2_xml_read_memory_fuzzer.cc)
|
|
[of](../expat/parse_fuzzer.cc) [examples](../zlib/zlib_uncompress_fuzzer.cc)
|
|
in this project repository.
|
|
|
|
## build.sh
|
|
|
|
This is where most of the work is done to build fuzzers for your target. The script will
|
|
be executed within an image built from a `Dockerfile`.
|
|
|
|
In general, this script will need to:
|
|
|
|
1. Build the target using its build system *with* correct compiler and its flags provided as
|
|
*environment variables* (see below).
|
|
2. Build the fuzzers, linking with the target and libFuzzer. Resulting fuzzers
|
|
should be placed in `/out`.
|
|
|
|
For expat, this looks like:
|
|
|
|
```bash
|
|
#!/bin/bash -eu
|
|
|
|
cd /src/expat/expat
|
|
./buildconf.sh
|
|
# configure scripts usually use correct environment variables.
|
|
./configure
|
|
|
|
make -j$(nproc) clean all
|
|
|
|
# build the fuzzer, linking with libFuzzer and libexpat.a
|
|
$CXX $CXXFLAGS -std=c++11 -Ilib/ \
|
|
/src/parse_fuzzer.cc -o /out/expat_parse_fuzzer \
|
|
-lfuzzer .libs/libexpat.a \
|
|
$FUZZER_LDFLAGS
|
|
```
|
|
|
|
### build.sh Script Environment
|
|
|
|
When build.sh script is executed, the following locations are available within the image:
|
|
|
|
| Path | Description
|
|
| ------ | -----
|
|
| `/src/<some_dir>` | Source code needed to build your target.
|
|
| `/usr/lib/libfuzzer.a` | Prebuilt libFuzzer library that need to be linked into all fuzzers (`-lfuzzer`).
|
|
|
|
You *must* use special compiler flags to build your target and fuzzers.
|
|
These flags are provided in following environment variables:
|
|
|
|
| Env Variable | Description
|
|
| ------------- | --------
|
|
| `$CC`, `$CXX`, `$CCC` | The C and C++ compiler binaries.
|
|
| `$CFLAGS`, `$CXXFLAGS` | C and C++ compiler flags.
|
|
| `$FUZZER_LDFLAGS` | Linker flags for fuzzer binaries.
|
|
|
|
Many well-crafted build scripts will automatically use these variables. If not,
|
|
passing them manually to a build tool might be required.
|
|
|
|
See [Provided Environment Variables](../infra/base-images/base-libfuzzer/README.md#provided-environment-variables) section in
|
|
`base-libfuzzer` image documentation for more details.
|
|
|
|
### Custom libFuzzer options for ClusterFuzz
|
|
|
|
By default ClusterFuzz will run your fuzzer without any options. You can specify
|
|
custom options by creating a `my_fuzzer.options` file next to a `my_fuzzer` executable in `/out`:
|
|
|
|
```
|
|
[libfuzzer]
|
|
max_len = 1024
|
|
```
|
|
|
|
[List of available options](http://llvm.org/docs/LibFuzzer.html#options)
|
|
|
|
At least `max_len` is highly recommended.
|
|
|
|
For out of tree fuzzers you will likely add options file using docker's
|
|
`COPY` directive and will copy it into output in build script.
|
|
([Woff2 example](https://github.com/google/oss-fuzz/blob/master/targets/woff2/convert_woff2ttf_fuzzer.options).)
|
|
|
|
### Seed Corpus
|
|
|
|
oss-fuzz uses evolutionary fuzzing algorithms. Supplying seed corpus consisting
|
|
of sample inputs is one of the best ways to improve fuzzer coverage.
|
|
|
|
To provide a corpus for `my_fuzzer`, put `my_fuzzer_seed_corpus.zip` file next
|
|
to the fuzzer binary in `/out` during the build. Individual files in the zip file
|
|
will be used as starting inputs for mutations. You can store the corpus next to
|
|
source files, generate during build or fetch it using curl or any other tool of
|
|
your choice.
|
|
([Boringssl example](https://github.com/google/oss-fuzz/blob/master/targets/boringssl/build.sh#L42).)
|
|
|
|
Seed corpus files will be used for cross-mutations and portions of them might appear
|
|
in bug reports or be used for further security research. It is important that corpus
|
|
has an appropriate and consistent license.
|
|
|
|
|
|
### Dictionaries
|
|
|
|
Dictionaries hugely improve fuzzer effectiveness for inputs with lots of similar
|
|
sequences of bytes. [libFuzzer documentation](http://llvm.org/docs/LibFuzzer.html#dictionaries)
|
|
|
|
Put your dict file in `/out` and specify in .options file:
|
|
|
|
```
|
|
[libfuzzer]
|
|
dict = dictionary_name.dict
|
|
```
|
|
|
|
It is common for several fuzzers to reuse the same dictionary if they are fuzzing very similar inputs.
|
|
([Expat example](https://github.com/google/oss-fuzz/blob/master/targets/expat/parse_fuzzer.options).)
|
|
|
|
## Jenkinsfile
|
|
|
|
This file will be largely the same for most libraries, and is used by our build
|
|
infrastructure. For expat, this is:
|
|
|
|
```groovy
|
|
// load libFuzzer pipeline definition.
|
|
def libfuzzerBuild = fileLoader.fromGit('infra/libfuzzer-pipeline.groovy',
|
|
'https://github.com/google/oss-fuzz.git')
|
|
|
|
libfuzzerBuild {
|
|
git = "git://git.code.sf.net/p/expat/code_git"
|
|
}
|
|
```
|
|
|
|
Simply replace the the "git" entry with the correct git url for the target.
|
|
|
|
*Note*: only git is supported right now.
|
|
|
|
## Testing locally
|
|
|
|
Helper script can be used to build images and fuzzers. Non-script
|
|
version using docker commands directly is documented [here](building_running_fuzzers_external.md).
|
|
|
|
```bash
|
|
$ cd /path/to/oss-fuzz
|
|
$ python infra/helper.py build_image $TARGET_NAME
|
|
$ python infra/helper.py build_fuzzers $TARGET_NAME
|
|
```
|
|
|
|
This should place the built fuzzers into `/path/to/oss-fuzz/build/out/$TARGET_NAME`
|
|
on your machine (`/out` in the container). You can then try to run these fuzzers
|
|
inside the container to make sure that they work properly:
|
|
|
|
```bash
|
|
$ python infra/helper.py run_fuzzer $TARGET_NAME name_of_a_fuzzer
|
|
```
|
|
|
|
If everything works locally, then it should also work on our automated builders
|
|
and ClusterFuzz.
|
|
|
|
It's recommended to look at coverage as a sanity check to make sure that fuzzer gets to the code you expect.
|
|
|
|
```bash
|
|
$ python infra/helper.py coverage $TARGET_NAME name_of_a_fuzzer
|
|
```
|
|
|
|
## Debugging Problems
|
|
|
|
[Debugging](debugging.md) document lists ways to debug your build scripts or fuzzers
|
|
in case you run into problems.
|
|
|
|
## Checking in to oss-fuzz repository
|
|
|
|
Fork oss-fuzz, commit and push to the fork, and then create a pull request with
|
|
your change! Follow the [Forking Project](https://guides.github.com/activities/forking/) guide
|
|
if you are new to contributing via GitHub.
|
|
|
|
### Copyright headers
|
|
|
|
Please include copyright headers for all files checked in to oss-fuzz:
|
|
|
|
```
|
|
# Copyright 2016 Google Inc.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
################################################################################
|
|
```
|
|
|
|
If porting a fuzzer from Chromium, keep the Chromium license header.
|
|
|
|
## The end
|
|
|
|
Once your change is merged, the fuzzers should be automatically built and run on
|
|
ClusterFuzz after a short while!
|
|
|
|
[oss-fuzz repo]: https://github.com/google/oss-fuzz
|
|
[dictionaries]: http://llvm.org/docs/LibFuzzer.html#dictionaries
|
|
[Install Docker]: https://docs.docker.com/engine/installation/linux/ubuntulinux/
|