2019-08-07 14:37:16 +00:00
|
|
|
---
|
|
|
|
layout: default
|
|
|
|
title: Code coverage
|
|
|
|
parent: Advanced topics
|
|
|
|
nav_order: 2
|
2019-08-22 18:33:39 +00:00
|
|
|
permalink: /advanced-topics/code-coverage/
|
2019-08-07 14:37:16 +00:00
|
|
|
---
|
|
|
|
|
2018-06-14 22:00:46 +00:00
|
|
|
# Code Coverage
|
2019-09-05 20:21:13 +00:00
|
|
|
{: .no_toc}
|
2018-06-14 22:00:46 +00:00
|
|
|
|
2021-09-03 18:47:22 +00:00
|
|
|
For projects written in C/C++, Rust, Go, Swift or Java and other JVM-based languages,
|
2021-05-25 14:30:21 +00:00
|
|
|
you can generate code coverage reports using Clang source-based code coverage.
|
|
|
|
This page walks you through the basic steps.
|
|
|
|
For more details on C/C++ coverage, see [Clang's documentation].
|
2020-08-12 16:32:05 +00:00
|
|
|
|
|
|
|
Code coverage reports generation for other languages is not supported yet.
|
2018-06-14 22:00:46 +00:00
|
|
|
|
2019-08-07 14:37:16 +00:00
|
|
|
- TOC
|
|
|
|
{:toc}
|
|
|
|
---
|
|
|
|
|
2018-07-23 15:02:28 +00:00
|
|
|
## Pull the latest Docker images
|
|
|
|
|
|
|
|
Docker images get regularly updated with a newer version of build tools, build
|
2019-09-05 20:21:13 +00:00
|
|
|
configurations, scripts, and other changes. We recommend you pull the most
|
|
|
|
recent images by running the following command:
|
2018-07-23 15:02:28 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py pull_images
|
2018-07-23 15:02:28 +00:00
|
|
|
```
|
|
|
|
|
2018-06-14 22:00:46 +00:00
|
|
|
## Build fuzz targets
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
Code coverage report generation requires a special build configuration to be
|
|
|
|
used. To create a code coverage build for your project, run these commands:
|
2018-06-14 22:00:46 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py build_image $PROJECT_NAME
|
|
|
|
$ python infra/helper.py build_fuzzers --sanitizer=coverage $PROJECT_NAME
|
2018-06-14 22:00:46 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
## Establish access to GCS
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
To get a good understanding of fuzz testing quality, you should generate code
|
|
|
|
coverage reports by running fuzz targets against the corpus
|
|
|
|
aggregated by OSS-Fuzz. Set up `gsutil` and ensure that you have access to the
|
|
|
|
corpora by doing the following:
|
2018-06-15 17:44:18 +00:00
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
* Install the [gsutil tool].
|
2018-06-15 17:44:18 +00:00
|
|
|
* Check whether you have access to the corpus for your project:
|
2018-06-14 22:00:46 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ gsutil ls gs://${PROJECT_NAME}-corpus.clusterfuzz-external.appspot.com/
|
2018-06-14 22:00:46 +00:00
|
|
|
```
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
If you see an authorization error from the command above, run this:
|
2018-06-14 22:00:46 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ gcloud auth login
|
2018-06-14 22:00:46 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
and try again. Once `gsutil` works, you can run the report generation.
|
|
|
|
|
2019-08-15 22:07:23 +00:00
|
|
|
## Generate code coverage reports
|
2018-06-14 22:00:46 +00:00
|
|
|
|
2018-06-18 21:19:48 +00:00
|
|
|
### Full project report
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
If you want to generate a code coverage report using the corpus aggregated on
|
|
|
|
OSS-Fuzz, run this command:
|
2018-06-14 22:00:46 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage $PROJECT_NAME
|
2018-06-14 22:00:46 +00:00
|
|
|
```
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
If you want to generate a code coverage report using the corpus you have
|
|
|
|
locally, copy the corpus into the
|
|
|
|
`build/corpus/$PROJECT_NAME/<fuzz_target_name>/` directories for each fuzz
|
|
|
|
target, then run this command:
|
2018-06-14 22:00:46 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage --no-corpus-download $PROJECT_NAME
|
2018-06-14 22:00:46 +00:00
|
|
|
```
|
|
|
|
|
2018-06-18 21:19:48 +00:00
|
|
|
### Single fuzz target
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
You can generate a code coverage report for a particular fuzz target by using
|
|
|
|
the `--fuzz-target` argument:
|
2018-06-18 21:19:48 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage --fuzz-target=<fuzz_target_name> $PROJECT_NAME
|
2018-06-18 21:19:48 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
In this mode, you can specify an arbitrary corpus location for the fuzz target
|
2019-09-05 20:21:13 +00:00
|
|
|
(instead of the corpus downloaded from OSS-Fuzz) by using `--corpus-dir`:
|
2018-06-18 21:19:48 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage --fuzz-target=<fuzz_target_name> \
|
|
|
|
--corpus-dir=<my_local_corpus_dir> $PROJECT_NAME
|
2018-06-18 21:19:48 +00:00
|
|
|
```
|
|
|
|
|
2021-05-25 14:30:21 +00:00
|
|
|
### Additional arguments for `llvm-cov` (C/C++ only)
|
2018-07-19 22:58:58 +00:00
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
You may want to use some of the options provided by the [llvm-cov tool], like
|
|
|
|
`-ignore-filename-regex=`. You can pass these to the helper script after `--`:
|
2018-07-19 22:58:58 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage $PROJECT_NAME -- \
|
|
|
|
-ignore-filename-regex=.*code/to/be/ignored/.* <other_extra_args>
|
2018-07-19 22:58:58 +00:00
|
|
|
```
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
If you want to specify particular source files or directories to show in the
|
|
|
|
report, list their paths at the end of the extra arguments sequence:
|
2018-08-12 17:19:40 +00:00
|
|
|
|
|
|
|
```bash
|
2019-08-15 22:07:23 +00:00
|
|
|
$ python infra/helper.py coverage zlib -- \
|
|
|
|
<other_extra_args> /src/zlib/inftrees.c /src/zlib_uncompress_fuzzer.cc /src/zlib/zutil.c
|
2018-08-12 17:19:40 +00:00
|
|
|
```
|
2018-06-14 22:00:46 +00:00
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
If you want OSS-Fuzz to use extra arguments when generating code coverage
|
|
|
|
reports for your project, add the arguments into your `project.yaml` file as
|
|
|
|
follows:
|
2018-08-30 16:46:14 +00:00
|
|
|
|
|
|
|
```yaml
|
|
|
|
coverage_extra_args: -ignore-filename-regex=.*crc.* -ignore-filename-regex=.*adler.* <other_extra_args>
|
|
|
|
```
|
|
|
|
|
2019-09-05 20:21:13 +00:00
|
|
|
[Clang's documentation]: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
|
2018-06-15 17:44:18 +00:00
|
|
|
[gsutil tool]: https://cloud.google.com/storage/docs/gsutil_install
|
2020-08-12 16:32:05 +00:00
|
|
|
[llvm-cov tool]: https://llvm.org/docs/CommandGuide/llvm-cov.html
|