This installs clang wrappers at /ccache/bin, and sets up a build cache
at /ccache/cache. To use this, inside the project container we just need
to do:
```
export PATH=/ccache/bin:$PATH
```
In another PR, we can store the /ccache/cache somewhere we can pull down
at runtime.
Some results:
Fresh compile:
real 0m49.249s
user 10m41.818s
sys 1m2.097s
With ccache cache:
real 0m9.877s
user 0m6.278s
sys 0m19.966s
Fresh compile:
real 1m17.214s
user 0m49.454s
sys 0m27.963s
With ccache:
real 0m34.962s
user 0m18.092s
sys 0m17.083s
The `PARALLEL_FUZZING` environment variable is used to enable parallel
fuzzing in CFLite. The variable is always defined by GH Action, but it
was not interpreted as a boolean value, so it was always enabled even if
the option is set to `false`.
Co-authored-by: Vitor Guidi <vitorguidi@gmail.com>
Upload introspector report even when build fails. This is to ensure
artifacts from FI light is uploaded for each build.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
This enables a light version of introspector, which performs analysis
prior to compilation, i.e. without use of LTO. At the current stage
light FI has implemented features to support test-to-harness migration
on OFG including harness source/executable pairing. Once this has been
rolled out in full the light analysis will continue to support more
interesting logic related to program analysis, all purely static.
Signed-off-by: David Korczynski <david@adalogics.com>
Follow-up on #12077 by @alexcrichton cc @maflcko
Main difference is to update
infra/base-images/base-runner/profraw_update.py so that oss-fuzz
converts profraw version 8 to 9 (and llvm-cov seems more tolerant in
older version reading cf
llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp
This way, it should be more transparent for projects, that can be
updated individually or not
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This caused a regression for experiments in OFG.
```sh
2024-09-20 22:39:56.906 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.6, 'max_output_tokens': 8192}
2024-09-20 22:39:57.088 ERROR builder_runner - _run_with_retry_control: Failed to evaluate /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c on cloud, attempt 1:
Traceback (most recent call last): File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 339, in <module> main() File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 332, in main run_experiment(args.project, args.target, args.args, args.upload_output_log, File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 288, in run_experiment extra_tags=[experiment_name, project_name] + tags) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ TypeError: can only concatenate list (not "NoneType") to list
2024-09-20 22:39:57.088 INFO evaluator - log: Fixing /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c with vertex_ai_gemini-1-5, attempt 1.
2024-09-20 22:39:57.088 WARNING prompt_builder - _format_fixer_problem: Unexpected empty error message in fix prompt for error_desc: None
2024-09-20 22:39:57.089 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.46, 'max_output_tokens': 8192}
```
From https://github.com/google/oss-fuzz-gen/pull/636
Ref: https://github.com/google/oss-fuzz/pull/12520
Makes it possible to extract all files from a given project. This is
particularly useful for bazel projects where we need a path prior to
bazel build in order to identify harness source.
Ref: https://github.com/google/oss-fuzz-gen/pull/577
The current bazel builds change the owner of the $OUT directory due to
`tar`. This can cause some issues depending on how Docker is run, but,
this is ultimately annoying since e.g. OSS-Fuzz-gen and end-to-end
OSS-Fuzz runs relies on reading the files in the OUT directory, which is
not possible when the owner changes. Furthermore, it's inconsistent with
the existing `compile_*` scripts to change the owner of $OUT/
This fixes it by not changing the owner.
@jonathanmetzman proposed a great idea about saving the machine state
just before compiling the fuzz target so that we can compile different
fuzz targets from that state later without having to go through the
earlier commands.
This is particularly beneficial for `OSS-Fuzz-Gen`.
This PR is an (incomplete) PoC at that.
Ideally, we:
1. [ ] Replace the fuzz target compilation command and all commands
after it with no-ops,
2. [x] Save them into a script (e.g., `$SRC/re-run.sh`), and
3. [x] Push the resulting image for later reuse.
In this way, we can reuse the image later by swapping the fuzz target
source code and executing `$SRC/re-run.sh`.
The script in the PR can do 2, but not 1.
This might be OK already because steps in 1 are normally at the end, and
there is unlikely any check to prevent them, but ideally, we should do
1, too.
To test this locally:
```bash
python infra/helper.py build_image libiec61850
docker run -ti --entrypoint=/bin/bash gcr.io/oss-fuzz/libiec61850
(in container) compile
cat /src/re-run.sh
```
`coverage_extra_args` are currently only used in cloud builds e.g.
7db2eae0b1/infra/build/functions/build_and_run_coverage.py (L128)
and not locally. Besides being confusing, this can cause some issues
with e.g. OSS-Fuzz-gen which relies on the `coverage` command to extract
coverage reports, as it may end up comparing apples to oranges.
This commit fixes it by applying the coverage extra args when coverage
is run by way of `infra/helper.py coverage`.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
Helps pin dependencies at the time of when `cargo-fuzz` itself was
published to avoid dependency updates breaking builds. This fixes a
current issue where all Rust projects using `cargo fuzz` are broken due
to the installation of `cargo-fuzz` failing due to failing to build a
newer dependency with an older nightly.
Closes#12168
This is a follow-up to the discussions held during our Monthly Fuzzing
Collaboration meetings and directly relates to issue #11967.
This Pull Request integrates Ruzzy support for Ruby fuzzing into
OSS-Fuzz. Ruzzy is a coverage-guided fuzzer for pure Ruby code and Ruby
C extensions, developed by Matt (@mschwager) at Trail of Bits. More
information on Ruzzy can be found in the blog post titled "[Introducing
Ruzzy, a coverage-guided Ruby
fuzzer](https://blog.trailofbits.com/2024/03/29/introducing-ruzzy-a-coverage-guided-ruby-fuzzer/)".
The first commit of this PR integrates Ruby support into the OSS-Fuzz
project via Ruzzy, while the second one includes the Ox project as an
example of its usage.
The first commit introduces changes in the infra directory, most notably
by adding the base-builder-ruby docker and the ruby install script.
Two scripts, ruzzy-build and ruzzy, have been added to base-builder-ruby
and base-runner respectively. The former creates scripts that start
harnesses with the latter one, and the latter is simply a wrapper for
ruby with LD_PRELOAD.
In order to prevent the duplication of many gigabytes of data, we use
separate installation directories for RubyGem. Technically, Ruzzy can be
installed in the default directory without any performance disadvantage,
but having a separate directory may facilitate troubleshooting.
This implementation was arrived at through testing a few ideas. If you
have suggestions for further improvements, please let me know. I am
currently addressing my concerns in the related issue.
Using the provided scripts isn't necessary but it does simplify the
process. Installation directories are set using environment variables in
the Dockerfiles, making it transparent for users.
The second commit simply adds a project to illustrate how
straightforward the integration process is. You can test it using the
standard helper commands.
Fixes: #11967
Co-authored-by: mschwager <matt.schwager@trailofbits.com>
---------
Co-authored-by: mschwager <matt.schwager@trailofbits.com>
Oliver Chang