This PR reduces the final `base-builder-jvm` image by 9GB.
Careful review and full integration testing across oss-fuzz projects are
required before merging, obviously.
This worked on two projects I was interested in. One that used
`@FuzzTest` and one that used `fuzzerTestOneInput`
Giving
https://github.com/google/oss-fuzz/pull/12592#issuecomment-2415803665 a
go using `bashlex`
Will place a script in `/out/replay-build-script.sh` with the script
that can be used for replaying.
This is still poc and will need some adjustments, e.g. we need some more
handling on the bash script's ast to reason about loops.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
Co-authored-by: Oliver Chang <ochang@google.com>
Create ccache wrappers for clang[++]-jcc, and remove the explicit path
from the CC/CXX variables in target_experiment.py.
Note: this is not fully tested end to end yet.
Part of https://github.com/google/oss-fuzz-gen/issues/682.
This installs clang wrappers at /ccache/bin, and sets up a build cache
at /ccache/cache. To use this, inside the project container we just need
to do:
```
export PATH=/ccache/bin:$PATH
```
In another PR, we can store the /ccache/cache somewhere we can pull down
at runtime.
Some results:
Fresh compile:
real 0m49.249s
user 10m41.818s
sys 1m2.097s
With ccache cache:
real 0m9.877s
user 0m6.278s
sys 0m19.966s
Fresh compile:
real 1m17.214s
user 0m49.454s
sys 0m27.963s
With ccache:
real 0m34.962s
user 0m18.092s
sys 0m17.083s
The `PARALLEL_FUZZING` environment variable is used to enable parallel
fuzzing in CFLite. The variable is always defined by GH Action, but it
was not interpreted as a boolean value, so it was always enabled even if
the option is set to `false`.
Co-authored-by: Vitor Guidi <vitorguidi@gmail.com>
Upload introspector report even when build fails. This is to ensure
artifacts from FI light is uploaded for each build.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
This enables a light version of introspector, which performs analysis
prior to compilation, i.e. without use of LTO. At the current stage
light FI has implemented features to support test-to-harness migration
on OFG including harness source/executable pairing. Once this has been
rolled out in full the light analysis will continue to support more
interesting logic related to program analysis, all purely static.
Signed-off-by: David Korczynski <david@adalogics.com>
Follow-up on #12077 by @alexcrichton cc @maflcko
Main difference is to update
infra/base-images/base-runner/profraw_update.py so that oss-fuzz
converts profraw version 8 to 9 (and llvm-cov seems more tolerant in
older version reading cf
llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp
This way, it should be more transparent for projects, that can be
updated individually or not
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This caused a regression for experiments in OFG.
```sh
2024-09-20 22:39:56.906 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.6, 'max_output_tokens': 8192}
2024-09-20 22:39:57.088 ERROR builder_runner - _run_with_retry_control: Failed to evaluate /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c on cloud, attempt 1:
Traceback (most recent call last): File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 339, in <module> main() File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 332, in main run_experiment(args.project, args.target, args.args, args.upload_output_log, File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 288, in run_experiment extra_tags=[experiment_name, project_name] + tags) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ TypeError: can only concatenate list (not "NoneType") to list
2024-09-20 22:39:57.088 INFO evaluator - log: Fixing /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c with vertex_ai_gemini-1-5, attempt 1.
2024-09-20 22:39:57.088 WARNING prompt_builder - _format_fixer_problem: Unexpected empty error message in fix prompt for error_desc: None
2024-09-20 22:39:57.089 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.46, 'max_output_tokens': 8192}
```
From https://github.com/google/oss-fuzz-gen/pull/636
Ref: https://github.com/google/oss-fuzz/pull/12520
Makes it possible to extract all files from a given project. This is
particularly useful for bazel projects where we need a path prior to
bazel build in order to identify harness source.
Ref: https://github.com/google/oss-fuzz-gen/pull/577
The current bazel builds change the owner of the $OUT directory due to
`tar`. This can cause some issues depending on how Docker is run, but,
this is ultimately annoying since e.g. OSS-Fuzz-gen and end-to-end
OSS-Fuzz runs relies on reading the files in the OUT directory, which is
not possible when the owner changes. Furthermore, it's inconsistent with
the existing `compile_*` scripts to change the owner of $OUT/
This fixes it by not changing the owner.
@jonathanmetzman proposed a great idea about saving the machine state
just before compiling the fuzz target so that we can compile different
fuzz targets from that state later without having to go through the
earlier commands.
This is particularly beneficial for `OSS-Fuzz-Gen`.
This PR is an (incomplete) PoC at that.
Ideally, we:
1. [ ] Replace the fuzz target compilation command and all commands
after it with no-ops,
2. [x] Save them into a script (e.g., `$SRC/re-run.sh`), and
3. [x] Push the resulting image for later reuse.
In this way, we can reuse the image later by swapping the fuzz target
source code and executing `$SRC/re-run.sh`.
The script in the PR can do 2, but not 1.
This might be OK already because steps in 1 are normally at the end, and
there is unlikely any check to prevent them, but ideally, we should do
1, too.
To test this locally:
```bash
python infra/helper.py build_image libiec61850
docker run -ti --entrypoint=/bin/bash gcr.io/oss-fuzz/libiec61850
(in container) compile
cat /src/re-run.sh
```
`coverage_extra_args` are currently only used in cloud builds e.g.
7db2eae0b1/infra/build/functions/build_and_run_coverage.py (L128)
and not locally. Besides being confusing, this can cause some issues
with e.g. OSS-Fuzz-gen which relies on the `coverage` command to extract
coverage reports, as it may end up comparing apples to oranges.
This commit fixes it by applying the coverage extra args when coverage
is run by way of `infra/helper.py coverage`.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
Helps pin dependencies at the time of when `cargo-fuzz` itself was
published to avoid dependency updates breaking builds. This fixes a
current issue where all Rust projects using `cargo fuzz` are broken due
to the installation of `cargo-fuzz` failing due to failing to build a
newer dependency with an older nightly.
Closes#12168
Tim Allison