Upload introspector report even when build fails. This is to ensure
artifacts from FI light is uploaded for each build.
---------
Signed-off-by: David Korczynski <david@adalogics.com>
This caused a regression for experiments in OFG.
```sh
2024-09-20 22:39:56.906 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.6, 'max_output_tokens': 8192}
2024-09-20 22:39:57.088 ERROR builder_runner - _run_with_retry_control: Failed to evaluate /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c on cloud, attempt 1:
Traceback (most recent call last): File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 339, in <module> main() File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 332, in main run_experiment(args.project, args.target, args.args, args.upload_output_log, File "/tmp/tmp5dlnpp7z/infra/build/functions/target_experiment.py", line 288, in run_experiment extra_tags=[experiment_name, project_name] + tags) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ TypeError: can only concatenate list (not "NoneType") to list
2024-09-20 22:39:57.088 INFO evaluator - log: Fixing /experiment/results/output-htslib-hfile_list_plugins/fixed_targets/02.c with vertex_ai_gemini-1-5, attempt 1.
2024-09-20 22:39:57.088 WARNING prompt_builder - _format_fixer_problem: Unexpected empty error message in fix prompt for error_desc: None
2024-09-20 22:39:57.089 INFO models - do_generate: vertex_ai_gemini-1-5 generating response with config: {'temperature': 0.46, 'max_output_tokens': 8192}
```
From https://github.com/google/oss-fuzz-gen/pull/636
Ref: https://github.com/google/oss-fuzz/pull/12520
Add a 3-hour timeout to `target_experiment.py`, which should be more
than enough given fuzzing timeout is 30 minutes and most successful
cloud builds finish in 2.5 hours empirically.
Some project (e.g.
[civetweb)](88dad2f6ef/Makefile (L84))
does not use jcc to build fuzz targets.
This pr:
1. Prevents cloud build fail at upload jcc err step.
2. Add hint text in err.log for uploading.
---------
Co-authored-by: Jim Choi <jimchoi@google.com>
Upload `JCC`'s error log to assist `OSS-Fuzz-Gen` analyzing and
classifying build errors:
1. Change JCC `err.log` location, because`/out` was removed before
`compile` by default and `err.log` cannot be saved there.
2. Add a step in `target_experiment.py` to upload `err.log` to gcloud
bucket.
---------
Co-authored-by: Jim Choi <jimchoi@google.com>
This reverts commit a80fd730e4.
The investigation shows that the slowness issues and `BrokenPipe` error
mentioned in
https://github.com/google/oss-fuzz/pull/11727#issue-2205034022 are not
caused by #11701, so this PR reverts #11727.
Queued cloud builds of a full experiment likely caused the slowness.
Specifically, some cloud builds' `BrokenPipe` error caused them to queue
forever, which blocks other cloud builds and causes the slowness.
This should have been solved by #11729, I will start another full
experiment shortly and verify that.
The GCE metadata server can be unresponsive sometimes, resulting in this
exception.
This should address the root cause of the stack trace pointed out in
https://github.com/google/oss-fuzz/pull/11727
Related: https://github.com/google/oss-fuzz-gen/pull/172, #11700.
This simplifies the link generation in benchmark `JSON`.
Previously, the binary directory name matches the binary name, now it is
always called `target_binary/`.
Helps https://github.com/google/oss-fuzz-gen/issues/156:
1. `target_experiment.py` takes a new parameter,
`upload_reproducer_path`.
2. `target_experiment.py` saves crash reproducer to
`local_artifect_path`.
3. `target_experiment.py` uploads the fuzz target binary and the crash
reproducer to bucket directory `upload_reproducer_path`.
Instead, tar up the project directory ourselves, upload this to GCS, and
create the Build ourselves.
This is also a lot more efficient, as we're excluding a lot of
unnecessary dirs from being uploaded.
Fixes: https://github.com/google/oss-fuzz-gen/issues/163
Instead of relying on `gcloud builds submit` to poll the status of the
build for us, we poll ourselves.
This is because `gcloud builds submit` has a fixed poll interval of 1
second.
Mitigates the known issue where we don't automatically change to the
`WORKDIR` defined in `Dockerfile` when running cloud experiments.
Question:
Would it be preferred if I introduce a flag for this?
(e.g., `--use_workdir` or `--workdir=/src/<project>`)
While this gives more flexibility, I feel `cd` to `WORKDIR` should
always be preferred if we want the cloud experiments to behave the same
as local ones.
This allows us to run arbitrary scripts inside project containers on
GCB.
All files in `/workspace/out` are upload to a specified GCS path at the
end.
Usage:
```bash
$ python project_experiment.py --project libxml2 --command 'python3 /workspace/hello.py' \
--experiment_name test-ochang \
--upload_output gs://BUCKET/to/upload/NAME
```
(Where /workspace is the OSS-Fuzz repository).
Previously the cloud build exited before we upload the logs. Now we
write a file to record build success and report the failure later after
uploading the log if there was one.
This launches a GCB build for a project, runs a specified fuzz target
with specified arguments, and uploads the output to a specified path.
e.g.
```
python infra/build/functions/target_experiment.py --project=libxml2 --target xml \
--output_log gs://oss-fuzz-gcb-experiment-run-logs/output.log -- -max_total_time=30
```
Oliver Chang