mirror of https://github.com/google/oss-fuzz.git
f716590249
The current number of parallel fuzzers running is set to the number of available CPUs. This is causing issues in Tensorflow: ``` Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4501 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: decode_compressed_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4873 Killed lvm-cov show -instr-profile=$profdata_file -object=$target -line-coverage-gt=0 $shared_libraries $BRANCH_COV_ARGS $LL VM_COV_COMMON_ARGS > ${TEXTCOV_REPORT_DIR}/$target.covreport Step #5: /usr/local/bin/coverage: line 75: 4897 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: saved_model_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4638 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:05,246 INFO] Finding shared libraries for targets (if any). ... Step #5: [2023-05-08 11:57:09,911 INFO] Finished finding shared libraries for targets. Step #5: /usr/local/bin/coverage: line 75: 4276 Killed llvm-cov expor -summary-only -instr-profile=$profdata_file -object=$target $shared_libraries $LLVM_COV_COMMON_ARGS > $FUZZER_STATS_DIR/$target.json Step #5: /usr/local/bin/coverage: line 75: 5450 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:40,282 INFO] Finding shared libraries for targets (if any). Step #5: [2023-05-08 11:57:40,323 INFO] Finished finding shared libraries for targets. Step #5: error: end_to_end_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! ``` [log](https://oss-fuzz-build-logs.storage.googleapis.com/log-050f4040-5009-4a23-81c4-9093922b4ffb.txt) (don't open in a browser but `wget`/`curl` it, as it's quite a large file and will probably annoy the browser). I assume this is because the fuzzers take up lots of the memory. A Tensorflow fuzzer can be ~3GB and there are ~50 fuzzers in Tensorflow, so I think the artifacts read by `llvm-profdata merge` will eat up memory, which consequently starts to crash processes on the system. I could imagine this happens for more projects with many fuzzers of large size? Signed-off-by: David Korczynski <david@adalogics.com> Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| base-builder | ||
| base-builder-fuzzbench | ||
| base-builder-go | ||
| base-builder-javascript | ||
| base-builder-jvm | ||
| base-builder-python | ||
| base-builder-rust | ||
| base-builder-swift | ||
| base-clang | ||
| base-image | ||
| base-runner | ||
| base-runner-debug | ||
| README.md | ||
| all.sh | ||
README.md
Building all infra images:
# run from project root
infra/base-images/all.sh