Rooba
/
oss-fuzz
mirror of https://github.com/google/oss-fuzz.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
oss-fuzz/infra/build/functions/test_data/expected_coverage_build_ste...

208 lines
6.0 KiB
JSON
Raw Normal View History

Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
[
{
"args": [
"clone",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"https://github.com/google/oss-fuzz.git",
"--depth",
"1"
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
],
"name": "gcr.io/cloud-builders/git"
},
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
Setup trial builds in CI. (#7415) This adds the script that will be used by the GCB trigger to do trial builds when we have a base-images change.
2022-04-04 21:47:21 +00:00
"--tag",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"gcr.io/oss-fuzz/test-project",
"."
],
"dir": "oss-fuzz/projects/test-project"
},
{
"name": "gcr.io/oss-fuzz/test-project",
"args": [
"bash",
"-c",
"srcmap > /workspace/srcmap.json && cat /workspace/srcmap.json"
],
"env": [
"OSSFUZZ_REVISION=$REVISION_ID",
"FUZZING_LANGUAGE=c++"
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
],
"id": "srcmap"
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
},
{
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
"name": "gcr.io/cloud-builders/docker",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"env": [
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"ARCHITECTURE=x86_64",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"FUZZING_ENGINE=libfuzzer",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"FUZZING_LANGUAGE=c++",
"HOME=/root",
"OUT=/workspace/out/libfuzzer-coverage-x86_64",
"SANITIZER=coverage"
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
],
"args": [
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
"run",
"--platform",
"linux/amd64",
"-v",
"/workspace:/workspace",
"-e",
"ARCHITECTURE=x86_64",
"-e",
"FUZZING_ENGINE=libfuzzer",
"-e",
"FUZZING_LANGUAGE=c++",
"-e",
"HOME=/root",
"-e",
"OUT=/workspace/out/libfuzzer-coverage-x86_64",
"-e",
"SANITIZER=coverage",
"-t",
"gcr.io/oss-fuzz/test-project",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"bash",
"-c",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"rm -r /out && cd /src && cd /src && mkdir -p /workspace/out/libfuzzer-coverage-x86_64 && compile || (echo \"********************************************************************************\nFailed to build.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer coverage --engine libfuzzer --architecture x86_64 test-project\n********************************************************************************\" && false)"
],
"id": "compile-libfuzzer-coverage-x86_64"
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
},
{
"url": "test_download"
},
{
"name": "gcr.io/oss-fuzz-base/base-runner",
"env": [
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"ARCHITECTURE=x86_64",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"FUZZING_ENGINE=libfuzzer",
"FUZZING_LANGUAGE=c++",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"HOME=/root",
"OUT=/workspace/out/libfuzzer-coverage-x86_64",
"SANITIZER=coverage",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"HTTP_PORT=",
"COVERAGE_EXTRA_ARGS="
],
"args": [
"bash",
"-c",
"for f in /corpus/*.zip; do unzip -q $f -d ${f%%.*} || (echo \"Failed to unpack the corpus for $(basename ${f%%.*}). This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.\" && exit 1); done && coverage || (echo \"********************************************************************************\nCode coverage report generation failed.\nTo reproduce, run:\npython infra/helper.py build_image test-project\npython infra/helper.py build_fuzzers --sanitizer coverage test-project\npython infra/helper.py coverage test-project\n********************************************************************************\" && false)"
],
"volumes": [
{
"name": "corpus",
"path": "/corpus"
}
]
},
{
"name": "gcr.io/cloud-builders/gsutil",
"entrypoint": "sh",
"args": [
"-c",
"gsutil -m rm -rf gs://oss-fuzz-coverage/test-project/reports/20200101 || exit 0"
]
},
{
"name": "gcr.io/cloud-builders/gsutil",
"args": [
"-m",
"cp",
"-r",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"/workspace/out/libfuzzer-coverage-x86_64/report",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"gs://oss-fuzz-coverage/test-project/reports/20200101"
]
},
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
{
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
"name": "gcr.io/cloud-builders/gsutil",
"entrypoint": "sh",
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
"args": [
"-c",
"gsutil -m rm -rf gs://oss-fuzz-coverage/test-project/reports-by-target/20200101 || exit 0"
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
]
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
},
{
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
"name": "gcr.io/cloud-builders/gsutil",
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
"args": [
"-m",
"cp",
"-r",
"/workspace/out/libfuzzer-coverage-x86_64/report_target",
"gs://oss-fuzz-coverage/test-project/reports-by-target/20200101"
[AArch64] Support building on GCB (#8197) Implemented using buildx to emulate aarch64 using QEMU Related: #8164
2022-08-11 20:34:13 +00:00
]
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
},
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
{
"name": "gcr.io/cloud-builders/gsutil",
"entrypoint": "sh",
"args": [
"-c",
"gsutil -m rm -rf gs://oss-fuzz-coverage/test-project/fuzzer_stats/20200101 || exit 0"
]
},
{
"name": "gcr.io/cloud-builders/gsutil",
"args": [
"-m",
"cp",
"-r",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"/workspace/out/libfuzzer-coverage-x86_64/fuzzer_stats",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"gs://oss-fuzz-coverage/test-project/fuzzer_stats/20200101"
]
},
FuzzIntrospector build steps (#7162) Adds build steps for fuzz introspector to the coverage build.
2022-01-31 00:56:22 +00:00
{
"name": "gcr.io/cloud-builders/gsutil",
"entrypoint": "sh",
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
"args": [
"-c",
"gsutil -m rm -rf gs://oss-fuzz-coverage/test-project/textcov_reports/20200101 || exit 0"
]
FuzzIntrospector build steps (#7162) Adds build steps for fuzz introspector to the coverage build.
2022-01-31 00:56:22 +00:00
},
{
"name": "gcr.io/cloud-builders/gsutil",
Reland "Upload per target coverage report. (#7389)" (#7400)" (#7407) Only upload per-target reports for C/C++. This reverts commit 1ae57df24d9d0146d6a2486bdb6e01ca7157ccef.
2022-03-21 03:57:55 +00:00
"args": [
"-m",
"cp",
"-r",
"/workspace/out/libfuzzer-coverage-x86_64/textcov_reports",
"gs://oss-fuzz-coverage/test-project/textcov_reports/20200101"
]
FuzzIntrospector build steps (#7162) Adds build steps for fuzz introspector to the coverage build.
2022-01-31 00:56:22 +00:00
},
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
{
"name": "gcr.io/cloud-builders/gsutil",
"entrypoint": "sh",
"args": [
"-c",
"gsutil -m rm -rf gs://oss-fuzz-coverage/test-project/logs/20200101 || exit 0"
]
},
{
"name": "gcr.io/cloud-builders/gsutil",
"args": [
"-m",
"cp",
"-r",
[build][infra] Improve build functions. (#6296) Important functional changes involve mostly improvements to the command line scripts (this doesn't affect the build infra, only local use): 1. Make sure scripts use the same builder as builds requested by infra, otherwise builds will be very slow and will fail for larger projects. 2. Allow users to specify --test-images to use base images with suffix "-testing" 3. Allow script users to specify --parallel for parallel builds. 4. Allow script users to specify --testing so that builds are uploaded to testing buckets. 5. Allow script users to specify --branch so that builds use specified branch instead of master. 6. Clone oss-fuzz with depth 1 for improved speed and space usage. 7. Use logging instead of writing to stderr or print. 8. Allow scripts to accept multiple projects. 9. Allow script to keep executing after failure to get build steps. 10. Change scripts to use python3. 11. Tag more so builds are easier to query. 12. Log the gcb page for each build. Other changes include major refactoring: 1. Don't construct image names from scratch using format strings each time they are used. Provide a helper function for this. 2. Provide a helper function, get_env instead of constructing the env from scratch each time. 3. Move compile step into its own function: get_compile_step. 4. Move upload steps into their own helper function get_upload_steps. 5. Don't misuse the name image_project when we really mean cloud project. 6. Move cleanup step into its own helper function: get_cleanup_step. 7. Exit with returncode of main function from build_project. 8. Add unittests for build_project. 9. Make request_build share run_build code with build_project. 10. Use proper spacing in comments. 11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures 12. Make build_and_run_coverage share more code with build_project. 13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file. 14. Use single quotes for strings. 15. Store state for a build in Build object instead of passing it everywhere. 16. Don't abuse project_yaml dict for storing project state. Use a Project object instead. 17. Better variable naming. 18. Use more classes instead of passing around arguments. 19. Use more f-strings. 20. Make scripts share main function. 21. Begin comments with uppercase and end with period. 22. Don't import functions or classes as dictated by style guide. 23. Share more test code in test_utils Related: #6180.
2021-08-25 18:44:52 +00:00
"/workspace/out/libfuzzer-coverage-x86_64/logs",
Adding build coverage cloud function, and refactoring test_utils.py (#4151) * Adding build coverage cloud function, and refactoring test_utils.py * Multiple changes will add a comment. * Catching sys.exit() exception and removing useless wrapper build_steps function * Comment update * Comment update * Comment formatting
2020-07-20 06:30:48 +00:00
"gs://oss-fuzz-coverage/test-project/logs/20200101"
]
},
{
"name": "gcr.io/cloud-builders/gsutil",
"args": [
"cp",
"/workspace/srcmap.json",
"gs://oss-fuzz-coverage/test-project/srcmap/20200101.json"
]
},
{
"name": "gcr.io/cloud-builders/curl",
"args": [
"-H",
"Content-Type: application/json",
"-X",
"PUT",
"-d",
"{\"fuzzer_stats_dir\": \"gs://oss-fuzz-coverage/test-project/fuzzer_stats/20200101\", \"html_report_url\": \"https://storage.googleapis.com/oss-fuzz-coverage/test-project/reports/20200101/linux/index.html\", \"report_date\": \"20200101\", \"report_summary_path\": \"gs://oss-fuzz-coverage/test-project/reports/20200101/linux/summary.json\"}",
"test_url"
]
}
[Ubuntu upgrade] Remove support for prebuilt MSAN libraries. (#6280) This is done in anticipation of the upgrade to Ubuntu 20.04 which wont support this. We'll do this first so we can handle any breakages caused by this step before needing to handle breakages caused by the upgrade. However, there shouldn't be any breakages due to #6281, but there may be some projects we overlooked. The only exception to this is libcxx. Related: #6180.
2021-08-25 15:36:05 +00:00
]