When a corpus zip file is unzipped the destination folder is set to be
the directory corresponding to the filepath of the zip file but without
".zip" in the name. This is achieved by `for f in /corpus/*.zip; do
unzip -q $f -d ${f%%.*}` where `f%%.*` substitutes the path of `f` based
on the first occurrence of ".". This causes some issues with fuzztest
fuzzers where a fuzzer name always has a "." in it, e.g.
`escaping_test@EscapingTest.EscapingAStringNeverTriggersUndefinedBehavior`
Substituting the name in this way causes issues for some coverage builds
e.g.
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53479
This changes it to substiute based on the last occurrence of ".", i.e.
just cutting off the ".zip".
Alternatively, we could substitute over
e.g. ".zip", however, this may cause some issues as a fuzztest fuzzer
may have ".zip" in the name.
Signed-off-by: David Korczynski <david@adalogics.com>
Signed-off-by: David Korczynski <david@adalogics.com>
* Fix bugs
1. Don't try to build the script name as a project.
2. Add a flag to force builds of projects that previously failed.
3. Make sure we build projects from our PR branch.
4. Wait on all builds not just builds of the last type.
5. Don't use test bucket for corpus or coverage (will corpus work or fail because of creds?)
Add tests for these features.
This adds ci.py a script that can be used to test changes to the base-images.
The script does the following:
1. Builds and pushes a test image for each base image. This will be used in step 3.
2. Finds which projects are buildable based on user specifications of: projects, build type (e.g. sanitizer/fuzzing engine) and which of these succeeded in production.
3. Submits builds for those projects.
4. Polls GCB to check which builds succeeded and which failed.
5. Returns 0 if all builds succeeded.
Currently this script is only useful to run locally.
Related: https://github.com/google/oss-fuzz/issues/7145
* Returning empty list of steps in case of error when downloading coverage data
* Better error handling
* Using logging.warning and more informative message
- Point `compile` to the right `fuzz-introspector` location (since it's no longer checked out in $SRC).
- Rename build tag to "introspector" to be more consistent with other
tags.
- Fix bad merge in deploy.sh script.
- Add introspector setup to project sync.
- Enable more logging for project sync cron.
* Helper flags for swift compilation
* Documentation for swift project integration
* Adds swift to the languages with coverage
* Only thread sanitizer is supported
* Fixes swift coverage target compilation
* fixup flags facotring
* swift: run on new ubuntu
* fixup
* swift: right copy for symbolizer
Important functional changes involve mostly improvements to
the command line scripts (this doesn't affect the build infra, only
local use):
1. Make sure scripts use the same builder as builds requested by infra, otherwise builds
will be very slow and will fail for larger projects.
2. Allow users to specify --test-images to use base images with suffix "-testing"
3. Allow script users to specify --parallel for parallel builds.
4. Allow script users to specify --testing so that builds are uploaded to testing buckets.
5. Allow script users to specify --branch so that builds use specified branch instead of master.
6. Clone oss-fuzz with depth 1 for improved speed and space usage.
7. Use logging instead of writing to stderr or print.
8. Allow scripts to accept multiple projects.
9. Allow script to keep executing after failure to get build steps.
10. Change scripts to use python3.
11. Tag more so builds are easier to query.
12. Log the gcb page for each build.
Other changes include major refactoring:
1. Don't construct image names from scratch using format strings each time they are used.
Provide a helper function for this.
2. Provide a helper function, get_env instead of constructing the env from scratch each time.
3. Move compile step into its own function: get_compile_step.
4. Move upload steps into their own helper function get_upload_steps.
5. Don't misuse the name image_project when we really mean cloud project.
6. Move cleanup step into its own helper function: get_cleanup_step.
7. Exit with returncode of main function from build_project.
8. Add unittests for build_project.
9. Make request_build share run_build code with build_project.
10. Use proper spacing in comments.
11. Test builds other than libfuzzer-ASAN-x86_64. Test other sanitizers, fuzzers and architectures
12. Make build_and_run_coverage share more code with build_project.
13. Move tests for build_and_run_coverage_test.py out of requst_coverage_test.py into their own file.
14. Use single quotes for strings.
15. Store state for a build in Build object instead of passing it everywhere.
16. Don't abuse project_yaml dict for storing project state. Use a Project object instead.
17. Better variable naming.
18. Use more classes instead of passing around arguments.
19. Use more f-strings.
20. Make scripts share main function.
21. Begin comments with uppercase and end with period.
22. Don't import functions or classes as dictated by style guide.
23. Share more test code in test_utils
Related: #6180.
- Remove gcb folder. Move the files that we still need into the
build/functions dir.
- Remove badge_images. They are now stored on GCS.
- Remove jenins configs.
Navidem