Commit Graph

1134 Commits

Author SHA1 Message Date
jonathanmetzman db8467bf30
[CI] Don't do coverage builds for engineless fuzzers. (#4374)
Coverage builds need to be special cased since they aren't specified
in sanitizers. Instead they are done for all C/C++ projects that
use libFuzzer.

Move all of this special casing to `should_build_coverage` and call that from
`should_build` so we have one place where we decide this.
Add tests as well.

Fixes: https://github.com/google/oss-fuzz/issues/4371
2020-08-26 11:59:56 -07:00
jonathanmetzman 13f7ba0f63
[presubmit] Allow 'none' as a fuzzing engine (#4370)
'none' is used by projects like spidermonkey that use the JS fuzzer.
2020-08-25 16:31:29 -07:00
Max Moroz eabcccf848
[infra] Fix reproduce command invoking run_fuzzer for a single testcase (#4338). (#4367)
* [infra] Fix reproduce command invoking run_fuzzer for a single testcase (#4338).

* review feedback

* do not introduce new variable, rely on SKIP_SEED_CORPUS only
2020-08-25 11:38:50 -07:00
Abhishek Arya a02946e8f4
Revert chromium clang roll to fix mismatched coverage data error (#4353)
Temporarily fixes #4348
2020-08-21 22:18:24 -07:00
Abhishek Arya 4628ee31a9
Revert "Revert "Compile + install LLVMgold.so so -flto works (#4332)" (#4351)" (#4352)
This reverts commit 7f8f5e0182.
2020-08-21 22:12:46 -07:00
Abhishek Arya 7f8f5e0182
Revert "Compile + install LLVMgold.so so -flto works (#4332)" (#4351)
This reverts commit d1ddd25937.
2020-08-21 16:52:23 -07:00
van Hauser d1ddd25937
Compile + install LLVMgold.so so -flto works (#4332) 2020-08-18 09:55:08 -07:00
Max Moroz d250f4ffbb
[infra] Upgrade reproduce command to re-use run_fuzzer functionality. (#4324)
* [infra] Upgrade reproduce command to re-use run_fuzzer functionality.

* fix a typo

* add missing quotes
2020-08-17 11:34:21 -07:00
Max Moroz a986d76ef4
[infra] Use 16 threads when building LLVM (#4270). (#4315)
* [infra] Use 16 threads when building LLVM (#4270).

* update the comment
2020-08-13 12:58:02 -07:00
Max Moroz cec66e9662
[infra] base-clang: limit the number of build threads to 4 (#4270). (#4311) 2020-08-12 16:44:29 -07:00
Max Moroz 8712207990
[infra] Make bad_build_check to use run_fuzzer when testing startup crash. (#4301)
* [infra] Make bad_build_check to use run_fuzzer when testing startup crash.

* use SKIP_SEED_CORPUS=1
2020-08-12 09:57:15 -07:00
Max Moroz e4d3fb0f8c
[infra][docs] Be more explicit about code coverage being supported for C/C++ only (#4284). (#4303)
* [infra][docs] Be more explicit about code coverage being supported for C/C++ only (#4284).

* fix typos and pass env variable
2020-08-12 09:32:05 -07:00
Oliver Chang 0987ddf994
even more fixes. (#4297) 2020-08-11 18:27:18 +10:00
Oliver Chang ade417ba20
Fix typo in project_sync.py (#4296)
* Fix typo in project_sync.py

* line length

* format
2020-08-11 16:25:08 +10:00
Abhishek Arya 103aca2e27
Add |builds_per_day| to supported section in project.yaml. (#4291) 2020-08-10 08:08:25 -07:00
Max Moroz 941ada694b
[infra] Add support for custom sanitizer options in run_fuzzer. (#4274)
* [infra] Add support for custom sanitizer options in run_fuzzer.

* address review feedback

* fix presubmit warnings
2020-08-07 12:48:23 -07:00
Abhishek Arya 751aa78373
Fix LLVM_REVISION parse regex (#4275)
Fix parse regex as per format change in
e6863f8783%5E%21/#F1
2020-08-06 20:12:58 -07:00
jonathanmetzman 571dbfb39f
Reland 55d9a81. (#4269)
The issue with 55d9a81 that caused it to be reverted by 3370f8f
was probably due to the build trigger on GCB being stale. I've
updated it so this commit should stick.
2020-08-06 14:33:08 -07:00
Evgeny Vereshchagin 3370f8f7e5
Revert "[CIFuzz] Allow supplying a manual source checkout (#4250)" (#4263)
This reverts commit 55d9a81cd7.

Closes https://github.com/google/oss-fuzz/issues/4262
2020-08-06 06:11:59 -07:00
Oliver Chang 65a83e2125
Split up builds status function. (#4261)
- Split into separate invocations for fuzzing, coverage, and badges.
- Also use threads to speed things up.
2020-08-06 13:59:08 +10:00
jonathanmetzman 55d9a81cd7
[CIFuzz] Allow supplying a manual source checkout (#4250)
Make CIFuzz building accept an env var `MANUAL_SRC_PATH` that points to a manually prepared checkout of the project-under-tests's source code. This allows projects like Skia which are not on OSS-Fuzz and/or need to make changes to the repo after checking out the right commit to use CIFuzz.
Note that for now we aren't supporting this in GitHub, so projects that need to modify the source after checkout can't use it until
we do support it.

Also, use the local copy of OSS-Fuzz when building cifuzz-base instead of cloning it from GitHub. This makes local debugging/development much easier since it allows one to use CIFuzz with local changes.
2020-08-05 18:45:12 -07:00
Oliver Chang 909db51a15
More build infra cleanup. (#4252)
- Deleted unused code.
- Consolidate some modules.
2020-08-06 09:36:47 +10:00
Oliver Chang 3f1d43115f
Clean up build infra. (#4251)
- Remove gcb folder. Move the files that we still need into the
  build/functions dir.
- Remove badge_images. They are now stored on GCS.
- Remove jenins configs.
2020-08-05 11:20:48 +10:00
kabeer27 21cd8de8dc
Skipping fuzzing build when project is disabled (#4239)
* Skipping fuzzing build when project is disabled

* Removing skip build and making it better
2020-07-31 17:20:09 +10:00
Oliver Chang 47b57ca1dd
Fix a typo from previous MSan function addition. (#4241) 2020-07-31 16:58:28 +10:00
Oliver Chang 95e66c619e
Move cmake install to base-clang. (#4240)
Clang now needs a newer cmake to build.
2020-07-31 16:55:30 +10:00
kabeer27 8e4c7b92fd
Adding msan builder to gcp (#4234)
* Adding msan builder to gcp

* Formatting changes

* Refactoring and reducing redundancy

* Moving msan builder entry point into base_images
2020-07-31 14:42:09 +10:00
Oliver Chang 5e3348cdb9
Create individual schedulers for requesting coverage builds. (#4237)
The single function for requesting all coverage builds was timing out.
2020-07-31 10:56:18 +10:00
kabeer27 a622daf7f5
better fix for template (#4226) 2020-07-30 18:01:07 +10:00
kabeer27 341b3d836a
Display historical logs ui change (#4197)
* Initial UI Change to display historical logs

* Minor changes

* More formatting changes

* Adding license header

* Changes, take a look Oliver

* Changes for handling empty build history/ no last successful build

* Reverting the logs folder path from /logs/log to /log

* Merged to latest master and other chnages to template

* Changing backend to reflect build history

* Fixing order of query

* Removed last build status

* More changes

* Adding unit tests and fixes found by unit testing

* Fixing lint errors

* Added more unit tests and fixed typos and formatting

* Fixing comment caps

* Minor formatting issue

Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2020-07-30 16:20:22 +10:00
Oliver Chang ccb5821f51
Fix build status update. (#4224)
Check for valid statuses rather than only excluding the WORKING status.
There are other failure statuses that need to be accounted for (e.g.
EXPIRED).
2020-07-30 15:10:52 +10:00
Oliver Chang 689f3500a2
Add missing dash to build tag. (#4221) 2020-07-30 08:31:01 +10:00
Oliver Chang ac4af2436f
build: Add queueTtl argument to one more place. (#4219) 2020-07-30 08:21:49 +10:00
Tom Ritter 284dad011e
[rnp] Add afl engine (#4212)
* [rnp] Add afl engine

* Add fuzz_dump and fuzz_keyring to the exception list

These binaries - from https://github.com/rnpgp/rnp/tree/master/src/fuzzing -
are very tiny and just call into the library with a single API,
and are not staticly compiled - so they have very few UBSAN calls.
2020-07-29 09:56:23 -07:00
kabeer27 5893c61c77
Fixing missing dash in datastore query (#4213) 2020-07-29 17:32:54 +10:00
Oliver Chang eb19f16419
Add queueTtl argument to cloud build request. (#4211)
To prevent expiry of queued builds.
2020-07-29 14:50:46 +10:00
Oliver Chang 7bb74407dc
Fix an Python 3 exception in build_project.py. (#4210) 2020-07-29 13:04:59 +10:00
Oliver Chang 4e2a7c07e5
Many build infra fixes. (#4209)
- Remove base image project argument from deploy.sh. Deploy all
  functions to the main oss-fuzz image.
- Reduce max instances of functions to 1 to avoid rate limiting issues.
- Fix missing ndb context initialization in request_build.py
- Fix incorrect tags in BuildHistory entities. "-" was doubled.
- Fix base build deployment with incorrect schedule variable.
- Add scripts for requesting builds locally.
2020-07-29 12:39:49 +10:00
Oliver Chang 24b4bb32cb
Fix builds status requirements.txt (#4203) 2020-07-28 10:13:46 +10:00
Oliver Chang cf1ce595ba
Disable coverage job sync. (#4195)
Migrating to new infratructure for requesting coverage jobs.
2020-07-27 16:50:45 +10:00
Oliver Chang bc7d478d4a
build: Project sync fixes. (#4194)
- Add some more logging.
- Use GitHub client ID/secret rather than personal access token.
- Fix function deploy wrt "--project" argument.
2020-07-27 13:19:44 +10:00
Abhishek Arya dbeab81f5d
Add corpus-dir to run_fuzzer, for later use in coverage cmd. (#4191)
* Add corpus-dir to run_fuzzer, for later use in coverage cmd.

* Update as per comment.
2020-07-24 16:41:45 -07:00
Max Moroz 70d278bb53
[infra] Rename msan-builder to msan-libs-builder (#3388). (#4190)
* [infra] Rename msan-builder to msan-libs-builder and add (broken) dfsan-libs-builder (#3388).

* remove dfsan-libs-builder for now

* presubmit format
2020-07-24 12:38:00 -07:00
Max Moroz a9d0062a55
[infra] Rename base-msan-builder image to base-sanitizer-libs-builder (#3388). (#4187)
* [infra] Rename base-msan-builder image to base-sanitizer-builder (#3388).

* rename to base-sanitizer-libs-builder
2020-07-23 16:36:58 -07:00
kabeer27 9d2381a621
Initial integration of builds_status (#4175) 2020-07-23 16:23:31 +10:00
kabeer27 769c6e80f2
Adding build coverage cloud function, and refactoring test_utils.py (#4151)
* Adding build coverage cloud function, and refactoring test_utils.py

* Multiple changes will add a comment.

* Catching sys.exit() exception and removing useless wrapper build_steps function

* Comment update

* Comment update

* Comment formatting
2020-07-20 16:30:48 +10:00
Max Moroz 05bafd4fcf
[infra] Build libc++ and libc++abi with DFSan (#3388, #1632). (#4153)
* [infra] Build libc++ and libc++abi with DFSan (#3388, #1632).

* create a common function to reduce code duplication
2020-07-17 13:54:36 -07:00
kabeer27 f0d54c33cb
Refactoring and request build cloud function (#4120)
Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-16 11:11:18 +10:00
Abhishek Arya 21ea9d6e3d
Remove sanitizer validation checks in CIFuzz, (#4131)
Sanitizer validation is not needed and it is hacky code with
no proper yaml parsing. sanitizer attribute is properly
documented.
Fixes #3996.
2020-07-15 13:00:57 -07:00
Abhishek Arya cfb18fbea7
CI fix. (#4139) 2020-07-15 09:46:24 -07:00