Commit Graph

1111 Commits

Author SHA1 Message Date
kabeer27 21cd8de8dc
Skipping fuzzing build when project is disabled (#4239)
* Skipping fuzzing build when project is disabled

* Removing skip build and making it better
2020-07-31 17:20:09 +10:00
Oliver Chang 47b57ca1dd
Fix a typo from previous MSan function addition. (#4241) 2020-07-31 16:58:28 +10:00
Oliver Chang 95e66c619e
Move cmake install to base-clang. (#4240)
Clang now needs a newer cmake to build.
2020-07-31 16:55:30 +10:00
kabeer27 8e4c7b92fd
Adding msan builder to gcp (#4234)
* Adding msan builder to gcp

* Formatting changes

* Refactoring and reducing redundancy

* Moving msan builder entry point into base_images
2020-07-31 14:42:09 +10:00
Oliver Chang 5e3348cdb9
Create individual schedulers for requesting coverage builds. (#4237)
The single function for requesting all coverage builds was timing out.
2020-07-31 10:56:18 +10:00
kabeer27 a622daf7f5
better fix for template (#4226) 2020-07-30 18:01:07 +10:00
kabeer27 341b3d836a
Display historical logs ui change (#4197)
* Initial UI Change to display historical logs

* Minor changes

* More formatting changes

* Adding license header

* Changes, take a look Oliver

* Changes for handling empty build history/ no last successful build

* Reverting the logs folder path from /logs/log to /log

* Merged to latest master and other chnages to template

* Changing backend to reflect build history

* Fixing order of query

* Removed last build status

* More changes

* Adding unit tests and fixes found by unit testing

* Fixing lint errors

* Added more unit tests and fixed typos and formatting

* Fixing comment caps

* Minor formatting issue

Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2020-07-30 16:20:22 +10:00
Oliver Chang ccb5821f51
Fix build status update. (#4224)
Check for valid statuses rather than only excluding the WORKING status.
There are other failure statuses that need to be accounted for (e.g.
EXPIRED).
2020-07-30 15:10:52 +10:00
Oliver Chang 689f3500a2
Add missing dash to build tag. (#4221) 2020-07-30 08:31:01 +10:00
Oliver Chang ac4af2436f
build: Add queueTtl argument to one more place. (#4219) 2020-07-30 08:21:49 +10:00
Tom Ritter 284dad011e
[rnp] Add afl engine (#4212)
* [rnp] Add afl engine

* Add fuzz_dump and fuzz_keyring to the exception list

These binaries - from https://github.com/rnpgp/rnp/tree/master/src/fuzzing -
are very tiny and just call into the library with a single API,
and are not staticly compiled - so they have very few UBSAN calls.
2020-07-29 09:56:23 -07:00
kabeer27 5893c61c77
Fixing missing dash in datastore query (#4213) 2020-07-29 17:32:54 +10:00
Oliver Chang eb19f16419
Add queueTtl argument to cloud build request. (#4211)
To prevent expiry of queued builds.
2020-07-29 14:50:46 +10:00
Oliver Chang 7bb74407dc
Fix an Python 3 exception in build_project.py. (#4210) 2020-07-29 13:04:59 +10:00
Oliver Chang 4e2a7c07e5
Many build infra fixes. (#4209)
- Remove base image project argument from deploy.sh. Deploy all
  functions to the main oss-fuzz image.
- Reduce max instances of functions to 1 to avoid rate limiting issues.
- Fix missing ndb context initialization in request_build.py
- Fix incorrect tags in BuildHistory entities. "-" was doubled.
- Fix base build deployment with incorrect schedule variable.
- Add scripts for requesting builds locally.
2020-07-29 12:39:49 +10:00
Oliver Chang 24b4bb32cb
Fix builds status requirements.txt (#4203) 2020-07-28 10:13:46 +10:00
Oliver Chang cf1ce595ba
Disable coverage job sync. (#4195)
Migrating to new infratructure for requesting coverage jobs.
2020-07-27 16:50:45 +10:00
Oliver Chang bc7d478d4a
build: Project sync fixes. (#4194)
- Add some more logging.
- Use GitHub client ID/secret rather than personal access token.
- Fix function deploy wrt "--project" argument.
2020-07-27 13:19:44 +10:00
Abhishek Arya dbeab81f5d
Add corpus-dir to run_fuzzer, for later use in coverage cmd. (#4191)
* Add corpus-dir to run_fuzzer, for later use in coverage cmd.

* Update as per comment.
2020-07-24 16:41:45 -07:00
Max Moroz 70d278bb53
[infra] Rename msan-builder to msan-libs-builder (#3388). (#4190)
* [infra] Rename msan-builder to msan-libs-builder and add (broken) dfsan-libs-builder (#3388).

* remove dfsan-libs-builder for now

* presubmit format
2020-07-24 12:38:00 -07:00
Max Moroz a9d0062a55
[infra] Rename base-msan-builder image to base-sanitizer-libs-builder (#3388). (#4187)
* [infra] Rename base-msan-builder image to base-sanitizer-builder (#3388).

* rename to base-sanitizer-libs-builder
2020-07-23 16:36:58 -07:00
kabeer27 9d2381a621
Initial integration of builds_status (#4175) 2020-07-23 16:23:31 +10:00
kabeer27 769c6e80f2
Adding build coverage cloud function, and refactoring test_utils.py (#4151)
* Adding build coverage cloud function, and refactoring test_utils.py

* Multiple changes will add a comment.

* Catching sys.exit() exception and removing useless wrapper build_steps function

* Comment update

* Comment update

* Comment formatting
2020-07-20 16:30:48 +10:00
Max Moroz 05bafd4fcf
[infra] Build libc++ and libc++abi with DFSan (#3388, #1632). (#4153)
* [infra] Build libc++ and libc++abi with DFSan (#3388, #1632).

* create a common function to reduce code duplication
2020-07-17 13:54:36 -07:00
kabeer27 f0d54c33cb
Refactoring and request build cloud function (#4120)
Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-16 11:11:18 +10:00
Abhishek Arya 21ea9d6e3d
Remove sanitizer validation checks in CIFuzz, (#4131)
Sanitizer validation is not needed and it is hacky code with
no proper yaml parsing. sanitizer attribute is properly
documented.
Fixes #3996.
2020-07-15 13:00:57 -07:00
Abhishek Arya cfb18fbea7
CI fix. (#4139) 2020-07-15 09:46:24 -07:00
Max Moroz 820e252dfa
[infra] Fix a typo in collect_dft bash script. 2020-07-13 10:19:57 -07:00
Max Moroz 2fe0d878ae
Revert "Refactoring gcb libraries for external use by Cloud functions (#4103)" (#4117)
This reverts commit 9ed73c1cd7.
2020-07-13 09:47:11 -07:00
kabeer27 9ed73c1cd7
Refactoring gcb libraries for external use by Cloud functions (#4103)
* Refactoring gcb libraries for external use

* Few changes done, a couple more left

* Fixed linting/formatting issues + other changes requested

* Fixing import order

* Fixing import order

* license header change

* Undo

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-13 14:45:39 +10:00
kabeer27 9413d10e08
[Sync cloud function] Cache file contents (#4104)
* Storing contents of project.yaml and dockerfile, to reduce api calls

* Fixing comment indentation

* Requested changes

* Minor changes

* Importing projectmetadata

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-13 14:21:56 +10:00
Max Moroz 33295d69f7
[infra] Do not pass empty port value to docker. (#4110) 2020-07-10 15:49:46 -07:00
Max Moroz 47484451d5
[infra] Build honggfuzz with -D_HF_LINUX_NO_BFD (#3886). (#4107)
* [infra] Build honggfuzz with -D_HF_LINUX_NO_BFD (#3886).

* dummy edit to trigger CI
2020-07-10 14:50:39 -07:00
Evgeny Vereshchagin c22a5b6db9
utils: ignore subprocess-run-check (#4102)
"check" is omitted intentionally there.

A follow-up to https://github.com/google/oss-fuzz/pull/4101
2020-07-09 14:18:21 -07:00
Oliver Chang 2816e8729c
Fix infra tests. (#4097) 2020-07-10 07:00:02 +10:00
Evgeny Vereshchagin ec269ac72f
CIFuzz: switch to systemd-detect-virt to detect docker (#4101)
Closes https://github.com/google/oss-fuzz/issues/4093
2020-07-09 12:51:34 -07:00
Oliver Chang 662260da6e
bisector: try older commits if old_commit has the same result as new_commit (#4096) 2020-07-09 20:04:13 +10:00
kabeer27 600c514958
Renaming schedule variable to builds_per_day for new feature (#4091)
* Renaming schedule variable to builds_per_day for new feature

* Minor formatting change

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 17:19:59 +10:00
kabeer27 0a712bb5d4
Base image builder (#4080)
* Base image builder

* Forgot to run formatter

* Fixing lint issues and removing unused imports

* Adding missing newline

* Refactoring code

* Fixing linting errors and changing name of cloud function entry

* Adding license header to build_base_images

* Changed logging and print_function

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 14:16:01 +10:00
kabeer27 277886dee9
Resetting emulators (#4079)
* Resetting datastore emulator

* Resetting datastore emulator

* setUp class

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 11:41:54 +10:00
Abhishek Arya 811add4dbf
Don't traverse symlinks in patch_build.py (#4086)
Should fix https://github.com/google/oss-fuzz/issues/4003
2020-07-07 08:11:23 -07:00
devtty1er d561c49ae5
Update Dockerfiles (#4070)
* Use LABEL in place of MAINTAINER

* Remove LABEL maintainer from Dockerfiles
2020-07-06 13:18:23 -07:00
Abhishek Arya b74211ecf9
Skip coverage build in CI for non C/C++ projects. (#4075)
* Skip coverage build in CI for non C/C++ projects.

Fixes https://github.com/google/oss-fuzz/issues/4074

* Test wasmtime.
2020-07-06 13:13:02 -07:00
kabeer27 e49a126000
Feature addition: Individual schedulers for projects (#4066)
This PR, integrates the new feature that allows users to specify their own schedules via project.yaml.
Parameter 'schedule : value' will dictate how many times a day should the project be built, i.e if value = 2
Then project will be built twice a day starting at 6 and interval 24/2 = 12 so 6:00 and 18:00
if parameter is unavailable DEFAULT_SCHEDULE = '0 6 * * *' i.e 6:00 will be used.

The cloud functions creates individual scheduler jobs for each project, with the name project_name + '-scheduler', and subscribes it to the topic 'request_build' with payload message 'project_name' which will be used by the next cloud function to request build for this project.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-06 17:52:24 +10:00
kabeer27 09c7ea778b
Added deploy file and restructured cloud function directory (#4057)
* Adding deploy.sh and restructuring directory

* Integrating review changes

* Passed project id as an argument to each command

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-01 16:13:08 +10:00
Max Moroz 05063afd88
Revert "Don't create empty corpus archive as it wont unpack. (#3903)" (#4048)
This reverts commit cb156403e9.
2020-06-30 13:03:44 -07:00
kabeer27 d15dc2a88f
Renaming infra/build/functions files (#4031)
Renamed main.py to sync.py
and test_sync.py to sync_test.py

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-29 11:10:47 +10:00
Evgeny Vereshchagin 25cff8db8e
[util-linux] cover mnt_table_parse_stream (#4032)
* [util-linux] cover mnt_table_parse_stream

Waiting for https://github.com/karelzak/util-linux/pull/1068

* temporarily point OSS-Fuzz to evverx/util-linux

* make sure it can be built with sanitizer=coverage
2020-06-25 15:35:44 -07:00
kabeer27 181812050d
Cloud function for syncing OSS-Fuzz projects (#3538). (#4023)
Integrating the first cloud function i implemented which syncs the project list from github and uploads the list to cloud datastore, which will be used by another cloud function to request builds.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-25 13:41:04 +10:00
jonathanmetzman cac6118315
[cifuzz] Temporary fix for sanitizer validation. (#3997)
Temporary fix for sanitizer validation.

Our sanitizer validation is hacky and isn't using pyyaml to actually
parse the project.yaml file. Temporarily work around #3996 by not
validating sanitizers if no sanitizers are specified in project.yaml.
2020-06-17 10:32:42 -07:00