Commit Graph

28 Commits

Author SHA1 Message Date
Oliver Chang 0987ddf994
even more fixes. (#4297) 2020-08-11 18:27:18 +10:00
Oliver Chang ade417ba20
Fix typo in project_sync.py (#4296)
* Fix typo in project_sync.py

* line length

* format
2020-08-11 16:25:08 +10:00
Oliver Chang 65a83e2125
Split up builds status function. (#4261)
- Split into separate invocations for fuzzing, coverage, and badges.
- Also use threads to speed things up.
2020-08-06 13:59:08 +10:00
Oliver Chang 909db51a15
More build infra cleanup. (#4252)
- Deleted unused code.
- Consolidate some modules.
2020-08-06 09:36:47 +10:00
Oliver Chang 3f1d43115f
Clean up build infra. (#4251)
- Remove gcb folder. Move the files that we still need into the
  build/functions dir.
- Remove badge_images. They are now stored on GCS.
- Remove jenins configs.
2020-08-05 11:20:48 +10:00
kabeer27 21cd8de8dc
Skipping fuzzing build when project is disabled (#4239)
* Skipping fuzzing build when project is disabled

* Removing skip build and making it better
2020-07-31 17:20:09 +10:00
Oliver Chang 47b57ca1dd
Fix a typo from previous MSan function addition. (#4241) 2020-07-31 16:58:28 +10:00
kabeer27 8e4c7b92fd
Adding msan builder to gcp (#4234)
* Adding msan builder to gcp

* Formatting changes

* Refactoring and reducing redundancy

* Moving msan builder entry point into base_images
2020-07-31 14:42:09 +10:00
Oliver Chang 5e3348cdb9
Create individual schedulers for requesting coverage builds. (#4237)
The single function for requesting all coverage builds was timing out.
2020-07-31 10:56:18 +10:00
kabeer27 341b3d836a
Display historical logs ui change (#4197)
* Initial UI Change to display historical logs

* Minor changes

* More formatting changes

* Adding license header

* Changes, take a look Oliver

* Changes for handling empty build history/ no last successful build

* Reverting the logs folder path from /logs/log to /log

* Merged to latest master and other chnages to template

* Changing backend to reflect build history

* Fixing order of query

* Removed last build status

* More changes

* Adding unit tests and fixes found by unit testing

* Fixing lint errors

* Added more unit tests and fixed typos and formatting

* Fixing comment caps

* Minor formatting issue

Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2020-07-30 16:20:22 +10:00
Oliver Chang ccb5821f51
Fix build status update. (#4224)
Check for valid statuses rather than only excluding the WORKING status.
There are other failure statuses that need to be accounted for (e.g.
EXPIRED).
2020-07-30 15:10:52 +10:00
Oliver Chang 689f3500a2
Add missing dash to build tag. (#4221) 2020-07-30 08:31:01 +10:00
Oliver Chang ac4af2436f
build: Add queueTtl argument to one more place. (#4219) 2020-07-30 08:21:49 +10:00
kabeer27 5893c61c77
Fixing missing dash in datastore query (#4213) 2020-07-29 17:32:54 +10:00
Oliver Chang 4e2a7c07e5
Many build infra fixes. (#4209)
- Remove base image project argument from deploy.sh. Deploy all
  functions to the main oss-fuzz image.
- Reduce max instances of functions to 1 to avoid rate limiting issues.
- Fix missing ndb context initialization in request_build.py
- Fix incorrect tags in BuildHistory entities. "-" was doubled.
- Fix base build deployment with incorrect schedule variable.
- Add scripts for requesting builds locally.
2020-07-29 12:39:49 +10:00
Oliver Chang bc7d478d4a
build: Project sync fixes. (#4194)
- Add some more logging.
- Use GitHub client ID/secret rather than personal access token.
- Fix function deploy wrt "--project" argument.
2020-07-27 13:19:44 +10:00
Max Moroz 70d278bb53
[infra] Rename msan-builder to msan-libs-builder (#3388). (#4190)
* [infra] Rename msan-builder to msan-libs-builder and add (broken) dfsan-libs-builder (#3388).

* remove dfsan-libs-builder for now

* presubmit format
2020-07-24 12:38:00 -07:00
kabeer27 9d2381a621
Initial integration of builds_status (#4175) 2020-07-23 16:23:31 +10:00
kabeer27 769c6e80f2
Adding build coverage cloud function, and refactoring test_utils.py (#4151)
* Adding build coverage cloud function, and refactoring test_utils.py

* Multiple changes will add a comment.

* Catching sys.exit() exception and removing useless wrapper build_steps function

* Comment update

* Comment update

* Comment formatting
2020-07-20 16:30:48 +10:00
kabeer27 f0d54c33cb
Refactoring and request build cloud function (#4120)
Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-16 11:11:18 +10:00
kabeer27 9413d10e08
[Sync cloud function] Cache file contents (#4104)
* Storing contents of project.yaml and dockerfile, to reduce api calls

* Fixing comment indentation

* Requested changes

* Minor changes

* Importing projectmetadata

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-13 14:21:56 +10:00
kabeer27 600c514958
Renaming schedule variable to builds_per_day for new feature (#4091)
* Renaming schedule variable to builds_per_day for new feature

* Minor formatting change

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 17:19:59 +10:00
kabeer27 0a712bb5d4
Base image builder (#4080)
* Base image builder

* Forgot to run formatter

* Fixing lint issues and removing unused imports

* Adding missing newline

* Refactoring code

* Fixing linting errors and changing name of cloud function entry

* Adding license header to build_base_images

* Changed logging and print_function

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 14:16:01 +10:00
kabeer27 277886dee9
Resetting emulators (#4079)
* Resetting datastore emulator

* Resetting datastore emulator

* setUp class

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 11:41:54 +10:00
kabeer27 e49a126000
Feature addition: Individual schedulers for projects (#4066)
This PR, integrates the new feature that allows users to specify their own schedules via project.yaml.
Parameter 'schedule : value' will dictate how many times a day should the project be built, i.e if value = 2
Then project will be built twice a day starting at 6 and interval 24/2 = 12 so 6:00 and 18:00
if parameter is unavailable DEFAULT_SCHEDULE = '0 6 * * *' i.e 6:00 will be used.

The cloud functions creates individual scheduler jobs for each project, with the name project_name + '-scheduler', and subscribes it to the topic 'request_build' with payload message 'project_name' which will be used by the next cloud function to request build for this project.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-06 17:52:24 +10:00
kabeer27 09c7ea778b
Added deploy file and restructured cloud function directory (#4057)
* Adding deploy.sh and restructuring directory

* Integrating review changes

* Passed project id as an argument to each command

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-01 16:13:08 +10:00
kabeer27 d15dc2a88f
Renaming infra/build/functions files (#4031)
Renamed main.py to sync.py
and test_sync.py to sync_test.py

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-29 11:10:47 +10:00
kabeer27 181812050d
Cloud function for syncing OSS-Fuzz projects (#3538). (#4023)
Integrating the first cloud function i implemented which syncs the project list from github and uploads the list to cloud datastore, which will be used by another cloud function to request builds.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-25 13:41:04 +10:00