Commit Graph

39 Commits

Author SHA1 Message Date
dependabot[bot] 54f0985f8e
Bump pyyaml from 5.2 to 5.4 in /infra/build/functions (#5512)
Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.2 to 5.4.
- [Release notes](https://github.com/yaml/pyyaml/releases)
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES)
- [Commits](https://github.com/yaml/pyyaml/compare/5.2...5.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-03-25 20:44:10 -07:00
Fabian Meumertzheim 6ca344a446
Generate badges for projects without coverage (#5459)
Generate badges also for projects with no coverage builds at all (e.g.
JVM and Python projects). For these projects, the badge only has the two
possible states "build passing" and "build failing".
2021-03-21 18:52:02 -07:00
Catena cyber c41e46ffc8
Rust coverage report (for Suricata) (#4697)
* Rust coverage test

* Workaround to get rust coverage for Suricata
2021-03-08 07:05:35 -08:00
jonathanmetzman 0ac841a1a4
Replace terms that are uninclusive. (#5045) 2021-01-25 08:41:34 -08:00
Oliver Chang 8fcb4659e3
Add a missing requirement for CI. (#5036) 2021-01-22 12:45:11 +11:00
jonathanmetzman 648242fdb4
[pylint] Enable no-member check (#5034) 2021-01-21 16:21:17 -08:00
Oliver Chang 5b51cfb00f
Fix build functions tests when invoked from unittest.TestSuite. (#5027) 2021-01-21 07:43:42 -08:00
Abhishek Arya 588336d665
s/test_all/test_all.py for google cloud build. (#4783)
Fixes https://github.com/google/oss-fuzz/issues/4781
2020-12-04 08:26:34 -08:00
Oliver Chang 5da57c9128
Fix requirements.txt for GCB functions. (#4670) 2020-11-20 12:07:09 +11:00
Max Moroz b97f6e296a
[infra] Allow coverage for Go project on GCB (#2817, #2714). (#4668) 2020-11-19 16:07:35 -08:00
Oliver Chang 8b10d4b705
Fix incorrect OSS-Fuzz build project deletion. (#4623) 2020-11-10 11:10:43 +11:00
Oliver Chang 0987ddf994
even more fixes. (#4297) 2020-08-11 18:27:18 +10:00
Oliver Chang ade417ba20
Fix typo in project_sync.py (#4296)
* Fix typo in project_sync.py

* line length

* format
2020-08-11 16:25:08 +10:00
Oliver Chang 65a83e2125
Split up builds status function. (#4261)
- Split into separate invocations for fuzzing, coverage, and badges.
- Also use threads to speed things up.
2020-08-06 13:59:08 +10:00
Oliver Chang 909db51a15
More build infra cleanup. (#4252)
- Deleted unused code.
- Consolidate some modules.
2020-08-06 09:36:47 +10:00
Oliver Chang 3f1d43115f
Clean up build infra. (#4251)
- Remove gcb folder. Move the files that we still need into the
  build/functions dir.
- Remove badge_images. They are now stored on GCS.
- Remove jenins configs.
2020-08-05 11:20:48 +10:00
kabeer27 21cd8de8dc
Skipping fuzzing build when project is disabled (#4239)
* Skipping fuzzing build when project is disabled

* Removing skip build and making it better
2020-07-31 17:20:09 +10:00
Oliver Chang 47b57ca1dd
Fix a typo from previous MSan function addition. (#4241) 2020-07-31 16:58:28 +10:00
kabeer27 8e4c7b92fd
Adding msan builder to gcp (#4234)
* Adding msan builder to gcp

* Formatting changes

* Refactoring and reducing redundancy

* Moving msan builder entry point into base_images
2020-07-31 14:42:09 +10:00
Oliver Chang 5e3348cdb9
Create individual schedulers for requesting coverage builds. (#4237)
The single function for requesting all coverage builds was timing out.
2020-07-31 10:56:18 +10:00
kabeer27 341b3d836a
Display historical logs ui change (#4197)
* Initial UI Change to display historical logs

* Minor changes

* More formatting changes

* Adding license header

* Changes, take a look Oliver

* Changes for handling empty build history/ no last successful build

* Reverting the logs folder path from /logs/log to /log

* Merged to latest master and other chnages to template

* Changing backend to reflect build history

* Fixing order of query

* Removed last build status

* More changes

* Adding unit tests and fixes found by unit testing

* Fixing lint errors

* Added more unit tests and fixed typos and formatting

* Fixing comment caps

* Minor formatting issue

Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
2020-07-30 16:20:22 +10:00
Oliver Chang ccb5821f51
Fix build status update. (#4224)
Check for valid statuses rather than only excluding the WORKING status.
There are other failure statuses that need to be accounted for (e.g.
EXPIRED).
2020-07-30 15:10:52 +10:00
Oliver Chang 689f3500a2
Add missing dash to build tag. (#4221) 2020-07-30 08:31:01 +10:00
Oliver Chang ac4af2436f
build: Add queueTtl argument to one more place. (#4219) 2020-07-30 08:21:49 +10:00
kabeer27 5893c61c77
Fixing missing dash in datastore query (#4213) 2020-07-29 17:32:54 +10:00
Oliver Chang 4e2a7c07e5
Many build infra fixes. (#4209)
- Remove base image project argument from deploy.sh. Deploy all
  functions to the main oss-fuzz image.
- Reduce max instances of functions to 1 to avoid rate limiting issues.
- Fix missing ndb context initialization in request_build.py
- Fix incorrect tags in BuildHistory entities. "-" was doubled.
- Fix base build deployment with incorrect schedule variable.
- Add scripts for requesting builds locally.
2020-07-29 12:39:49 +10:00
Oliver Chang bc7d478d4a
build: Project sync fixes. (#4194)
- Add some more logging.
- Use GitHub client ID/secret rather than personal access token.
- Fix function deploy wrt "--project" argument.
2020-07-27 13:19:44 +10:00
Max Moroz 70d278bb53
[infra] Rename msan-builder to msan-libs-builder (#3388). (#4190)
* [infra] Rename msan-builder to msan-libs-builder and add (broken) dfsan-libs-builder (#3388).

* remove dfsan-libs-builder for now

* presubmit format
2020-07-24 12:38:00 -07:00
kabeer27 9d2381a621
Initial integration of builds_status (#4175) 2020-07-23 16:23:31 +10:00
kabeer27 769c6e80f2
Adding build coverage cloud function, and refactoring test_utils.py (#4151)
* Adding build coverage cloud function, and refactoring test_utils.py

* Multiple changes will add a comment.

* Catching sys.exit() exception and removing useless wrapper build_steps function

* Comment update

* Comment update

* Comment formatting
2020-07-20 16:30:48 +10:00
kabeer27 f0d54c33cb
Refactoring and request build cloud function (#4120)
Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-16 11:11:18 +10:00
kabeer27 9413d10e08
[Sync cloud function] Cache file contents (#4104)
* Storing contents of project.yaml and dockerfile, to reduce api calls

* Fixing comment indentation

* Requested changes

* Minor changes

* Importing projectmetadata

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-13 14:21:56 +10:00
kabeer27 600c514958
Renaming schedule variable to builds_per_day for new feature (#4091)
* Renaming schedule variable to builds_per_day for new feature

* Minor formatting change

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 17:19:59 +10:00
kabeer27 0a712bb5d4
Base image builder (#4080)
* Base image builder

* Forgot to run formatter

* Fixing lint issues and removing unused imports

* Adding missing newline

* Refactoring code

* Fixing linting errors and changing name of cloud function entry

* Adding license header to build_base_images

* Changed logging and print_function

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 14:16:01 +10:00
kabeer27 277886dee9
Resetting emulators (#4079)
* Resetting datastore emulator

* Resetting datastore emulator

* setUp class

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-08 11:41:54 +10:00
kabeer27 e49a126000
Feature addition: Individual schedulers for projects (#4066)
This PR, integrates the new feature that allows users to specify their own schedules via project.yaml.
Parameter 'schedule : value' will dictate how many times a day should the project be built, i.e if value = 2
Then project will be built twice a day starting at 6 and interval 24/2 = 12 so 6:00 and 18:00
if parameter is unavailable DEFAULT_SCHEDULE = '0 6 * * *' i.e 6:00 will be used.

The cloud functions creates individual scheduler jobs for each project, with the name project_name + '-scheduler', and subscribes it to the topic 'request_build' with payload message 'project_name' which will be used by the next cloud function to request build for this project.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-06 17:52:24 +10:00
kabeer27 09c7ea778b
Added deploy file and restructured cloud function directory (#4057)
* Adding deploy.sh and restructuring directory

* Integrating review changes

* Passed project id as an argument to each command

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-07-01 16:13:08 +10:00
kabeer27 d15dc2a88f
Renaming infra/build/functions files (#4031)
Renamed main.py to sync.py
and test_sync.py to sync_test.py

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-29 11:10:47 +10:00
kabeer27 181812050d
Cloud function for syncing OSS-Fuzz projects (#3538). (#4023)
Integrating the first cloud function i implemented which syncs the project list from github and uploads the list to cloud datastore, which will be used by another cloud function to request builds.

Co-authored-by: Kabeer Seth <kabeerseth@google.com>
2020-06-25 13:41:04 +10:00