Commit Graph

984 Commits

Author SHA1 Message Date
Leo Neat 6f02735d0c
[CIFuzz] Dynamic fuzzer scheduling (#3453)
Allocates time for fuzzers based on amount of fuzzing time left. This is beneficial when a fuzzer fails early in its allotted time because of a reproducible OSS-Fuzz bug. It will then give the remaining portion of its allotted time to the following fuzzers.
2020-03-13 10:35:33 -07:00
Leo Neat b6a1d4dcb1
Revert "Revert "[CIFuzz] Affected fuzzers (#3450)" (#3488)" (#3496)
This reverts commit e58ee49e20.
2020-03-12 13:51:33 -07:00
Leo Neat b0cd13e3bb
Affected fuzzer fix (#3494)
Fixing json decode error with affected fuzzer functionality.
2020-03-12 12:01:57 -07:00
Max Moroz 130710d518
[infra] Change language attribute from "cpp" to "c++" in build scripts. (#3493)
* [infra] Change language attribute from "cpp" to "c++" in coverage job runner.

Should've done this in https://github.com/google/oss-fuzz/pull/3477. Good news is that nothing failed, the coverage job was just skipped for pretty much all projects.

* also remove default value from build_project script
2020-03-12 09:06:46 -07:00
Jakub Jelen 3c248cdc57
[infra] helper: Add possibility to pass source directory to gdb/shell commands to ease local debugging (#3465)
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
2020-03-10 16:49:10 -07:00
Leo Neat e58ee49e20
Revert "[CIFuzz] Affected fuzzers (#3450)" (#3488)
This reverts commit e8ebda10ee.
2020-03-10 15:59:43 -07:00
Leo Neat e8ebda10ee
[CIFuzz] Affected fuzzers (#3450)
Add feature to only run fuzzers that are affected by the code change. This change prevents valuable time being wasted on fuzzing unchanged code.
2020-03-10 13:22:11 -07:00
Max Moroz 71f4914c45
[presubmit] Enforce language attribute in project.yaml to be always set. (#3477)
* [presubmit] Enforce language attribute in projectt.yaml to be always set.

* Update documentation, better presubmit check, new project template.

* add docstring to templates.py

* Add example values in the project.yaml template and remove python value for now

* Add "project: c++" to 256 projects

* format

* Add labels and selective_unpack sections to the presubmit check

* fix incorrect auto_ccs format in three projects

* fix nss emails after rebase
2020-03-10 11:08:01 -07:00
Leo Neat 5869bb7341
[CIFuzz] Download code coverage (#3434)
This is the foundation for affected fuzzers. It provides the ability to map fuzzers to files. In the future we will use this functionality to pick which fuzzers to run during CI.
2020-03-05 11:22:44 -08:00
Leo Neat ffa49091d6
[infra] Presubmit fix - test (#3443)
Fixes issue that presubmit test was not returning failure even when a test failed.
2020-03-05 08:44:18 -08:00
Leo Neat 7fb97cb3d6
[CIFuzz] Timeout orphan process fix (#3462)
This fix adds the -max_total_time argument to the cifuzz run_fuzzers command. The bug that was occurring was the docker process was being killed but the fuzzing was continuing wasting valuable compute resources. So a project with a large number of fuzzers would have all of them running by the end of CIFUzz leading to no actual fuzzing getting done because of resource scarcity. This patch should fix that.
2020-03-04 14:58:09 -08:00
Catena cyber 4231987eee
Adds locally installed includes for coverage build (#3448) 2020-03-02 07:39:12 -08:00
Leo Neat 8d905b08c2
[CIFuzz] Adding dry_run mode to check_build (#3444)
Prevents crash from surfacing when dry run mode is enabled and check fuzzers fails.
2020-02-28 09:41:44 -08:00
Leo Neat f798890b66
[Infra] Add git diff to repo manager class (#3427)
This adds the functionality of git diff --name-only to the repo manager module. This functionality will be used for the affected fuzzer feature of CIFuzz. Associated tests have also been added.
2020-02-28 08:30:42 -08:00
Leo Neat f8a4ad1a8f
[CIFuzz] Reproduce bug fix (#3442)
This patch should fix the reproduce bug that has been occurring. It should distinguish between novel and old bugs better.
2020-02-27 15:33:00 -08:00
Leo Neat 12a2e86d46
[CIFuzz] Check fuzzers (#3419)
Checks that the fuzzer build was a success in the build action.
2020-02-27 08:54:52 -08:00
Leo Neat 2aa7a0d558
[CIFuzz] Use backup corpus for fuzzing (#3380)
This allows fuzzing using fuzz target's backup corpora. It will make fuzzing more efficient for targets that have a backup OSS-Fuzz corpus avaliable to the public. It will not effect targets that don't have backups.
2020-02-26 08:47:13 -08:00
Leo Neat f16f391a18
Docs fix (#3416)
Small change to example file to reflect the current action state.
2020-02-21 10:48:22 -08:00
Leo Neat 1522a7428c
[Infra] Add tests to presubmit (#3405)
Adds unit/integration tests to travis presubmit.
2020-02-21 08:47:13 -08:00
Max Moroz dd5ad3b20e
[infra] Loose the limits for dataflow traces collection (#1632). (#3411)
* [infra] Loose the limits for dataflow traces collection (#1632).

* format
2020-02-19 16:40:23 -08:00
Leo Neat 39fe0d725b
[CIFuzz] Check crash on most recent OSS-Fuzz build (#3385)
Adds functionality to check if a crash exists in the most recent OSS-Fuzz build. This is necessary to determine if a crash was introduced in the current pull request or existed in the project already.  Crashes that are surfaced to the user will be both reproducible and novel to the OSS-Fuzz project.
2020-02-19 15:32:30 -08:00
Max Moroz 6cd7db7758
[infra] GSB: ignore failing "gsutil rm -rf" step. (#3412) 2020-02-19 10:43:21 -08:00
Leo Neat e661c8c4a8
[CIFuzz] log bug fix (#3407)
* Fixing log bug

* Fixing log bug
2020-02-18 15:45:31 -08:00
Max Moroz a0ade7d8e5
[infra] Perform "gsutil rm -rf" if we need to overwrite coverage data. (#3404)
* [infra] Perform "gsutil rm -rf" if we need to overwrite coverage data.

* format
2020-02-18 09:25:16 -08:00
Max Moroz 6e8232078a
[infra] Use wildcard in gsutil cp command as it lacks -T option. (#3396) 2020-02-14 13:23:20 -08:00
Max Moroz dfaf951f35
[infra] Upload latest.version metadata file for most recent build. (#3379)
* [infra] Upload latest.version metadata file for most recent build.

* move common functionality into build_lib

* pass content_type to get_signed_url

* format
2020-02-13 07:31:12 -08:00
Leo Neat 9f52d142aa
[CIFuzz] Only report reproducible crashes (#3376)
* Tests for Reproduce

* Leo comments

* Maxs comments pt.2

* Olivers comments

* Olivers comments

* Add fuzz target module tests

* Formatting

* Small punct and spelling

* Test update

* Format
2020-02-12 14:44:11 -08:00
Leo Neat 0ed0f37ca2
[CIFuzz] Change 'bug_report' directory to 'artifacts'. (#3369) 2020-02-11 08:42:10 -08:00
Leo Neat ffc6af6d1d
[CIFuzz] Add parse fuzzer output functionality (#3342) 2020-02-10 10:00:54 -08:00
Philip Withnall b036df0ce1
glib: Fix build due to Meson option changing type (#3358)
* glib: Fix build due to Meson option changing type

In GLib master commit b220033c we changed the `libmount` option from a
`boolean` to a `feature`, which means it now takes
`enabled`/`disabled`/`auto` rather than `true`/`false`.

Signed-off-by: Philip Withnall <philip@tecnocode.co.uk>

Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=20552

* glib: Fix homepage URI

The FSF page on GLib is not official, loaded with their advertising, and
quite out of date.

Signed-off-by: Philip Withnall <philip@tecnocode.co.uk>

* infra: Add help_url to the valid section names list in Travis checks

It’s documented as valid:
https://google.github.io/oss-fuzz/getting-started/new-project-guide/#help_url

Signed-off-by: Philip Withnall <philip@tecnocode.co.uk>
2020-02-07 05:03:31 -08:00
Max Moroz a32769e89a
[infra] Dump full coverage JSONs separately (#1632). (#3356) 2020-02-06 17:45:57 -08:00
Max Moroz 4092bebcb5
[infra] Fix broken batching in the corpora download helper (#3351). (#3353)
* [infra] Fix broken batching in the corpora download helper (#3351).

* one more
2020-02-06 15:44:36 -08:00
Leo Neat b96d2186ae
[CIFuzz] Go support (#3352)
* Add go support to detecting the main repo

* Go support test update

* Format

* Format

* Removing debug statment

* Format
2020-02-06 15:35:42 -08:00
Leo Neat 460661c700
Fixing entrypoint (#3354) 2020-02-06 14:00:55 -08:00
Leo Neat 5cefaac000
[CIFuzz] Separate build and run actions (#3336)
* Separating actions into build and run

* Formatting

* Formatting

* Removing project-name from run action

* Jonathan comments

* Maxs comments pt.1

* Updating example_main.yml

* Switch 1 0 to true false

* Add base Dockerfile for CIFuzz

* Adding license to dockerfile
2020-02-06 13:39:42 -08:00
Max Moroz 69f75431fe [infra] Dummy change in the base-builder to test something. 2020-02-06 11:11:24 -08:00
Max Moroz 67353e882d [cifuzz] Dummy change to test something. 2020-02-06 10:47:18 -08:00
Leo Neat 52e640af34
[CIFuzz] Fix .strip bug. (#3349) 2020-02-06 09:33:54 -08:00
Leo Neat f0df2d675c
Move detect_repo.py script into /opt (#3337) 2020-02-05 11:32:37 -08:00
Max Moroz c57465770a
[infra] Travis: rely on the presence of Dockerfile to identify buildable projects. (#3328)
* [infra] Travis: rely on the presese of Dockerfile to identify buildable projects.

* fix the comment
2020-02-05 09:14:59 -08:00
jonathanmetzman 27409e0fbc
[infra] Make travis_build presubmit clean (#3334) 2020-02-05 09:10:05 -08:00
Max Moroz 8d3d290c61
[infra] Generate full coverage JSONs for dataflow enabled targets (#1632). (#3325) 2020-02-04 17:42:25 -08:00
Leo Neat 53cb4f7935
[Infra] Update execute function in utils.py (#3319) 2020-02-04 11:51:18 -08:00
Leo Neat da04f5e431
[CIFuzz] Add seed and len control to fuzzer arguments (#3321) 2020-02-04 10:54:28 -08:00
Leo Neat 83d951bd1d
[CIFuzz] Prevent action from failing option (#3315)
* Adding opt to workflow to prevent failures

* fclose bug

* Undoing double negitive

* Undoing double negitive

* Undoing double negitive

* Syntax and formatting

* Olivers comments

* Presubmit fix

* Trying boolean argument

* cleaning

* cleaning fuzz disassembles

* Maxs comments pt.2

* Updating dry_run parse

* Comments added

* Formatting

* Olivers comments
2020-02-03 15:35:04 -08:00
Leo Neat 1a87da68c8
[cifuzz] - commit and pull request hook functionality (#3310) 2020-01-31 15:31:18 -08:00
Leo Neat f2588ea5a8
Moving execute to utils.py (#3313) 2020-01-31 10:19:12 -08:00
Max Moroz 7751ab5a87
[infra] Introduce "language" attribute in the project.yaml (#3297). (#3299)
* [infra] Introduce "language" attribute in the project.yaml (#3297).

* follow up

* enable the attribute for more projects

* trailing newline
2020-01-30 15:36:44 -08:00
Leo Neat 007226e5cc
[infra] unit/integration test updates (#3300)
* Merging build_specified_commit.py and utils.py

* undo dataflow change

* Updating RepoManager tests

* Updated detect_repo_tests

* Build specified commit updated

* Updated test_repos

* updating bisector tests

* formatting updates

* Switch inversions

* Rebase master
2020-01-30 10:27:56 -08:00
Leo Neat 4d5e5016c0
[Infra] Update print statements (#3305) 2020-01-29 14:10:04 -08:00