Commit Graph

26 Commits

Author SHA1 Message Date
Oliver Chang 4a51ecb61f
build_specified_commit: Handle build.sh which are part of upstream repo. (#3932)
This is a best effort attempt to parse the relevant copy command from
the Dockerfile.

Also add a main function to build_specified_commit to make it easier to
test.
2020-06-10 17:12:28 +10:00
Oliver Chang 2f1d2ed756
Retry building project images in build_specified_commit. (#3915)
To mitigate transient network issues.
2020-06-01 17:16:15 +10:00
Oliver Chang 1778067c4b
build_specified_commit: Don't replace gitdir if already relative. (#3875)
Upgraded git broke this.
2020-05-26 16:08:59 +10:00
Oliver Chang 9ac361c149
build_specified_commit: revert to using cp. (#3871)
rsync seems to have disappeared in the latest images somehow, and we
can't rely on it always existing.
2020-05-25 15:49:37 +10:00
Oliver Chang 40c64add94
bisector: Increase rsync performance. (#3757) 2020-05-04 12:23:05 +10:00
Oliver Chang 174e03249a
bisector: Re-copy /src in between runs. (#3720)
Some builds won't work if there are remaining artifacts in /src.

Also fix a bug from last commit.
2020-04-28 14:26:08 +10:00
Oliver Chang 47068dff0a
Use base-builder close to commit date. (#3715) 2020-04-28 11:31:02 +10:00
Oliver Chang 06b6d9c8b3
Address comment from last commit. (#3714) 2020-04-27 11:30:22 +10:00
Oliver Chang e939fe3ced
bisector: Don't attempt to checkout old OSS-Fuzz revision if it does not exist. (#3707) 2020-04-24 15:37:22 +10:00
Oliver Chang 22d254a445
build_specified_commit: Make submodules work. (#3691)
Submodules weren't working correctly due to absolute paths being set.
We add a step to turn those paths to relative after copying them to
host.
2020-04-23 10:49:22 +10:00
Oliver Chang ed292c0b40
Bisector: Be a bit smarter about picking which OSS-Fuzz commit to build with. (#3665)
When the build fails against HEAD OSS-Fuzz, we find the date of the
commit for the project, and use the latest revision of OSS-Fuzz before
it to rebuild the project builder container.

Subsequent runs will use the last built container, and if that fails
that will again find the closest revision of OSS-Fuzz.

Also factor BaseRepoManager out of RepoManager to provide a generic repo
manager class for dealing with existing checkouts (which don't need a
clone).
2020-04-21 10:11:29 +10:00
Oliver Chang 3d6a6f546b
bisector: Don't clean when checking out commits. (#3652) 2020-04-15 16:17:13 +10:00
Oliver Chang 0f22065c16
bisector: raise exceptions when building new or old commit fails. (#3653)
Let build failures continue during the actual bisection for now to keep
the bisection going. A future PR can improve the logic of that when it
happens.
2020-04-15 16:03:52 +10:00
Oliver Chang d22704877b
bisector: Preserve file metadata when copying /src to host. (#3649)
Otherwise changed timestamps may cause build issues.
2020-04-15 12:24:10 +10:00
Oliver Chang aa045b9b37
Bisector fixes. (#3601)
- Copy /src from host instead of checking out repo on host. This fixes
  issues with dependencies in the main repo.
- Add some more logging.
2020-04-14 11:38:23 +10:00
Oliver Chang 81febde566
Log output when we fail to detect main repo. (#3599) 2020-04-09 12:38:46 +10:00
Leo Neat f0df2d675c
Move detect_repo.py script into /opt (#3337) 2020-02-05 11:32:37 -08:00
Leo Neat 53cb4f7935
[Infra] Update execute function in utils.py (#3319) 2020-02-04 11:51:18 -08:00
Leo Neat f2588ea5a8
Moving execute to utils.py (#3313) 2020-01-31 10:19:12 -08:00
Leo Neat 4d5e5016c0
[Infra] Update print statements (#3305) 2020-01-29 14:10:04 -08:00
Leo Neat 8ffc6db00c
[Infra] CIFuzz pipeline complete. (#3281)
* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working fuzzers with out error surface

* Working fuzzers with out error surface

* Working fuzzers with out error surface

* Printing std err

* Adding fuzzer timeout

* Adding fuzzer timeout

* Changing fuzzer timeout to fuzz time

* Formatting and refactoring

* Spelling in fuzz_target.py

* Spelling in fuzz_target.py

* Spelling in fuzz_target.py

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Refactoring error codes.

* reverting helper.py

* reverting helper.py

* reverting helper.py

* chaning method to static

* moving cifuzz file

* Jonathan changes

* Oliver and Jonathan comments

* Oliver and Jonathan comments

* Oliver and Jonathan comments

* Utils unit tests

* Test formatting and documentation

* Build fuzzer test added

* Changed repo manager errors

* Unit and integration tests complete

* Jonathan comments pt.1

* Jonathan comments pt.1

* Jonathan comments pt.1

* adding cifuzz_test

* Build fuzzer test completed

* Run fuzzers test finished.

* Removed SRC dependency

* Jonathan comments pt.2

* Max comments pt.1

* Max comments pt.2

* removing log specified out stream

* Max comments pt.3

* Adding OSS_FUZZ_HOME env var

* Jonathan comments pt.3

* Formatting

* Olivers comments

* Jonathan comments
2020-01-29 11:03:43 -08:00
jonathanmetzman 2bcc2bfe2b
[cifuzz][bisection] Make BuildData a namedtuple (#3282) 2020-01-23 09:48:09 -08:00
Leo Neat 14582175d0 [infra] Scripts for building fuzzers with CIFuzz (#3207) 2020-01-15 13:30:57 -08:00
Leo Neat 9532d4781a [infra] Fixing bisection to work with more OSS-Fuzz projects (#3152)
* Max comments

* Added bisection tests

* Added test cases

* Docker run command added to build_specific commit

* Infer main repo uses docker image rather than docker file

* Added mores tests for repo infer

* With verbosity

* Formatting

* Bisection fixed

* Formatting updates

* Oliver small comments.

* Script infer main repo

* Detect repo and test module finished

* Detect main repo from script

* Regex searching

* Regex searching

* Max comments + bisector test script

* Oliver comments

* Max's comments

* String concat update

* Jonathan comments
2020-01-06 12:17:26 -08:00
Leo Neat 2c05fb229f [infra] Add support for Bisecting Git Commits Introducing Crashes(#3119) 2019-12-16 16:48:49 -08:00
Leo Neat 2dd2a0fdcc [infra] Add functionality to check out a specific commit for a project (#3092)
* Starting the git python API

* repo manager working with tests

* Docker repo manager working, can update image with specific commit

* Working bisection

* Formatting and linting

* Working bisection

* Specific commit checkout

* Specific commit checkout

* Removed bisector and updated helper

* remove branch bug

* Max comments

* Added tests for infer main repo

* Oliver comments

* helper.py function names change

* Oliver comments pt. 2

* Olivers comments

* Formatting updates

* Removing DockerRepoManager class

* Removing DockerRepoManager class

* Changing from copying repo to docker image to mounting repo on docker image

* Jonathan comments

* Build image from commit tests

* Oliver comments pt.3

* Oliver comments pt.4

* Linting

* Max comments pt. 2

* change check commit exists

* Max comments pt. 3

* Regex updated

* Formatting updates
2019-12-12 09:26:37 -08:00