Commit Graph

35 Commits

Author SHA1 Message Date
Fabian Meumertzheim cbc4293c37
[infra] Make mount location configurable in build_fuzzers (#5744)
Exposing the mount location for local sources to a command-line
argument makes it possible to reproduce findings with local source
even for umbrella projects with WORKDIR $SRC/.
2021-05-18 09:15:14 -07:00
jonathanmetzman ffe4f892b1
[helper] Fix build_image --pull and refactor (#5642)
Fix behavior of build_image --pull (Fixes #5640)

Also refactor helper.py:

1. Change behavior of functions so that most return True on success and False on failure.
2. Only main will return 1 on failure and 0 on success now. Previous behavior was very error prone.
3. Rename _get_output_dir to _get_out_dir.
4. Make function docstrings use descriptive tense.
5. Make helper.py print help when no argument is specified.
2021-04-21 14:28:26 +00:00
Oliver Chang 1ee46825bf
build_specified_commit: Add call to fetch_all_remotes on cleanup. (#5485)
We need to redo fetch_all_remotes as we re-copy the repo from the docker
container.
2021-03-24 08:59:33 +11:00
Oliver Chang 6e41513dbb
build_specified_commit: Remove an exception. (#5009)
Return None rather than exceptioning out when a suitable base-builder
cannot be found to allow more graceful error handling.
2021-01-20 16:52:49 +11:00
Oliver Chang 0612668e8c
build_specified_commit: Return failure rather than raise exception when rebuild fails. (#4964) 2021-01-14 15:49:15 +11:00
Oliver Chang a23d9bc6fb
Rename BaseRepoManager -> RepoManager. (#4800)
This was renamed in b0b99d5ccd
2020-12-08 10:01:38 +11:00
jonathanmetzman b0b99d5ccd
Cifuzz external build (#4656)
* Support building fuzzers for projects outside of OSS-Fuzz
* Use retry wrapper
* Fix some tests.
2020-12-07 10:50:11 -08:00
Oliver Chang 8bd567697d
build_specified_commit: fallback to oldest integration commit. (#4779)
When we fail a build, we attempt to identify the first OSS-Fuzz commit
prior to the upstream commit date.

If this does not exist, we bailed out. This commit changes it such that
we at least try on the oldest integration commit.
2020-12-04 17:51:31 +11:00
jonathanmetzman ec5491853d
[infra] Add retry decorator and use it. (#4702) 2020-11-24 09:51:56 -08:00
Oliver Chang 4a51ecb61f
build_specified_commit: Handle build.sh which are part of upstream repo. (#3932)
This is a best effort attempt to parse the relevant copy command from
the Dockerfile.

Also add a main function to build_specified_commit to make it easier to
test.
2020-06-10 17:12:28 +10:00
Oliver Chang 2f1d2ed756
Retry building project images in build_specified_commit. (#3915)
To mitigate transient network issues.
2020-06-01 17:16:15 +10:00
Oliver Chang 1778067c4b
build_specified_commit: Don't replace gitdir if already relative. (#3875)
Upgraded git broke this.
2020-05-26 16:08:59 +10:00
Oliver Chang 9ac361c149
build_specified_commit: revert to using cp. (#3871)
rsync seems to have disappeared in the latest images somehow, and we
can't rely on it always existing.
2020-05-25 15:49:37 +10:00
Oliver Chang 40c64add94
bisector: Increase rsync performance. (#3757) 2020-05-04 12:23:05 +10:00
Oliver Chang 174e03249a
bisector: Re-copy /src in between runs. (#3720)
Some builds won't work if there are remaining artifacts in /src.

Also fix a bug from last commit.
2020-04-28 14:26:08 +10:00
Oliver Chang 47068dff0a
Use base-builder close to commit date. (#3715) 2020-04-28 11:31:02 +10:00
Oliver Chang 06b6d9c8b3
Address comment from last commit. (#3714) 2020-04-27 11:30:22 +10:00
Oliver Chang e939fe3ced
bisector: Don't attempt to checkout old OSS-Fuzz revision if it does not exist. (#3707) 2020-04-24 15:37:22 +10:00
Oliver Chang 22d254a445
build_specified_commit: Make submodules work. (#3691)
Submodules weren't working correctly due to absolute paths being set.
We add a step to turn those paths to relative after copying them to
host.
2020-04-23 10:49:22 +10:00
Oliver Chang ed292c0b40
Bisector: Be a bit smarter about picking which OSS-Fuzz commit to build with. (#3665)
When the build fails against HEAD OSS-Fuzz, we find the date of the
commit for the project, and use the latest revision of OSS-Fuzz before
it to rebuild the project builder container.

Subsequent runs will use the last built container, and if that fails
that will again find the closest revision of OSS-Fuzz.

Also factor BaseRepoManager out of RepoManager to provide a generic repo
manager class for dealing with existing checkouts (which don't need a
clone).
2020-04-21 10:11:29 +10:00
Oliver Chang 3d6a6f546b
bisector: Don't clean when checking out commits. (#3652) 2020-04-15 16:17:13 +10:00
Oliver Chang 0f22065c16
bisector: raise exceptions when building new or old commit fails. (#3653)
Let build failures continue during the actual bisection for now to keep
the bisection going. A future PR can improve the logic of that when it
happens.
2020-04-15 16:03:52 +10:00
Oliver Chang d22704877b
bisector: Preserve file metadata when copying /src to host. (#3649)
Otherwise changed timestamps may cause build issues.
2020-04-15 12:24:10 +10:00
Oliver Chang aa045b9b37
Bisector fixes. (#3601)
- Copy /src from host instead of checking out repo on host. This fixes
  issues with dependencies in the main repo.
- Add some more logging.
2020-04-14 11:38:23 +10:00
Oliver Chang 81febde566
Log output when we fail to detect main repo. (#3599) 2020-04-09 12:38:46 +10:00
Leo Neat f0df2d675c
Move detect_repo.py script into /opt (#3337) 2020-02-05 11:32:37 -08:00
Leo Neat 53cb4f7935
[Infra] Update execute function in utils.py (#3319) 2020-02-04 11:51:18 -08:00
Leo Neat f2588ea5a8
Moving execute to utils.py (#3313) 2020-01-31 10:19:12 -08:00
Leo Neat 4d5e5016c0
[Infra] Update print statements (#3305) 2020-01-29 14:10:04 -08:00
Leo Neat 8ffc6db00c
[Infra] CIFuzz pipeline complete. (#3281)
* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Testing action build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working build

* Working fuzzers with out error surface

* Working fuzzers with out error surface

* Working fuzzers with out error surface

* Printing std err

* Adding fuzzer timeout

* Adding fuzzer timeout

* Changing fuzzer timeout to fuzz time

* Formatting and refactoring

* Spelling in fuzz_target.py

* Spelling in fuzz_target.py

* Spelling in fuzz_target.py

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Upload artifact fix

* Refactoring error codes.

* reverting helper.py

* reverting helper.py

* reverting helper.py

* chaning method to static

* moving cifuzz file

* Jonathan changes

* Oliver and Jonathan comments

* Oliver and Jonathan comments

* Oliver and Jonathan comments

* Utils unit tests

* Test formatting and documentation

* Build fuzzer test added

* Changed repo manager errors

* Unit and integration tests complete

* Jonathan comments pt.1

* Jonathan comments pt.1

* Jonathan comments pt.1

* adding cifuzz_test

* Build fuzzer test completed

* Run fuzzers test finished.

* Removed SRC dependency

* Jonathan comments pt.2

* Max comments pt.1

* Max comments pt.2

* removing log specified out stream

* Max comments pt.3

* Adding OSS_FUZZ_HOME env var

* Jonathan comments pt.3

* Formatting

* Olivers comments

* Jonathan comments
2020-01-29 11:03:43 -08:00
jonathanmetzman 2bcc2bfe2b
[cifuzz][bisection] Make BuildData a namedtuple (#3282) 2020-01-23 09:48:09 -08:00
Leo Neat 14582175d0 [infra] Scripts for building fuzzers with CIFuzz (#3207) 2020-01-15 13:30:57 -08:00
Leo Neat 9532d4781a [infra] Fixing bisection to work with more OSS-Fuzz projects (#3152)
* Max comments

* Added bisection tests

* Added test cases

* Docker run command added to build_specific commit

* Infer main repo uses docker image rather than docker file

* Added mores tests for repo infer

* With verbosity

* Formatting

* Bisection fixed

* Formatting updates

* Oliver small comments.

* Script infer main repo

* Detect repo and test module finished

* Detect main repo from script

* Regex searching

* Regex searching

* Max comments + bisector test script

* Oliver comments

* Max's comments

* String concat update

* Jonathan comments
2020-01-06 12:17:26 -08:00
Leo Neat 2c05fb229f [infra] Add support for Bisecting Git Commits Introducing Crashes(#3119) 2019-12-16 16:48:49 -08:00
Leo Neat 2dd2a0fdcc [infra] Add functionality to check out a specific commit for a project (#3092)
* Starting the git python API

* repo manager working with tests

* Docker repo manager working, can update image with specific commit

* Working bisection

* Formatting and linting

* Working bisection

* Specific commit checkout

* Specific commit checkout

* Removed bisector and updated helper

* remove branch bug

* Max comments

* Added tests for infer main repo

* Oliver comments

* helper.py function names change

* Oliver comments pt. 2

* Olivers comments

* Formatting updates

* Removing DockerRepoManager class

* Removing DockerRepoManager class

* Changing from copying repo to docker image to mounting repo on docker image

* Jonathan comments

* Build image from commit tests

* Oliver comments pt.3

* Oliver comments pt.4

* Linting

* Max comments pt. 2

* change check commit exists

* Max comments pt. 3

* Regex updated

* Formatting updates
2019-12-12 09:26:37 -08:00