Proposal: Quality Assurance workflow for BOINC

Motivation

To increase the quality and stability of all BOINC components it is useful to run automated tests and validation tools on a regular basis. This way one can quickly see if a specific change breaks another component/part of the system or introduces a possible security risk. Overall this should help to identify weak spots in the BOINC source code. Using codesniffers (like the PHP CodeSniffer) could help enforcing the BOINC coding guidelines. This has the side-effect of a syntax check and can also generate ToDo/Wish lists from comments in the code.

Requirements

This is a first shot and open for discussion:

automated system that runs every night
automatically run unit tests that are in VCS
automatically run third-party tools on portions of the codebase (e.g. cppcheck, doxygen)
compile a report of the automated runs (shown via web)
support multiple nodes to test on different operating systems
support multiple git branches (e.g. run the same tests on master as on client_release/7/7.2)
support testing multiple language on portions of the VCS (C++, PHP, Java)

Travis CI

Travis-CI is a continuous integration service that is free for Open Source projects:

tracks all commits and pull request made in the BOINC github mirror repository
installs needed software in a VM (Ubuntu 12.04) and tries to build the different BOINC components separately
failures are recognized (exit code != 0) and a notification is send to the boinc_cvs mailinglist
supports only Linux and Mac OS X at the moment

problems

It is difficult to handle the android build as it needs a complete different .travis.yml configuration file. A solution would be to have a separate branch 'android_build' with a modified .travis.yml file that only builds the android client. This would require to merge in the android specific changes by hand without overwriting the .travis.yml file.
Since the build is triggered on every commit or pull request it might not be ideal to generate documentation or reports within the VM

future changes

In order to enhance the usefulness of the automated build the following changes are possible:

add a simple syntax test for php Code
add a more elaborate test that creates a project inside the VM
explore if it's useful to add a Coverty scan (free but limited in builds per week)

On creating test cases

To create a test case one has to know what the function/class/component is supposed to do. Especially for white box tests this is crucial. Documenting the preconditions, postconditions and invariants of functions and classes is important. The documentation should be unified in regard to the used languages (C++, PHP, Java) and the used documentation tool (doxygen). It should be possible to distinguish between public and internal APIs.

Home