Testing BOINC

1) PHP-based testing framework

doc/test.php is a library of PHP classes that make it easy to write "test scripts" that perform end-to-end tests of BOINC.

1.1) Goals of the testing framework

The goal of the framework is to support automated testing of BOINC itself, and of BOINC applications. Each test is performed by a PHP script that initializes the system to a deterministic state, executes the system, detects termination, and returns success or failure depending on the final state.

Many BOINC features involve multiple projects and/or multiple hosts. It would be prohibitively complex to do automated testing on multiple physical hosts. Fortunately the BOINC architecture makes it possible for multiple projects and (virtual) hosts to reside on a single physical host. So the testing framework uses this approach.

The framework lets you write scripts with any of the following attributes:

Multiple projects
Multiple applications per project
Multiple participants per project
Multiple hosts, with arbitrary enrollment in projects

1.2) Directory structure

Each project has various files that must be kept separate. You must designate a "BOINC projects" directory on your test host. The framework will create subdirectories as follows:

boinc_projects/
    proj1/
        cgi/
        download/
        html_ops/
        html_user/
        keys/
        upload/
    proj2/
    ...

where proj1, proj2 etc. are the names of the projects in your scripts.

Similarly, each virtual host has various files that must be kept separate. You must designate a "BOINC hosts" directory on your test host. The framework will create subdirectories as follows:

boinc_hosts/
    host1/
        projects/
        slots/
        client_state.xml
        log_flags.xml
    host2/
    ...

where host1, host2 etc. are the names of the hosts in your scripts.

1.3) Classes

The framework provides the following classes

class Project {                 // represents a project
    var $name;                  // defaults to "test"; override if needed
    function add_user($user);   // add a User to project's DB
    function add_app($app);     // add an application
    function add_app_version($app_version);     // add an app version
    function install();         // set up directories and DB
    function start();           // start feeder
    function stop();            // stop feeder
    function check_results_done();  // verify that all results are in state DONE
    function compare_file($out, $correct);  // verify that a result file
                                // matches a known-correct file
}

class User {                    // represents an account on a project
    var $name;                  // override if needed
    var $email_addr;
    var $authenticator;
}

class Host {                    // represents a (virtual) host
    function Host($user);
    function add_project($project);
    function install();
    function run($flags);
}

class App {                     // represents an application
    function App($name);
}

class App_Version {             // represents an application version
    var $version;               // defaults to 1
    var $exec_name;             // name of executable; default to app name
    function App_Version($app);
}

class Work {                    // represents a workunit and some results
    var $wu_template;           // name of workunit template file
    var $result_template;       // name of result template file
    var $nresults;              // number of results for this WU
    var $input_files;           // array of input file names
    function Work($project, $app);
    function install();         // install in DB and directories
}

A test script instantiates one or more instances of these classes, links them as needed, installs them, starts the projects, runs the hosts. It then checks the final state (typically, that results are done and result files are correct).

1.4) Environment vars

Before using the test framework, you must define the following environment variables (example values are given - you must supply your own):

setenv BOINC_PROJECTS_DIR   /home/david/boinc_projects
# BOINC projects directory
setenv BOINC_HOSTS_DIR      /home/david/boinc_hosts
# BOINC hosts directory
setenv BOINC_USER_NAME      david
# prepended to web error log entries
setenv BOINC_SRC_DIR        /home/david/boinc_cvs/boinc
# BOINC source directory
setenv BOINC_CGI_DIR        /home/david/cgi-bin
# path of a CGI directory
setenv BOINC_CGI_URL        http://localhost/cgi-bin
# URL of that directory
setenv BOINC_HTML_DIR       /home/david/html
# path of a web-page directory
setenv BOINC_HTML_URL       http://localhost
# URL of that directory
setenv BOINC_KEY_DIR        /home/david/boinc_keys
# path of some pre-generated security keys
setenv BOINC_PLATFORM       i686-pc-linux-gnu
# platform name of this machine

1.5) Example script

The following script (test/test_uc.php) illustrates the use of the testing framework.

#! /usr/local/bin/php
<?php
    include_once("test.inc");

    $project = new Project;
    $user = new User();
    $host = new Host($user);
    $app = new App("upper_case");
    $app_version = new App_Version($app);

    $project->add_user($user);
    $project->add_app($app);
    $project->add_app_version($app_version);
    $project->install();      // must install projects before adding to hosts

    $host->add_project($project);
    $host->install();

    echo "adding work\n";

    $work = new Work($project, $app);
    $work->wu_template = "uc_wu";
    $work->result_template = "uc_result";
    $work->nresults = 2;
    array_push($work->input_files, "input");
    $work->install();

    $project->start();
    $host->run("-exit_when_idle");
    $project->stop();

    $project->check_results_done();
    $project->compare_file("uc_wu_0_0", "uc_correct_output");
    $project->compare_file("uc_wu_1_0", "uc_correct_output");
?>

1.6) Implementation

2) Test applications

The apps directory contains the following test applications:

concat: concatenates its input files. Input and output filenames are passed on the command line.
upper_case: reads from stdin, converts to upper case, writes to stdout.
uc_slow: like upper_case, but processes only one character per second. Checkpoints every 5 characters. Restartable.

3) Test scripts

The test directory contains PHP scripts, together with XML templates and sample input files, for initializing and testing the entire system:

test_uc.php: tests I/O connection using descriptors.
test_concat.php: tests I/O connection using command-line args and filenames.
test_uc_slow.php: tests checkpoint/restart. You have to run the client yourself, kill and restart it a few times.
test_prefs.php: tests some aspects of preferences.
test_api.php: tests to ensure the api is working properly.
test_water.php: tests some aspects of water marks.
test_rsc.php: tests that scheduling server only sends feasible work units.

These scripts use functions defined in the PHP include file init.inc. You can use these functions to easily write test scripts for your own applications.