Testing BOINC

1) PHP-based testing framework

doc/test.php is a library of PHP classes that make it easy to write scripts that perform end-to-end tests of BOINC.

1.1) Goals of the testing framework

The goal of the framework is to support automated testing of BOINC itself, and of BOINC applications. Each test is performed by a PHP script that initializes the system to a deterministic state, executes the system, detects termination, and returns success or failure depending on the final state.

Many BOINC features involve multiple projects and/or multiple hosts. It would be prohibitively complex to do automated testing on multiple physical hosts. Fortunately the BOINC architecture makes it possible for multiple projects and (virtual) hosts to reside on a single physical host. So the testing framework uses this approach.

The framework lets you test systems with any of the following attributes:

The following system attributes are planned but not implemented yet:

1.2) Directory structure

To use the testing framework, you must designate a "BOINC projects" directory on your test host. The framework will create subdirectories as follows:
boinc_projects/
    proj1/
        cgi/
        download/
        html_ops/
        html_user/
        keys/
        upload/
    proj2/
    ...
where proj1, proj2 etc. are the names of the projects in your scripts.

Similarly, you must designate a "BOINC hosts" directory on your test host. The framework will create subdirectories as follows:

boinc_hosts/
    host1/
        projects/
        slots/
        client_state.xml
        log_flags.xml
    host2/
    ...
where host1, host2 etc. are the names of the hosts in your scripts.

1.3) Classes

The framework provides the following classes
class Project {                 // represents a project
    var $name;                  // defaults to "test"; override if needed
    var $generate_keys;         // set to true if you want to generate
                                // encryption keys for this project (slow)
    function add_user($user);   // add a User to project's DB
    function add_app($app);     // add an application
    function add_app_version($app_version);     // add an app version
    function install();         // set up directories and DB
    function start();           // start feeder
    function stop();            // stop feeder
    function compare_file($out, $correct);  // verify that a result file
                                // matches a known-correct file
    function check_results($n, $result);    // check that there are n results
                                // and that they match "result"
                                // (for all fields that are defined)
}

class User {                    // represents an account on a project
    var $name;                  // override if needed
    var $email_addr;
    var $authenticator;
}

class Host {                    // represents a (virtual) host
    function Host($user);
    function add_project($project);
    function install();
    function run($flags);
    function check_file_present($project, $name);
                                // check that a file exists
}

class App {                     // represents an application
    function App($name);
}

class App_Version {             // represents an application version
    var $version;               // defaults to 1
    var $exec_name;             // name of executable; default to app name
    function App_Version($app);
}

class Work {                    // represents a workunit and some results
    var $wu_template;           // name of workunit template file
    var $result_template;       // name of result template file
    var $nresults;              // number of results for this WU
    var $input_files;           // array of input file names
    function Work($project, $app);
    function install();         // install in DB and directories
}
A test script instantiates one or more instances of these classes, links them as needed, installs them, starts the projects, runs the hosts. It then checks the final state (typically, that results are done and result files are correct).

1.4) Environment vars

Before using the test framework, you must define the following environment variables (example values are given - you must supply your own):
setenv BOINC_PROJECTS_DIR   /home/david/boinc_projects
# BOINC projects directory
setenv BOINC_HOSTS_DIR      /home/david/boinc_hosts
# BOINC hosts directory
setenv BOINC_USER_NAME      david
# part of DB name, and prepended to web error log entries
setenv BOINC_SRC_DIR        /home/david/boinc_cvs/boinc
# BOINC source directory
setenv BOINC_CGI_DIR        /home/david/cgi-bin
# path of a CGI directory
setenv BOINC_CGI_URL        http://localhost/cgi-bin
# URL of that directory
setenv BOINC_HTML_DIR       /home/david/html
# path of a web-page directory
setenv BOINC_HTML_URL       http://localhost
# URL of that directory
setenv BOINC_KEY_DIR        /home/david/boinc_keys
# path of some pre-generated security keys
setenv BOINC_PLATFORM       i686-pc-linux-gnu
# platform name of this machine
setenv BOINC_SHMEM_KEY      0x3abc1234
# shared-memory key; must be less than 2^31

1.5) Example script

The following script (test/test_uc.php) illustrates the use of the testing framework.
#! /usr/local/bin/php
<?php
    include_once("test.inc");

    $project = new Project;
    $user = new User();
    $host = new Host($user);
    $app = new App("upper_case");
    $app_version = new App_Version($app);

    $project->add_user($user);
    $project->add_app($app);
    $project->add_app_version($app_version);
    $project->install();      // must install projects before adding to hosts

    $host->log_flags = "log_flags.xml";
    $host->add_project($project);
    $host->install();

    echo "adding work\n";

    $work = new Work($app);
    $work->wu_template = "uc_wu";
    $work->result_template = "uc_result";
    $work->nresults = 2;
    array_push($work->input_files, "input");
    $work->install($project);

    $project->start();
    $host->run("-exit_when_idle");
    $project->stop();

    $result->state = RESULT_STATE_DONE;
    $result->stderr_out = "APP: upper_case: starting, argc 1";
    $result->exit_status = 0;
    $project->check_results(2, $result);
    $project->compare_file("uc_wu_0_0", "uc_correct_output");
    $project->compare_file("uc_wu_1_0", "uc_correct_output");
?>

1.6) Implementation

Project->Install() does the following:

Host->install() does the following:

2) Test applications

The apps directory contains the following test applications:

3) Test scripts

The test directory contains PHP scripts, together with XML templates and sample input files, for initializing and testing the entire system: