Testing BOINC

1) PHP-based testing framework

doc/test.php is a library of PHP classes that make it easy to write scripts that perform end-to-end tests of BOINC.

1.1) Goals of the testing framework

The goal of the framework is to support automated testing of BOINC itself, and of BOINC applications. Each test is performed by a PHP script that initializes the system to a deterministic state, executes the system, detects termination, and returns success or failure depending on the final state.

Many BOINC features involve multiple projects and/or multiple hosts. It would be prohibitively complex to do automated testing on multiple physical hosts. Fortunately the BOINC architecture makes it possible for multiple projects and (virtual) hosts to reside on a single physical host. So the testing framework uses this approach.

The framework lets you test systems with any of the following attributes:

The following system attributes are planned but not implemented yet:

1.2) Directory structure

To use the testing framework, you must designate a "BOINC projects" directory on your test host. The framework will create subdirectories as follows: 
boinc_projects/
    proj1/
        cgi/
        download/   /*this is where the real download directory*/
	download0/  /*these are optional, they are soft links to the
	              real download directory*/
	download1/
	...
        html_ops/
        html_user/
        keys/
        upload/
    proj2/
    ...
where proj1, proj2 etc. are the names of the projects in your scripts.

Similarly, you must designate a "BOINC hosts" directory on your test host. The framework will create subdirectories as follows: 

boinc_hosts/
    host1/
        projects/
        slots/
        client_state.xml
        log_flags.xml
    host2/
    ...
where host1, host2 etc. are the names of the hosts in your scripts.

1.3) Classes

The framework provides the following classes 
class Project {                 // represents a project
    var $name;                  // defaults to "test"; override if needed
    var $generate_keys;         // set to true if you want to generate
                                // encryption keys for this project (slow)
    function add_user($user);   // add a User to project's DB
    function add_app($app);     // add an application
    function add_app_version($app_version);     // add an app version
    function install(optional: $scheduler_file);         
        //set up directories and DB. If $scheduler_file is provided, then
        //there will be installed schedulers at the directories pointed by
        //the schedulers
    function start();           // start feeder
    function stop();            // stop feeder
    function compare_file($out, $correct);
                // verify that a result file matches a known-correct file
    function check_results($n, $result);
                // check that there are n results
                // and that they match "result"
                // (for all fields that are defined)

    // The functions below are used to interrupt or remove certain
    // functionalities of the project for a desired time in order to be
    // able to test persistence across these failures. All the
    // parameters of the functions below are optional.
    // $cgi_num,$download_dir_num,
    // $handler_num refer to the number of the corresponding parts of
    // the system  to be disabled temporarily. if they are not
    // the default cgi, download, and file_upload_handler respectively.
    // multiple cgis can be handles with passing in a file with
    // scheduler_urls to project->install. Multiple download URLS are
    // handled by adding a numer in between the download_url and the
    // file to be downloaded in the wu_template file of the Work class 
    // for example: 0/, in which
    // a link from project_dir/download0/ will be added to project_dir/download
    // Similiar action can be taken by modifying the result_template file
    // of a Work class to handle multiple upload URLs: 
    // for example: 0, in which case
    // file_upload_handler0 will be added to project_dir/cgi
 
    function delete_masterindex()
                //deletes the index.php for this project 
    function reestablish_masterindex()
                //reestablished the master index.php
    function delete_scheduler($cgi_num) 
                 //deletes project_dir/cgi/cgi$cgi$num
    function reinstall_scheduler($cgi_num)
                 //copies the cgi program into project_dir/cgi/cgi$cgi$num
    function delete_downloaddir($download_dir_number)
                 //removes project_dir/download$download_dir_num
    function reinstall_downloaddir($download_dir_num)
                 //reestablished project_dir/download$download_dir_num
    function remove_file_upload_handler($handler_num)
                 //deletes project_dir/cgi/file_upload_handler$handler_num	
    function reinstall_file_upload_handler($handler_num)	 
                 //reinstalls project_dir/cgi/file_upload_handler$handler_num
    function kill_file_upload_handler() //blocks until a file_upload_handler is 
                 //running in the system, kills it and returns
}

class User {                    // represents an account on a project
    var $name;                  // override if needed
    var $email_addr;
    var $authenticator;
}

class Host {                    // represents a (virtual) host
    function Host($user);
    function add_project($project);
    function install();

    //There are two run functions, one that blocks until the client
    //being run by this host finishes execution: run() and then there
    //run_asynch which spawns a new php process running the client 
    //and returns to the thread of execution of the parent php process
    //the php process_id of the child. This process_id can then be
    //used for a call to pcntl_waitpid() to block until the execution 
    //of the client.
    function run($args);
    function run_asynch($args);

    function get_new_client_pid($client_pid)
    //returns a pid for a client process running in the system that is
    //different from $client_pid. This call blocks until such process is started.
    
    function check_file_present($project, $name);
                                // check that a file exists
}

class App {                     // represents an application
    function App($name);
}

class App_Version {             // represents an application version
    var $version;               // defaults to 1
    var $exec_name;             // name of executable; default to app name
    function App_Version($app);
}

class Work {                    // represents a workunit and some results
    var $wu_template;           // name of workunit template file
    var $result_template;       // name of result template file
    var $nresults;              // number of results for this WU
    var $input_files;           // array of input file names
    function Work($project, $app);
    function install();         // install in DB and directories
}
A test script instantiates one or more instances of these classes, links them as needed, installs them, starts the projects, runs the hosts. It then checks the final state (typically, that results are done and result files are correct).

1.4) Environment vars

Before using the test framework, you must define the following environment variables (example values are given - you must supply your own): 
setenv BOINC_PROJECTS_DIR   /home/david/boinc_projects
# BOINC projects directory
setenv BOINC_HOSTS_DIR      /home/david/boinc_hosts
# BOINC hosts directory
setenv BOINC_USER_NAME      david
# part of DB name, and prepended to web error log entries
setenv BOINC_SRC_DIR        /home/david/boinc_cvs/boinc
# BOINC source directory
setenv BOINC_CGI_DIR        /home/david/cgi-bin
# path of a CGI directory
setenv BOINC_CGI_URL        http://localhost/cgi-bin
# URL of that directory
setenv BOINC_HTML_DIR       /home/david/html
# path of a web-page directory
setenv BOINC_HTML_URL       http://localhost
# URL of that directory
setenv BOINC_KEY_DIR        /home/david/boinc_keys
# path of some pre-generated security keys
setenv BOINC_PLATFORM       i686-pc-linux-gnu
# platform name of this machine
setenv BOINC_SHMEM_KEY      0x3abc1234
# shared-memory key; must be less than 2^31

1.5) Example script

The following script (test/test_uc.php) illustrates the use of the testing framework. 
#! /usr/local/bin/php
<?php
    include_once("test.inc");

    $project = new Project;
    $user = new User();
    $host = new Host($user);
    $app = new App("upper_case");
    $app_version = new App_Version($app);

    $project->add_user($user);
    $project->add_app($app);
    $project->add_app_version($app_version);
    $project->install();      // must install projects before adding to hosts

    $host->log_flags = "log_flags.xml";
    $host->add_project($project);
    $host->install();

    echo "adding work\n";

    $work = new Work($app);
    $work->wu_template = "uc_wu";
    $work->result_template = "uc_result";
    $work->nresults = 2;
    array_push($work->input_files, "input");
    $work->install($project);

    $project->start();
    $host->run("-exit_when_idle");
    $project->stop();

    $result->state = RESULT_STATE_DONE;
    $result->stderr_out = "APP: upper_case: starting, argc 1";
    $result->exit_status = 0;
    $project->check_results(2, $result);
    $project->compare_file("uc_wu_0_0", "uc_correct_output");
    $project->compare_file("uc_wu_1_0", "uc_correct_output");
?>

1.6) Implementation

Project->Install() does the following:

Host->install() does the following:

2) Test applications

The apps directory contains the following test applications:

3) Test scripts

The test directory contains PHP scripts, together with XML templates and sample input files, for initializing and testing the entire system: