Using volunteers to test applications

Testing new apps and app versions has two phases

making sure they doesn't crash;
making sure they computes the correct results.

It's important to test new applications on a wide range of hosts, since bugs may appear only with particular OS versions, memory sizes, usage patterns, and so on. It's handy to use volunteers to do this testing, since they provide the needed diversity of hosts.

Beta-test project

One way to implement this is to create a separate test project, attach your own hosts to it, and get a limited number of volunteers to attach to it.

This is useful, but it has two limitations:

There is overhead in creating and maintaining a separate project.
The credit earned by volunteer testers goes to a different project.

BOINC provides two ways to do beta testing in the context of your existing project. You can let users volunteer run test versions, warning them in advance that these versions are more likely to crash. These "beta-test users" will get a mixture of regular and test results, and they'll get credit for both.

To use these mechanisms, include the line

$project_has_beta = true;

in your html/project/project_specific_prefs.inc file. This will add a 'Run test applications?' option to the project-specific preferences page.

Then publicize this on your web site and wait for some users to set their preferences to allow test apps. (Note: this flag is stored in XML in the project_prefs field of the user table; to find beta-test users, scan for <allow_beta_work>1</allow_beta_work>).

Beta-test applications

You can designate application as "beta", and only beta-test users will be sent jobs for these apps. To do this:

Create a new application using xadd or the admin web interface. Mark the app as "beta", e.g. by including <beta>1</beta> in the <app> element.
Add a validator and assimilator for the test application.
Create application versions for the test application.
Create jobs (as needed for testing) for the test application. To prevent test jobs from dominating regular work, either use the --allapps feeder option (and give the test app a small weight) or make a work generator for the test app that maintains only a small number of unsent results.

Note: for beta applications, the scheduler ignores which apps a user has allowed to run in their project specific preferences (the <app_id> element in the XML). If a project has several beta applications, a user can only either run all or none.

Beta-test app versions

You may want to verify that new app versions compute the same results as old ones, i.e. that your validator accepts combinations of old and new.

You can do this by designating app versions as "beta". These app versions will be sent only to beta-test users. For results processed using beta app versions, usually the other replicas will be processed using non-beta app versions; you can examine these workunits and check whether they validated.

When you create a new app version you can mark it as beta by including <beta/> in its version.xml file. You can toggle the beta flag for existing app versions using the admin web interface.

Home