Rooba
/
boinc
mirror of https://github.com/BOINC/boinc.git
1
0
Fork 0
Code Issues 1 Packages Projects Releases Wiki Activity

multi-file program support 

svn path=/trunk/boinc/; revision=46
This commit is contained in:
David Anderson 2002-05-17 22:33:57 +00:00
parent ff1e199372
commit 79b17d7e0d
37 changed files with 867 additions and 459 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

163
TODO
View File

@ -1,54 +1,41 @@
- test environment-var mechanism
- design alpha/beta/prod mechanism for core and apps
Should have flag in host record
(maybe separate flags for core/app).
Project should control which hosts are alpha/beta.
Test.
- include shared memory allocation with key data values (current running projects, est. time, etc) to allow easier development of third party addons (Eric)
HIGH-PRIORITY (must be done to support SETI@home)
- test versioning mechanisms for core
message, deprecation
send messages if either
1) some WU couldn't be sent because core is too old
2) no WUs could be sent because core is too old
- test incompatible-core mechanism for apps
same as above?
- test reporting of app errors
- test sticky files
- test checksumming of executables
- test on a multiprocessor
- mechanism for returning app stderr output to server? store in blob?
- define logging system on client
- proxy support
- put core client version in client state file
- add country, postal code to user
collect on login
- implement scheme for distributing scheduling server addresses
each project provides a URL for file with list of address,
each project provides a URL for file with list of scheduler address,
and email address of "problems" contact
- server-side estimates of WU time requirements
- DB entry for WU has int/FP/RAM/disk/net
- implement and test hi/lo water mark scheme
- core client must estimate how much work it has left,
based on %done (if available) and time estimates
- server sends only feasible WUs
- limit by RAM, disk
- test
- CPU accounting in the presence of checkpoint/restart
core client periodically gets CPU time, accumulates in state file
- review and complete documentation
- scalable/efficient server architecture: alternatives
use our own process framework, shared memory for result/WU lists
use fast CGI w/ shared mem
- define a notion of "app preferences": arbitrary XML edited on server,
passed to app client.
This could be used to e.g. select a color scheme for graphics
Could pass entire prefs?
- measure hardware parameters: CPU speed, #CPUs, memory, disk
- define CPU benchmarks
- do this for other platforms
- Get info periodically rather than always at startup
- core client / app trigger file interface
core->app
@ -86,12 +73,66 @@
capable of offscreen drawing.
OpenGL version?
- CPU accounting in the presence of checkpoint/restart
core client periodically gets CPU time, accumulates in state file
- GUI and screensaver around core client
design default (core client) display
screensaver options
move around screen?
system tray icon behavior
- hi/lo water mark scheme
- core client must estimate how much work it has left,
based on %done (if available) and time estimates
- get idle-only behavior without screensaver
- Network retry policies
can't download file: when to give up? how to retry?
exponential backoff
can't upload file: when to give up/retry?
can't connect to sched server
error return from sched server
- sched server should return total credit info (user, team?)
could display in default core client display
- edit user account
- team system
--------------------------
MEDIUM-PRIORITY (must be done for CM)
- review and complete documentation
- test sticky files
- test multiple-file applications
change add.C to support multi-file applications
- implement WU/result sequences
--------------------------
LOW-PRIORITY
- implement checkpoint/restart for file transfers
use features of HTTP 1.1
Return an error to server if transfer fails
(store in DB in server)
- implement and test the batch mechanism
- test environment-var mechanism
- test alpha/beta/prod for apps
- test on a multiprocessor
- define logging system on client
- server-side estimates of WU time requirements
- DB entry for WU has int/FP/RAM/disk/net
- server sends only feasible WUs
- limit by RAM, disk
- test
- implement file upload/download requests
- preferences
finish PHP web interface
@ -107,52 +148,24 @@
implement in client
control over priorities?
- scalable/efficient server architecture: alternatives
use our own process framework, shared memory for result/WU lists
use fast CGI w/ shared mem
- add size to FILE_INFO
server, client sides
- implement checkpoint/restart for file transfers
use features of HTTP 1.1
- measure bandwidth on each network xfer
maintain exponential average, weighted by #bytes xferred
- measure hardware parameters: CPU speed, #CPUs, memory, disk
- define CPU benchmarks
- do this for other platforms
- Get info periodically rather than always at startup
- define a notion of "app preferences": arbitrary XML edited on server,
passed to app client.
This could be used to e.g. select a color scheme for graphics
Could pass entire prefs?
- NET_XFER_SET:: do_select: prevent one stream from starving others
- GUI and screensaver around core client
design default (core client) display
screensaver options
move around screen?
system tray icon behavior
- get idle-only behavior without screensaver
- Network retry policies
can't download file: when to give up? how to retry?
exponential backoff
can't upload file: when to give up/retry?
can't connect to sched server
error return from sched server
- implement WU/result sequences
- add the ability to store input data directly in WU,
and output directly in result (blob).
For projects with small data.
- implement file upload/download requests
--------------------------
DONE (may need test)
- sched server should return total credit info (user, team?)
could display in default core client display
- mechanism for returning app stderr output to server? store in blob?
- edit user account
- add size to FILE_INFO
server, client sides
- team system
- measure bandwidth on each network xfer
maintain exponential average, weighted by #bytes xferred

91
checkin_notes
View File

@ -9,50 +9,49 @@ David A. April 11 2002
- Pass in platform name, version number from Makefile
(platform name comes from configure)
David A. April 12 2002
- Think through application-version logic. Here's how it works:
On the server side, WUs don't have a version number,
either in the DB or in the XML doc.
When the server sends a WU,
it precedes it with an <app> tag containing a version number.
This instructs the client to process all WUs included in
the reply using that app version.
On the client side, WU records DO have a version number
(obtained as above).
- Change the "add" program so that command-line arguments are named
(e.g., "-app_name" foo instead of just "foo")
Modify test scripts accordingly.
- Have the scheduler read static tables (platform, app, app_version)
into memory at the beginning (new class: DB_CACHE)
This is a first step towards making the server efficient
(also need to convert it to fast CGI)
- Have the scheduler use files of the form
/tmp/boinc_req_PID and /tmp/boinc_reply_PID
rather than fixed-named files
- Have the scheduler send back error message when it can't open files
client
Makefile.in
client_state.C
cs_scheduler.C
hostinfo_unix.C
types.C,h
db
db.h
db_mysql.C
html_user
db.inc
sched
handle_request.C,h
main.C
server_types.C,h
test
*wu
init.inc
tools
add.C
David A. April 15, 2002
Change name from BOINC to BONC :-(
David A. May 15 2002
- Allow applications to have multiple files
(e.g. scripts, libraries, pre- and post-processing programs).
The app_version table now has an xml_doc field,
which contains a list of <file_info> and <file_ref> elements.
Note: the utility for adding applications (tools/add.C)
still handles only single-file applications.
- Changed IO_FILE_DESC to FILE_REF since it now represents
an association of a file to workunit, result or application
David A. April 25, 2002
Change name from BONC back to BOINC :-)
client
app.C
client_state.C,h
cs_apps.C
cs_scheduler.C
main.C
message.h (new)
parse.C,h
scheduler_reply.C,h
types.C,h
Makefile.in
db
db.h
db_mysql.C
schema.sql
doc
app.html (new)
files.html
index.html
intro.html
project.gif
project.html
project.png
protocol.html
result.html (new)
work.html
html_user
db.inc
sched
handle_request.C
server_types.C,h
test
test_uc.php
init.inc
tools
add.C

7
client/Makefile.in
View File

@ -1,10 +1,10 @@
VERSION = 1
CFLAGS = -g -Wall @DEFS@ \
-DHOSTTYPE=\"@host@\" \
-DHOST=\"@host@\" \
-DVERSION=$(VERSION)
CC = g++ $(CFLAGS) -I ../lib
CC = @CC@ $(CFLAGS) -I ../lib
CLIBS = @LIBS@
@ -35,7 +35,6 @@ OBJS = \
time_stats.o \
types.o \
util.o \
version.o \
../lib/md5_file.o \
../lib/md5.o
@ -74,7 +73,7 @@ tar:
tar cf boinc_client.tar *.C *.h Makefile.in
clean:
rm -f *.o $(PROGS) core dependencies config.cache
rm -f *.o $(PROGS) core dependencies Makefile config.cache
dependencies: *.C
$(CC) -M *.C > dependencies

58
client/app.C
View File

@ -24,7 +24,6 @@
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <fcntl.h>
#include <ctype.h>
@ -91,7 +90,8 @@ int ACTIVE_TASK::start(bool first_time) {
char exec_name[256], file_path[256], link_path[256];
char* argv[100];
unsigned int i;
IO_FILE_DESC ifd;
FILE_REF file_ref;
FILE_INFO* fip;
int fd, retval;
@ -104,28 +104,34 @@ int ACTIVE_TASK::start(bool first_time) {
// knows there was a problem.
// make a link to the executable
get_pathname(app_version->file_info, file_path);
strcpy(exec_name, app_version->file_info->name);
if (first_time) {
sprintf(link_path, "%s/%s", dirname, exec_name);
retval = link(file_path, link_path);
if (log_flags.task_debug) {
printf("link %s to %s\n", file_path, link_path);
//
for (i=0; i<app_version->app_files.size(); i++) {
fip = app_version->app_files[i].file_info;
get_pathname(fip, file_path);
if (i == 0) {
strcpy(exec_name, fip->name);
}
if (retval) {
perror("link");
exit(retval);
if (first_time) {
sprintf(link_path, "%s/%s", dirname, fip->name);
retval = link(file_path, link_path);
if (log_flags.task_debug) {
printf("link %s to %s\n", file_path, link_path);
}
if (retval) {
perror("link");
exit(retval);
}
}
}
// create symbolic links, and hook up descriptors, for input files
//
for (i=0; i<wup->input_files.size(); i++) {
ifd = wup->input_files[i];
get_pathname(ifd.file_info, file_path);
if (strlen(ifd.open_name)) {
file_ref = wup->input_files[i];
get_pathname(file_ref.file_info, file_path);
if (strlen(file_ref.open_name)) {
if (first_time) {
sprintf(link_path, "%s/%s", dirname, ifd.open_name);
sprintf(link_path, "%s/%s", dirname, file_ref.open_name);
if (log_flags.task_debug) {
printf("link %s to %s\n", file_path, link_path);
}
@ -137,10 +143,10 @@ int ACTIVE_TASK::start(bool first_time) {
}
} else {
fd = open(file_path, O_RDONLY);
if (fd != ifd.fd) {
retval = dup2(fd, ifd.fd);
if (fd != file_ref.fd) {
retval = dup2(fd, file_ref.fd);
if (retval < 0) {
fprintf(stderr, "dup2 %d %d returned %d\n", fd, ifd.fd, retval);
fprintf(stderr, "dup2 %d %d returned %d\n", fd, file_ref.fd, retval);
exit(retval);
}
close(fd);
@ -151,12 +157,12 @@ int ACTIVE_TASK::start(bool first_time) {
// hook up the output files
//
for (i=0; i<result->output_files.size(); i++) {
ifd = result->output_files[i];
get_pathname(ifd.file_info, file_path);
if (strlen(ifd.open_name)) {
file_ref = result->output_files[i];
get_pathname(file_ref.file_info, file_path);
if (strlen(file_ref.open_name)) {
if (first_time) {
creat(file_path, 0660);
sprintf(link_path, "%s/%s", dirname, ifd.open_name);
sprintf(link_path, "%s/%s", dirname, file_ref.open_name);
if (log_flags.task_debug) {
printf("link %s to %s\n", file_path, link_path);
}
@ -168,10 +174,10 @@ int ACTIVE_TASK::start(bool first_time) {
}
} else {
fd = open(file_path, O_WRONLY|O_CREAT, 0660);
if (fd != ifd.fd) {
retval = dup2(fd, ifd.fd);
if (fd != file_ref.fd) {
retval = dup2(fd, file_ref.fd);
if (retval < 0) {
fprintf(stderr, "dup2 %d %d returned %d\n", fd, ifd.fd, retval);
fprintf(stderr, "dup2 %d %d returned %d\n", fd, file_ref.fd, retval);
exit(retval);
}
close(fd);

55
client/client_state.C
View File

@ -37,6 +37,8 @@ CLIENT_STATE::CLIENT_STATE() {
exit_when_idle = false;
contacted_sched_server = false;
activities_suspended = false;
version = VERSION;
platform_name = HOST;
}
int CLIENT_STATE::init(ACCOUNTS& accounts) {
@ -139,7 +141,7 @@ int CLIENT_STATE::parse_state_file() {
if (!f) {
if (log_flags.state_debug) {
printf("can't open state file: %s\n", STATE_FILE_NAME);
printf("No state file; will create one\n");
}
return ERR_FOPEN;
}
@ -187,6 +189,11 @@ int CLIENT_STATE::parse_state_file() {
net_stats.parse(f);
} else if (match_tag(buf, "<active_task_set>")) {
active_tasks.parse(f, this);
} else if (match_tag(buf, "<platform_name>")) {
// should match out current platform name
} else if (match_tag(buf, "<version>")) {
// could put logic here to detect incompatible state files
// after core client update
} else {
fprintf(stderr, "CLIENT_STATE::parse_state_file: unrecognized: %s\n", buf);
}
@ -246,6 +253,12 @@ int CLIENT_STATE::write_state_file() {
}
}
active_tasks.write(f);
fprintf(f,
"<platform_name>%s</platform_name>\n"
"<version>%d</version>\n",
platform_name,
version
);
fprintf(f, "</client_state>\n");
fclose(f);
retval = rename(STATE_FILE_TEMP, STATE_FILE_NAME);
@ -307,7 +320,7 @@ FILE_INFO* CLIENT_STATE::lookup_file_info(PROJECT* p, char* name) {
}
// functions to create links between state objects
// (which, in their XML form, refer to one another by name)
// (which, in their XML form, reference one another by name)
//
int CLIENT_STATE::link_app(PROJECT* p, APP* app) {
app->project = p;
@ -322,6 +335,10 @@ int CLIENT_STATE::link_file_info(PROJECT* p, FILE_INFO* fip) {
int CLIENT_STATE::link_app_version(PROJECT* p, APP_VERSION* avp) {
APP* app;
FILE_INFO* fip;
FILE_REF file_ref;
unsigned int i;
avp->project = p;
app = lookup_app(p, avp->app_name);
if (!app) {
@ -330,30 +347,34 @@ int CLIENT_STATE::link_app_version(PROJECT* p, APP_VERSION* avp) {
);
return 1;
}
fip = lookup_file_info(p, avp->file_name);
if (!fip) {
fprintf(stderr,
"app_version refers to nonexistent file: %s\n", avp->file_name
);
return 1;
}
avp->project = p;
avp->app = app;
avp->file_info = fip;
for (i=0; i<avp->app_files.size(); i++) {
file_ref = avp->app_files[i];
fip = lookup_file_info(p, file_ref.file_name);
if (!fip) {
fprintf(stderr,
"app_version refers to nonexistent file: %s\n",
file_ref.file_name
);
return 1;
}
avp->app_files[i].file_info = fip;
}
return 0;
}
int CLIENT_STATE::link_io_file_desc(PROJECT* p, IO_FILE_DESC* ifd) {
int CLIENT_STATE::link_file_ref(PROJECT* p, FILE_REF* file_refp) {
FILE_INFO* fip;
fip = lookup_file_info(p, ifd->file_name);
fip = lookup_file_info(p, file_refp->file_name);
if (!fip) {
fprintf(stderr,
"I/O desc links to nonexistent file: %s\n", ifd->file_name
"I/O desc links to nonexistent file: %s\n", file_refp->file_name
);
return 1;
}
ifd->file_info = fip;
file_refp->file_info = fip;
return 0;
}
@ -382,7 +403,7 @@ int CLIENT_STATE::link_workunit(PROJECT* p, WORKUNIT* wup) {
wup->app = app;
wup->avp = avp;
for (i=0; i<wup->input_files.size(); i++) {
retval = link_io_file_desc(p, &wup->input_files[i]);
retval = link_file_ref(p, &wup->input_files[i]);
if (retval) return retval;
}
return 0;
@ -401,7 +422,7 @@ int CLIENT_STATE::link_result(PROJECT* p, RESULT* rp) {
rp->wup = wup;
rp->app = wup->app;
for (i=0; i<rp->output_files.size(); i++) {
retval = link_io_file_desc(p, &rp->output_files[i]);
retval = link_file_ref(p, &rp->output_files[i]);
if (retval) return retval;
}
return 0;

4
client/client_state.h
View File

@ -47,6 +47,8 @@ private:
vector<WORKUNIT*> workunits;
vector<RESULT*> results;
int version;
char* platform_name;
HTTP_OP scheduler_op;
PROJECT* scheduler_op_project;
NET_XFER_SET* net_xfers;
@ -68,7 +70,7 @@ private:
int write_state_file_if_needed();
int link_app(PROJECT*, APP*);
int link_file_info(PROJECT*, FILE_INFO*);
int link_io_file_desc(PROJECT*, IO_FILE_DESC*);
int link_file_ref(PROJECT*, FILE_REF*);
int link_app_version(PROJECT*, APP_VERSION*);
int link_workunit(PROJECT*, WORKUNIT*);
int link_result(PROJECT*, RESULT*);

8
client/cs_apps.C
View File

@ -82,8 +82,14 @@ bool CLIENT_STATE::input_files_available(RESULT* rp) {
WORKUNIT* wup = rp->wup;
FILE_INFO* fip;
unsigned int i;
APP_VERSION* avp;
avp = wup->avp;
for (i=0; i<avp->app_files.size(); i++) {
fip = avp->app_files[i].file_info;
if (!fip->file_present) return false;
}
if (!wup->avp->file_info->file_present) return false;
for (i=0; i<wup->input_files.size(); i++) {
fip = wup->input_files[i].file_info;
if (!fip->file_present) return false;

10
client/cs_scheduler.C
View File

@ -25,8 +25,8 @@
#include "file_names.h"
#include "parse.h"
#include "log_flags.h"
#include "message.h"
#include "scheduler_reply.h"
#include "version.h"
#include "client_state.h"
@ -104,8 +104,8 @@ int CLIENT_STATE::make_scheduler_request(PROJECT* p, int work_req) {
p->authenticator,
p->hostid,
p->rpc_seqno,
gplatform_name,
gversion,
platform_name,
version,
work_req
);
@ -211,6 +211,10 @@ void CLIENT_STATE::handle_scheduler_reply(PROJECT* project) {
f = fopen(SCHED_OP_RESULT_FILE, "r");
retval = sr.parse(f);
if (strlen(sr.message)) {
show_message(sr.message, sr.message_priority);
}
// copy new entities to client state
//

10
client/main.C
View File

@ -17,6 +17,8 @@
// Contributor(s):
//
// command-line version of the BOINC core client
#include <unistd.h>
#include "accounts.h"
@ -24,6 +26,14 @@
#include "log_flags.h"
#include "client_state.h"
void show_message(char* message, char* priority) {
if (!strcmp(priority, "high")) {
fprintf(stderr, "BOINC core client: %s\n", message);
} else {
printf("BOINC core client: %s\n", message);
}
}
int main(int argc, char** argv) {
CLIENT_STATE cs;
ACCOUNTS accounts;

4
client/message.h Normal file
View File

@ -0,0 +1,4 @@
// show a message.
// Implemented in different ways in cmdline client versus GUI client
//
extern void show_message(char* message, char* priority);

24
client/parse.C
View File

@ -17,6 +17,13 @@
// Contributor(s):
//
// A very crude interface for parsing XML files;
// assumes all elements are either single-line or
// have start and end tags on separate lines.
// This is meant to be used ONLY for parsing XML files produced
// by the BOINC scheduling server or client.
// Could replace this with a more general parser.
#include <string.h>
#include <stdlib.h>
@ -44,12 +51,27 @@ bool parse_double(char* buf, char* tag, double& x) {
bool parse_str(char* buf, char* tag, char* x) {
char* p = strstr(buf, tag);
if (!p) return false;
p = strchr(p, '>');
char* q = strchr(p+1, '<');
*q = 0;
strcpy(x, p+strlen(tag));
strcpy(x, p+1);
return true;
}
void parse_attr(char* buf, char* name, char* out) {
char* p, *q;
strcpy(out, "");
p = strstr(buf, name);
if (!p) return;
p = strchr(p, '"');
if (!p) return;
q = strchr(p+1, '"');
if (!q) return;
*q = 0;
strcpy(out, p+1);
}
void copy_stream(FILE* in, FILE* out) {
char buf[1024];
int n, m;

1
client/parse.h
View File

@ -23,5 +23,6 @@ extern bool parse(char*, char*);
extern bool parse_int(char*, char*, int&);
extern bool parse_double(char*, char*, double&);
extern bool parse_str(char*, char*, char*);
extern void parse_attr(char* buf, char* attrname, char* out);
extern bool match_tag(char*, char*);
extern void copy_stream(FILE* in, FILE* out);

6
client/scheduler_reply.C
View File

@ -27,7 +27,8 @@
int SCHEDULER_REPLY::parse(FILE* in) {
char buf[256];
strcpy(error_message, "");
strcpy(message, "");
strcpy(message_priority, "");
request_delay = 0;
hostid = 0;
@ -70,7 +71,8 @@ int SCHEDULER_REPLY::parse(FILE* in) {
RESULT result;
result.parse(in, "</result_ack>");
result_acks.push_back(result);
} else if (parse_str(buf, "<error_message>", error_message)) {
} else if (parse_str(buf, "<message", message)) {
parse_attr(buf, "priority", message_priority);
continue;
} else {
fprintf(stderr, "SCHEDULER_REPLY::parse: unrecognized %s\n", buf);

3
client/scheduler_reply.h
View File

@ -23,7 +23,8 @@
struct SCHEDULER_REPLY {
int hostid;
int request_delay;
char error_message[256];
char message[1024];
char message_priority[256];
PREFS prefs;
vector<APP> apps;
vector<FILE_INFO> file_infos;

51
client/types.C
View File

@ -176,17 +176,23 @@ int FILE_INFO::delete_file() {
int APP_VERSION::parse(FILE* in) {
char buf[256];
FILE_REF file_ref;
strcpy(app_name, "");
strcpy(file_name, "");
//strcpy(file_name, "");
version_num = 0;
app = NULL;
project = NULL;
file_info = NULL;
//file_info = NULL;
while (fgets(buf, 256, in)) {
if (match_tag(buf, "</app_version>")) return 0;
else if (parse_str(buf, "<app_name>", app_name)) continue;
else if (parse_str(buf, "<file_name>", file_name)) continue;
//else if (parse_str(buf, "<file_name>", file_name)) continue;
else if (match_tag(buf, "<file_ref>")) {
file_ref.parse(in, "</file_ref>");
app_files.push_back(file_ref);
continue;
}
else if (parse_int(buf, "<version_num>", version_num)) continue;
else fprintf(stderr, "APP_VERSION::parse(): unrecognized: %s\n", buf);
}
@ -194,34 +200,44 @@ int APP_VERSION::parse(FILE* in) {
}
int APP_VERSION::write(FILE* out) {
unsigned int i;
fprintf(out,
"<app_version>\n"
" <app_name>%s</app_name>\n"
" <file_name>%s</file_name>\n"
" <version_num>%d</version_num>\n"
"</app_version>\n",
app_name, file_name, version_num
//" <file_name>%s</file_name>\n"
" <version_num>%d</version_num>\n",
app_name,
//file_name,
version_num
);
for (i=0; i<app_files.size(); i++) {
app_files[i].write(out, "file_assoc");
}
fprintf(out,
"</app_version>\n"
);
return 0;
}
int IO_FILE_DESC::parse(FILE* in, char* end_tag) {
int FILE_REF::parse(FILE* in, char* end_tag) {
char buf[256];
strcpy(file_name, "");
strcpy(open_name, "");
fd = -1;
main_program = false;
while (fgets(buf, 256, in)) {
if (match_tag(buf, end_tag)) return 0;
else if (parse_str(buf, "<file_name>", file_name)) continue;
else if (parse_str(buf, "<open_name>", open_name)) continue;
else if (parse_int(buf, "<fd>", fd)) continue;
else fprintf(stderr, "IO_FILE_DESC::parse(): unrecognized: %s\n", buf);
else if (match_tag(buf, "<main_program/>")) main_program = true;
else fprintf(stderr, "FILE_REF::parse(): unrecognized: %s\n", buf);
}
return 1;
}
int IO_FILE_DESC::write(FILE* out, char* tag) {
int FILE_REF::write(FILE* out, char* tag) {
fprintf(out, " <%s>\n", tag);
if (strlen(open_name)) {
fprintf(out, " <open_name>%s</open_name>\n", open_name);
@ -233,13 +249,16 @@ int IO_FILE_DESC::write(FILE* out, char* tag) {
if (fd >= 0) {
fprintf(out, " <fd>%d</fd>\n", fd);
}
if (main_program) {
fprintf(out, " <main_program/>\n");
}
fprintf(out, " </%s>\n", tag);
return 0;
}
int WORKUNIT::parse(FILE* in) {
char buf[256];
IO_FILE_DESC ifd;
FILE_REF file_ref;
strcpy(name, "");
strcpy(app_name, "");
@ -255,8 +274,8 @@ int WORKUNIT::parse(FILE* in) {
else if (parse_str(buf, "<command_line>", command_line)) continue;
else if (parse_str(buf, "<env_vars>", env_vars)) continue;
else if (match_tag(buf, "<input_file>")) {
ifd.parse(in, "</input_file>");
input_files.push_back(ifd);
file_ref.parse(in, "</input_file>");
input_files.push_back(file_ref);
continue;
}
else fprintf(stderr, "WORKUNIT::parse(): unrecognized: %s\n", buf);
@ -284,7 +303,7 @@ int WORKUNIT::write(FILE* out) {
int RESULT::parse(FILE* in, char* end_tag) {
char buf[256];
IO_FILE_DESC ifd;
FILE_REF file_ref;
strcpy(name, "");
strcpy(wu_name, "");
@ -302,8 +321,8 @@ int RESULT::parse(FILE* in, char* end_tag) {
else if (parse_str(buf, "<name>", name)) continue;
else if (parse_str(buf, "<wu_name>", wu_name)) continue;
else if (match_tag(buf, "<output_file>")) {
ifd.parse(in, "</output_file>");
output_files.push_back(ifd);
file_ref.parse(in, "</output_file>");
output_files.push_back(file_ref);
continue;
}
else if (parse_double(buf, "<cpu_time>", cpu_time)) continue;

37
client/types.h
View File

@ -85,33 +85,36 @@ struct FILE_INFO {
int delete_file(); // attempt to delete the underlying file
};
struct APP_VERSION {
char app_name[256];
char file_name[256];
int version_num;
APP* app;
PROJECT* project;
FILE_INFO* file_info;
int parse(FILE*);
int write(FILE*);
};
// describes a connection between a file and an application.
// Either the app will use open() or fopen() to access the file
// Describes a connection between a file and a workunit, result, or application.
// In the first two cases,
// the app will either use open() or fopen() to access the file
// (in which case "open_name" is the name it will use)
// or the app will be connected by the given fd (in which case fd is nonzero)
//
struct IO_FILE_DESC {
struct FILE_REF {
char file_name[256];
char open_name[256];
int fd;
bool main_program;
FILE_INFO* file_info;
int parse(FILE*, char*);
int write(FILE*, char*);
};
struct APP_VERSION {
char app_name[256];
//char file_name[256];
int version_num;
APP* app;
PROJECT* project;
//FILE_INFO* file_info;
vector<FILE_REF> app_files;
int parse(FILE*);
int write(FILE*);
};
struct WORKUNIT {
char name[256];
char app_name[256];
@ -121,7 +124,7 @@ struct WORKUNIT {
// (TODO: use alpha/beta/prod scheme)
char command_line[256];
char env_vars[256]; // environment vars in URL format
vector<IO_FILE_DESC> input_files;
vector<FILE_REF> input_files;
PROJECT* project;
APP* app;
APP_VERSION* avp;
@ -134,7 +137,7 @@ struct WORKUNIT {
struct RESULT {
char name[256];
char wu_name[256];
vector<IO_FILE_DESC> output_files;
vector<FILE_REF> output_files;
bool is_active; // an app is currently running for this
bool is_compute_done; // computation finished
bool is_server_ack; // ack received from scheduling server

13
db/db.h
View File

@ -71,9 +71,16 @@ struct APP_VERSION {
int appid;
int version_num;
int platformid;
char url[256];
char md5_cksum[256];
double nbytes;
// describes app files. format:
// <file_info>...</file_info>
// ...
// <file_assocs>
// <io_file_desc>...</io_file_desc>
// ...
// </file_assocs>
//
char xml_doc[MAX_BLOB_SIZE];
// The following defined for apps other than core client.
// They let you handle backwards-incompatible changes to

10
db/db_mysql.C
View File

@ -85,7 +85,7 @@ void struct_to_str(void* vp, char* q, int type) {
avp = (APP_VERSION*)vp;
sprintf(q,
"id=%d, create_time=%d, appid=%d, version_num=%d, platformid=%d, "
"url='%s', md5_cksum='%s', nbytes=%f, "
"xml_doc='%s', "
"min_core_version=%d, max_core_version=%d, "
"message='%s', deprecated=%d",
avp->id,
@ -93,9 +93,7 @@ void struct_to_str(void* vp, char* q, int type) {
avp->appid,
avp->version_num,
avp->platformid,
avp->url,
avp->md5_cksum,
avp->nbytes,
avp->xml_doc,
avp->min_core_version,
avp->max_core_version,
avp->message,
@ -243,9 +241,7 @@ void row_to_struct(MYSQL_ROW& r, void* vp, int type) {
avp->appid = atoi(r[i++]);
avp->version_num = atoi(r[i++]);
avp->platformid = atoi(r[i++]);
strcpy(avp->url, r[i++]);
strcpy(avp->md5_cksum, r[i++]);
avp->nbytes = atof(r[i++]);
strcpy(avp->xml_doc, r[i++]);
avp->min_core_version = atoi(r[i++]);
avp->max_core_version = atoi(r[i++]);
strcpy(avp->message, r[i++]);

4
db/schema.sql
View File

@ -24,9 +24,7 @@ create table app_version (
appid integer not null,
version_num integer not null,
platformid integer not null,
url varchar(254),
md5_cksum varchar(254),
nbytes double not null,
xml_doc blob,
min_core_version integer not null,
max_core_version integer not null,
message varchar(254),

5
db/temp
View File

@ -1,5 +0,0 @@
struct ENUM {
int active;
MYSQL_RES *rp;
};

44
doc/app.html Normal file
View File

@ -0,0 +1,44 @@
<h2>Platforms, applications, and versions</h2>
<p>
A <b>platform</b> is a compilation target
(Windows/Intel, Linux/Intel, MacOS/PPC, etc.).
Platforms are maintained in the <b>platform</b> table in the BOINC DB,
and can be created using the <a href=tools.html>add</a> utility program.
<p>
An <b>application</b> is a computational task;
in concrete terms it consists of an executable program
(perhaps in versions for various platforms)
and a set of input files to be processed by the program.
A project can operate many applications.
Applications are maintained in the <b>application</b> table in the BOINC DB,
and can be created using the <a href=tools.html>add</a> utility.
<p>
An application program may go through a sequence of
<a href=version.html>versions</a>.
A particular version, compiled for a particular platform,
is called an <b>application version</b>.
An application version can consist of multiple files:
for example, a controller script,
pre- and post-processing programs, and a primary.
<p>
Application version are maintained in the <b>app_version</b>
table in the BOINC DB,
Each entry includes an XML document describing the files
that make up the application version:
<pre>
&lt;file_info> ... &lt;/file_info>
[ ... ]
&lt;app_version>
&lt;app_name>foobar&lt;/app_name>
&lt;version_num>4&lt;/version_num>
&lt;file_ref>
&lt;file_name>program_1&lt;/file_name>
&lt;main_program/>
&lt;/file_ref>
&lt;file_ref>
&lt;file_name>library_12&lt;/file_name>
&lt;/file_ref>
&lt;/app_version>
</pre>
The <a href=tools.html>add</a> utility program provides
a simplified interface for creating application versions.

98
doc/files.html
View File

@ -1,35 +1,83 @@
<h2>Compute model: files</h2>
<h2>Compute model: storage</h2>
<h3>Files</h3>
<p>
<b>Files</b> are inputs and outputs to applications,
and the executable application programs.
Each file has a <b>name</b> that is unique within the project.
A file may also have one or more URLs,
indicating where it is (or will be) located on data servers.
Once a file is created (on the project server or on a participant host)
it is immutable.
The BOINC storage model is based on <b>files</b>.
The inputs and outputs of applications,
and the applications executables, are files.
<p>
Files have the following Boolean attributes:
A file is described by an XML element of the form
<pre>
&lt;file_info>
&lt;name>foobar&lt;/name>
&lt;url>http://a.b.c/foobar&lt;/url>
&lt;url>http://x.y.z/foobar&lt;/url>
...
&lt;md5_cksum>123123123123&lt;/md5_cksum>
&lt;nbytes>134423&lt;/nbytes>
[ &lt;executable/> ]
[ &lt;upload_when_present/> ]
[ &lt;sticky/> ]
&lt;/file_info>
</pre>
The components are as follows:
<ul>
<li> Sticky
<li> Upload-when-present
<li>
The <b>&lt;name></b> element gives the file's name,
which must be unique within the project.
<li> Each <b>&lt;url></b> element gives a URL
where the file is (or will be) located on a data server.
<li> The <b>&lt;md5_cksum></b> element
is the MD5 checksum of the file,
providing protection against corruption and spoofing.
<li> The <b>&lt;nbytes></b> element is the size of the file in bytes
(may be greater than 2^32).
<li> The <b>&lt;sticky></b> element, if present,
indicates that the file should be retained on the client
after its initial use.
<li> The <b>&lt;upload_when_present></b> element, if present,
indicates that the file should be uploaded after it is created.
<li> The <b>&lt;executable></b> element,
if present, indicates that the file protections
should be set to allow execution.
</ul>
Some files originate on hosts;
i.e. as output files of applications.
If the Upload-when-present flag is set,
the file (when available) is uploaded to one of its URLs.
If the Sticky flag is set, the file remains on the host
(where, e.g., it will serve as input to a later computation).
These attributes allow the specification of various types of files:
for example, input or output files that are retained for
use as input to later computations.
<p>
Other files originate in the back-end servers,
and are placed on data servers.
If the Sticky flag is set,
the file is retained and a participant host after it is used.
Once a file is created (on a data server or a participant host)
it is immutable.
<h3>File references</h3>
<p>
The core agent and scheduling server exchange descriptions of files.
These descriptions include an MD5 checksum of the file,
providing protection against corruption and spoofing.
Files may be associated with
<a href=work.html>workunits</a>,
<a href=result.html>results</a> and
<a href=app.html>application versions</a>.
Each such association is represented by an XML element of the form
<pre>
&lt;file_ref>
&lt;file_name>foobar&lt;/file_name>
[ &lt;open_name>input&lt;/open_name> ]
[ &lt;fd>2&lt;/fd> ]
[ &lt;main_program/> ]
&lt;/file_ref>
</pre>
The components are as follows:
<ul>
<li> The <b>&lt;file_name></b> element specifies a file.
<li> The <b>&lt;open_name></b> element, if present,
is the name by which the application will refer to the file
(in a call to <b>fopen()</b>, for example).
<li> The <b>&lt;fd></b> element, if present,
indicates the file descriptor number by which the application
will access the file.
<li> The <b>&lt;main_program></b> element is used for
files associated with application versions.
It indicates that this file is the application's main program.
</ul>

15
doc/index.html
View File

@ -1,10 +1,4 @@
<center>
<h2>Berkeley Open Infrastructure for Network Computing (BOINC)</h2>
<i>BOINC documentation is under development.
Please do not copy or distribute the URL.</i>
</center>
<ul>
<li><a href=intro.html>Introduction</a>
@ -12,9 +6,11 @@ Please do not copy or distribute the URL.</i>
<h3>Developing distributed applications with BOINC</h3>
<ul>
<li><a href=project.html>Projects and applications</a>
<li><a href=files.html>Compute model: files</a>
<li><a href=work.html>Compute model: workunits and results</a>
<li><a href=project.html>Projects</a>
<li><a href=files.html>Compute model: files and file references</a>
<li><a href=app.html>Compute model: applications, platforms and versions</a>
<li><a href=work.html>Compute model: workunits</a>
<li><a href=result.html>Compute model: results</a>
<li><a href=batch.html>Compute model: batches</a>
<li><a href=flow.html>Compute model: work distribution</a>
<li><a href=affinity.html>Compute model: handling very long computations</a>
@ -64,3 +60,4 @@ Please do not copy or distribute the URL.</i>
<li><a href=client_debug.html>Core client: debugging</a>
</ul>
<A href="http://sourceforge.net"><IMG src="http://sourceforge.net/sflogo.php?group_id=52437&amp;type=5" width="210" height="62" border="0" alt="SourceForge Logo"></A>

15
doc/intro.html
View File

@ -1,7 +1,7 @@
<h2>Introduction</h2>
<p>
Berkeley Open Infrastructure for Network Computing (BOINC)
Berkeley Open Infrastructure for Distributed Computing (BOINC)
is a software platform for public-participation
distributed computing projects like distributed.net and SETI@home.
BOINC is distributed under the Mozilla license.
@ -20,16 +20,17 @@ Participants can control how their resources are divided among the projects.
<li> BOINC supports applications that produce or consume large amounts of data.
It allows data distribution and collection to be
spread across large numbers of servers,
and it allows participant hosts to transfer data transparently
over long time periods.
spread across many servers,
and it allows participant hosts to do large data transfers unobtrusively.
<li> BOINC supports applications with extreme requirements for
memory or other resources.
Work is dispatched only to hosts able to handle it.
Work units are dispatched only to hosts able to handle them.
<li> BOINC applications can be developed in any compiled language
(C, C++, Fortran).
<li> BOINC applications can be developed in any language
(C++, Fortran, Perl).
An application can consist of several files
(e.g. several programs and a coordinating script).
New versions of applications can be released without participant download.
Separate alpha, beta, and production versions
are distributed to the appropriate set of hosts.

201
doc/project.fig
View File

@ -7,58 +7,177 @@ Letter
Single
-2
1200 2
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
3600 2400 6900 2400 6900 3300 3600 3300 3600 2400
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
3600 2400 6900 2400 6900 3300 3600 3300 3600 2400
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
3600 1500 6900 1500 6900 1875 3600 1875 3600 1500
0 32 #c7b696
0 33 #effbff
0 34 #dfcba6
0 35 #414141
0 36 #868286
0 37 #c7c3c7
0 38 #e7e3e7
0 39 #8e8e8e
0 40 #aeaaae
0 41 #515551
0 42 #414141
0 43 #868286
0 44 #c7c3c7
0 45 #868286
0 46 #c7c3c7
0 47 #e7e3e7
0 48 #8e8e8e
0 49 #8e8e8e
0 50 #414141
0 51 #868286
0 52 #c7c3c7
0 53 #e7e3e7
0 54 #414141
0 55 #868286
0 56 #c7c3c7
0 57 #e7e3e7
0 58 #868286
0 59 #c7c3c7
0 60 #e7e3e7
0 61 #c7b696
0 62 #effbff
0 63 #dfcba6
0 64 #c7b696
0 65 #effbff
0 66 #dfcba6
0 67 #aeaaae
0 68 #515551
0 69 #8e8e8e
0 70 #414141
0 71 #868286
0 72 #c7c3c7
0 73 #e7e3e7
0 74 #414141
0 75 #868286
0 76 #c7c3c7
0 77 #e7e3e7
0 78 #868286
0 79 #c7c3c7
0 80 #e7e3e7
0 81 #414141
0 82 #868286
0 83 #c7c3c7
0 84 #414141
0 85 #c7c3c7
0 86 #e7e3e7
0 87 #414141
0 88 #868286
0 89 #c7c3c7
0 90 #8e8e8e
0 91 #414141
0 92 #868286
0 93 #c7c3c7
0 94 #e7e3e7
0 95 #414141
0 96 #868286
0 97 #c7c3c7
0 98 #e7e3e7
0 99 #bebebe
0 100 #515151
0 101 #000049
0 102 #797979
0 103 #303430
0 104 #414541
0 105 #414141
0 106 #868286
0 107 #c7c3c7
0 108 #e7e3e7
6 3225 2625 4650 3300
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
3225 2625 4650 2625 4650 3300 3225 3300 3225 2625
4 0 0 50 0 18 14 0.0000 4 210 1155 3375 2925 scheduling\001
4 0 0 50 0 18 14 0.0000 4 120 660 3600 3150 server\001
-6
6 4950 3675 5925 4950
5 1 0 1 -1 -1 0 0 -1 0.000 0 1 0 0 5400.000 4125.000 4950 4725 5400 4875 5850 4725
1 2 0 1 -1 11 0 0 20 0.000 1 0.0000 5421 3894 450 150 4971 3894 5871 3894
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2
4950 3900 4950 4725
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2
5850 3975 5850 4725
4 1 -1 0 0 18 12 0.0000 6 135 600 5400 4350 BOINC\001
4 1 -1 0 0 18 12 0.0000 6 135 270 5400 4575 DB\001
-6
6 7650 750 8775 1500
2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5
7650 750 8775 750 8775 1500 7650 1500 7650 750
4 0 0 50 0 18 14 0.0000 4 120 780 7800 1350 servers\001
4 0 0 50 0 18 14 0.0000 4 165 450 7950 1050 data\001
-6
6 2100 4050 2775 4800
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
2400 4575 2400 4050
4 0 0 50 0 16 14 0.0000 4 120 630 2100 4800 server\001
-6
6 2100 5250 2625 6075
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
2400 5550 2400 6075
4 0 0 50 0 16 14 0.0000 4 165 510 2100 5475 client\001
-6
6 4800 2625 6225 3300
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
4800 2625 6225 2625 6225 3300 4800 3300 4800 2625
4 0 0 50 0 18 14 0.0000 4 210 585 5175 2925 utility\001
4 0 0 50 0 18 14 0.0000 4 165 1005 5025 3150 programs\001
-6
6 6375 2625 7800 3300
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
6375 2625 7800 2625 7800 3300 6375 3300 6375 2625
4 0 0 50 0 18 14 0.0000 4 165 465 6900 2925 Web\001
4 0 0 50 0 18 14 0.0000 4 165 1020 6600 3150 interfaces\001
-6
2 2 0 1 0 30 50 0 20 0.000 0 0 -1 0 0 5
3600 750 6900 750 6900 1500 3600 1500 3600 750
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
5250 1875 5250 2400
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
3900 3300 3900 5925
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
5550 5400 5550 5925
2 1 1 1 0 3 50 0 20 4.000 0 0 -1 0 0 2
2700 3750 9600 3750
3900 3300 3900 7200
2 2 0 1 0 30 50 0 20 0.000 0 0 -1 0 0 5
9825 750 10350 750 10350 1200 9825 1200 9825 750
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
9825 1350 10350 1350 10350 1800 9825 1800 9825 1350
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 0 2
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
8400 3300 8400 2775
2 1 0 1 0 3 50 0 20 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
8400 4200 8400 4725
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
4214 5025 6975 5025 6975 5400 4214 5400 4214 5025
5550 6675 5550 7200
2 1 1 1 0 3 50 0 20 4.000 0 0 -1 0 0 2
2700 5025 9600 5025
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
4214 6300 6975 6300 6975 6675 4214 6675 4214 6300
2 2 0 1 0 30 50 0 20 0.000 0 0 -1 0 0 5
4214 4275 6975 4275 6975 5025 4214 5025 4214 4275
2 2 0 1 0 3 50 0 20 0.000 0 0 -1 0 0 5
3600 5925 7650 5925 7650 6825 3600 6825 3600 5925
4214 5550 6975 5550 6975 6300 4214 6300 4214 5550
2 2 0 1 0 11 50 0 20 0.000 0 0 -1 0 0 5
3600 7200 8850 7200 8850 8100 3600 8100 3600 7200
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
7350 5925 7350 1500
2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5
6975 750 8100 750 8100 1500 6975 1500 6975 750
4 0 0 50 0 18 14 0.0000 4 210 2355 4125 2925 BOINC server complex\001
8250 7200 8250 1500
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
4350 3300 4950 3750
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
5400 3300 5400 3675
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
6825 3300 6000 3750
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
6309 1488 7125 2625
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
5409 1488 5400 2625
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
1 1 1.00 60.00 120.00
10081 6123 7500 3300
4 0 0 50 0 18 14 0.0000 4 210 1740 4350 1275 project back end\001
4 0 0 50 0 18 14 0.0000 4 165 1365 4500 1800 back end API\001
4 0 0 50 0 16 14 0.0000 4 210 2400 10500 1200 project-specific software\001
4 0 0 50 0 16 14 0.0000 4 165 1545 10500 1800 BOINC software\001
4 0 0 50 0 16 14 0.0000 4 165 510 8100 4125 client\001
4 0 0 50 0 16 14 0.0000 4 120 630 8100 3525 server\001
4 0 0 50 0 18 14 0.0000 4 215 1826 4751 4725 application client\001
4 0 0 50 0 18 14 0.0000 4 169 1012 4981 5325 client API\001
4 0 0 50 0 18 14 0.0000 4 169 1120 4981 6450 core client\001
4 0 0 50 0 18 14 0.0000 4 120 780 7125 1350 servers\001
4 0 0 50 0 18 14 0.0000 4 165 450 7275 1050 data\001
4 0 0 50 0 16 14 0.0000 4 210 2775 10500 1200 project-specific components\001
4 0 0 50 0 16 14 0.0000 4 210 1920 10500 1800 BOINC components\001
4 0 0 50 0 18 14 0.0000 4 210 1785 4751 6000 application client\001
4 0 0 50 0 18 14 0.0000 4 165 990 4981 6600 client API\001
4 0 0 50 0 18 14 0.0000 4 165 1095 5550 7725 core client\001
4 0 0 50 0 18 14 0.0000 4 210 2355 2100 2400 BOINC server complex\001
4 0 0 50 0 18 14 0.0000 4 210 1245 9450 6525 participants\001

39
doc/project.html
View File

@ -1,12 +1,9 @@
<h2>Projects and applications</h2>
<p>
A <b>project</b> is an organization
(e.g., an academic research project) that uses BOINC to get computing power.
A <b>project</b> is an organization that uses BOINC.
Projects are independent;
each one manages its own applications, databases and servers,
and is not affected by the status of other projects.
<p>
The components of a project are shown below.
<br>
<img vspace=10 src=project.png>
@ -14,11 +11,17 @@ The components of a project are shown below.
<p>
The server side of a project consists of two parts:
<ul>
<li> A <b>project back end</b> part that generates
computation input files, and handles the output of computations,
in a way that depends on the particular application.
<li> A <b>project back end</b> that supplies
applications and work units,
and that handles the computational results.
Input and output files are distributed by <b>data servers</b>,
which are HTTP servers able to handle uploads.
These servers need not be owned or operated by the project.
A project might, for example, recruit other organizations to
donate network bandwidth by hosting data servers;
data could be moved on tape between the project back end and the data servers.
<li> A <b>BOINC server complex</b>
that handles data distribution and collection.
that manages data distribution and collection.
</ul>
The BOINC server complex includes the following components:
@ -26,23 +29,7 @@ The BOINC server complex includes the following components:
<li> A <b>scheduling server</b> that communicates with participant hosts.
<li> A relational database storing information
about work, results, and participants.
<li> C and scripting-language interfaces for
communication with the project back end.
<li> Utility programs and libraries that allow the
project back end to interact with the server complex.
<li> Web interfaces for participants and developers.
</ul>
<p>
Input and output files are distributed by <b>data servers</b>,
which are HTTP servers able to handle uploads.
These servers need not be owned or operated by the project.
A project might, for example, recruit other organizations to
donate network bandwidth by hosting data servers;
data could be moved on tape between the project back end and the data servers.
<h3>Applications and platforms</h3>
<p>
A project can operate one or more <b>applications</b>.
Versions of an application are compiled for various <b>platforms</b>
(Windows/Intel, Linux/Intel, MacOS/PPC, etc.).
The BOINC core client is available for many platforms;
a project can compile an application for any or all of these.

BIN
doc/project.png
View File

Binary file not shown.

47
doc/protocol.html
View File

@ -1,7 +1,13 @@
<h2>The BOINC client/server protocol</h2>
<h3>Protocol nucleus</h3>
<p>
The core client communicates with scheduling servers using HTTP.
The following "nucleus" of the protocol should be considered immutable.
This will allow any version of the core client to talk to
any version of the scheduling server,
and for the server, if needed,
to tell the core client that it's out of date.
The form of the request is:
<pre>
&lt;scheduler_request>
@ -20,39 +26,46 @@ The form of the reply is:
&lt;scheduler_reply>
[ &lt;message priority="low"> arbitrary text &lt;/message> ... ]
[ &lt;request_delay>3600&lt;/request_delay> ]
[ &lt;redirect>URL&lt;/redirect> ]
... other elements
&lt;/scheduler_reply>
</pre>
<p>
The "message" element provides message for the core client.
If the priority is "high", the core client should do what it can
to ensure the the participant sees the message
(e.g., display it in a modal dialog).
<p
Each <b>message</b> element is a message to the participant.
<ul>
<li>
If the priority is "high", the core client should try
to ensure that the participant sees the message;
for example, by displaying it in a popup window.
It should not, however, wait for user input, as this
could starve other projects.
This should be reserved for situations where definitive action
is required, e.g. the user must download a new version of the
core client in order to continue participating in this project.
<li>
If the priority is "low" (default) the core client should
allow the participant to see the message, but not require it;
e.g., by writing the message to a log file.
allow the participant to see the message, but should not require it.
For example, it could the message to a log file.
</ul>
<p>
A reply message can contain multiple message elements.
<p>
The optional "request_delay" tag instructs the client
A <b>request_delay</b> element instructs the client
to not issue another request until the indicated number
of seconds has elapsed.
<p>
The above portion of the protocol should be considered immutable.
This will allow any version of the core client to talk to
any version of the scheduling server,
and for the server, if needed,
to tell the core client that it's hopelessly out of date.
A <b>redirect</b> element gives the URL for a scheduling
server for this project.
If present, the core client should replace its list
of scheduling servers for this project.
The reply may contain multiple <b>redirect</b> elements.
<h3>Extensible protocol</h3>
<p>
The remaining protocol may evolve over time.
<p>
Request elements include
<pre>

75
doc/result.html Normal file
View File

@ -0,0 +1,75 @@
<h2>Compute model: results</h2>
<p>
A <b>result</b> describes an instance of a computation,
either to be performed, in progress, or completed.
Results are stored in the <b>result</b> table of the BOINC DB.
The attributes of a result include:
<ul>
<li> The name of the result (unique across all results in the project).
<li> The name of the associated workunit.
<li> An XML document listing the names of its output files; see below.
<li> An XML document giving the sizes and checksums of its output files
(this is filled in after the result is completed).
<li> The stderr output of the result.
<li> The host that computed the result.
<li> The times when the result was dispatched and received.
<li> The exit status of the application.
<li> The reported CPU time.
<li> Its <b>state</b>. Values include:
<ul>
<li> Inactive (not ready to dispatch)
<li> Unsent (ready to dispatch, but not dispatched)
<li> In progress (dispatched, not done)
<li> Done successfully
<li> Timed out
<li> Done with error
</ul>
</ul>
The XML document listing the output files has the form:
<pre>
&lt;file_info>...&lt;/file_info>
[ ... ]
&lt;result>
&lt;name>foobar&lt;/name>
&lt;wu_name>blah&lt;/wu_name>
&lt;file_ref>...&lt;/file_ref>
[ ... ]
&lt;/result>
</pre>
The components are:
<ul>
<li> The <b>&lt;name></b> element is the result name.
<li> The <b>&lt;wu_name></b> element is the workunit name.
<li> Each <b>&lt;file_ref></b> element is an association
to an output file, described by a corresponding
<b>&lt;file_info></b> element.
</ul>
<p>
The XML document describing the sizes and checksums
of the output files is just a list of
<b>&lt;file_info></b> elements, with the
<b>nbytes</b> and <b>md5_cksum</b> fields present.
The project back end must parse this field to
find the locations and checksums of output files.
<p>
Several results may be associated with a single workunit.
Results may be generated in either of two ways
(selected as part of the application):
<ul>
<li> <b>Advance generation</b> of results.
One or more result records are stored in the database
when the workunit is produced.
The scheduling server dispatches each result to a single participant host.
When all result records have been dispatched,
participants hosts are "turned away".
<li> <b>On-demand generation</b> of results.
The application specifies a "result template",
which has place-holder tokens for the output filenames.
The scheduling server, in response to a host request,
generates a new result record and sends the result template.
The host generates unique output filenames,
and returns them along when it the computation is done.
</ul>

94
doc/work.html
View File

@ -1,75 +1,55 @@
<body bgcolor=ffffff>
<h2>Compute model: workunits and results</h2>
<h2>Compute model: workunits</h2>
<p>
A <b>workunit</b> describes a computation to be performed.
Workunits are maintained in the <b>workunit</b> table in the BOINC DB.
The attributes of a workunit include:
<ul>
<li> The application it's associated with.
<li> Its application.
<li> Its name (unique across all workunits in the project).
<li> A list of its input files.
Each has an <b>input file association</b>
that specifies the filename or file descriptor number
by which the application will refer to the file.
<li> The command-line arguments to be passed to the application.
<li> The environment variables to be set for the application.
<li> An XML document describing its input files and other parameters
(see below).
<li> The estimated resource requirements of the work unit
(computation, memory, disk space, network traffic).
<li> Counts of how many times this workunit should be dispatched,
how many times it has been dispatched, how many results have
been returned, and how many failures have occurred.
</ul>
<p>
The inputs of a workunit are described by an XML document of the form
<pre>
[ &lt;file_info>...&lt;/file_info> ]
[ ... ]
&lt;workunit>
&lt;name>foobar&lt;/name>
&lt;app_name>blah&lt;/app_name>
[ &lt;command_line>-flags xyz&lt;/command_line> ]
[ &lt;env_vars>name=val&name=val&lt;/env_vars> ]
[ &lt;file_ref>...&lt;/file_ref> ]
[ ... ]
&lt;workunit>
</pre>
The components are:
<ul>
<li> The <b>&lt;name></b> element is the name of the workunit.
<li> The <b>&lt;app_name></b> element is the name of the application.
<li> The <b>&lt;command_line></b> element, if present,
is the command-line arguments to be passed to the main program.
<li> The <b>&lt;env_vars></b> element, if present,
is a list of environment variables to be passed to the main program.
<li> Each <b>&lt;file_ref></b> element describes a
<a href=files.html>reference</a> to an input file,
each of which is described by a <b>&lt;file_info></b> element.
</ul>
<p>
A workunit is associated with an application,
not with a particular version or set of versions.
If the format of your input data changes in a way that
is incompatible with older versions,
you must create a new application.
This can often be avoided by using XML-like data representations.
This can often be avoided by using XML data format.
<p>
A <b>result</b> describes a particular instance of a computation,
either to be performed or already performed.
The attributes of a result include:
<ul>
<li> The workunit it's associated with.
<li> Its name (unique across all results in the project).
<li> A list of its output files.
Each has an <b>output file association</b>
that specifies the filename or file descriptor number
by which the application will refer to the file.
<li> Its <b>state</b>. Values include:
<ul>
<li> Inactive (not ready to dispatch)
<li> Unsent (ready to dispatch, but not dispatched)
<li> In progress (dispatched, not done)
<li> Done successfully
<li> Timed out
<li> Done with error
</ul>
</ul>
<p>
Several results may be associated with a single workunit.
Results may be generated in either of two ways
(selected as part of the application):
<ul>
<li> <b>Advance generation</b> of results.
One or more result records are stored in the database
when the workunit is produced.
The scheduling server dispatches each result to a single participant host.
When all result records have been dispatched,
participants hosts are "turned away".
<li> <b>On-demand generation</b> of results.
The application specifies a "result template",
which has place-holder tokens for the output filenames.
The scheduling server, in response to a host request,
generates a new result record and sends the result template.
The host generates unique output filenames,
and returns them along when it the computation is done.
</ul>
<p>
<hr>
The following diagram illustrates the relationship between
workunits, results, files, and I/O assocations.
<p>
<img src=work.gif>
The <a href=tools.html>create_work</a> utility program
provides a simplified interface for creating workunits.

7
html/user/db.inc
View File

@ -39,8 +39,11 @@ function show_app_version($app_version) {
row("app ID", $app_version->appid);
row("version num", $app_version->version_num);
row("platform ID", $app_version->platformid);
row("url", $app_version->url);
row("MD5", $app_version->md5_cksum);
row("XML doc", "<pre>".htmlspecialchars($app_version->xml_doc)."</pre>");
row("min_core_version", $app_version->min_core_version);
row("max_core_version", $app_version->max_core_version);
row("message", $app_version->message);
row("deprecated", $app_version->deprecated);
echo "</table>";
}

24
sched/handle_request.C
View File

@ -77,20 +77,22 @@ int authenticate_user(
if (sreq.hostid) {
retval = db_host(sreq.hostid, host);
if (retval) {
strcpy(reply.error_message, "invalid hostid");
strcpy(reply.message, "Can't find host record");
strcpy(reply.message_priority, "low");
return -1;
}
retval = db_user(host.userid, user);
if (retval) {
strcpy(reply.error_message, "internal error - no such user");
strcpy(reply.message, "Can't find user record");
strcpy(reply.message_priority, "low");
return -1;
}
if (strcmp(sreq.authenticator, user.authenticator)) {
strcpy(reply.error_message,
strcpy(reply.message,
"Invalid or missing authenticator. "
"Visit this project's web site to request your authenticator, "
"then cut and paste it into your accounts file."
"Visit this project's web site to get an authenticator."
);
strcpy(reply.message_priority, "low");
return -1;
}
@ -107,7 +109,8 @@ int authenticate_user(
strcpy(user.authenticator, sreq.authenticator);
retval = db_user_lookup_auth(user);
if (retval) {
strcpy(reply.error_message, "invalid or missing authenticator");
strcpy(reply.message, "Invalid or missing authenticator");
strcpy(reply.message_priority, "low");
return -1;
}
new_host:
@ -120,7 +123,8 @@ new_host:
host.rpc_time = time(0);
retval = db_host_new(host);
if (retval) {
strcpy(reply.error_message, "DB operation failed");
strcpy(reply.message, "server database error");
strcpy(reply.message_priority, "low");
db_print_error("db_host_new");
return -1;
}
@ -304,7 +308,8 @@ int send_work(
}
if (nresults == 0) {
strcpy(reply.error_message, "no work available");
strcpy(reply.message, "no work available");
strcpy(reply.message_priority, "low");
reply.request_delay = 10;
}
return 0;
@ -330,7 +335,8 @@ void process_request(
platform = db.lookup_platform(sreq.platform_name);
if (!platform) {
sprintf(buf, "platform %s not found", sreq.platform_name);
strcpy(reply.error_message, buf);
strcpy(reply.message, buf);
strcpy(reply.message_priority, "low");
return;
}

31
sched/server_types.C
View File

@ -78,7 +78,8 @@ int SCHEDULER_REQUEST::parse(FILE* fin) {
SCHEDULER_REPLY::SCHEDULER_REPLY() {
request_delay = 0;
hostid = 0;
strcpy(error_message, "");
strcpy(message, "");
strcpy(message_priority, "");
send_prefs = false;
}
@ -95,8 +96,12 @@ int SCHEDULER_REPLY::write(FILE* fout) {
if (request_delay) {
fprintf(fout, "<request_delay>%d</request_delay>\n", request_delay);
}
if (strlen(error_message)) {
fprintf(fout, "<error_message>%s</error_message>\n", error_message);
if (strlen(message)) {
fprintf(fout,
"<message priority=\"%s\">%s</message>\n",
message_priority,
message
);
}
if (hostid) {
@ -193,25 +198,7 @@ int APP::write(FILE* fout) {
}
int APP_VERSION::write(FILE* fout, APP& app) {
fprintf(fout,
"<file_info>\n"
" <name>%s_%d</name>\n"
" <executable/>\n"
" <url>%s</url>\n"
" <md5_cksum>%s</md5_cksum>\n"
"</file_info>\n"
"<app_version>\n"
" <app_name>%s</app_name>\n"
" <file_name>%s_%d</file_name>\n"
" <version_num>%d</version_num>\n"
"</app_version>\n",
app.name, id,
url,
md5_cksum,
app.name,
app.name, id,
version_num
);
fputs(xml_doc, fout);
return 0;
}

3
sched/server_types.h
View File

@ -44,7 +44,8 @@ struct SCHEDULER_REQUEST {
struct SCHEDULER_REPLY {
int request_delay; // don't request again until this time elapses
char error_message[256]; // if present, don't return anything else
char message[1024];
char message_priority[256];
int hostid; // send this only if nonzero.
// this tells client to reset rpc_seqno
bool send_prefs;

25
test/init.inc
View File

@ -1,16 +1,15 @@
<?php
define("HOME", "/disks/jill/a/inet_services/boinc_www/share/htdocs");
define("DOWNLOAD_DIR", "$HOME/download");
define("UPLOAD_DIR", "$HOME/upload");
define("PLATFORM_NAME", "sparc-sun-solaris2.7");
define("DOWNLOAD_URL", "http://localhost/download/");
define("HOME", "home/david");
define("DOWNLOAD_DIR", "$HOME/html/download");
define("UPLOAD_DIR", "$HOME/html/upload");
define("PLATFORM_NAME", "i686-pc-linux-gnu");
define("VERSION", "1");
define("CORE_CLIENT", "boinc_".VERSION."_".PLATFORM_NAME);
define("PROJECTS", "localhost zoot");
define("PROJECTS", "localhost.localdomain zoot");
// directories to empty
define("EMAIL_ADDR", "david@localhost");
define("URL_BASE", "http://localhost/download");
define("EMAIL_ADDR", "david@localhost.localdomain");
define("URL_BASE", "http://localhost.localdomain/download");
// data server URL root
include_once("../html_user/db.inc");
@ -47,12 +46,16 @@ function add_platform() {
PassThru("../tools/add platform -platform_name ".PLATFORM_NAME);
}
function add_core_client() {
function add_core_client_message($message, $priority) {
PassThru("../tools/add app -app_name core_client -version ".VERSION);
PassThru("../tools/add app_version -app_name core_client -platform_name ".PLATFORM_NAME." -version ".VERSION." -exec_dir ../client -exec_file ".CORE_CLIENT." -download_dir ".DOWNLOAD_DIR." -url_base ".URL_BASE);
PassThru("../tools/add app_version -app_name core_client -platform_name ".PLATFORM_NAME." -version ".VERSION." -exec_dir ../client -exec_file ".CORE_CLIENT." -download_dir ".DOWNLOAD_DIR." -url_base ".URL_BASE." -message '$message' -message_priority '$priority'");
PassThru("cp ../client/".CORE_CLIENT." ".DOWNLOAD_DIR);
}
function add_core_client() {
add_core_client_message("", "");
}
function add_app($name) {
PassThru("../tools/add app -app_name $name -version ".VERSION);
PassThru("../tools/add app_version -app_name $name -platform_name ".PLATFORM_NAME." -version ".VERSION." -exec_dir ../apps -exec_file $name -download_dir ".DOWNLOAD_DIR." -url_base ".URL_BASE);
@ -60,7 +63,7 @@ function add_app($name) {
}
function create_work($x) {
PassThru("../tools/create_work $x -download_url " .DOWNLOAD_URL);
PassThru("../tools/create_work $x");
}
function run_client() {

1
test/test_uc.php
View File

@ -11,7 +11,6 @@
init_client_dirs("account1.xml");
copy_to_download_dir("input");
add_platform();
add_core_client();
add_user();
add_app("upper_case");
create_work("-appname upper_case -wu_name uc_wu -wu_template uc_wu -result_template uc_result -nresults 2 input input");

43
tools/add.C
View File

@ -48,10 +48,12 @@ int main(int argc, char** argv) {
USER user;
PREFS prefs;
int i, version, retval;
char buf[256];
double nbytes;
char buf[256], md5_cksum[64];
char *app_name=0, *platform_name=0, *exec_dir=0, *exec_file=0;
char *email_addr=0, *user_name=0, *web_password=0, *authenticator=0;
char *prefs_file=0, *download_dir, *url_base;
char *message=0, *message_priority=0;
db_open("boinc");
for (i=2; i<argc; i++) {
@ -94,6 +96,12 @@ int main(int argc, char** argv) {
} else if (!strcmp(argv[i], "-version")) {
i++;
version = atoi(argv[i]);
} else if (!strcmp(argv[i], "-message")) {
i++;
message = argv[i];
} else if (!strcmp(argv[i], "-message_priority")) {
i++;
message_priority = argv[i];
}
}
if (!strcmp(argv[1], "app")) {
@ -117,7 +125,6 @@ int main(int argc, char** argv) {
}
} else if (!strcmp(argv[1], "app_version")) {
memset(&app_version, 0, sizeof(app_version));
sprintf(app_version.url, "%s/%s", url_base, exec_file);
strcpy(app.name, app_name);
retval = db_app_lookup_name(app);
@ -134,6 +141,8 @@ int main(int argc, char** argv) {
}
app_version.platformid = platform.id;
app_version.version_num = version;
if (message) strcpy(app_version.message, message);
if (message_priority) strcpy(app_version.message, message_priority);
// copy executable to download directory
//
@ -151,7 +160,35 @@ int main(int argc, char** argv) {
// compute checksum of executable
//
sprintf(buf, "%s/%s", exec_dir, exec_file);
md5_file(buf, app_version.md5_cksum, app_version.nbytes);
md5_file(buf, md5_cksum, nbytes);
// generate the XML doc directly.
// TODO: use a template, as in create_work
//
sprintf(app_version.xml_doc,
"<file_info>\n"
" <name>%s</name>\n"
" <url>%s/%s</url>\n"
" <executable/>\n"
" <md5_cksum>%s</md5_cksum>\n"
" <nbytes>%f</nbytes>\n"
"</file_info>\n"
"<app_version>\n"
" <app_name>%s</app_name>\n"
" <version_num>%d</version_num>\n"
" <file_ref>\n"
" <file_name>%s</file_name>\n"
" <main_program/>\n"
" </file_ref>\n"
"</app_version>\n",
exec_file,
url_base, exec_file,
md5_cksum,
nbytes,
app_name,
version,
exec_file
);
app_version.create_time = time(0);
retval = db_app_version_new(app_version);