Files and data servers
The BOINC storage model is based on files.
Examples of files:
- The inputs and outputs of applications;
- Application executables, libraries, etc.
The BOINC core client transfers files to and from project-operated
data servers using HTTP.
A file is described by an XML element of the form
".html_text("
foobar
http://a.b.c/foobar
http://x.y.z/foobar
...
123123123123
134423
200000
1
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
")."
The elements are as follows:
";
list_start();
list_item(
"name",
"The file's name, which must be unique within the project.
If you want to use participant hosts on which
filenames are case-insensitive (e.g. Windows)
this uniqueness is case-insensitive."
);
list_item("url",
"a URL where the file is (or will be) located on a data server."
);
list_item("md5_cksum", "The MD5 checksum of the file."
);
list_item("nbytes",
"the size of the file in bytes."
);
list_item("max_nbytes",
"The maximum allowable size of the file in bytes (may be greater than 2^32).
This is used to prevent flooding data servers with bogus data."
);
list_item("status",
"0 if the file is not present,
1 if the file is present, or a negative error code if there was a
problem in downloading or generating the file."
);
list_item("generated_locally",
"If present, indicates that the file will be generated by an application on
the client, as opposed to being downloaded."
);
list_item("executable",
"If present, indicates that the file protections should be set to allow
execution."
);
list_item("upload_when_present",
"If present, indicates that the file should be uploaded
when the application finishes.
The file is uploaded even if the application doesn't
finish successfully.
API functions are available for
uploading files prior to
finishing computation.
");
list_item("sticky",
"If present, indicates that the file should be retained
on the client after its initial use."
);
list_item("signature_required",
"If present, indicates that the file should be verified with an
RSA signature.
This generally only applies to executable files."
);
list_item("no_delete",
"If present for an input (workunit) file,
indicates that the file should NOT be removed from the data server's
download directory when the workunit is completed.
Use this if a particular input file or files are used by more than one
workunit, or will be used by future workunits.
If present for an output (result) file,
indicates that the file should NOT be removed from the data server's upload
directory when the corresponding workunit is completed.
Use with caution - this may cause your upload directory to overflow."
);
list_item("report_on_rpc",
"Include a description of this file in scheduler RPC requests,
so that the scheduler may send appropriate work
using locality scheduling."
);
list_end();
echo "
Once a file is created (on a data server or a participant host) it
is immutable.
This means that all replicas of that file are assumed to be identical.
File references
Files may be associated with workunits,
results and
application versions.
Each such association is represented by an XML element of the form
".html_text("
foobar
[ input ]
[ ]
")."
The elements are as follows:
";
list_start();
list_item("file_name", "Specifies a file.");
list_item("open_name",
"The name by which the application will refer to the file.
Applications access files using
the following functions:
char physical_name[256];
boinc_resolve_filename(\"input\", physical_name, 256);
fopen(physical_name, \"r\")
In this example, open_name is 'input'.
It is mapped, at runtime, to a path that includes
the filename ('foobar' in the example above).
");
list_item("main_program",
"Relevant only for files associated with application versions.
It indicates that this file is the application's main program.
");
list_end();
echo "
File management
BOINC's default behavior is to delete files around
when they aren't needed any more.
Specifically:
- On the client, input files are deleted when no workunit refers to them,
and output files are deleted when no result refers to them.
Application-version files are deleted when they are referenced
only from superceded application versions.
- On the client, the 'sticky' flag overrides the above mechanisms
and suppresses the deletion of the file.
The file may deleted by an explicit
server request.
The file may also be deleted at any time by the core client
in order to honor limits on disk-space usage.
- On the server, the file deleter daemon
deletes input and output files that are no longer needed.
This can be suppressed using the 'no_delete' flag,
or using command-line options to the file deleter.
File compression
Starting with version 5.4, the BOINC client
is able to handle HTTP Content-Encoding types 'deflate' (zlib algorithm)
and 'gzip' (gzip algorithm).
The client decompresses these files 'on the fly' and
stores them on disk in uncompressed form.
Projects can set this encoding in two ways:
-
Use the Apache 2.0 mod_deflate module to automatically
compress files on the fly.
This method will work with all BOINC clients,
but it will do compression only for 5.4+ clients.
-
Compress their workunits when they create them and use
a filename suffix such as '.gz'.
In httpd.conf make sure that the following line is present:
AddEncoding x-gzip .gz
This will add the content encoding to the header so that
the client will decompress the file automatically.
This method has the advantage of reducing server disk usage
and server CPU load,
but it will only work with 5.4+ clients.
Use the 'min_core_version' field of the app_version table to enforce this.
You can also use these in conjunction because the mod_deflate module
allows you to exempt certain filetypes from on-the-fly compression.
Neither of these methods stores files in compressed form on the client.
For this, you must do compression at the application level.
The BOINC source distribution includes
a version of the zip library
designed for use by BOINC applications on any platform.
";
page_tail();
?>