Persistent File Transfer

Since BOINC may deal with uploading or downloading very large files, or servers which are not always available, the notion of a persistent file transfer is necessary.

The logic to handle resumption of downloads or uploads is embedded in the HTTP and file transfer layers. The HTTP 1.1 byte range option is used to support download resumption. Therefore, data servers must support HTTP 1.1 in order for download resumption to work. The upload resumption feature is handled in the file_upload_handler program.

A persistent file transfer is described by an XML element of the form

<persistent_file_xfer>
    <num_retries>2</num_retries>
    <first_request_time>1030665600</first_request_time>
    <next_request_time>1030665725</next_request_time>
</persistent_file_xfer>

• The num_retries element is the number of times the client has attempted to download/upload the file.
• The first_request_time element is the first time (in UNIX time) that the client attempted to download/upload the file.
• The next_request_time element is the earliest time (in UNIX time) that the client will retry the download/upload.

When there is a failure to download or upload a file, the client will increment num_retries by 1 and calculate a new next_request_time based on randomized exponential backoff. The equation for this is:

next_request_time = current_time+max(MIN_DELAY,min(MAX_DELAY,exp(rand(0,1)*num_retries)))

The client will consider the download/upload a complete failure if the current time becomes much later than this (default is two weeks). In this event, the file will be deleted and the failure reported to the scheduling server.