If a remote job submission system wants to maintain a constant
level of unsent jobs, it needs a way to get this value.
This commit adds a Python API for this.
It also returns other counts such as in-progress results,
and WUs needing validation/assimilation/file delete.
I could add PHP and C++ interfaces too if needed.
The RPC handler wasn't looking in per-job templates for rsc_fpops_est.
This commit alters the API for job submission. Please see
https://boinc.berkeley.edu/trac/wiki/RemoteJobs for the current API
details
Change the submit_batch RPC, and the various bindings of it (PHP, Python, C++)
to support an app_version_num arg,
specifying which app version number should be used to process the jobs in the batch.
In some cases of file staging (both remote and via stage_file)
we'd do the following:
1) create the .md5 file (in check_download_file())
2) move or copy the file into the download dir
This can result in the file having a later mod time than the .md5 file,
which causes process_input_template() to reject the .md5 file.
Solution: touch the .md5 file after the move or copy
Files in the download dir can have accompanying ".md5" files
containing their MD5 and size.
This eliminates the need to calculate these when creating a job using the file.
The .md5 files were being created by stage_file (local staging)
but not by remote file management.
In fact, the latter wasn't checking for file immutability violations.
I changed remote file management to add this check,
and to create the .md5 file.
The latter is done in a new function shared with stage_file.
- The RPC handler was expecting the "BOINC names" to be the file MD5.
This is no longer true; the BOINC name can be anything as long as it's unique.
- To reflect this, use <phys_name> instead of <md5> in request messages.
This means that you'll need to update both RPC client and server software.
- BoincJobFile::insert() needed to return the insert ID
This supports the TACC use case,
in the jobs in a batch can use different Docker images
and different input and output file signatures,
none of which are known in advance.
Python API binding:
- A JOB_DESC object can optionally contain wu_template and result_template
elements, which are the templates (the actual XML) to use for that job.
Add these to the XML request message if present.
- Added the same capability to the PHP binding, but not C++.
- Added and debugged test cases for both languages.
Also, submit_batch() can take either a batch name (in which case
the batch is created) or a batch ID
(in which the batch was created prior to remotely staging files).
RPC handler:
- in submit_batch(), check for jobs with templates specified
and store them in files.
For input templates (which are deleted after creating jobs)
we put them in /tmp,
and use a map so that if two templates are the same we use 1 file.
For output templates (which have to last until all jobs are done)
we put them in templates/tmp, with content-based filenames
to economize.
- When creating jobs, or generating SQL strings for multiple jobs,
use these names as --wu_template_filename
and --result_template_filename args to create_work
(either cmdline args or stdin args)
- Delete WU templates when done
create_work.cpp:
handle per-job --wu_template and --result_template args in stdin job lines
(the names of per-job WU and result templates).
Maintain a map mapping WU template name to contents,
to avoid repeatedly reading them.
For jobs that don't specify templates, use the ones specified
at the batch level, or the defaults.
barton26