Commit Graph

9 Commits

Author SHA1 Message Date
David Anderson 0681baad54 remote job submission: add <get_job_details> option to query_batch() 2017-02-25 15:05:14 -08:00
David Anderson ef49109a88 remote job submission: fix error if no input files
also use hashlib instead of md5 in python code to avoid deprecation warning
2017-01-29 15:19:28 -08:00
David Anderson 23a43c7c5d remove file management: fixes; add python interface to query_files 2017-01-27 14:21:03 -08:00
David Anderson 37854b5262 remote job submission, python interface: fix typo 2017-01-27 13:55:26 -08:00
David Anderson 01533dc9e4 remote file management: bug fixes
- The RPC handler was expecting the "BOINC names" to be the file MD5.
  This is no longer true; the BOINC name can be anything as long as it's unique.
- To reflect this, use <phys_name> instead of <md5> in request messages.
  This means that you'll need to update both RPC client and server software.
- BoincJobFile::insert() needed to return the insert ID
2017-01-25 16:17:42 -08:00
David Anderson 381e0caf14 Remote job submission: add support for per-job templates in submit requests
This supports the TACC use case,
in the jobs in a batch can use different Docker images
and different input and output file signatures,
none of which are known in advance.

Python API binding:
    - A JOB_DESC object can optionally contain wu_template and result_template
        elements, which are the templates (the actual XML) to use for that job.
        Add these to the XML request message if present.
    - Added the same capability to the PHP binding, but not C++.
    - Added and debugged test cases for both languages.

    Also, submit_batch() can take either a batch name (in which case
    the batch is created) or a batch ID
    (in which the batch was created prior to remotely staging files).

RPC handler:
    - in submit_batch(), check for jobs with templates specified
        and store them in files.
        For input templates (which are deleted after creating jobs)
        we put them in /tmp,
        and use a map so that if two templates are the same we use 1 file.
        For output templates (which have to last until all jobs are done)
        we put them in templates/tmp, with content-based filenames
        to economize.
    - When creating jobs, or generating SQL strings for multiple jobs,
        use these names as --wu_template_filename
        and --result_template_filename args to create_work
        (either cmdline args or stdin args)
    - Delete WU templates when done

create_work.cpp:
    handle per-job --wu_template and --result_template args in stdin job lines
    (the names of per-job WU and result templates).
    Maintain a map mapping WU template name to contents,
    to avoid repeatedly reading them.

    For jobs that don't specify templates, use the ones specified
    at the batch level, or the defaults.
2017-01-21 00:24:11 -08:00
David Anderson 799126a039 Mostly finish Python interface to remote job submission and file handling 2016-10-18 00:38:02 -07:00
David Anderson 2ce773d320 Add Python interface for remove job submission 2016-10-16 16:30:16 -07:00
David Anderson d19c366692 update CERN URL 2016-10-16 16:28:55 -07:00