boinc/tools/submit_api_test.py

181 lines
4.8 KiB
Python
Raw Normal View History

2016-10-16 23:28:55 +00:00
# This file is part of BOINC.
# http://boinc.berkeley.edu
# Copyright (C) 2016 University of California
#
# BOINC is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License
# as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# BOINC is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with BOINC. If not, see <http://www.gnu.org/licenses/>.
# test functions for submit_api.py
2016-10-16 23:28:55 +00:00
from submit_api import *
project_url = 'http://isaac.ssl.berkeley.edu/test/'
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
# read auth from a file so we don't have to include it here
2016-10-16 23:28:55 +00:00
#
def get_auth():
with open("test_auth", "r") as f:
return (f.readline()).strip()
2016-10-19 20:20:34 +00:00
# make a batch description, to be passed to estimate_batch() or submit_batch()
#
def make_batch_desc():
2016-10-16 23:28:55 +00:00
file = FILE_DESC()
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
file.mode = 'local_staged'
file.source = 'input'
2016-10-16 23:28:55 +00:00
job = JOB_DESC()
job.files = [file]
batch = BATCH_DESC()
batch.project = project_url
2016-10-16 23:28:55 +00:00
batch.authenticator = get_auth()
batch.app_name = "uppercase"
batch.batch_name = "blah25"
2016-10-16 23:28:55 +00:00
batch.jobs = []
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
for i in range(2):
2016-10-16 23:28:55 +00:00
job.command_line = '-i %s' %(i)
if True:
2017-01-27 20:35:35 +00:00
job.wu_template = """
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
<input_template>
<file_info>
</file_info>
<workunit>
<file_ref>
<open_name>in</open_name>
</file_ref>
<target_nresults>1</target_nresults>
<min_quorum>1</min_quorum>
<credit>2</credit>
<rsc_fpops_est> 60e9 </rsc_fpops_est>
</workunit>
</input_template>
"""
if True:
2017-01-27 20:35:35 +00:00
job.result_template = """
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
<output_template>
<file_info>
<name><OUTFILE_0/></name>
<generated_locally/>
<upload_when_present/>
<max_nbytes>4000000</max_nbytes>
<url><UPLOAD_URL/></url>
</file_info>
<result>
<file_ref>
<file_name><OUTFILE_0/></file_name>
<open_name>out</open_name>
</file_ref>
</result>
</output_template>
"""
job.rsc_fpops_est = (i+1)*1e9
2016-10-16 23:28:55 +00:00
batch.jobs.append(copy.copy(job))
return batch
def test_estimate_batch():
2016-10-19 20:20:34 +00:00
batch = make_batch_desc()
2016-10-16 23:28:55 +00:00
#print batch.to_xml("submit")
r = estimate_batch(batch)
if check_error(r):
return
print 'estimated time: ', r[0].text, ' seconds'
def test_submit_batch():
2016-10-19 20:20:34 +00:00
batch = make_batch_desc()
r = submit_batch(batch)
if check_error(r):
return
print 'batch ID: ', r[0].text
2016-10-16 23:28:55 +00:00
def test_query_batches():
req = REQUEST()
req.project = project_url
2016-10-16 23:28:55 +00:00
req.authenticator = get_auth()
req.get_cpu_time = True
r = query_batches(req)
if check_error(r):
return
2016-10-16 23:28:55 +00:00
print ET.tostring(r)
def test_query_batch():
req = REQUEST()
req.project = project_url
2016-10-16 23:28:55 +00:00
req.authenticator = get_auth()
req.batch_id = 271
2016-10-16 23:28:55 +00:00
req.get_cpu_time = True
r = query_batch(req)
if check_error(r):
return
2016-10-16 23:28:55 +00:00
print ET.tostring(r)
print 'njobs: ', r.find('njobs').text
print 'fraction done: ', r.find('fraction_done').text
print 'total CPU time: ', r.find('total_cpu_time').text
# ... various other fields
print 'jobs:'
for job in r.findall('job'):
print ' id: ', job.find('id').text
print ' n_outfiles: ', job.find('n_outfiles').text
# ... various other fields
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
def test_create_batch():
req = CREATE_BATCH_REQ()
req.project = project_url
req.authenticator = get_auth()
req.app_name = 'uppercase'
req.batch_name = 'foobar'
req.expire_time = 0
r = create_batch(req)
if check_error(r):
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
return
print 'batch ID: ', r[0].text
def test_abort_batch():
req = REQUEST()
req.project = project_url
req.authenticator = get_auth()
req.batch_id = 271
Remote job submission: add support for per-job templates in submit requests This supports the TACC use case, in the jobs in a batch can use different Docker images and different input and output file signatures, none of which are known in advance. Python API binding: - A JOB_DESC object can optionally contain wu_template and result_template elements, which are the templates (the actual XML) to use for that job. Add these to the XML request message if present. - Added the same capability to the PHP binding, but not C++. - Added and debugged test cases for both languages. Also, submit_batch() can take either a batch name (in which case the batch is created) or a batch ID (in which the batch was created prior to remotely staging files). RPC handler: - in submit_batch(), check for jobs with templates specified and store them in files. For input templates (which are deleted after creating jobs) we put them in /tmp, and use a map so that if two templates are the same we use 1 file. For output templates (which have to last until all jobs are done) we put them in templates/tmp, with content-based filenames to economize. - When creating jobs, or generating SQL strings for multiple jobs, use these names as --wu_template_filename and --result_template_filename args to create_work (either cmdline args or stdin args) - Delete WU templates when done create_work.cpp: handle per-job --wu_template and --result_template args in stdin job lines (the names of per-job WU and result templates). Maintain a map mapping WU template name to contents, to avoid repeatedly reading them. For jobs that don't specify templates, use the ones specified at the batch level, or the defaults.
2017-01-21 08:24:11 +00:00
r = abort_batch(req)
if check_error(r):
return
print 'success'
def test_upload_files():
req = UPLOAD_FILES_REQ()
req.project = project_url
req.authenticator = get_auth()
req.batch_id = 283
req.local_names = ('updater.cpp', 'kill_wu.cpp')
req.boinc_names = ('dxxxb_updater.cpp', 'dxxxb_kill_wu.cpp')
r = upload_files(req)
if check_error(r):
return
print 'upload_files: success'
2016-10-16 23:28:55 +00:00
def test_query_files():
req = QUERY_FILES_REQ()
req.project = project_url
req.authenticator = get_auth()
req.batch_id = 271
req.boinc_names = ('dxxx_updater.cpp', 'dxxx_kill_wu.cpp')
r = query_files(req)
if check_error(r):
return
print 'absent files:'
for f in r[0]:
print f.text
test_upload_files()