A "remote input file" is located on a data server other than the project server.
Previously these could be specified only in the input template,
which was of limited utility.
Add new ways of specifying remote input files:
1) in the create_work program, a remote input file can be specified
with command-line args
--remote_file name URL nbytes MD5
or by the same syntax in stdin when creating multiple jobs
2) add a variant of create_work() called create_work2(),
which takes a vector of INFILE_DESC structures that can specify
either local or remote files
The job submission RPC handler (PHP) originally ran the
create_work program once per job.
This took about 1.5 minutes to create 1000 jobs.
Recently I changed this so that create_work only is run once;
it does one SQL insert per job.
Disappointingly, this was only slightly faster: 1 min per 1000 jobs.
This commit changes create_work to create multiple jobs per SQL insert
(as many as will fit in a 1 MB query, which is the default limit).
This speeds things up by a factor of 100: 1000 jobs in 0.5 sec.
Previously if you wanted to create lots of jobs from a script (e.g. PHP)
you had to run create_work once per job.
With the --stdin option you run it once,
passing it a file (view stdin) with one line per job.
Each line can specify a command line and/or a set of input files.
On my server this gives a performance of about 1000 jobs per minute,
which is less than I would have expected,
but all the time is spent in doing MySQL inserts
so that's as good as we can do for now.
Also fix a bug in stage_file.
The docs said that putting <gzip/> for a file in your input template
would cause it to be transferred in gzip form.
But most of the server-side implementation was missing.
- in process_input_template(), parse <gzip/>,
and add <gzipped_url> elements to the output.
- stage_file was generating MD5 cache files containing only the MD5,
but process_input_template() expected them to contain file size as well.
Change stage_file to write both,
and change process_input_template() to write an error message
if it finds a bad MD5 file.
- remove stuff from process_input_template() related to
"generated_locally", a feature that doesn't exist anymore.
- add delete button
- show # of badges assigned
- don't use separate head/tail function are admin pages
- add badge assign task for new projects
- turn on db_dump, update_uotd, update_forum_activities,
update_profile_pages, notify for new projects
Add script (compare_pot) for comparing templates module comments,
and sending reminder emails if they differ.
Add script (buid_pos) for generating all templates and comparing
them with current versions.
See http://boinc.berkeley.edu/trac/wiki/MultiSize
The components of this include:
- DB changes:
add size_class to workunit and result
n_size_classes to app; >1 means multi-size
- size_regulator daemon program: change results states
from INACTIVE to UNSENT carefully
- size_census program; writes quantile info in flat files
- transitioner: when creating results for multi-size apps,
set server state to INACTIVE
- sched shmem (feeder): read quantile info from flat files,
store in shared memory
- scheduler (score-based scheduling): for multi-size apps,
add component to score function for size class.
- show_shmem: show result size class
- make_work (and other callers of count_unsent_results()):
count both INACTIVE and UNSENT
- create_work: add --size_class cmdline option
Also:
- if get MySQL errors in upgrade, don't rewrite db_version
has an invalid URL, type, or app
- server, create_work() function: if a <file_info> in input template
lists URLs, they're directories; append filename to each one
to account for systematic errors in FLOP count
- adjust_user_priority: get total project RAC by summing RAC
of app versions where RAC has been updated in past week
- feeder: add --priority_asc option
(for when wu.priority is a logical time)
David Anderson