A "remote input file" is located on a data server other than the project server.
Previously these could be specified only in the input template,
which was of limited utility.
Add new ways of specifying remote input files:
1) in the create_work program, a remote input file can be specified
with command-line args
--remote_file name URL nbytes MD5
or by the same syntax in stdin when creating multiple jobs
2) add a variant of create_work() called create_work2(),
which takes a vector of INFILE_DESC structures that can specify
either local or remote files
The job submission RPC handler (PHP) originally ran the
create_work program once per job.
This took about 1.5 minutes to create 1000 jobs.
Recently I changed this so that create_work only is run once;
it does one SQL insert per job.
Disappointingly, this was only slightly faster: 1 min per 1000 jobs.
This commit changes create_work to create multiple jobs per SQL insert
(as many as will fit in a 1 MB query, which is the default limit).
This speeds things up by a factor of 100: 1000 jobs in 0.5 sec.
Previously if you wanted to create lots of jobs from a script (e.g. PHP)
you had to run create_work once per job.
With the --stdin option you run it once,
passing it a file (view stdin) with one line per job.
Each line can specify a command line and/or a set of input files.
On my server this gives a performance of about 1000 jobs per minute,
which is less than I would have expected,
but all the time is spent in doing MySQL inserts
so that's as good as we can do for now.
Also fix a bug in stage_file.
The docs said that putting <gzip/> for a file in your input template
would cause it to be transferred in gzip form.
But most of the server-side implementation was missing.
- in process_input_template(), parse <gzip/>,
and add <gzipped_url> elements to the output.
- stage_file was generating MD5 cache files containing only the MD5,
but process_input_template() expected them to contain file size as well.
Change stage_file to write both,
and change process_input_template() to write an error message
if it finds a bad MD5 file.
- remove stuff from process_input_template() related to
"generated_locally", a feature that doesn't exist anymore.
- add delete button
- show # of badges assigned
- don't use separate head/tail function are admin pages
- add badge assign task for new projects
- turn on db_dump, update_uotd, update_forum_activities,
update_profile_pages, notify for new projects
Add script (compare_pot) for comparing templates module comments,
and sending reminder emails if they differ.
Add script (buid_pos) for generating all templates and comparing
them with current versions.
See http://boinc.berkeley.edu/trac/wiki/MultiSize
The components of this include:
- DB changes:
add size_class to workunit and result
n_size_classes to app; >1 means multi-size
- size_regulator daemon program: change results states
from INACTIVE to UNSENT carefully
- size_census program; writes quantile info in flat files
- transitioner: when creating results for multi-size apps,
set server state to INACTIVE
- sched shmem (feeder): read quantile info from flat files,
store in shared memory
- scheduler (score-based scheduling): for multi-size apps,
add component to score function for size class.
- show_shmem: show result size class
- make_work (and other callers of count_unsent_results()):
count both INACTIVE and UNSENT
- create_work: add --size_class cmdline option
Also:
- if get MySQL errors in upgrade, don't rewrite db_version
has an invalid URL, type, or app
- server, create_work() function: if a <file_info> in input template
lists URLs, they're directories; append filename to each one
to account for systematic errors in FLOP count
- adjust_user_priority: get total project RAC by summing RAC
of app versions where RAC has been updated in past week
- feeder: add --priority_asc option
(for when wu.priority is a logical time)
(but not all) wasn't finished.
New logic: if the project has an NCI app then:
- make a list of NCI apps for which the client doesn't have
a job in progress.
- try to send one job for each of these apps
- do this even if no work is being requested.
- don't send jobs for NCI apps by other mechanisms
NOTE: the client logic isn't quite right for mixed NCI projects.
If there's no job for a given NCI app,
the client should do a scheduler RPC.
This isn't critical so we won't do this now.
svn path=/trunk/boinc/; revision=26068
cmdline tool for remote job submission (not done)
- remote job submission: support the 4 file modes described
in the documentation (not done)
svn path=/trunk/boinc/; revision=26067
- Fix various #include issues.
CODING STYLE LAW (minimal inclusion principle):
If foo.cpp requires <blah.h>,
#include <blah.h> in foo.cpp, NOT foo.h
svn path=/trunk/boinc/; revision=25837
(or at least the last double).
This accommodates a particular application (LAMMPS)
that can only append to this file.
- CAS@home stuff
svn path=/trunk/boinc/; revision=25557
to be sent to non-targeted hosts.
The feeder was erroneously putting targeted jobs
in the shared mem cache.
Changes:
- The feeder only enumerates jobs for which
workunit.transitioner_flags is zero.
NOTE: this field is nonzero iff the job is assigned.
- create_work: when creating an assigned jobs,
set workunit.transitioner_flags appropriately
svn path=/trunk/boinc/; revision=25314
This now supports two main use cases:
1) there's a job that you want to run once on all hosts,
present and future
(or all hosts belonging to a user, or to a team).
The job is never transitioned, validated, or assimilated.
2) There's a normal job for which you want to use only
hosts belonging to a specific user (e.g. cluster or cloud hosts).
This restriction can be made either when the job is created,
or on the fly,
e.g. as part of a scheme for accelerating batch completion.
For the latter purposes we now provide a function
restrict_wu_to_user(DB_WORKUNIT&, int userid);
The job goes through the standard
transitioner/validator/assimilator path.
These cases are enabled by config flags
<enable_assignment_multi/>
<enable_assignment/>
respectively.
Assignment of type 2) are no longer stored in shared mem,
so there is no limit on their number.
There is no longer a rule that assigned job names must contain "asgn".
NOTE: this requires a database update.
svn path=/trunk/boinc/; revision=25169
send the size of the compressed file as well.
- client: parse and write the compressed size (FILE_INFO::gzipped_nbytes).
For get_transfer GUI RPCs, if it's a compressed download send
the compressed size.
That way the manager will show the fraction done correctly.
svn path=/trunk/boinc/; revision=25152
(It turns out that the compression schemes supported by
Apache and libcurl, suprisingly, aren't restartable.)
if a <file_info> from the server contains <gzipped_url> tags,
use those instead of the <url> tags,
and flag the file as "download_gzipped".
If this is the case, download NAME.gz and save it as NAME.gzt.
When the download is complete, rename NAME.gzt to NAME.gz,
and uncompress it to NAME.
(this ensures that if NAME.gz is present, it's complete).
Also do the uncompression, if needed, in verify_file().
This ensures that the uncompression will eventually get done
even if the client quits are crashes in the middle.
- update_versions: if <gzip> is present in a <file_info>,
add a gzipped copy in the download directory
and add a <gzipped_url> elements to the app version's xml_doc.
svn path=/trunk/boinc/; revision=25112
- prompt user for DB user/passwd if DB updates needed
- error out immediately if any DB update fails
Fixes#1169, #1170
- create_work: check for duplicate <file_info>s in input template
NOTE: the input template syntax is pretty clunky
svn path=/trunk/boinc/; revision=24969
Report it (along with disk usage) in scheduler request messages.
This will allow the scheduler to send file-delete commands
if the project is using more than its share.
- client: add <disk_usage_debug> log flag
- create_work: add --help, show --command_line option
svn path=/trunk/boinc/; revision=24968
(manage_privileges and get_output.php).
From Nico Schlitter.
- change things so that
make_project --test_app
will produce input/output templates that are compatible
with the remote job submission tools
svn path=/trunk/boinc/; revision=24774
says whether the client is using account-based sandboxing.
Needed by vboxwrapper.
- web: don't show authenticator in Your Account page.
The problem with authenticators is that if a bad guys gets yours,
they can log in to your account even if you change your password.
Eventually we should not use them at all
(i.e. in cookies and sent to client).
- make_project: copy python stuff (e.g. bin/start) for web-only projects;
generate appropriate project.readme file for web-only projects.
svn path=/trunk/boinc/; revision=24447
for canceling jobs
- added program cancel_jobs for canceling jobs
- DB interface: it's not an error if update_fields_noid()
affects != 1 rows
svn path=/trunk/boinc/; revision=24413
and other operations.
You can now designate a user as "manager" for a particular app.
They can then:
- control job-submit permissions for that app
- deprecate/undeprecate versions of the app.
- abort jobs for that app
You can also designate a user a manage for the project.
They can then edit permissions and quotas,
as well as performing the app-specific functions for all apps.
This is described here:
http://boinc.berkeley.edu/trac/wiki/MultiUser#Accesscontrol
This required some changes to the DB schema.
svn path=/trunk/boinc/; revision=24250
Add parsed_tag and is_tag to the class,
so that parsing functions don't need to declare them
and pass them around.
- Complete the task of using XML_PARSER as the argument
to all parsing functions.
(Internally, many of these functions still use the old XML parser;
that's the next step.)
svn path=/trunk/boinc/; revision=23978
are not processed correctly
- remote job submission: debug
- create_work: --rsc_fpops_est etc. should override the template file
svn path=/trunk/boinc/; revision=23942
If present, "file_prefix/" is prepended to the logical names
of input and output files of jobs using that app version.
I.e. for Vbox wrapper based app versions, file_prefix is "share",
so that I/O files are put in a "share" subdirectory of the slot dir.
- update_versions: add support for
<dont_throttle>
<file_prefix>x</file_prefix>
in version.xml
svn path=/trunk/boinc/; revision=23924
<file> element in version.xml.
If present, update_versions will gzip the file in the download dir.
Use this only if you have gzip-encoding enabled in Apache.
svn path=/trunk/boinc/; revision=23915
as described here: http://boinc.berkeley.edu/trac/wiki/ClientDataModel
Compatibility: if your project is using upload signatures:
- set ignore_upload_certificates
- disable job creation
- let your job queue drain
- upgrade to new server software
- clear ignore_upload_certificates
- enable job creation
svn path=/trunk/boinc/; revision=23863
David Anderson