This now supports two main use cases:
1) there's a job that you want to run once on all hosts,
present and future
(or all hosts belonging to a user, or to a team).
The job is never transitioned, validated, or assimilated.
2) There's a normal job for which you want to use only
hosts belonging to a specific user (e.g. cluster or cloud hosts).
This restriction can be made either when the job is created,
or on the fly,
e.g. as part of a scheme for accelerating batch completion.
For the latter purposes we now provide a function
restrict_wu_to_user(DB_WORKUNIT&, int userid);
The job goes through the standard
transitioner/validator/assimilator path.
These cases are enabled by config flags
<enable_assignment_multi/>
<enable_assignment/>
respectively.
Assignment of type 2) are no longer stored in shared mem,
so there is no limit on their number.
There is no longer a rule that assigned job names must contain "asgn".
NOTE: this requires a database update.
svn path=/trunk/boinc/; revision=25169
If set, and a WU has nonzero batch,
it is interpreted as a user ID,
and the job will be sent only to hosts with that user ID.
Note: the use of workunit.batch is arbitrary;
we could also use workunit.opaque or other deprecated field.
svn path=/trunk/boinc/; revision=23556
for some WUs
- back end: fix the way "report grace period" is implemented
old: result.report_deadline (i.e. what's in the DB) and
the deadline sent to the client are the same.
Some confusing and incorrect logic in the transitioner
tries to provide the desired semantics.
new: result.report_deadline is the deadline sent to the client,
plus the grace period.
No logic in the transitioner is needed.
svn path=/trunk/boinc/; revision=23040
(in config.xml) to include DB name, user, and password.
- back end: add read-only replica info to SCHED_CONFIG,
so that C++ programs can use the replica
(currently only PHP code can use it)
- db_dump: use the read-only DB replica if it exists.
svn path=/trunk/boinc/; revision=22958
If set, the feeder doesn't read jobs into shmem,
and the scheduler doesn't send jobs.
Intended for use when a project wants to process
a backlog of completed jobs and not issue more.
svn path=/trunk/boinc/; revision=22601
- scheduler: add max_download_urls_per_file config option
(to limit the length of workunit.xml_doc,
which is currently capped at 64KB).
From Bernd.
svn path=/trunk/boinc/; revision=22082
This feature lets you run the BOINC client as a job on grid systems
that handle only 1-CPU jobs;
it disables various mechanisms that prevent multiple clients per host
(which is normally a bad thing).
Old:
- Run the client with a --allow_multiple_clients flag.
This tells it not to use a mutex that prevents
multiple clients per host.
- Run the project with the <multiple_clients_per_host> config flag.
This suppresses two mechanisms:
- (avoid duplicate host records)
on a scheduler request with no host ID,
looks for a host with same domain name, OS type,
and mem size, and assumes the request is from that host
- (job retry)
If we get a request that doesn't have a host ID
but does have a host CPID,
mark its in-progress results as over
NOTE: I CAN'T REMEMBER WHY WE SUPPRESS THIS;
MARK S, DO YOU REMEMBER?
Problem:
if the grid clients attach to a project that
doesn't use <multiple_clients_per_host>, bad things happen.
E.g., if there are several requests at about the same time,
most of them will fail with
"another RPC already in progress" errors.
If a project does include this flag,
it loses protection from duplicate host records.
New:
- If the client is run with --allow_multiple_clients flag,
it passes a <allow_multiple_clients> element
in scheduler requests.
- The scheduler skips the duplicate-host check on
requests that include this flag.
- There is no more <multiple_clients_per_host> scheduler option.
Note: if a project using the old mechanism upgrades to this change,
it will need to use new clients for its grid deployment.
svn path=/trunk/boinc/; revision=21839
That produced a messed-up query that assigned garbage values to:
host_app_version.turnaround_var
host_app_version.turnaround_q
host_app_version.max_jobs_per_day
host_app_version.consecutive_valid
To repair these:
- set turnaround_var and turnaround_q to zero
- if max_jobs_per_day is outside of
(0..config.daily_result_quota)
set it to config.daily_result_quota
- if consecutive_valid is outside (0..1000), set it to zero
I added a script, html/ops/repair_21812.php, that does this;
if you ran server code between [21181] and [21812], run this script.
- scheduler/transitioner: add <debug_quota> log flag
- changed the build system to always use -Wall
(if we'd done this before, this bug wouldn't have happened)
- fixed a bunch of other compile warnings
svn path=/trunk/boinc/; revision=21812
Old: back off until random time in 1st hour of next day
New: no server-dictated backoff; rely on client backoff
This is needed to let hosts recover in a reasonable amount of time
after a burst of errors.
- scheduler config: it turns out we can't put arbitrary XML in config.xml;
The Python code is set up to parse only 1 level of tags (??),
and I'm not up to the task of changing this.
So the fine-grained job limit feature [21674] needs to use
a different file, namely config_aux.xml
svn path=/trunk/boinc/; revision=21686
You can now specify limits for specific apps,
and/or for the project as a whole.
Within each of these, you can specify limits on
CPU jobs, GPU jobs, or total jobs.
In the case of CPU and GPU limits, you can specify
whether the limit should be scaled by the number of devices.
Note: the enforcement of this is done in get_app_version(),
since per-resource-type limits may dictate what app versions
we can use for a particular job.
svn path=/trunk/boinc/; revision=21674
Add more info to "project in-progress job list".
Old: entries included only job name and app plan class;
this was used to resend lost jobs,
and to count the # of CPU and GPU jobs.
But it's not usable e.g. for per-app in-progress limits.
New: send the client's app versions (including usage info)
and for each in-progress job, which app version it uses.
(This reduces request-message size compared with sending
usage info and app name per job).
- client and scheduler RPC:
Add more info to "all in-progress job list", and make it optional.
This list is used by schedulers that do deadline checks
using EDF workload simulation.
Old: the list is always sent, and it contains no info
about job resource usage
New: the list is sent only if the scheduler asked for it
in a previous reply,
and each entry now contains resource usage (CPU, GPUs)
Note: the scheduler's EDF simulator is outdated;
it doesn't know about GPU jobs.
But we may as well get the info in place.
svn path=/trunk/boinc/; revision=21513
- whether host is "reliable" for an app version
- whether host is eligible for single replication for an app version
- whether to use host scaling
In each case, the answer is yes if the number of
consecutive valid results is above a threshold.
This replaces existing "error rate" and "scale probation" mechanisms.
TODO: the # of consecutive valid results should also determine
a limit on jobs in progress for an app version.
Namely, if N is the threshold for host scaling, the limit should be
ndevices*(max(1, consecutive_valid - N))
The client currently doesn't supply enough
app version info to do this.
It could be approximated; that would give some protection
against cherry-picking.
- credit: more conservative formulas for combining claimed credit
among replicas.
If there are normal replicas, we use a "low average"
that weights each sample by the sum of the other samples.
Otherwise we use the min (not the average) of the approximate samples.
NOTE: a DB update is required
svn path=/trunk/boinc/; revision=21230
see http://boinc.berkeley.edu/trac/wiki/CreditNew
Projects will need to update DB and recompile all back-end programs.
Summary:
- new way of computing credit
- "reliable host" mechanism is per app version
- "host punishment" mechanism is per app version
- adjustment of wu.rsc_fpops_est provides the
equivalent of per app version DCF
- max jobs in progress is now per app
- max jobs per RPC is now per app
TODO:
- reliable mechanism:
- populate and use host_app_version.error_rate
- populate host_app_version.turnaround
- host punishment:
- populate host_app_version.max_jobs_per_day
- populate host_app_version.n_jobs_today
- use app.max_jobs_per_day_init
- job limits:
- use app.max_jobs_in_progress, max_gpu_jobs_in_progress
- use app.max_jobs_per_rpc
- adjust wu.rsc_fpops_est
- remove old credit stuff
fpops_cumulative, credit_multiplier
credit computation in scheduler
- AVERAGE class: use the Knuth algorithm (Wikipedia)
svn path=/trunk/boinc/; revision=21021
The limit on jobs in progress is now
max_wus_in_progress * NCPUS
+ max_wus_in_progress * NGPUS
where NCPUS and NGPUS reflect prefs and are capped.
Furthermore: if the client reports plan class for in-progress jobs
(see checkin of 31 May 2009)
then these limits are enforced separately;
i.e. the # of in-progress CPU jobs is <= max_wus_in_progress*NCPUS,
and the # of in-progress GPU jobs is <= max_wus_in_progress_gpu*NGPUS
- scheduler config: rename <cuda_multiplier> to <gpu_multiplier>
- scheduler: <max_wus_to_send> is now scaled by
(NCPUS + gpu_multiplier*NGPUS)
- scheduler: don't keep scanning array if !work_needed()
- scheduler: moved array-scan logic from sched_send.cpp to sched_array.cpp
- scheduler: don't say "no work available" if jobs are available
but work_needed() is initially false
svn path=/trunk/boinc/; revision=18255
limits the # of completed results handled per scheduler RPC.
This may be needed to avoid crashes due to memory allocation
failure (each reported result uses about 128KB memory).
- web: In showing result lists,
include "Validate error" results in the "Invalid" category.
(Previously they didn't appear in any category)
svn path=/trunk/boinc/; revision=18104
- client (Unix): if client crashes while benchmark processes are going,
make sure they detect this and exit.
- back-end programs: remove hardwired assumptions about
what directory they run in, and hence where config.xml is.
E.g., daemons look for it in "..", others expect it in current dir.
New approach: all the programs look for the project dir as follows:
1) the environment var BOINC_PROJECT_DIR, if defined
2) the current dir, if config.xml is there.
3) else ".."
This means you can run programs in either proj/bin/ or proj/,
or (using BOINC_PROJECT_DIR) you can keep executables
outside of the project dir.
svn path=/trunk/boinc/; revision=18042
which of those files to include
- Modified MAC address check to work on some non-Linux unixes.
(mac_address.cpp)
- Added suggested change to "already attached to project" checking.
(ProjectInfoPage.cpp)
- changed includes of standard c header files to their c++ equivalents
(i.e. replaced <stdio.h> with <cstdio>) for namespace protection.
- replaced "using namespace std;" with more explicit "using std::function" in
several files.
- Fixed bug in checking whether the os is OS/2 and added conditional OS_OS2
to the build environment. (boinc_platform.m4,configure.ac)
- Changed build environment to not use -nostandardlibs unless we are using
G++ and static linkage is specified. (configure.ac)
- Added makefiles and package building files for solaris CSW package manager.
- Fixed bug with attempting to find login name using logname. (configure.ac)
- Added ifdef HAVE_* protection around some include files commonly found in
sys.
- Added support for unified binary for x86_64/i686-pc-solaris.
(cs_platforms.cpp)
- generate_host_cpid() now uses MAC address on non-linux unix.
(hostinfo_network.cpp)
- Macro BOINC_SET_COMPILE_FLAGS now doesn't check gcc only flags on non-gcc
compilers. (boinc_set_compile_flags.m4)
- Library compiles no longer depend upon the library extension or require
the library to be prefixed with lib.
- More fixes for fcgi builds.
- Added declaration of "struct ether_addr" and ether_ntoa(). Have not yet
implemented ether_ntoa() for machines that don't have it, or where it is
buggy. (unix_util.h)
- Added FCGI::perror() which calls FCGI_perror(). (boinc_fcgi.{h,cpp})
- Fixed library Makefiles so that all required headers get installed.
svn path=/trunk/boinc/; revision=17388
and add <cuda_multiplier>.
The latter is used in calculating max jobs/day for a host;
namely, it's host.max_results_day * (NCPUS + NCUDA*cuda_multiplier).
Set it to 10 or so if you have CUDA apps.
- scheduler: don't overload effective_ncpus();
instead, add two new functions,
max_results_day_multiplier() and max_wus_in_progress_multiplier()
- scheduler: don't reduce max_results_day if we get an aborted job
(it might have been aborted by the project;
not appopriate to punish host in this case)
svn path=/trunk/boinc/; revision=16959
If set the "effective NCPUS" (which is used to scale
daily_result_quota and max_wus_in_progress)
is max'd with the # of CUDA GPUs.
svn path=/trunk/boinc/; revision=16246
- API: in APP_INIT_DATA, enclose project preferences in tags
so that it's legal XML
- scheduler: add <multiple_clients_per_host> option.
Use this if your project runs on Condor or grids
and (to use multicore machines) you're running
multiple clients per host.
This will skip the host lookup based on IP address.
svn path=/trunk/boinc/; revision=15954
- scheduler: fix bug in adaptive replication:
if send an unreplicated job to untrusted host,
set both wu.target_nresults and wu.min_quorum to app.target_nresults.
svn path=/trunk/boinc/; revision=15762
mostly so that the parse function could assume
that everything was initially zero.
However, various back-end functions pass around SCHED_CONFIG&
as an argument (also named "config").
This creates a shadow, which is always bad.
Worse is the possibility that some projects have back-end programs
that have a SCHED_CONFIG variable that's automatic,
and therefore isn't zero initially,
and therefore isn't parsing correctly.
To fix this, I changed the 2 vectors in SCHED_CONFIG into pointers,
and have the parse routine zero the structure.
I was tempted to remove the SCHED_CONFIG& args to back-end functions,
but this would have broken some projects' code.
I did, however, change the name from config to config_loc
to avoid shadowing.
Also fixed various other compiler warnings.
svn path=/trunk/boinc/; revision=15541
- config option <matchmaker> for matchmaker scheduling
- config options <mm_min_slots>, <mm_max_slots>, <job_size_matching>
to control matchmaker scheduling
- scheduler: tweaks to matchmaker scheduling from Kevin Reed
- web: fixes to alternative stylesheet from Simek
svn path=/trunk/boinc/; revision=15281
for getting info on EDF simulation;
change output from seconds to hours
- API: remove extern "C" from graphics API
(convince me it's needed before restoring)
svn path=/trunk/boinc/; revision=15148
(attempt to send big jobs to fast hosts, small jobs to slow hosts).
- have "census" compute mean/stdev of host speeds,
write it to a file perf_info.txt
- have feeder compute mean/stdev of sizes of jobs in shmem
- have feeder read perf_info.txt into shmem
- scheduler: add some debugging messages for app version selection
- Add LGPL license to a few files
- upgrade/setup scripts: copy census to bin/
svn path=/trunk/boinc/; revision=15136
Merge redundant messages.
Condition various messages on config flags.
- client (Unix) fix to CUDA detection if LD_LIBRARY_PATH is ""
svn path=/trunk/boinc/; revision=15122
- scheduler: condition lots of log file writes on config flags
(i.e. divide "debug" output into a bunch of categories, individually selectable)
svn path=/trunk/boinc/; revision=15101
- DB code: remove "is_high_priority" stuff.
- scheduler: merge find_app_version() into get_app_version().
Have the latter memoize its results (both positive and negative).
Have it call app_plan() for apps with nonempty plan_class.
- scheduler: first steps towards improved selectability of log messages.
It will eventually be like the client,
where you can select among various types of messages.
- feeder: if can't unlink the reread_db trigger file, exit
(else we'd go into an infinite loop)
svn path=/trunk/boinc/; revision=14940
David Anderson