Various bad things could happen when CPU throttling was used together w/ GPU apps.
Examples:
- on a multi-GPU system, several GPU tasks are assigned to the same GPU
- a suspended GPU task remains in memory (tying up its GPU resources)
while other tasks try to use the GPU.
The problem was that parts of the code assumed that suspended
GPU processes don't exist - i.e. that when a GPU task is suspended
it's always removed from memory.
This isn't true in the presence of CPU throttling.
So I made the following changes:
- When assigning GPUs to tasks, treat suspended tasks like running tasks
(i.e. reserve their GPUs)
- At the end of the CPU-scheduling logic, if there are any GPU tasks
that are suspended and not scheduled, remove them from memory,
and trigger a reschedule so we can reallocate their GPUs.
Also, a cosmetic change: in the resource usage string shown in the GUI,
include "(device X)" even if the task is suspended (i.e. because of throttling).
Also: zero out COPROC::opencl_device_indexes[] so we don't write
a garbage number to init_data.xml for non-OpenCL jobs
This can lead to starving the CPUs if there are both GPU and MT jobs.
The basic problem is that a host with GPUs will never have all its CPUs
available for MT jobs.
It should probably advertise fewer CPUs, or something.
In enforce_run_list(), don't count the RAM usage of NCI tasks.
NCI tasks run sporadically, so it doesn't make to count it;
doing so can starve regular jobs in some cases.
The basic problem: the way we assign GPU instances when creating
the "run list" is slightly different from the way we assign them
when we actually run the jobs;
the latter assigns a running job to the instance it's using,
but the former doesn't.
Solution (kludge): when building the run list,
don't reserve instances for currently running jobs.
This will result in more jobs in the run list, and avoid starvation.
For efficiency, do this only if there are exclusions for this type.
Comment: this is yet another complexity that would be eliminated
if GPU instances were modeled separately.
I wish I had time to do that.
- client emulator: change default latency bound from 1 day to 10 days
http://boinc.berkeley.edu/trac/wiki/ClientAppConfig
This lets users do the following:
1) limit the number of concurrent jobs of a given app
(e.g. for WCG apps that are I/O-intensive)
2) Specify the CPU and GPU usage parameters of GPU versions
of a given app.
Implementation notes:
- max app concurrency is enforced in 2 places:
1) when building the initial job run list
2) when enforcing the final job run list
Both are needed to avoid possible starvation.
- however, we don't enforce it during RR simulation.
Doing so could cause erroneous shortfall and work fetch.
This means, however, that work buffering will not work
as expected if you're using max concurrency.
on each request.
- client: when showing how much work a scheduler request returned,
scale by availability (as is done to show the amount of the request)
- client in account manager request, <not_started_dur> and
<in_progress_dur> are in wall time, not run time
(i.e. scale them by availability)
Note: there's some confusion in the code between runtime and wall time,
where in general wall time = runtime / availability.
New convention: let's use "runtime" for the former,
and "duration" for the latter.
svn path=/trunk/boinc/; revision=25597
not just when a result becomes ready to upload.
Fix bug where a scheduler RPC to report results is done
even though uploads are active.
- client: cpu_sched_debug enables messages about not scheduling jobs
because of insufficient RAM
svn path=/trunk/boinc/; revision=25493
- fix bug that could greatly overcommit CPUs
if there are several EDF jobs and several non-EDF GPU jobs.
- don't overcommit CPUs if any job is MT (MT means avg_ncpus > 1).
For example, on a 4-CPU machine we will run:
a 0.5-CPU GPU job and 4 1-CPU jobs
but not
a 0.5-CPU GPU job and 1 4-CPU job
svn path=/trunk/boinc/; revision=25442
The old policy avoided running an N-CPU job unless N CPUs were free.
This could result in idle CPUs for long periods; for example:
on a 4-CPU machine, suppose you have a long 1-CPU job in EDF mode,
and some 4-CPU jobs.
3 CPUs will be idle until the 1-CPU job finishes.
Furthermore, the work fetch mechanism won't try to get
jobs (possibly non-MT) from other projects,
because the RR simulation doesn't reflect the scheduling
policy's exclusion principle.
The change: schedule jobs until ncpus_used >= ncpus.
E.g. in the above situation run the 1- and 4-CPU jobs together.
In extreme cases we might run 3 1-CPU jobs and the 4-CPU job.
This will degrade the performance of the 4-CPU job,
but that's probably better than having idle CPUs.
svn path=/trunk/boinc/; revision=25312
in which the tiebreaker is MD5 of name.
That way the order is stable
(it doesn't change from one run of the client to the next)
and it doesn't grep results with similar names
(and hence for the same app).
This ordering is used for
1) the order of display in the manager
2) the job scheduler's notion of FIFO
svn path=/trunk/boinc/; revision=25300
old: RR simulation marks some jobs as missing their deadline,
and the job scheduler runs those jobs as "high priority".
problem: those generally aren't the ones we should run.
E.g. if the client has a lot of jobs from a project,
typically the ones with later deadlines are the ones
whose deadlines are missed in the simulation.
But in this case the EDF policy says we should run
the ones with earliest deadlines.
new: if a project has N deadline misses,
run its N earliest-deadline jobs,
regardless of whether they missed their deadline in the sim.
Note: this is how it used to be (as designed by John McLeod).
I attempted to improve it, and got it wrong.
svn path=/trunk/boinc/; revision=25188
Report it (along with disk usage) in scheduler request messages.
This will allow the scheduler to send file-delete commands
if the project is using more than its share.
- client: add <disk_usage_debug> log flag
- create_work: add --help, show --command_line option
svn path=/trunk/boinc/; revision=24968
work fetch (e.g. to report completed jobs)
only request work if it's the project we would have chosen
if we were fetching work.
- client: the way in which project priorities were adjusted
in work fetch to reflected currently queued work was wrong.
- client: fix bug in the way project priorities are adjusted
in RR simulator
- client emulator: if there are results in the state file
with states DOWNLOADING or UPLOADING,
change them to DOWNLOADED or UPLOADED.
Otherwise they're stuck.
svn path=/trunk/boinc/; revision=24737
If we're contacting a project to report results,
only piggyback work requests for resources for which
that project is the highest priority that may have work.
- client: compute result.not_started more efficiently
TODO: continue efficiency work. There's still some quadratic stuff
svn path=/trunk/boinc/; revision=24523
(e.g. when editing it via the Manager).
Include only the GPUs that were specified in the original cc_config.xml,
not those detected by the client.
- client: fix bug that failed to require authorization for
GUI RPCs that are supposed to be authorized
- client: report parse errors in acct_mgr_url.xml and acct_mgr_login.xml
- fix compile warnings
- user web: in sample project_specific_prefs.inc,
get app names from the DB instead of listing them in the PHP code.
svn path=/trunk/boinc/; revision=24518
so that they do what they're supposed to
(i.e. enforce resource shares)
- client: change log flag <debt_debug> to <priority_debug>
- client simulator: update REC even with large delta-t.
- client simulator: handle "no new work" apps correctly
svn path=/trunk/boinc/; revision=24429
- Job scheduling: the baseline policy is to schedule based on "project priority",
which is how much processing P should receive based on resource share
minus how much it actually has received recently.
This policy tends to run jobs from the same project together,
so we modified it by adding a priority adjustment as jobs are scheduled.
The idea is that if 2 projects have about the same priority
they should split the processors.
The problem: the adjustment was too large on hosts that are on
only a small fraction of the time,
thus tending to run 1 job from each project, regardless of priority.
Solution: make an adjustment that reflects the host's actual throughput.
See adjust_rec_sched() for details.
- Work fetch: similar situation.
We were making an adjustment based on how much work the project currently has queued,
but the adjustment drowned out the project priority,
so we'd tend to always get work from the project that has least work queued.
Solution: make a smaller adjustment (-.3 ... .3)
- client: in message announcing app start, show the plan class
- client: don't show "unrecognized XML" messages for account files.
It's typically project-specific prefs that the client doesn't know about.
svn path=/trunk/boinc/; revision=24403
- client: if an app version can't be used because the GPUs it needs
are all excluded, mark it and all its results as "coproc missing"
so that they won't be looked at in scheduling logic.
svn path=/trunk/boinc/; revision=24317
in the presence of GPU exclusions.
The problem was in the job-selection phase,
which picks enough jobs to use all devices.
It was ignoring GPU exclusions, so for example on
a 2 GPU system it could pick 2 jobs from a project
for which 1 GPU is excluded,
and as a result 1 GPU would be idle.
Solution: during job selection,
keep track of GPU usage on a per-instance basis.
Select a job only if it can run on a non-excluded GPU.
- client: in computing ncprocs_excluded (which is used in
work fetch policy) don't count exclusions of non-existent devices
svn path=/trunk/boinc/; revision=24316
- measure the available RAM of each GPU when BOINC starts up.
If this fails, set available = physical.
Show available RAM in startup messages.
- use available RAM rather than physical RAM in selecting
the "best" GPU instance
- report available RAM to the scheduler
TODO: change the scheduler to use available rather than physical
if it's reported
svn path=/trunk/boinc/; revision=24210
David Anderson