A result with a lot of failed uploads could overflow a 4K buffer.
Change report_result_error() so you just pass it the error message,
rather than va_args nonsense.
From PVS Studio:
V808
'preemptable_tasks' object of 'vector' type was created but was not utilized.
https://www.viva64.com/en/w/V808/print
Signed-off-by: Vitalii Koshura <lestat.de.lionkur@gmail.com>
In estimating the WSS of a job, we were using
- the observed WSS of the job itself
- if not available, the max WSS of current jobs of same app version
However, if neither is available we need a backup, namely WU.rsc_memory_bound.
Otherwise we can schedule jobs that exceed available RAM.
Review: job scheduling has 2 phases:
1) Make a list of jobs to run. Add enough jobs to use all resources.
2) Actually run the jobs and preempt existing jobs.
The problem: checking for RAM usage limits
(i.e. making sure the sum of working sets is < RAM usage prefs)
is done in 2) but not 1).
So on a 1 CPU machine we might make a run list consisting of a single job,
which turns out not to fit in available RAM,
and we end up running nothing.
Solution: when we add a job to the run list that previously
exceeded RAM limits, don't count its resource usage.
That way we'll add more jobs to the run list,
and we'll have something to run in the end.
This is for my own use in BOINC-wide teams.
It must work even if account/team creation are disabled
(as they are in the BOINC-wide teams site).
To do this, I moved the <disable_team_creation> check out of make_team()
and moved it to the existing places that call make_team().
The logic now matches that of make_user().
The following should apply to GPUs but not other coprocs (e.g. miner ASICs):
- "suspend GPUs" command in GUI
- prefs for suspending GPUs
- always removing app from memory when suspended
- Remove code that tries to keep track of available GPU RAM
and defer jobs that don't fit.
This never worked, it relied on project estimates of RAM usage,
and it's been replaced by having the app do temporary exit
if alloc fails.
- Move logic for checking for deferred jobs from CPU
to work fetch.
- Rename rsc_defer_sched to has_deferred_job,
and move it from PROJECT to RSC_PROJECT_WORK_FETCH
- tweak work_fetch_debug output
Vboxwrapper detects known buggy versions of Vbox and calls
boinc_temporary_exit().
The "Incompatible version" message appears in the task status
in the BOINC Manager, where some users may never see it.
It needs to appear as a notice, telling the user to upgrade VBox.
To do this, I added an optional argument to boinc_temporary_exit()
saying that the message should be delivered as a notice.
This is conveyed to the client by adding
a line containing "notice" to the temp exit file.
I changed the client and vboxwrapper to use this.
Scheduling: if a resource has exclusions, put all jobs in the run list;
otherwise we might fail to have a job for a GPU instance, and starve it.
Work fetch: allow work fetch from zero-share projects if the resource
has instances that are idle because of GPU exclusion
Job scheduling has 2 phases:
make_run_list(): build a sorted list of runnable jobs
enforce_run_list() go through the list and run jobs
The run list in general contains more jobs than can actually be run.
This is intentional.
There are lots of reasons why enforce_run_list() might not be able
to run a particular job, and we don't know these during make_run_list().
So we need to give enforce_run_list() a surplus of choices.
The problem: make_run_list() was accounting RAM usage of jobs in the list,
and stopping when this exceeded physical RAM.
This led to a situation where we added a bunch of GPU jobs to the list -
more than could actually be run -
and this caused too few CPU jobs to be put in the list.
Oddly, the comment at the start of cpu_sched.cpp said that RAM usage
was ignored by make_run_list(); this was not the case.
Anyway, I removed RAM accounting from make_run_list().
I forgot that the wrapper has a 1-second poll for suspend and resume,
so sub-second throttling won't work properly for wrapper apps.
Revert to a variant of the old scheme,
in which the min of the suspended and resumed periods is 1 sec.
Also, fix task start/suspend/resume log messages.
- Don't throttle GPU apps. GPU apps spend all their time in a
critical section, during which they can't be suspended.
They length of these critical sections (i.e. of GPU kernels)
may be a significant part of a second, or more,
so sub-second throttling isn't possible.
- Account elapsed time correctly when throttling is used
- Also (not related to throttling)
don't schedule a job in QUIT_PENDING or ABORT_PENDING state.
Doing so results in 2 processes in the slot dir,
and can cause the job to fail.
- allow overcommitment by > 1 CPU.
E.g. If there are two 6-CPU jobs on an 8 CPU machine, run them both.
- Prefer MT jobs to ST jobs in general.
When reorder the run list (i.e. converting "preliminary" to "final" list),
prefer job J1 to J2 if:
1) J1 is EDF and J2 isn't
2) J1 uses GPUs and J2 doesn't
3) J1 is in the middle of a timeslice and J2 isn't
4) J1 uses more CPUs than J2
5) J1's project has higher scheduling priority than J2's
... in that order.
4) is new; it replaces the function promote_multi_thread_jobs(),
which did something similar but didn't work in some cases.
Various bad things could happen when CPU throttling was used together w/ GPU apps.
Examples:
- on a multi-GPU system, several GPU tasks are assigned to the same GPU
- a suspended GPU task remains in memory (tying up its GPU resources)
while other tasks try to use the GPU.
The problem was that parts of the code assumed that suspended
GPU processes don't exist - i.e. that when a GPU task is suspended
it's always removed from memory.
This isn't true in the presence of CPU throttling.
So I made the following changes:
- When assigning GPUs to tasks, treat suspended tasks like running tasks
(i.e. reserve their GPUs)
- At the end of the CPU-scheduling logic, if there are any GPU tasks
that are suspended and not scheduled, remove them from memory,
and trigger a reschedule so we can reallocate their GPUs.
Also, a cosmetic change: in the resource usage string shown in the GUI,
include "(device X)" even if the task is suspended (i.e. because of throttling).
Also: zero out COPROC::opencl_device_indexes[] so we don't write
a garbage number to init_data.xml for non-OpenCL jobs
This can lead to starving the CPUs if there are both GPU and MT jobs.
The basic problem is that a host with GPUs will never have all its CPUs
available for MT jobs.
It should probably advertise fewer CPUs, or something.
In enforce_run_list(), don't count the RAM usage of NCI tasks.
NCI tasks run sporadically, so it doesn't make to count it;
doing so can starve regular jobs in some cases.
The basic problem: the way we assign GPU instances when creating
the "run list" is slightly different from the way we assign them
when we actually run the jobs;
the latter assigns a running job to the instance it's using,
but the former doesn't.
Solution (kludge): when building the run list,
don't reserve instances for currently running jobs.
This will result in more jobs in the run list, and avoid starvation.
For efficiency, do this only if there are exclusions for this type.
Comment: this is yet another complexity that would be eliminated
if GPU instances were modeled separately.
I wish I had time to do that.
- client emulator: change default latency bound from 1 day to 10 days
http://boinc.berkeley.edu/trac/wiki/ClientAppConfig
This lets users do the following:
1) limit the number of concurrent jobs of a given app
(e.g. for WCG apps that are I/O-intensive)
2) Specify the CPU and GPU usage parameters of GPU versions
of a given app.
Implementation notes:
- max app concurrency is enforced in 2 places:
1) when building the initial job run list
2) when enforcing the final job run list
Both are needed to avoid possible starvation.
- however, we don't enforce it during RR simulation.
Doing so could cause erroneous shortfall and work fetch.
This means, however, that work buffering will not work
as expected if you're using max concurrency.
David Anderson