Commit Graph

77 Commits

Author SHA1 Message Date
David Anderson 6205ffed08 - scheduler: add extra check for not sending homogeneous app version
jobs to anonymous platform clients
- remote job submission: add DB table for keeping track of files
2013-03-04 15:16:58 +01:00
David Anderson 6ba0dd5722 svn path=/trunk/boinc/; revision=26078 2012-09-06 22:46:02 +00:00
David Anderson d83b882851 svn path=/trunk/boinc/; revision=26077 2012-09-06 19:41:49 +00:00
David Anderson 455dd3bb95 - scheduler: fix bug in homogeneous app version.
We were using a static BEST_APP_VERSION in
    check_homogeneous_app_version(),
    and it wasn't being initialized on each call
    (e.g. its HOST_USAGE was not being cleared).


svn path=/trunk/boinc/; revision=26076
2012-09-06 19:25:05 +00:00
David Anderson d085bc4ee6 - Client/manager: there was a bug because some code was writing
"cpu" in XML, and other code was looking for "CPU".
    To fix this and prevent similar problems,
    processor type names are now encapsulated in proc_type_name_xml().
    Code should use this rather than having hard-wired names.
    Redefine: GPU_TYPE_* as macros that call proc_type_name_xml().


svn path=/trunk/boinc/; revision=25996
2012-08-08 23:09:43 +00:00
Eric J. Korpela d7b7609059 - Altered randomization factor in version selection to remove slower versions
from distributions more rapidly.


svn path=/trunk/boinc/; revision=25958
2012-08-02 23:18:51 +00:00
David Anderson 6b5bb0a3e6 - Unix build tweak
svn path=/trunk/boinc/; revision=25818
2012-06-28 06:27:37 +00:00
David Anderson 8c71f6d59a - scheduler: add support for Intel GPUs, and restructure things
to make it easier to add other GPU types in the future


svn path=/trunk/boinc/; revision=25792
2012-06-25 23:09:45 +00:00
David Anderson fd0983b991 - web: server status page should show elapsed time, not CPU time
svn path=/trunk/boinc/; revision=25785
2012-06-22 07:35:54 +00:00
Bernd Machenschalk ac24fafb4c - scheduler: moved client version check to be last of the checks
performed for a particular app version. It is not necessary
        to tell the user to upgrade the client just to suite the needs of
        a particular app version if this app version requires resources
        that the host dosn't have or didn't request work for.

        Actually I don't think it's good to tell the user he needs to
        upgrade the client if there is only one particular app version
        that requires a more recent one than he has. I think that the
        purpose of the g_wreq->outdated_client flag was checking the                                                                                                                                     
        min_core_version in the project configuration. For this the                                                                                                                                      
        flag and the notice/message that it triggers is still ok. But                                                                                                                                    
        in the app version checks setting this flag leads to misleading                                                                                                                                  
        messages in most cases, so I commented that out for now.

        I'm not sure, though, that both of these measurements are needed.

svn path=/trunk/boinc/; revision=25742
2012-06-06 09:11:41 +00:00
David Anderson d2d9110584 - add script to repair damaged user.global_prefs fields
- compile warning fixes


svn path=/trunk/boinc/; revision=25707
2012-05-23 00:25:19 +00:00
David Anderson 42c909ae03 - scheduler: change ET_RATIO_LIMIT to 250
svn path=/trunk/boinc/; revision=25606
2012-04-27 06:17:59 +00:00
David Anderson 86f50ba080 - admin web: when resetting app statistics,
clear elapsed time stats as well as PFC stats


svn path=/trunk/boinc/; revision=25530
2012-04-05 11:01:38 +00:00
David Anderson fb20906923 - scheduler: ignore app_version.max_core_version if it's zero
svn path=/trunk/boinc/; revision=25505
2012-03-29 07:58:47 +00:00
David Anderson d3f2d6318f - scheduler: enforce app_version.max_core_version
svn path=/trunk/boinc/; revision=25491
2012-03-26 19:26:12 +00:00
David Anderson dc4584156d - scheduler: when choosing the best app version,
by default we skip app versions that use a resource
    for which work has not been requested.
    This is determined by the "check_req" arg to get_app_version().
    This flag is cleared whenever we want to send a job
    regardless of whether a requested resource can be used:
    namely, when resending lost jobs, and when sending assigned jobs.

    Fix a bug that could skip unrequested versions even
    when check_req is false.

    NOTES:

    1) The current semantics aren't right.
    When check_req is false, we selected the fastest of all app versions,
    including those for which no work is requested.
    Instead, we should select the fastest of the versions
    for which work is requested if there are any;
    otherwise, select the fastest version.

    2) The mechanism isn't implemented for anonymous platform.
    It should be.

    3) If we've cached an answer (including NULL) for a given
    value of check_req, that answer may be wrong for a different value.


svn path=/trunk/boinc/; revision=25342
2012-02-25 03:54:23 +00:00
David Anderson c4d1229830 - scheduler: in version selection, when deciding which version is fastest,
we multiple projected FLOPS by a normal random var
    with mean 1 and stddev 0.1.
    Make the stddev configurable; in particular it can be zero.


svn path=/trunk/boinc/; revision=25311
2012-02-22 19:51:09 +00:00
David Anderson dd16170fc1 - scheduler: the p_fpops value reported by clients can't be trusted.
Some credit cheats (e.g. with credit_by_runtime) can be done
    by reporting a huge value.
    Fix this by capping the value at 1.1 times the 95th percentile
    of host.p_fpops, taken over active hosts.


svn path=/trunk/boinc/; revision=25017
2012-01-09 17:35:48 +00:00
David Anderson 0777ab174a - scheduler: if using homogeneous app version and a WU is committed
to a superceded or deprecated app version, use it anyway.
    The current app version may not validate against the old one.


svn path=/trunk/boinc/; revision=24823
2011-12-17 22:11:26 +00:00
David Anderson 5c02170d5a - storage simulator: add stats for network load and fault tolerance.
- client: msg tweak
- client: minimum work buffer lower bound is 180 sec
- scheduler: in computing HOST_USAGE::project_flops for a job,
    if we don't have sufficient elapsed_time statistics
    for either the (host, app_version) or the app_version,
    use a conservative estimate (p_fpops*(#cpus+#ngpus))
    rather than the number returned by app_plan().
    This avoids "time limit exceeded" errors when the latter is way off.


svn path=/trunk/boinc/; revision=24820
2011-12-16 19:45:31 +00:00
David Anderson 1366dc1cdc - scheduler: if using homogeneous app version,
and a WU is committed to an app version that's been superceded,
    treat it as committed to the later version.


svn path=/trunk/boinc/; revision=24776
2011-12-12 22:57:58 +00:00
David Anderson 4e95e690f8 - client: tweak parameters of file xfer backoff to
reduce backoff intervals somewhat
- vboxwrapper: fix buffer size typo (from Attila)
- scheduler: fix crash if using homogeneous app version,
    and a WU is committed to an old or deprecated app version.
    From Kevin Reed.


svn path=/trunk/boinc/; revision=24775
2011-12-12 22:07:37 +00:00
David Anderson 690e18bbe3 - server: plan class names containing 'nvidia' or 'cuda'
are assumed to be for NVIDIA GPU apps;
    plan class names containing 'ati' are assumed to be for AMD GPU apps.
    Clauses for 'nvidia' were missing in a couple of places.


svn path=/trunk/boinc/; revision=24512
2011-11-03 05:26:19 +00:00
David Anderson 836e8aacf7 - scheduler: in cuda_check(), ati_check() and opencl_check()
(in sched_customize.cpp)
    the flops_scale argument is intended to express the
    GPU efficiency (actual/peak).
    Pass appropriate values.


svn path=/trunk/boinc/; revision=24405
2011-10-16 06:04:13 +00:00
David Anderson b1456cfc0b - scheduler: fix a bug that would choose app versions erroneously.
The problem: the choice of app version was based on
    the "projected FLOPS" return by estimate_flops(av).
    If usage stats exist for the host / app version,
    this returns a number X such that
    WU.rsc_fpops_est/X approximates the runtime of a job
    using the given app version..
    (If WU.rsc_fpops_est is way off, this will be correspondingly way off
    from the actual FLOPS the app version will get.)
    However, if there are no usage stats,
    it return an estimate based on host hardware speed,
    which might be 100X less.
    Hence, in some cases a new app version would never get used.

    Solution: choose app versions based on the values
    returned by the app plan functions.
    Use estimate_flops() AFTER choosing the version.
- scheduler: improve the accuracy of FLOPS estimation for GPU apps.
    The "flops_scale" argument to coproc_perf
    (which expresses the difference between peak GPU FLOPS
    and actual FLOPS) should be used to scale GPU FLOPS
    prior to calling coproc_perf(),
    rather than scaling the estimate returned by coproc_perf().
- show_shmem: show have_X_apps flags


svn path=/trunk/boinc/; revision=24385
2011-10-12 23:59:38 +00:00
David Anderson e00b080b5e - scheduler: fix crashing bug when using HR. From Kevin Reed.
svn path=/trunk/boinc/; revision=24355
2011-10-08 08:16:24 +00:00
David Anderson a4cccec2cc - scheduler: revise [21428] to include non-anonymous-platform,
and change the ratio limit from 2 to 10.


svn path=/trunk/boinc/; revision=24217
2011-09-15 06:53:01 +00:00
David Anderson 249435f0d8 - scheduler: fix crashing bug
svn path=/trunk/boinc/; revision=24139
2011-09-07 17:37:50 +00:00
David Anderson 7c81d72378 - web: fix warnings in forum pages
- scheduler: when using elapsed time stats to predict runtime,
    cap the estimated FLOPS at twice the peak FLOPS;
    otherwise, if a host has received a lot of very short jobs
    recently, it will get a too-high FLOPS estimate and
    will exceed the rsc_fpops_bound limit.


svn path=/trunk/boinc/; revision=24128
2011-09-05 17:29:53 +00:00
David Anderson 1bf54d11ff - client: send all running jobs a "reread app info" message
when global prefs change
    (they might have to suspend or resume network activity)


svn path=/trunk/boinc/; revision=24084
2011-08-30 21:34:27 +00:00
David Anderson 0059d6bf78 - scheduler: don't send user a message when there is no
app version for their platform for a particular app.
    The may be versions for other apps which don't have jobs right now.
    TODO: send a message if there are no versions of ANY app
    for any platform.
- fix makefile indentation, caused manager to not be built


svn path=/trunk/boinc/; revision=24052
2011-08-27 02:54:39 +00:00
David Anderson 436415cfe1 - scheduler, back end: add "homogeneous app version" feature.
Lets you specify, on a per-app basis,
    that all instances should be done using the same app version.
    This is for validation in the presence of GPUs.
- scheduler: code cleanup
    - Instead of adding a bunch of non-DB fields to RESULT,
        used a derived class SCHED_DB_RESULT.
    - Instead of storing a pointer to BEST_APP_VERSION in RESULT,
        store the structure itself.
        This simplifies the memory allocation situation.
- client: condition "Got server request to delete file" messages
    on <file_xfer_debug>


svn path=/trunk/boinc/; revision=23636
2011-06-06 03:40:42 +00:00
David Anderson 86205059cd - scheduler: app version FLOPS estimates were wrong
in the case where we don't have enough elapsed-time stats
    for the host/app_version.
    The right formula is (peak FLOPS)/app_version.avg_pfc


svn path=/trunk/boinc/; revision=23634
2011-06-03 19:53:52 +00:00
David Anderson 8a4c3dccf3 - scheduler: if an in-progress limit is given in config_aux.xml,
and <per_proc> is not specified, default it to false.
- scheduler: add some log messages


svn path=/trunk/boinc/; revision=23555
2011-05-17 19:11:44 +00:00
David Anderson 597320db39 - scheduler: compile fixes
svn path=/trunk/boinc/; revision=23281
2011-03-25 22:47:49 +00:00
David Anderson 18f2e90929 - client: work fetch: if the chosen project is currently uploading a file,
and an upload started in the last 5 min, don't fetch work from it.
    The goal is to merge the 2 scheduler RPCs
    (fetch work, report completed taskS) into a single RPC.
    Note: this may result in idleness in some cases.
- scheduler: if client doesn't handle plan class (pre-5.10),
    check plan-class app versions anyway,
    but only use if it's a single-CPU app.
    This allows single-CPU app versions with specific requirements
    (like SSE) to be issued to old clients.
    From Bernd Machenschalk


svn path=/trunk/boinc/; revision=22841
2010-12-13 22:58:15 +00:00
David Anderson 864ee7e3a3 - scheduler: in some cases the system may have a too-low estimate
of the performance of an app version on a host.
    It will then stop using that app version,
    so the estimate never has a chance to converge to its correct value.
    Fix: multiply by a random factor (mean 1, stddev .1)
    when comparing the FLOPS estimates of app versions.

svn path=/trunk/boinc/; revision=22837
2010-12-09 00:32:50 +00:00
David Anderson f8e2d07cf9 - scheduler: add vbox32 and vbox64 plan classes for VirtualBox apps.
svn path=/trunk/boinc/; revision=22778
2010-11-30 19:36:07 +00:00
David Anderson 40c50852f5 - scheduler: fix logic that deals with jobs that need > 2GB RAM.
My change of 1 Oct ([22440]) required that such jobs
    be processed with 64-bit apps,
    on the assumption that 32-bit apps have a 2 GB user address space limit.
    However, it turns out this limit applies only to Windows
    (kernel and user mode share the 4GB address space; each gets half).
    On Linux, the split is 3GB user / 1 GB kernel.
    On Mac OS X, user mode and kernel mode have separate address spaces,
    each of them 4 GB.


svn path=/trunk/boinc/; revision=22599
2010-10-27 22:58:16 +00:00
David Anderson 7bd620e6b5 - scheduler: instead of "app is not available for your type of computer",
say "app is not available for Microsoft Windows (98 or later) running on an Intel x86-compatible CPU" (or whatever)


svn path=/trunk/boinc/; revision=22537
2010-10-15 20:25:51 +00:00
David Anderson be14996a1e - scheduler: deal correctly with jobs that need > 2GB RAM.
Such jobs fail on 32-bit machines, even if they have sufficient RAM,
    because 32-bit OSs don't support address spaces > 2GB.

    In general, we want to support the following scenario:
    - an app has a mixture of small (< 2GB) and big (> 2GB) jobs.
    - there are app versions for both 32b and 64b platforms
    - one of the 32b versions is faster than the 64b version
        (say, it's a 32b GPU app)

    Goals:
    If the client is 32b, send it only small jobs,
        using the fast 32b version if possible
    If the client is 64b and has sufficient RAM,
        send it large jobs using the 64b version;
        send it small jobs using the fast 32b version if possible,
        else the 64b version

    Solution: extend get_app_version() so that it detects big jobs,
        and uses only 64b versions for them.
        Add a "for_64b_jobs" field to BEST_APP_VERSION
        so that we maintain a separate memoized set of
        BEST_APP_VERSIONs for big jobs.

- client: don't set report_results_immediately inappropriately

svn path=/trunk/boinc/; revision=22440
2010-10-01 19:54:09 +00:00
David Anderson 18b5b46aab - scheduler: fix "prefer_primary_platform" logic (I hope).
svn path=/trunk/boinc/; revision=22332
2010-09-09 20:01:28 +00:00
David Anderson 84679f482a - scheduler: change the "primary_platform_only" config option
to "prefer_primary_platform".
    If an app has only only 32-bit versions, use the for 64-bit clients.


svn path=/trunk/boinc/; revision=22282
2010-08-22 19:13:25 +00:00
David Anderson d79ca6a9f2 - scheduler: add <primary_platform_only> config option:
send only 64-bit app versions to 64-bit hosts 
    (the default is to send whatever app version is fastest)

svn path=/trunk/boinc/; revision=22183
2010-08-10 22:17:59 +00:00
David Anderson 6b8a569d6d - client/scheduler: fix a group of bugs related to the new mechanism
where the client tells the scheduler which app versions
    its queued jobs use
    (this is needed, e.g., to enforce per-app or per-resource job limits).
    In this mechanism, the client sends an array of <app_version>s,
    and each <other_result> includes an index into this array.

    - The wrong index was being sent (client).
    - If an <app_version> had a non-existent app name
        (e.g. because that app had been deprecated)
        it wasn't getting put in the array, invalidating array indices
        Furthermore, an erroneous message was being sent to the user

        Fix: if parse error for <app_version>,
        put it in the array anyway, but with cav.app = NULL,
        meaning that it's a place-holder.
        Send a message to user only if anon platform.

- manager: increase notice buffers to 64K

svn path=/trunk/boinc/; revision=22052
2010-07-23 17:43:20 +00:00
David Anderson faab0991f7 - scheduler: fix and restore fpops scaling for anonymous platform jobs
svn path=/trunk/boinc/; revision=21962
2010-07-15 21:38:24 +00:00
David Anderson 55e0e86c90 - scheduler: make messages translatable
svn path=/trunk/boinc/; revision=21896
2010-07-13 02:49:35 +00:00
David Anderson e53e9710e8 - scheduler: make some "notice"-priority messages translatable
- scheduler: add a clause to wu_is_infeasible_custom() for SETI@home:
    don't process VLAR jobs using CUDA apps.
    Note: this is implemented in a slightly non-optimal way.
    If the request asks for both GPU and CPU jobs,
    the scheduler will first decide to use the GPU version.
    It will scan jobs, skipping over VLAR jobs.
    When the GPU request is satisfied, it will switch to the CPU version
    and continue scanning, accepting VLAR jobs.
    But the jobs that were skipped initially won't be rescanned.
    Also, it would be slightly nice to preferentially send
    VLAR jobs to hosts asking for CPU work.
    (This could be done in the scoring function).

svn path=/trunk/boinc/; revision=21895
2010-07-12 22:43:53 +00:00
David Anderson b1851ce02c - user web: PHP 5.3 compatibility fix, from Nicolas. Fixes #787
svn path=/trunk/boinc/; revision=21878
2010-07-06 23:31:26 +00:00
David Anderson d756994bda - scheduler and back end: message tweaks and fixes
svn path=/trunk/boinc/; revision=21835
2010-06-29 03:20:19 +00:00