Commit Graph

32 Commits

Author SHA1 Message Date
David Anderson 7878d24f95 Add header comments to sched/*.cpp 2016-06-24 15:42:11 -07:00
David Anderson 8cd8c8e7ee server software: handle 64-bit database IDs
The SETI@home result table is about to run out of 32-bit IDs,
so we need to move to 64-bit result IDs.
This will happen to the workunit table at some point too.

I changed the server C++ code to use the "long" type for all DB IDs
(and to use appropriate conversion codes like %lu).
"long" is 64 bit on 64-bit machines.
For uniformity I did this for all tables,
even ones (like app) that will never get big.

I chose NOT to change the DB schema for now.
The new code will work with 32-bit ID fields in the DB.
As projects approach the 32-bit limit on a table they can change
its ID field, and fields that reference this table, to BIGINT.
This is likely to happen only on the result and workunit tables.
I put functions in html/ops/db_update.php
to change the IDs of these tables.
2015-07-23 10:11:08 -07:00
David Anderson 638503e1ee server: add restrict_wu_to_host() function 2015-03-30 11:32:01 -07:00
David Anderson aca1aead5f server: shuffle code so that the file upload handler doesn't need MySQL
Also (client): remove notices about app_config.xml after problem is fixed
2014-06-17 18:07:45 -07:00
David Anderson 9889ee8fb6 scheduler: enforce GPU job limits separately for each GPU type
Previously, if a project specified a limit on GPU jobs in progress,
it would be enforced across GPU types.
This could lead to starvation for hosts with multiple GPU types.
E.g. the limit is 10, and a host has 10 NVIDIA jobs and no AMD jobs.

Fix this by enforcing limits separately for each GPU type.
2014-03-08 11:17:16 -08:00
David Anderson 0d8a22e75c Server: add optional size_class parameter to count_unsent_results().
This lets you write work generators that maintain min levels of
unsent jobs for each size class.
2014-02-20 13:44:56 -08:00
David Anderson 2e4d561647 sample work generator: wait until transitioner has processed jobs before creating any more
Work generators create jobs (workunits);
the transitioner creates instances (results).
If a work generator tries to maintain a certain number of unsent results
(as the sample work generator does)
it must wait for a bit, after creating jobs,
to let the transitioner create instances of those jobs.
The example work generator waited 5 seconds.

Problem: on a heavily loaded project, the transitioner can fall behind -
minutes or hours behind.
So the above policy can create way too many jobs.

Solution: after creating jobs, the sample work generator
notes the current time X,
then waits until the transitioner catches up to time X
(i.e., until the min workunit.transition_time exceeds X).
This ensures that instances have been created for all the new jobs.

Other work generators the limit the number of unsent jobs
should use the same technique;
use min_transition_time(x) to get the min transition time.

Code cleanup: get_double should be a member of DB_CONN, not DB_BASE.
2013-12-14 16:36:18 -08:00
David Anderson ef82d5d9fb server: fix compile error on systems that don't define MAXPATHLEN 2013-08-22 17:01:45 -07:00
Eric J Korpela 03e64f720b SCHED: Added "intel_gpu" to app_plan_uses_gpu() 2013-06-25 19:31:23 -07:00
David Anderson 0c430ce1fa Add support for multi-size apps
See http://boinc.berkeley.edu/trac/wiki/MultiSize
The components of this include:
- DB changes:
    add size_class to workunit and result
    n_size_classes to app; >1 means multi-size
- size_regulator daemon program: change results states
    from INACTIVE to UNSENT carefully
- size_census program; writes quantile info in flat files
- transitioner: when creating results for multi-size apps,
    set server state to INACTIVE
- sched shmem (feeder): read quantile info from flat files,
    store in shared memory
- scheduler (score-based scheduling): for multi-size apps,
    add component to score function for size class.
- show_shmem: show result size class
- make_work (and other callers of count_unsent_results()):
    count both INACTIVE and UNSENT
- create_work: add --size_class cmdline option

Also:
- if get MySQL errors in upgrade, don't rewrite db_version
2013-04-25 00:27:35 -07:00
David Anderson 0129c1c53d - server: fix bug in restrict_wu_to_user() that caused it
to go into infinite loop


svn path=/trunk/boinc/; revision=26006
2012-08-11 04:20:48 +00:00
David Anderson d41f79588d - server daemons: add daemon_sleep(n), which sleeps for n secs
but checks for the "stop_daemons" trigger file every 1 sec.
    Use this instead of sleep() in daemons.
    This will speed up bin/stop.


svn path=/trunk/boinc/; revision=25708
2012-05-23 18:11:59 +00:00
David Anderson 32a08d27d9 - C++ code: use MAXPATHLEN for char arrays that hold paths
svn path=/trunk/boinc/; revision=25659
2012-05-09 16:11:50 +00:00
David Anderson 9c154484ee - fix many problems with validator_test
svn path=/trunk/boinc/; revision=25582
2012-04-19 08:47:38 +00:00
Wenjing Wu ccad62b912 - wrapper: when reading fraction-done file, read the last line
(or at least the last double).
    This accommodates a particular application (LAMMPS)
    that can only append to this file.
- CAS@home stuff


svn path=/trunk/boinc/; revision=25557
2012-04-13 09:44:01 +00:00
David Anderson f3df549482 - scheduler: fix a couple of assigned-job bugs
(need "where" at start of enumerate() clause!)


svn path=/trunk/boinc/; revision=25299
2012-02-20 21:54:31 +00:00
David Anderson 130d6ed4f0 - server: revamp the "assigned job" mechanism.
This now supports two main use cases:
    1) there's a job that you want to run once on all hosts,
        present and future
        (or all hosts belonging to a user, or to a team).
        The job is never transitioned, validated, or assimilated.
    2) There's a normal job for which you want to use only
        hosts belonging to a specific user (e.g. cluster or cloud hosts).
        This restriction can be made either when the job is created,
        or on the fly,
        e.g. as part of a scheme for accelerating batch completion.
        For the latter purposes we now provide a function
            restrict_wu_to_user(DB_WORKUNIT&, int userid);

        The job goes through the standard
        transitioner/validator/assimilator path.

    These cases are enabled by config flags
        <enable_assignment_multi/>
        <enable_assignment/>
    respectively.

    Assignment of type 2) are no longer stored in shared mem,
    so there is no limit on their number.

    There is no longer a rule that assigned job names must contain "asgn".

    NOTE: this requires a database update.


svn path=/trunk/boinc/; revision=25169
2012-01-30 22:39:13 +00:00
David Anderson dd16170fc1 - scheduler: the p_fpops value reported by clients can't be trusted.
Some credit cheats (e.g. with credit_by_runtime) can be done
    by reporting a huge value.
    Fix this by capping the value at 1.1 times the 95th percentile
    of host.p_fpops, taken over active hosts.


svn path=/trunk/boinc/; revision=25017
2012-01-09 17:35:48 +00:00
David Anderson dd93780787 - API and client: add "ncpus" field to APP_INIT_DATA.
Tells multicore apps how many cores to use.
    The --nthreads command line arg to the app is now deprecated
    though we'll keep it around for the time being.


svn path=/trunk/boinc/; revision=24708
2011-12-01 18:44:19 +00:00
David Anderson d2e5ed17cf - client: smoothed working-set size wasn't being computed correctly.
It was always just the most recent size.


svn path=/trunk/boinc/; revision=24500
2011-10-26 23:23:01 +00:00
David Anderson 8625e87285 - server: fix for EmBOINC
svn path=/trunk/boinc/; revision=22933
2011-01-20 21:32:00 +00:00
David Anderson eeab2aee92 - simulator work
- fix some indentation

svn path=/trunk/boinc/; revision=22891
2011-01-07 20:23:22 +00:00
David Anderson b169e5ab0f - server programs: print error message instead of numeric retval
in log messages

svn path=/trunk/boinc/; revision=22647
2010-11-08 17:51:57 +00:00
David Anderson b2451544e1 - server: change the following from per-host to per-(host, app version):
- daily quota mechanism
    - reliable mechanism (accelerated retries)
    - "trusted" mechanism (adaptive replication)
- scheduler: enforce host scale probation only for apps with
    host_scale_check set.
- validator: do scale probation on invalid results
    (need this in addition to error and timeout cases)
- feeder: update app version scales every 10 min, not 10 sec
- back-end apps: support --foo as well as -foo for options

Notes:
- If you have, say, cuda, cuda23 and cuda_fermi plan classes,
    a host will have separate quotas for each one.
    That means it could error out on 100 jobs for cuda_fermi,
    and when its quota goes to zero,
    error out on 100 jobs for cuda23, etc.
    This is intentional; there may be cases where one version
    works but not the others.
- host.error_rate and host.max_results_day are deprecated

TODO:
    - the values in the app table for limits on jobs in progress etc.
        should override rather than config.xml.

Implementation notes:
scheduler:
    process_request():
        read all host_app_versions for host at start;
        Compute "reliable" and "trusted" for each one.
        write modified records at end
    get_app_version():
        add "reliable_only" arg; if set, use only reliable versions
        skip over-quota versions
    Multi-pass scheduling: if have at least one reliable version,
        do a pass for jobs that need reliable,
        and use only reliable versions.
        Then clear best_app_versions cache.
    Score-based scheduling: for need-reliable jobs,
        it will pick the fastest version,
        then give a score bonus if that version happens to be reliable.
    When get back a successful result from client:
        increase daily quota
    When get back an error result from client:
        impose scale probation
        decrease daily quota if not aborted
Validator:
    when handling a WU, create a vector of HOST_APP_VERSION
        parallel to vector of RESULT.
        Pass it to assign_credit_set().
        Make copies of originals so we can update only modified ones
    update HOST_APP_VERSION error rates
Transitioner:
    decrease quota on timeout


svn path=/trunk/boinc/; revision=21181
2010-04-15 03:13:56 +00:00
David Anderson b300519444 svn path=/trunk/boinc/; revision=18825 2009-08-10 04:49:02 +00:00
David Anderson 12eb6057e5 - client, Mac: don't do res_init(). It causes a crash.
- client (Unix): if client crashes while benchmark processes are going,
    make sure they detect this and exit.
- back-end programs: remove hardwired assumptions about
    what directory they run in, and hence where config.xml is.
    E.g., daemons look for it in "..", others expect it in current dir.
    New approach: all the programs look for the project dir as follows:
    1) the environment var BOINC_PROJECT_DIR, if defined
    2) the current dir, if config.xml is there.
    3) else ".."
    This means you can run programs in either proj/bin/ or proj/,
    or (using BOINC_PROJECT_DIR) you can keep executables
    outside of the project dir.


svn path=/trunk/boinc/; revision=18042
2009-05-07 13:54:51 +00:00
David Anderson 5cf568a180 - client: don't allow coproc apps in app_info.xml.
Otherwise we'll get stuck in a loop where the client asks for CPU work,
    and the scheduler sends jobs for what it thinks is a CPU app
    but is actually a coproc app.

    Eventually we should add coproc info to the app descriptions
    send in scheduler request,
    so that you can use anonymous platform for coproc apps.
    But let's wait on this.

- scheduler: compile fix for gcc 4.4.  Fixes #854

svn path=/trunk/boinc/; revision=17502
2009-03-04 22:12:16 +00:00
Eric J. Korpela 8f3abcc835 - Added checks for net/*.h, arpa/*.h, netinet/*.h and code to figure out
which of those files to include
    - Modified MAC address check to work on some non-Linux unixes.
      (mac_address.cpp)
    - Added suggested change to "already attached to project" checking.
      (ProjectInfoPage.cpp)
    - changed includes of standard c header files to their c++ equivalents
      (i.e. replaced <stdio.h> with <cstdio>) for namespace protection.
    - replaced "using namespace std;" with more explicit "using std::function" in
      several files.
    - Fixed bug in checking whether the os is OS/2 and added conditional OS_OS2
      to the build environment. (boinc_platform.m4,configure.ac)
    - Changed build environment to not use -nostandardlibs unless we are using
      G++ and static linkage is specified. (configure.ac)
    - Added makefiles and package building files for solaris CSW package manager.
    - Fixed bug with attempting to find login name using logname. (configure.ac)
    - Added ifdef HAVE_* protection around some include files commonly found in
      sys.
    - Added support for unified binary for x86_64/i686-pc-solaris.
      (cs_platforms.cpp)
    - generate_host_cpid() now uses MAC address on non-linux unix.
      (hostinfo_network.cpp)
    - Macro BOINC_SET_COMPILE_FLAGS now doesn't check gcc only flags on non-gcc
      compilers. (boinc_set_compile_flags.m4)
    - Library compiles no longer depend upon the library extension or require
      the library to be prefixed with lib.
    - More fixes for fcgi builds.
    - Added declaration of "struct ether_addr" and ether_ntoa().  Have not yet
      implemented ether_ntoa() for machines that don't have it, or where it is
      buggy.  (unix_util.h)
    - Added FCGI::perror() which calls FCGI_perror(). (boinc_fcgi.{h,cpp})
    - Fixed library Makefiles so that all required headers get installed.


svn path=/trunk/boinc/; revision=17388
2009-02-26 00:23:23 +00:00
David Anderson 91e120b3f4 - scheduler: improve message formatting; add <debug_locality> flag
for locality scheduling messages

svn path=/trunk/boinc/; revision=16921
2009-01-15 20:23:20 +00:00
David Anderson 9bca753fd5 - scheduler, file upload handler: fix server runtime message in FCGI case
svn path=/trunk/boinc/; revision=16890
2009-01-12 23:05:49 +00:00
David Anderson 57b92fb40a - scheduler: #ifdef'd tweaks for server simulator
svn path=/trunk/boinc/; revision=16097
2008-09-30 18:21:41 +00:00
David Anderson 98cfb8d3b0 - rename .C files to .cpp so that Doxygen will work
svn path=/trunk/boinc/; revision=16069
2008-09-26 18:20:24 +00:00