Commit Graph

607 Commits

Author SHA1 Message Date
David Anderson 761fb3f4c1 - admin web: add a function for "revalidating" a given set of jobs.
This reruns validation for instances that are successful
    but marked as invalid or inconclusive.
    Use this if you changed your validator to be more permissive,
    and you want to grant credit for instances that were
    originally marked as invalid.


svn path=/trunk/boinc/; revision=25714
2012-05-25 23:49:17 +00:00
Bernd Machenschalk 8b5b765bb7 scheduler: get app_version info for validator items
svn path=/trunk/boinc/; revision=25658
2012-05-09 08:04:21 +00:00
David Anderson 759c23ed27 - server: create a harness for testing validator code.
If you link your functions (init_result(), compare_results(),
    cleanup_result()) with validate_test.cpp,
    you'll get a program that you can run as
        validate_test file1 file2
    and it will compare the two files
    (this works only for validators that expect 1 file per result).

    I added a makefile, sched/makefile_validator_test,
    that you can use for this.
- server: shuffle code so that the above doesn't need to
    link MySQL libraries
- client: if we fetch a master file and it contains no scheduler URLs,
    show a message of class INTERNAL_ERROR
- client/scheduler: make CUDA_DEVICE_PROP.totalGlobalMem a double,
    and remove dtotalGlobalMem.
    Although NVIDIA reports RAM size as a size_t,
    there's no reason to store it as an integer after that.


svn path=/trunk/boinc/; revision=25542
2012-04-10 00:32:35 +00:00
David Anderson 86f50ba080 - admin web: when resetting app statistics,
clear elapsed time stats as well as PFC stats


svn path=/trunk/boinc/; revision=25530
2012-04-05 11:01:38 +00:00
David Anderson c703b68090 - server: allow <db_host> to include a :port
svn path=/trunk/boinc/; revision=25405
2012-03-12 21:45:29 +00:00
David Anderson 4a50b2b2e2 - wrapper: compute final CPU time correctly for multi-process apps
- storage stuff


svn path=/trunk/boinc/; revision=25356
2012-02-29 20:58:45 +00:00
David Anderson 127e905e0d - storage stuff. Getting there.
svn path=/trunk/boinc/; revision=25355
2012-02-29 07:22:59 +00:00
David Anderson 516e5ad798 - storage stuff
svn path=/trunk/boinc/; revision=25354
2012-02-29 01:11:28 +00:00
David Anderson ce52c9cf3e - storage stuff
svn path=/trunk/boinc/; revision=25341
2012-02-24 22:55:11 +00:00
David Anderson a8f883d2fa - server: split out the "antique file deletion" feature of
file_deleter.cpp into a separate program,
    since it blocks normal file deletion while it's running.
    From Bernd.
- storage stuff


svn path=/trunk/boinc/; revision=25321
2012-02-24 03:09:56 +00:00
David Anderson 2ed1cfbbb2 - scheduler and create_work: fix bugs that caused targeted jobs
to be sent to non-targeted hosts.
    The feeder was erroneously putting targeted jobs
    in the shared mem cache.
    Changes:
    - The feeder only enumerates jobs for which
        workunit.transitioner_flags is zero.
        NOTE: this field is nonzero iff the job is assigned.
    - create_work: when creating an assigned jobs,
        set workunit.transitioner_flags appropriately


svn path=/trunk/boinc/; revision=25314
2012-02-22 22:13:08 +00:00
David Anderson c4d1229830 - scheduler: in version selection, when deciding which version is fastest,
we multiple projected FLOPS by a normal random var
    with mean 1 and stddev 0.1.
    Make the stddev configurable; in particular it can be zero.


svn path=/trunk/boinc/; revision=25311
2012-02-22 19:51:09 +00:00
David Anderson d03f697456 - storage stuff
svn path=/trunk/boinc/; revision=25307
2012-02-21 20:55:09 +00:00
David Anderson 24d386e511 - db_purge: when deleting a workunit record,
delete any assignments that refer to it


svn path=/trunk/boinc/; revision=25284
2012-02-17 18:26:36 +00:00
David Anderson 1b8d6b098d - storage stuff (work in progress)
- small code shuffle


svn path=/trunk/boinc/; revision=25274
2012-02-16 23:59:26 +00:00
David Anderson caf56b8b6b - lib: change get_mac_address() to avoid sprintf(buf, "%s...", buf);
use strcat instead
- client: don't use get_mac_address() to create host CPIDs
    (we have plenty of other info to make them unique)
- storage stuff


svn path=/trunk/boinc/; revision=25269
2012-02-16 00:08:40 +00:00
David Anderson 540a16e2f0 - transitioner: fix bug that cause invalid SQL query
svn path=/trunk/boinc/; revision=25197
2012-02-04 00:18:37 +00:00
David Anderson 480e28b54c - web: fix the user search feature
- scheduler: parse d_project_share
- scheduler: if vbox and vbox_mt are both available,
    use vbox for a 1-CPU machine


svn path=/trunk/boinc/; revision=25176
2012-02-01 03:30:14 +00:00
David Anderson 130d6ed4f0 - server: revamp the "assigned job" mechanism.
This now supports two main use cases:
    1) there's a job that you want to run once on all hosts,
        present and future
        (or all hosts belonging to a user, or to a team).
        The job is never transitioned, validated, or assimilated.
    2) There's a normal job for which you want to use only
        hosts belonging to a specific user (e.g. cluster or cloud hosts).
        This restriction can be made either when the job is created,
        or on the fly,
        e.g. as part of a scheme for accelerating batch completion.
        For the latter purposes we now provide a function
            restrict_wu_to_user(DB_WORKUNIT&, int userid);

        The job goes through the standard
        transitioner/validator/assimilator path.

    These cases are enabled by config flags
        <enable_assignment_multi/>
        <enable_assignment/>
    respectively.

    Assignment of type 2) are no longer stored in shared mem,
    so there is no limit on their number.

    There is no longer a rule that assigned job names must contain "asgn".

    NOTE: this requires a database update.


svn path=/trunk/boinc/; revision=25169
2012-01-30 22:39:13 +00:00
David Anderson 10c79a7166 - scheduler: initialize COPROC_ATI::version to zero;
avoid sending spurious "update driver" messages


svn path=/trunk/boinc/; revision=25131
2012-01-23 21:59:12 +00:00
David Anderson c05444ad1e - GUI RPC: switching to the new XML parser
(which won't parse a double as an int)
    revealed a type mismatch in FILE_TRANSFER::next_request_time
    between client and server.


svn path=/trunk/boinc/; revision=25125
2012-01-23 05:03:52 +00:00
David Anderson dd16170fc1 - scheduler: the p_fpops value reported by clients can't be trusted.
Some credit cheats (e.g. with credit_by_runtime) can be done
    by reporting a huge value.
    Fix this by capping the value at 1.1 times the 95th percentile
    of host.p_fpops, taken over active hosts.


svn path=/trunk/boinc/; revision=25017
2012-01-09 17:35:48 +00:00
David Anderson e8657adfd2 - scheduler: change vbox_mt app plan function to use 1, 2 or 3 CPUs
depending on how many the host has,
    and whether CPU VM extensions are present
    (this reflects the requirements of CernVM).


svn path=/trunk/boinc/; revision=25009
2012-01-08 01:28:39 +00:00
David Anderson 95ebb112c2 - client: for VBox apps, check stderr for "ERR_CPU_VM_EXTENSIONS_DISABLED".
If found, set HOST_INFO::p_vm_extensions_disabled,
    and pass this to the scheduler.
- scheduler (VBox app plan function) if a host has p_vm_extensions_disabled
    set, don't sent it multicore VBox jobs.

Note: if you have a host with VM extensions, and they're disabled
    in the BIOS, and you enable them, you can remove the
    <p_vm_extensions_disabled> line from client_state.xml
    and you'll be eligible to get multicore VM jobs again.


svn path=/trunk/boinc/; revision=24944
2011-12-30 09:43:58 +00:00
David Anderson 8877aa5183 - web: in GPU model list page,
look for plan classes containing "nvidia" as well as "cuda".


svn path=/trunk/boinc/; revision=24614
2011-11-16 19:47:40 +00:00
David Anderson 22a911516c - server: more fixes to DB to handle unsigned result IDs
svn path=/trunk/boinc/; revision=24563
2011-11-09 17:27:50 +00:00
David Anderson 7c201eba3f - DB: use %u when writing result IDs in SQL queries;
this is to support SETI@home, which ran out of result IDs
    and changed the DB field type to int unsigned.
    Note: eventually I'll make this change official
    and change the .h types as well.
- web: put <apps_selected> tags around <app_id> elements
    in project-specific prefs.


svn path=/trunk/boinc/; revision=24555
2011-11-09 07:41:49 +00:00
David Anderson ae04b50a71 - client: don't crash if trickle up exceeds 64KB
(this bug was introduced Sept 20)
- scheduler: truncate long trickle-ups to 256KB; don't crash


svn path=/trunk/boinc/; revision=24535
2011-11-06 06:25:48 +00:00
David Anderson 4b826b52a0 - scheduler: fix bug in the "homogeneous app version" (HAV) feature
(reported by Kevin Reed).
    The problem: cache inconsistency.
    If there are 2 results for the same WU in shared mem,
    and 2 scheduler instances get them around the same time,
    they can send them with different app versions.
    We already fixed this problem for HR by
    1) rereading the relevant WU fields while deciding
        whether to send the result
    2) doing a "careful update" of the WU field using a where clause
        to make sure it wasn't modified in the (short) interval
        since rereading it.
    I fixed the HAV problem in the same way,
    and merged the two mechanisms to combine the DB queries.

    Also:
    - The rereads are done in slow_check() (see below).
    - The careful updates are done in update_wu_on_send(),
        and this is called *before* doing careful updates on result fields.
        That way, if the WU updates fail, we don't have orphaned results.
    - already_sent_to_different_platform_careful() (sic)
        no longer does DB stuff, so it's merged with
        already_send_to_different_hr_class() (better name)

    NOTE: slow_check() is used in array scheduling only.
        Score-based scheduling uses other code,
        in which this bug is not yet fixed.
        Locality scheduling doesn't support HR or HAV at all.
        This should be unified.


svn path=/trunk/boinc/; revision=24484
2011-10-26 07:15:22 +00:00
David Anderson 3410abef6f - backend API: added function cancel_jobs(minid, maxid)
for canceling jobs
- added program cancel_jobs for canceling jobs
- DB interface: it's not an error if update_fields_noid()
    affects != 1 rows


svn path=/trunk/boinc/; revision=24413
2011-10-18 07:15:04 +00:00
David Anderson e279b59913 - Updates Linux notifications to use current libnotify.
- Fix build problems on Mac OS X using autotools
- Consistently use #if HAVE_X for platform checks,
    rather than #ifdef HAVE_X or #if defined(HAVE_X)
- In Unix build, make lots of compiler checks standard
- Fix some compile warnings

From Matt Arsenault.

Note: there are now lots of compile warnings in clientgui/ on Unix,
    mostly in WxWidgets code


svn path=/trunk/boinc/; revision=24303
2011-09-27 19:45:27 +00:00
David Anderson 75c6032043 - fix typo in DB schema
- msg tweaks in make_project


svn path=/trunk/boinc/; revision=24253
2011-09-22 03:15:21 +00:00
David Anderson 9d2ba11e09 - back end: extend the access control system for remote job submission
and other operations.
    You can now designate a user as "manager" for a particular app.
    They can then:
    - control job-submit permissions for that app
    - deprecate/undeprecate versions of the app.
    - abort jobs for that app

    You can also designate a user a manage for the project.
    They can then edit permissions and quotas,
    as well as performing the app-specific functions for all apps.

    This is described here:
    http://boinc.berkeley.edu/trac/wiki/MultiUser#Accesscontrol

    This required some changes to the DB schema.


svn path=/trunk/boinc/; revision=24250
2011-09-21 21:26:00 +00:00
David Anderson a886e0be7c - transitioner: fix bug related to new runtime_outlier field
svn path=/trunk/boinc/; revision=24228
2011-09-16 20:42:45 +00:00
David Anderson e49f945908 - Validator: allow project-specific code to mark a result
is a "runtime outlier", i.e. its runtime does
    not correspond to the job's rsc_fpops_est.
    Runtime outliers are not counted in the statistics for
    elapsed time, turnaround time, and peak FLOPs count.

    The is intended for applications like SETI@home,
    some of whose jobs finish more or less instantly
    (this happens if the data contains a lot of interference).
    If a host happens to get a bunch of these short jobs,
    its statistics will get skewed: in essence, the server
    will think that the host is extremely fast,
    and will send it too many jobs.


svn path=/trunk/boinc/; revision=24225
2011-09-16 16:43:15 +00:00
David Anderson 176b0a4327 - validator: add a --credit_from_runtime option.
This assigns credit proportional to runtime*p_fpops.
    To prevent cheating, p_fpops is capped at the 95th percentile value
    among active hosts,
    and runtime is capped at a specified limit.
    This option supports apps, like LHC's CERNvm app,
    that run for a certain amount of time and then exit.
    The CreditNew system doesn't work for such apps.
- trickle_credit:
    To prevent cheating,
    cap p_fpops at the 95th percentile value among active hosts,
    and require a limit on runtime.
- require that trickle handlers supply an initialization function


svn path=/trunk/boinc/; revision=24182
2011-09-13 21:01:42 +00:00
David Anderson b80f1525f6 - feeder: change the DB query to skip jobs for deprecated apps.
Otherwise, if you have a deprecated app with >= 200 jobs
    (200 is the query's limit)
    it could always get jobs for that app,
    and never put anything into the cache.


svn path=/trunk/boinc/; revision=24142
2011-09-07 19:57:46 +00:00
David Anderson 8c49c174c3 - preliminary stuff for mechanism where privileged users
can create apps and app versions
- crontab commands should be preceded by cd to project root


svn path=/trunk/boinc/; revision=24137
2011-09-07 06:54:44 +00:00
David Anderson 7c81d72378 - web: fix warnings in forum pages
- scheduler: when using elapsed time stats to predict runtime,
    cap the estimated FLOPS at twice the peak FLOPS;
    otherwise, if a host has received a lot of very short jobs
    recently, it will get a too-high FLOPS estimate and
    will exceed the rsc_fpops_bound limit.


svn path=/trunk/boinc/; revision=24128
2011-09-05 17:29:53 +00:00
David Anderson c5c5975b44 - Improve interface of XML_PARSER.
Add parsed_tag and is_tag to the class,
    so that parsing functions don't need to declare them
    and pass them around.
- Complete the task of using XML_PARSER as the argument
    to all parsing functions.
    (Internally, many of these functions still use the old XML parser;
    that's the next step.)


svn path=/trunk/boinc/; revision=23978
2011-08-10 17:11:08 +00:00
David Anderson 271699ea0a - server: fix typo
svn path=/trunk/boinc/; revision=23904
2011-07-30 22:42:05 +00:00
David Anderson 6e5acbbe60 - web: remote job submission:
- add fields to batch table, extend APIs accordingly
    - require that example web interface run on BOINC server
        (this makes many things easier;
        an actual remote interface would require a bit more work)


svn path=/trunk/boinc/; revision=23881
2011-07-27 06:20:48 +00:00
David Anderson 83965db576 - web: more remote job submission code. Not finished.
svn path=/trunk/boinc/; revision=23871
2011-07-25 21:45:53 +00:00
David Anderson 2177a6bd95 - server: restore fpops/intops_cumulative to RESULT
(structure, not table) for AQUA
- client, Windows: when wake up from hibernation,
    get the time before printing log msg


svn path=/trunk/boinc/; revision=23784
2011-06-29 23:00:39 +00:00
David Anderson 4403df42d8 - client: add <type> element to <exclude_gpu> config option,
in case of multiple GPU types


svn path=/trunk/boinc/; revision=23777
2011-06-25 05:13:56 +00:00
David Anderson 8a9605e48c - web: add a web-service interface for remotely submitting, querying
and controlling batches of jobs
- web: add an administrative interface for controlling
    user permissions for submitting jobs
- web: add an interface where users can view and control
    their submitted jobs
See: http://boinc.berkeley.edu/trac/wiki/RemoteJobs
This is at a functional but rough stage.


svn path=/trunk/boinc/; revision=23762
2011-06-21 22:56:15 +00:00
David Anderson 61cbe110ef - web: when creating an item in News forum,
show "Export as Notice?" checkbox, and default to off.


svn path=/trunk/boinc/; revision=23721
2011-06-14 05:42:52 +00:00
David Anderson 93add14614 - backend: use new XML parser for input template files
(so that they don't have to be 1 element/line)
    and also allow optional <input_template> root element
- fix bug in WORKUNIT DB interface


svn path=/trunk/boinc/; revision=23648
2011-06-07 04:12:49 +00:00
David Anderson 436415cfe1 - scheduler, back end: add "homogeneous app version" feature.
Lets you specify, on a per-app basis,
    that all instances should be done using the same app version.
    This is for validation in the presence of GPUs.
- scheduler: code cleanup
    - Instead of adding a bunch of non-DB fields to RESULT,
        used a derived class SCHED_DB_RESULT.
    - Instead of storing a pointer to BEST_APP_VERSION in RESULT,
        store the structure itself.
        This simplifies the memory allocation situation.
- client: condition "Got server request to delete file" messages
    on <file_xfer_debug>


svn path=/trunk/boinc/; revision=23636
2011-06-06 03:40:42 +00:00
David Anderson b6140088e3 - update_versions: trim XML strings
svn path=/trunk/boinc/; revision=23569
2011-05-21 06:22:15 +00:00