Commit Graph

143 Commits

Author SHA1 Message Date
Christian Beer a5dad9b895 Build: whitespace only changes 2019-04-20 13:31:13 +02:00
Christian Beer 6dbf55fd9a Tests: add unit testing framework using googletest
* enable coverage reports from gcc and disable optimizations via option to configure
* install googletest library into buildcache
* script to compile and run unit tests using cmake
* first set of unit tests originaly contributed by Keith Uplinger

There are several hardcoded paths and assumptions made in order to get this working on Travis CI. New tooling is using cmake for cross platform builds and as such is not easy to use with an autotools based system.
It's not ideal but better than nothing.
2019-04-20 13:30:03 +02:00
Bernd Machenschalk ac5d87331f scheduler: linking db_dump requires -lz 2018-11-15 14:29:03 +00:00
Bernd Machenschalk 546ff94872 db_purge: added compression type zlib
- The current db_purge pipes its output through external programs for
  compression, which leads to bad error handling and possible data loss
  (e.g. if the disk runs full). Add an option to use zlib to produce
  gz files without an extenal program / pipe.

(cherry picked from commit dd838ea53b326588af9f78cf9778b7b7a94e55db)
2017-08-13 23:37:53 +02:00
David Anderson 20d07be2b8 back end: add keyword-based component to job scheduling score.
- add DB field for storing job keywords: workunit.keywords
    add this to various DB parse/write functions
- add --keywords option to create_work for specifying job keywords
- add <keyword_sched> option in config.xml for enabling keyword score
    (it's disabled by default).
    If set, increment score for "yes" keyword matches,
    and disallow jobs with "no" matches
- in scheduler, add array job_keywords_array for parsed versions
    of job keywords (vector<int>)

also:
- use symbols instead of numbers for slow_check() return values
- parse unused fields in req message to remove unparsed-XML warnings
2017-07-22 00:48:38 -07:00
Fabrice Fontaine bf807980c1 Fix LDFLAGS for cross-compilation
Remove "-L$(libdir) -rapth $(libdr)" from LDFLAGS of boinc libraries in
Makefile.am of api, lib, sched and zlib directories to be able to
cross-compile boinc.

Indeed, libdir is not equal to the path where BOINC will be installed
but to exec_prefix/lib. The full installation path is
$(DESTDIR)/$(libdir).

To cross-compile boinc, exec_prefix will be set to the target path (for
example: /usr) and DESTDIR will be set (during make install) to the
staging or target directory on the host (for example /home/xxx/target).
The issue of adding -L$(libdir) is that it is not allowed by the
compiler, the error "unsafe header/library path used in
cross-compilation: '-L/usr/lib'" will be raised.

As a result, remove "-L$(libdir) -rapth $(libdr)" from LDFLAGS, the
default library search paths are sufficient for "standard" compilation
or can be updated manually by passing the additional search path to
LDFLAGS during the configure call for cross-compilation.

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
2017-03-04 16:46:41 +01:00
Christian Beer 7a39be8342 install sched_util_basic.h
sched_util.h includes sched_util_basic.h and both are needed to build project specific daemons
2015-10-08 15:22:49 +02:00
David Anderson ea7edd75b3 Move sched_msgs.* from sched/ to lib/
Lets SCHED_MSG_LOG be used including only -lboinc, not -lsched.
This is a tactical move; SETI@home back-end code
uses SCHED_MSG_LOG and I need to make this work without -lsched,
which pulls in MySQL libraries which I don't want.
2015-09-08 11:28:06 -07:00
David Anderson 9f3fe162d0 make_project, upgrade: copy new server programs 2014-10-23 11:57:05 -07:00
David Anderson d2488ef87f server: add script_assimilator.cpp: support for script-based assimilators.
Also fix bug in script_validator.cpp
2014-10-22 10:24:40 -07:00
David Anderson 302aa8c7aa server: add script_validator: a validator framework for Python, Perl, bash etc.
Added script_validator, a validator that invokes scripts of your choice
to check and compare results.
These can be in any language you want.
2014-10-21 22:14:57 -07:00
David Anderson aca1aead5f server: shuffle code so that the file upload handler doesn't need MySQL
Also (client): remove notices about app_config.xml after problem is fixed
2014-06-17 18:07:45 -07:00
David Anderson 52bd196c4f scheduler: fix bugs in sending non-compute-intensive jobs 2014-05-26 21:07:07 -07:00
David Anderson 834ac11661 server: add sample validator that checks for string in stderr 2014-03-18 19:12:13 -07:00
David Anderson b2e06e0704 Server: various fixes for "make install" 2013-08-24 20:36:49 -07:00
David Anderson 95d12b76e7 server: add code for extending deadlines via trickle-ups; from Christian 2013-08-23 00:34:37 -07:00
David Anderson 8e2524f55f Unix build: Makefile changes for "make install", from Steffen Moeller
"make install" followed by make_project should now work
2013-05-20 15:19:13 -07:00
David Anderson 0c430ce1fa Add support for multi-size apps
See http://boinc.berkeley.edu/trac/wiki/MultiSize
The components of this include:
- DB changes:
    add size_class to workunit and result
    n_size_classes to app; >1 means multi-size
- size_regulator daemon program: change results states
    from INACTIVE to UNSENT carefully
- size_census program; writes quantile info in flat files
- transitioner: when creating results for multi-size apps,
    set server state to INACTIVE
- sched shmem (feeder): read quantile info from flat files,
    store in shared memory
- scheduler (score-based scheduling): for multi-size apps,
    add component to score function for size class.
- show_shmem: show result size class
- make_work (and other callers of count_unsent_results()):
    count both INACTIVE and UNSENT
- create_work: add --size_class cmdline option

Also:
- if get MySQL errors in upgrade, don't rewrite db_version
2013-04-25 00:27:35 -07:00
David Anderson c9c9f2bae0 - scheduler: code shuffle; new file sched_check.cpp contains functions
that decide whether a job can be sent to a host
2013-04-09 12:19:00 -07:00
David Anderson 450c5592ae - scheduler: add feature for deleting no-longer-used sticky files.
Create a file "file_delete_regex" in your project dir.
    Each line is a regular expression.
    Any sticky file whose name matches one of the expressions is deleted.
2013-03-04 17:39:24 +01:00
David Anderson 2ded3ff67d - fix typo in GUI RPC
- check in some code for multi-user job prioritization
2013-03-04 15:23:39 +01:00
David Anderson 9a84980792 - lib: treat MINGW32 like CYGWIN32 (in 1 place - should do everywhere?)
from Oliver


svn path=/trunk/boinc/; revision=25874
2012-07-17 03:59:12 +00:00
David Anderson a16032dee5 - scheduler: add an alternative way of defining app plan functions
that uses an XML configuration file;
    see http://boinc.berkeley.edu/trac/wiki/AppPlanConfig
    From Bernd.


svn path=/trunk/boinc/; revision=25426
2012-03-14 19:53:16 +00:00
David Anderson a8f883d2fa - server: split out the "antique file deletion" feature of
file_deleter.cpp into a separate program,
    since it blocks normal file deletion while it's running.
    From Bernd.
- storage stuff


svn path=/trunk/boinc/; revision=25321
2012-02-24 03:09:56 +00:00
David Anderson 61e169f270 - server: add volunteer data archival to the build system
svn path=/trunk/boinc/; revision=25285
2012-02-17 19:16:49 +00:00
David Anderson c529c3c50d - server: rename process_wu_template() to process_input_template(),
and move it to its own file


svn path=/trunk/boinc/; revision=24107
2011-09-01 19:58:27 +00:00
David Anderson 661fec8333 - client: simplify the semantics of sticky files:
- All sticky files are reported on each scheduler RPC
    - If a scheduler reply says to delete a file, clear its sticky flag
    In particular:
    - remove the "send file list" tag in scheduler RPC replies
    - remove FILE_INFO::marked_for_delete
    - remove FILE_INFO::report_on_rpc
- remove the request_file_list program


svn path=/trunk/boinc/; revision=23431
2011-04-24 21:33:51 +00:00
David Anderson c70cd934b4 - server: rename send_file to put_file.
Factor out put_file() and get_file() functions
    so they have a C++ API as well as command-line


svn path=/trunk/boinc/; revision=23425
2011-04-24 02:00:27 +00:00
David Anderson 732866b8aa - back end: add two example trickle handlers:
trickle_credit: grants credit based on CPU time reported in msg
    trickle_echo: echoes trickle-up as a trickle-down

svn path=/trunk/boinc/; revision=23118
2011-02-27 00:10:14 +00:00
David Anderson b677f0c25e - validator: remove app and app_versions arguments from check_set().
These weren't used, and I'm not sure why they were added.
- include sched_limit.h in "make install" list

svn path=/trunk/boinc/; revision=21894
2010-07-12 21:35:05 +00:00
David Anderson cf7fb29227 - scheduler: add fine-grained "max jobs in progress" control.
You can now specify limits for specific apps,
    and/or for the project as a whole.
    Within each of these, you can specify limits on
    CPU jobs, GPU jobs, or total jobs.
    In the case of CPU and GPU limits, you can specify
    whether the limit should be scaled by the number of devices.

    Note: the enforcement of this is done in get_app_version(),
    since per-resource-type limits may dictate what app versions
    we can use for a particular job.

svn path=/trunk/boinc/; revision=21674
2010-06-01 23:41:07 +00:00
David Anderson 1764c6f7d4 - scheduler: preliminary checkin of new limit code
svn path=/trunk/boinc/; revision=21621
2010-05-24 23:14:48 +00:00
David Anderson fb851311e0 - server: various changes;
see http://boinc.berkeley.edu/trac/wiki/CreditNew

    Projects will need to update DB and recompile all back-end programs.

    Summary:
    - new way of computing credit
    - "reliable host" mechanism is per app version
    - "host punishment" mechanism is per app version
    - adjustment of wu.rsc_fpops_est provides the
        equivalent of per app version DCF
    - max jobs in progress is now per app
    - max jobs per RPC is now per app

    TODO:
    - reliable mechanism:
        - populate and use host_app_version.error_rate
        - populate host_app_version.turnaround
    - host punishment:
        - populate host_app_version.max_jobs_per_day
        - populate host_app_version.n_jobs_today
        - use app.max_jobs_per_day_init
    - job limits:
        - use app.max_jobs_in_progress, max_gpu_jobs_in_progress
        - use app.max_jobs_per_rpc
    - adjust wu.rsc_fpops_est
    - remove old credit stuff
        fpops_cumulative, credit_multiplier
        credit computation in scheduler

- AVERAGE class: use the Knuth algorithm (Wikipedia)


svn path=/trunk/boinc/; revision=21021
2010-03-29 22:28:20 +00:00
David Anderson 295d4b54ea - server: major improvements to locality scheduling from Einstein@home.
Triggering the work generator is now done via the DB
    instead of flat files.

    Since only E@h uses locality scheduling,
    I kept the DB changes in a separate file (db/schema_locality.sql).
    There's a new field in the workunit table,
    and that's a required update (in db_update.php)
- manager: compile fix


svn path=/trunk/boinc/; revision=20807
2010-03-05 22:55:16 +00:00
David Anderson 67bc7dfc9b - unix: build fixes
svn path=/trunk/boinc/; revision=19755
2009-12-01 17:04:28 +00:00
David Anderson 8caa2cf3d5 - test code for new credit system
svn path=/trunk/boinc/; revision=19462
2009-11-04 21:23:56 +00:00
Eric J. Korpela 208956257a - Moved credit.cpp into libboinc_sched where it should have gone in the first
place.
- Added a separate GPU memory requirement for the CUDA23 plan


svn path=/trunk/boinc/; revision=18887
2009-08-21 02:12:31 +00:00
David Anderson fb443e5c31 - compile fixes
svn path=/trunk/boinc/; revision=18832
2009-08-13 03:35:26 +00:00
David Anderson a525453b5e - code shuffling
svn path=/trunk/boinc/; revision=18826
2009-08-10 04:56:46 +00:00
David Anderson b300519444 svn path=/trunk/boinc/; revision=18825 2009-08-10 04:49:02 +00:00
David Anderson e3363c7eb8 - scheduler: on second thought, it would be better to add the above
feature without requiring use of score-based scheduling.
    So add a new customizable function, wu_is_infeasible_custom(),
    where projects can put job-specific checks.

    Also, move customizable functions (of which there are now 4)
    to a new file, sched_customize.cpp.

svn path=/trunk/boinc/; revision=18767
2009-07-29 18:55:50 +00:00
David Anderson cc47b6238f - server: improve the Python assimilator framework;
add an assimilator for PyMW (Python master/worker);
    remove old C++ PyMW assimilator
    (from Jeremy Cowles)

svn path=/trunk/boinc/; revision=18389
2009-06-12 03:06:01 +00:00
David Anderson 65d52137ec - server: Tools that should run by hand are installed under $(bindir).
Daemons that are managed by 'start' go to $(libexecdir)/sched.
    The (F)CGI stuff goes to $(libexecdir)/cgi-bin.
    Finally, example applications go under $(libexecdir)/examples.

svn path=/trunk/boinc/; revision=18378
2009-06-11 16:11:45 +00:00
David Anderson 84afd18450 - scheduler: move app-version selection and score-based scheduling
to new files.

svn path=/trunk/boinc/; revision=17630
2009-03-19 16:35:35 +00:00
Eric J. Korpela 8f3abcc835 - Added checks for net/*.h, arpa/*.h, netinet/*.h and code to figure out
which of those files to include
    - Modified MAC address check to work on some non-Linux unixes.
      (mac_address.cpp)
    - Added suggested change to "already attached to project" checking.
      (ProjectInfoPage.cpp)
    - changed includes of standard c header files to their c++ equivalents
      (i.e. replaced <stdio.h> with <cstdio>) for namespace protection.
    - replaced "using namespace std;" with more explicit "using std::function" in
      several files.
    - Fixed bug in checking whether the os is OS/2 and added conditional OS_OS2
      to the build environment. (boinc_platform.m4,configure.ac)
    - Changed build environment to not use -nostandardlibs unless we are using
      G++ and static linkage is specified. (configure.ac)
    - Added makefiles and package building files for solaris CSW package manager.
    - Fixed bug with attempting to find login name using logname. (configure.ac)
    - Added ifdef HAVE_* protection around some include files commonly found in
      sys.
    - Added support for unified binary for x86_64/i686-pc-solaris.
      (cs_platforms.cpp)
    - generate_host_cpid() now uses MAC address on non-linux unix.
      (hostinfo_network.cpp)
    - Macro BOINC_SET_COMPILE_FLAGS now doesn't check gcc only flags on non-gcc
      compilers. (boinc_set_compile_flags.m4)
    - Library compiles no longer depend upon the library extension or require
      the library to be prefixed with lib.
    - More fixes for fcgi builds.
    - Added declaration of "struct ether_addr" and ether_ntoa().  Have not yet
      implemented ether_ntoa() for machines that don't have it, or where it is
      buggy.  (unix_util.h)
    - Added FCGI::perror() which calls FCGI_perror(). (boinc_fcgi.{h,cpp})
    - Fixed library Makefiles so that all required headers get installed.


svn path=/trunk/boinc/; revision=17388
2009-02-26 00:23:23 +00:00
David Anderson 91e120b3f4 - scheduler: improve message formatting; add <debug_locality> flag
for locality scheduling messages

svn path=/trunk/boinc/; revision=16921
2009-01-15 20:23:20 +00:00
Eric J. Korpela 4e60ef3003 - STILL WORK TO BE DONE TO GET locale STUFF INSTALLED PROPERLY!!!
- Update to libtool 1.5.24
- build environment:  Major automake changes that I've been warning about
  for some time.
- Now uses libtool to build libraries.
- Builds separate boinc_fcgi and sched_fcgi libraries for use with 
  FCGI server components.
- New macro "BOINC_CHECK_LIB_WITH" that executes a "AC_CHECK_LIB" on
  a library only if --with-libname[=DIR] is specified on the configure
  command line.  This is to allow inclusion of libraries when the 
  ssl, gtk, wxWidgets, or other configuration is incorrect for static
  libraries.
- Added a lot of "--with-*" for some libraries that might be required for
  static builds.
- The sea directory has been moved to packages/generic.  Changes to sea
  and the associated scripts might be required to better make use of the
  staging mechanism and shared libraries.
- Fixed includes of boinc_fcgi.h in many files.
- Fixed places where FCGI_FILE needs to be used implicitly.
- Fixed missing define of _SC_PAGESIZE on hosts that define only
  _SC_PAGE_SIZE.
- Moved build of boinc_cmd (and source file) from lib to client



svn path=/trunk/boinc/; revision=16904
2009-01-13 23:06:02 +00:00
David Anderson 98cfb8d3b0 - rename .C files to .cpp so that Doxygen will work
svn path=/trunk/boinc/; revision=16069
2008-09-26 18:20:24 +00:00
Eric J. Korpela 40e243412d - Fixed fcgi builds to use an installed version of fcgi_stdio.h rather than
a modified boinc version.
    - Added new header "boinc_fcgi.h" to be used instead of "fcgi_stdio.h".
      This header defines I/O functions in the namespace FCGI rather than using
      redefined functions the way "fcgi_stdio.h" does.  This was causing a lot
      of headaches when both <cstdio> and "fcgi_stdio.h" was called.  Using
      overloaded functions fixes this problem, except when the only difference
      between functions is the return type (for example ::fopen() returns FILE*
      and FCGI::fopen() returns FCGI_FILE*).
    - Fixed some missing "#ifdef _WIN32" blocks in filesys.C



svn path=/trunk/boinc/; revision=15984
2008-09-09 19:10:42 +00:00
David Anderson 95fdff2fd3 - server: add assimilator for Python Master/Worker system,
and change setup script to copy its files

svn path=/trunk/boinc/; revision=15702
2008-07-28 22:56:39 +00:00