boinc/checkin_notes

David A.	April 10 2002
    Initial checkin of all files.
    The system is fairly feature-complete and
    runs a number of test cases correctly.
    Currently runs only on Linux.

David A.	April 11 2002
    - Got things to compile on Solaris.
    - Pass in platform name, version number from Makefile
	(platform name comes from configure)

David A.        May 15 2002
    - Allow applications to have multiple files
        (e.g. scripts, libraries, pre- and post-processing programs).
        The app_version table now has an xml_doc field,
        which contains a list of <file_info> and <file_ref> elements.
        Note: the utility for adding applications (tools/add.C)
        still handles only single-file applications.
    - Changed IO_FILE_DESC to FILE_REF since it now represents
        an association of a file to workunit, result or application

    client
        app.C
        client_state.C,h
        cs_apps.C
        cs_scheduler.C
        main.C
        message.h (new)
        parse.C,h
        scheduler_reply.C,h
        types.C,h
        Makefile.in
    db
        db.h
        db_mysql.C
        schema.sql
    doc
        app.html (new)
        files.html
        index.html
        intro.html
        project.gif
        project.html
        project.png
        protocol.html
        result.html (new)
        work.html
    html_user
        db.inc
    sched
        handle_request.C
        server_types.C,h
    test
        test_uc.php
        init.inc
    tools
        add.C
David A.        May 23 2002
    - Change the scheduling server to reduce database accesses.
        There's now a shared-memory segment that contains
        1) the platform, app, and app_version tables in their entirety;
        2) a fixed-size set of results ready to be sent,
            and their corresponding workunit.
        This segment is initialized and maintained by a new program
        called the "feeder" (sched/feeder.C) that should run
        whenever the scheduling server is up.
        Note: the scheduler still needs to be converted to fast CGI
    - Added handy interfaces for dealing with shared memory and semaphores.
    - Changed to utilities and scripts used for testing so that all
        server-specific names (paths and URLs) come from environment
        variables instead of being hardwired in the code.
        You'll want to add these to your .tcshrc or whatever.
        See the doc/install.html for details.

    Files:
    TODO
    apps/
        Makefile.in
    client/
        app.C
    db/
        db.h
    doc/
        index.html
        install.html
        intro.html
        tools.html (removed)
        tools_other.html
        tools_work.html
        work.html
    lib/
        Makefile.in
        md5_file.C
        shmem.C,h (new)
        shmem_test.C (new)
        synch.C,h (new)
        synch_test.C (new)
    sched/
        feeder.C (new)
        Makefile.in
        handle_request.C,h
        main.C
        sched_shmem.C,h (new)
        server_types.C,h
    test/
        account.xml (deleted)
        account1.xml
        init.inc
        test_uc.php
    tools/
        add.C
        create_work.C

Hiram C.	Fri May 24 23:34:59 PDT 2002
	files db/dependencies and client/config.status and client/config.log
		removed.  These are build time files and should not belong
		in the source tree.  They confuse subsequent builds.
	files updated: configure configure.in api/Makefile.in
		apps/Makefile.in client/Makefile.in db/Makefile.in
		db/mysql.h lib/Makefile.in sched/Makefile.in
		tools/Makefile.in

	These Makefile.in changes will allow the build to be performed
	outside the source tree.  This is convenient for several
	reasons.  The number one reason is that this allows the source
	tree to remain untouched by the build and therefore clean.
	This will prevent garbage build files from being checked in
	during a 'cvs commit'

	To run a build outside the source tree, simply make an object
	directory anywhere else that you would like to work, then just
	execute the configure script in the top-level boinc directory.
		$ mkdir boincobj
		$ cd boincobj
		$ <... path to source tree ...>/boinc/configure
		$ make > make.out 2>&1 &

	Note the examples in apps/Makefile.in and sched/Makefile.in
	of how to use the top_srcdir variable to locate include files
	in the source tree outside of the current directory.  Local
	references to other object files and libraries produced by
	the build remain as local .. references.

	The change to configure.in, (and thus configure) and db/mysql.h
	allow a proper identification of where the mysql .h files live.
	The SSL location of /usr/local/mysql/include is a bit non-standard.
	Most systems will have these in /usr/local/include/mysql

Hiram C.		Sat May 25 09:20:25 PDT 2002
	Continue updating:
	Makefile.in config.guess configure configure.in api/Makefile.in
	apps/Makefile.in client/Makefile.in client/config.guess
	client/configure client/configure.in sched/Makefile.in
	tools/Makefile.in

	I have almost clean builds now on Solaris, Mac OS X,
	and UnixWare.  There appears to be a missing sched_shmem.h
	file from the source tree.

	There is still cruft in this configure system.  I will continue
	to work on it to clean it up.

David A.        May 29, 2002
    - Mostly implemented support for editing preferences,
        including project-specific preferences.
        Simplified the preferences scheme.
        Abandoned the idea of multiple preference sets per user;
        each user now has a single set of preferences.
        Got rid of the "prefs" database table;
        prefs are now just an XML and mod_time field in the user table.
        Haven't tested much yet.
    
    NOTE: these new features use the XML parsing features of PHP.
    This means that, on the BOINC web server,
    you must configure PHP with the --with-xml option,
    and you must make and install Apache.
    Also, you probably need to use a recent version of PHP;
    XML parsing was broken in PHP 4.0.4, but it works in 4.2.1.

    client/
        Makefile.in
        speed_stats.C  (fixed code formatting a bit; doesn't compile on linux)
    db/
        Makefile.in
        db.h
        db_mysql.C
        schema.sql
    doc/
        prefs.html
    html_user/
        db.inc
        prefs.inc
        prefs.php
        prefs_action.php (removed)
        prefs_add_project_action.php (new)
        prefs_add_project_form.php (new)
        prefs_delete_project.php (new)
        prefs_delete_project_confirm.php (new)
        prefs_edit.php (removed)
        prefs_edit_disk_action.php (new)
        prefs_edit_disk_form.php (new)
        prefs_edit_project_action.php (new)
        prefs_edit_project_form.php (new)
        prefs_edit_projects.php (new)
        prefs_edit_work_action.php (new)
        prefs_edit_work_form.php (new)
        test.php (removed)
    sched/
        handle_request.C
        server_types.C,h
    test/
        init.inc
        test_1sec.php
        test_concat.php
        test_dynamic.php
        test_prefs.php
        test_projects.php
        test_stderr.php
        test_uc.php
        test_uc_slow.php
    tools/
        add.C

David A.        May 29, 2002
    - forgot to add a couple of files
    sched/
        feeder.C
        sched_shmem.C,h

Hiram C.  Thu May 30 01:28:47 PDT 2002
	updating: client/configure client/configure.in
	client/net_xfer.C sched/Makefile.in sched/feeder.C

	This now builds just fine on Solaris, Linux and OpenUnix8.
	And only two tiny errors remain for a Mac OS X build.
	That being the specification of g++ in the sched/Makefile
	and statvfs() is not available on the Mac for
	the client/hostinfo_unix.C compile.  Need a substitute for
	that function on the Mac.

Eric H.  May 30, 2002
     Added safeguard to avoid removing root directory when
     BOINC_DOWNLOAD_DIR and BOINC_UPLOAD_ DIR environment
     variables aren't declared.
     test/
        init.inc

David A.    June 1, 2002
    - Ensure that preferences propagate from the server
        where they are updated to all hosts,
        and from there to the servers of other project.
        Eliminated concept of "home project".
        Not tested yet.
        Also to do: replace the "accounts.xml" file with
        a file containing preferences.
    - Changed the way the client handles preferences.
        It now saves saves the exact XML it gets from the server,
        regardless of whether it can parse it all.
    - Fixed client parsing code to reflect new names for preferences
    - Changed the way the scheduling server handles prefs in
        a client/server interaction.
        If it gets newer prefs, it updates its own DB.
        If its DB has newer prefs, it sends them to client.
    - Fixed the test scripts so that they start/stop the feeder also.
        This required adding a "ready" flag to the shared-mem structure
        so that the scheduling server can find out if the feeder hasn't
        finished initializing the structure.
        Also required adding a "-synch" cmdline option to feeder.
    - added utility program (sched/show_shmem.C) for looking at
        scheduler shared memory
    - renamed "types.*" to "client_types.*" to avoid name conflict
        with system include files

    /Makefile (removed - this is derived from Makefile.in)
    client/
        Makefile.in
        accounts.C,h
        app.C
        client_state.C,h
        client_types.C,h (new)
        cs_files.C
        cs_scheduler.C
        file_names.h
        file_xfer.h
        hostinfo_unix.C
        net_xfer.C
        prefs.C,h
        scheduler_reply.C,h
        speed_stats.C
        types.C,h (removed)
    db/
        constraints.sql
    doc/
        protocol.html
    html_user
        prefs.inc
    sched/
        feeder.C
        handle_request.C
        main.C
        sched_shmem.h
        server_types.C,h
        show_shmem.C (new)
        ss_reply_file (removed)
        ss_req_file (removed)
    test/
        account1.xml
        init.inc
        sched_reply.xml (removed)
        sched_request.xml (removed)
        test_uc.php
    tools/
        add.C

Eric Heien    June 03, 2002
    - Consolidated XML parsing functions into lib
    - Fixed compile bug on Solaris in shmem.C

    client/
        parse.C (moved to lib)
        parse.h (moved to lib)
        Makefile.in
    lib/
        parse.C (moved from client)
        parse.h (moved from client)
        shmem.C
    sched/
        parse.C (removed, uses lib version now)
        parse.h (removed, uses lib version now)
        Makefile.in

Michael Gary	June 04, 2002
    - Added server side water level functionality, now sends as many work units
	as necessary to fill the time requested. 

    sched/
	handle_request.C

Eric Heien   June 06, 2002
    - Changes and additions made to begin porting to Windows.
    - This code will compile on Windows, but is not quite fully functional.

    client/
	hostinfo_unix.C
	hostinfo_win.C (added)
	http.C
	main.C
	net_xfer.C
	prefs.C
	speed_stats.C
	test_file_xfer.C
	test_http.C
	test_net_xfer.C
	util.C
	util.h
	windows_cpp.h (added)
	win_main.C (added)
    lib/
	md5.c
	parse.C

Michael Gary	June 06, 2002
    - Added client side water level functionality.
    - Added rsc_fpops and rsc_iops to the client WORKUNIT struct, bun not yet
	functional since not initialized.
    - Test scripts to check water level functionality, including minima and
	maxima. Test scripts are based on test_prefs.php. 

    client/
	client_state.h
	cs_scheduler.C
	client_types.h
    test/
	max_water_prefs.xml (new)
	min_water_prefs.xml (new)
	normal_water_prefs.xml (new)
	test_max_water_prefs.php (new)
	test_min_water_prefs.php (new)
	test_normal_water_prefs.php (new)

Michael Gary	June 07, 2002
    - Converted scheduling server to allow for Fast CGI. Fast CGI will only be
	used if Makefile.in is replaced by Makefile.fcgi.in. Makefile.nofcgi.in
	is a copy of the current Makefile.in.

    sched/
	feeder.C
	handle_request.C
	main.C
	sched_shmem.C
	main.C
	sched_shmem.C
	server_types.C
	show_shmem.C
	Makefile.nofcgi.in (new)
	Makefile.fcgi.in (new)
    lib/
	parse.C

Eric Heien    June 07, 2002
    - Added initial functionality for passing graphics preferences between core and app

    client/
        Makefile.in
        app.C
    lib/
        gfx_interface.C
        gfx_interface.h

Michael Gary		June 08, 2002
    - Moved fast cgi Makefile.in to boinc/sched_fcgi
    - make now makes fcgi in boinc/sched_fcgi
    - Added links to existing code where necessary for making fast cgi
	scheduling server.

    configure
    Makefile.in
    sched/
	Makefile.nofcgi.in (removed)
	Makefile.fcgi.in (removed)
	Makefile.in
	parse.C (added)
	parse.h (added)
    sched_fcgi/ (added)
	Makefile.in (added)
	feeder.C (added)
	handle_request.C (added)
	handle_request.h (added)
	main.C (added)
	parse.C (added)
	parse.h (added)
	sched_shmem.C (added)
	sched_shmem.h (added)
	server_types.C (added)
	server_types.h (added)
	show_shmem.C (added)

David A     June 9 2002
    - added support for multiple URLs in a FILE_INFO
        (e.g. multiple servers from which the file can be downloaded)
    - started work on "persistent file transfer": a layer on top of
        FILE_XFER that manages restarting from failed connections,
        and that implements a give-up policy
    - added offset arguments to GET and PUT HTTP operations.
        NOTE: this will work fine for downloading files (GET)
        but we'll have to use something else for upload,
        since the standard PUT handler doesn't do offsets,
        and we need security functionality in any case.
    - added preliminary version of application API for
        communicating with core client.
    - use <file_ref> tags instead of <input_file> and <output_file>
        (makes things simpler)

    TODO
    notes
    api/
        api.C,h
    client/
        configure.in
        client_state.C
        client_types.C,h
        cs_files.C
        error_numbers.h
        file_xfer.h
        http.C,h
        pers_file_xfer.C,h
        test_http.C
    html_ops/
        db.php
    html_user/
        db.inc
    test/
        1sec_result
        account2.xml
        *_result
        *_wu
        init.inc
        laptop_prefs.xml
        test_*.php
    tools/
        add.C

David A     June 13 2002
    - added the RSAEuro library to the project,
        and some interface routines for it.

    RSAEuro/ (new)
        source/*
        demo/*
    lib/
        crypt.C,h (new)
        crypt_prog.C
        Makefile.in
    sched/
        file_upload_handler.C

Eric H       June 19, 2002
    - Removed redundant files

    sched/
        parse.C (removed)
        parse.h (removed)
        
Eric H       June 20, 2002
    - added soft link support via XML tags
    - fixed Windows ASCII/binary bug
    - Added Windows networking support
    - Fixed some bad indenting

    api/
        api.C
        api.h
    lib/
        md5_file.C
    client/
        client_state.C
        cs_scheduler.C
        net_xfer.C
        win_net.cpp (added)
        win_net.h (added)
        win_main.C (changed to win_main.cpp)
        win_main.cpp (changed from win_main.C)
        windows_cpp.h

David A     June 20 2002
    - Replaced the "accounts.xml" file with the user preferences ("prefs.xml").
        All non-host-specific project info is stored in this file;
        all host-specific project info is in client_state.xml.
        The PROJECT class is a union of the two.
        The logic for dealing with inconsistencies between
        prefs.xml and client_state.xml, and with updating in-memory
        and on-disk project lists in response to an update from a server,
        are a little tricky and are described in the code.
    - The prefs file can be overwritten by <preferences> in a scheduling
        server reply.  To prevent buggy servers from zeroing out
        users' project lists, the client makes sure there's at least
        one project, and backs up the old prefs.xml into a timestamped file.
    - The command-line client, if prefs.xml is absent,
        prompts the user for a project URL and authenticator,
        and creates an initial prefs.xml.
    - Each project now has a "master URL", with is its home page
        and also contains <scheduler> elements giving the URLs of
        its scheduling servers.
    - Added a class SCHEDULER_OP which encapsulates fetching and
        parsing a project's master page (if necessary),
        then making an RPC to one of its scheduling servers.
        TODO: add retry and failure logic.
    - A project can have more than one scheduling server.
        TODO: use all of them.
    - Project directories are stored in URL-encoded form.
        This allows project master URLs to have slashes, which is a necessity.
        
    client/
        Makefile.in
        accounts.C,h (deleted)
        app.C,h
        client_state.C,h
        client_types.C,h
        cs_apps.C
        cs_scheduler.C
        file_names.C,h
        main.C
        prefs.C,h
        scheduler_op.C,h (new)
        scheduler_reply.C,h (deleted)
    doc/
        project.html
    lib/
        parse.C,h
    sched/
        server_types.C
    test/
        account1.xml, account2.xml (deleted)
        init.inc
        prefs1.xml, prefs2.xml (new)
        test_*.php

David A     June 21 2002
    - top-level Makefile now compiles RSAEuro/,
        and doesn't refer to sched_fcgi
    - Added <scheduler> element to html_user/index.html,
        making it the "master file" for test project.
        This file must be placed in the directory referred to by
        http://localhost/

    Makefile.in
    html_user/
        index.html

Michael Gary	June 21 2002
    - added install to the make system to put executables
        in /usr/local/boinc
    - simplified make system for fast cgi scheduling server
    - completed implementation of high/low water system and
        testing of host.
    - checkpoints and timekeeping now implemented in api
    - timekeeping through api done in client state

    Makefile.in
    configure
    api/
        Makefile.in
        api.C
        api.h
        api_test.C
    apps/
        Makefile.in
    client/
        Makefile.in
        client_state.C
        client_state.h
        client_types.C
        client_types.h
        cs_scheduler.C
        hostinfo.C
        hostinfo.h
        app.h
        app.C
    db/
        db.h
    lib/
        Makefile.in
    sched/
        Makefile.in
        feeder.C
        handle_request.C
        main.C
        parse.C (removed)
        parse.h (removed)
        sched_shmem.C
        server_types.C
        show_shmem.C
    test/
        log_flags.php
    sched_fcgi/ (removed)

Micahel Gary	6/23/2002
    - Checkpoint timer initialized in boinc_init.
    - Test script added for api.

    api/
        api.C
        test_api.C
    test/
        init.inc
        test_api.php (new)
        ta_correct_atc (new)
        ta_correct_f (new)
        core_to_app.xml.in (new)

Michael Gary	6/25/2002
    - Maggie is now the master url for boinc.

    test/
        prefs1.html
    doc/
        test.html

Michael Gary	6/27/2002
    - Fixed a bug in add_work and added water level testing to test_uc.php

    test/
        test_uc.php
    tools/
        add_work.C

Michael Gary	6/28/2002
    - Added an explicit test for water level
    - Added a -no_time_test argument to the boinc client to stop the time tests
        from running. 

    client/
        main.C
        client_state.h
        client_state.C
    test/
        test_water.php (new)

Eric Heien 7/01/2002
    - Added S@H test scripts, added other testing functionality.
    - Added random WU generation.
    - Added concat_slow application.
    - Altered functionality of API checkpoint_completed.
  
    api/
        api.C
        api.h
        api_test.C
    apps/
        concat_slow.C (added)
        Makefile.in
        uc_slow.C
        upper_case.C
    test/
        sah_result (added)
        sah_wu (added)
        sah_test.php (added)
        init.inc
        test_uc_slow.php
    tools/
        create_work.C

Michael Gary	7/01/2002
    - Fixed fast cgi scheduling server
    - If low water mark is higher than high water mark, water marks are
        switched
    - Updated documentation

    doc/
        flow.html
        protocol.html
    client/
        client_state.C
        cs_scheduler.C
    sched/
        main.C
    test/
        prefs1.xml

Michael Gary 	7/03/2002
    - api test is now more thorough, tests time accounting and restarting
    - added app_completed function to api
    - api_test.C was moved to api_app.C
    - removed #ifdef solaris and #ifdef unix from all files

    api/
        Makefile.in
        api.C
        api.h
        api_test.C
        api_app.C (new)
    test/
        init.inc
        ta_correct_f
        test_api.php
    client/
        app.C
        hostinfo_unix.C
        configure
        configure.in
        filesys.C
    doc/
        api.html
    configure
    configure.in

David Anderson      July 4, 2002
    - Added support for upload authentication.
        This prevents bad guys from filling up data servers with trash.
        In this scheme, each <file_info> element sent from
        server to client includes a <max_nbytes> field limiting
        the size of the file, and includes a digital signature
        based on the project's "upload authentication" key pair.
        File uploads, instead of being done by PUT, are now done by POST 
        to a CGI program, "file_upload_handler".
        The request header includes the signed <file_info>,
        and the CGI program verifies the signature and enforces the size limit.

        The affected pieces of code:
        - Added a function create_keys() in PHP test scripts
            to create encryption keys.  Call it from all script.
        - Added environment var BOINC_KEY_DIR saying where keys are kept.
        - The client must maintain an exact copy of each <file_info> XML,
            and of the signature, so that it can send to upload server.
        - Added a new variant of HTTP operation, HTTP_OP_POST2.
            The existing variants all use single files for request and reply.
            The new variant (used for file upload) has a request
            consisting of a memory block followed by (part of) a file;
            the reply is in memory.
            This avoid copying possibly huge upload files.
        - FILE_XFER objects now take a FILE_INFO as initialization argument;
            needed to convey authentication info.
            The upload variant creates and sends the authentication header.
        - Result templates now include a <max_nbytes> in each
            <file_info> element, and the URLs refer to the
            file_upload_handler (with no filename)
        - process_result_template() works differently, since it must
            generate a digital signature at the end of each <file_info>
        - create_work expects the name of a private key file.
        - Added crypt/md5 functions to sign/verify in memory,
            encode/decode ASCII data in memory, checksum in memory
    - Change "gmake" to "make" in top-level makefile.
        (alias make to gmake if this is a problem)

    boinc/
        Makefile.in
        TODO
    RSAEuro/source/
        rsaeuro.h
    client/
        Makefile.in
        client_state.C
        client_types.C,h
        cs_files.C
        file_names.C
        file_xfer.C,h
        http.C,h
        main.C
        scheduler_op.C
        test_file_xfer.C
    db/
        mysql_util.C
    doc/
        index.html
        intro.html
        master_url.html (new)
        project_startup.html (new)
        tools_security.html (new)
    html_user/
        index.html
    lib/
        Makefile.in
        crypt.C,h
        crypt_prog.C
        md5_file.C,h
        parse.C,h
    sched/
        Makefile.in
        file_upload_handler.C
        server_types.C
    test/
        1sec_result
        concat_result
        init.inc
        master.html (new)
        sah_result
        test_1sec.php
        test_concat.php
        test_dynamic.php
        test_max_water_prefs.php
        test_min_water_prefs.php
        test_normal_water_prefs.php
        test_prefs.php
        test_projects.php
        test_stderr.php
        test_uc.php
        test_uc_slow.php
        uc_result
        ucs_result
    tools/
        Makefile.in
        add.C
        backend_lib.C,h
        create_work.C
        process_result_template.C

David   July 4, 2002
    - fixed compile problems on linux.
        use kill() instead of sigsend()
    
    client/app.C
    sched/file_upload_handler.C

Michael Gary	July 5, 2002
    - fixed fast cgi crypto
        use fgets and sscanf instead of fscanf, which 
        is not implemented in fcgi_stdio.h

    sched/Makefile.in
    lib/crypt.C

David A.    July 7, 2002
    - Added code signing.  All files associated with an app version
        (i.e. all executable files) must now be signed
        with the project's "code sign key".  The components include:
        - PROJECT has new field "code_sign_key", keep track of project's
            public code-signing key.
            Send this on each scheduler RPC.
        - The scheduling server stores the public code-signing key in memory.
            Send it to any client who doesn't have it.
            If the client has an old key,
            send them the new key signed with the old key.
            NOTE: this uses a scheme in which signatures are kept
            in files on the server, to avoid keeping private keys
            in online machines.  This should be documented.
        - The utility to add new app versions (add.C) adds a signature
            to the FILE_INFO element.
            NOTE: eventually the signing should be done offline.
        - FILE_INFO has a new boolean field "signature_required";
            set for all files associated with an app version
        - FILE_INFO has a new char* field "file_signature",
            which is the digital signature.
            Renamed upload authenticator from "signature" to "xml_signature"
            to avoid confusion.
        - Add function verify_downloaded_file(), called after every download;
            does MD5 or signature checking as needed.
        - Added some new functions for signature checking etc.
        - the client is now linked with the crypt library
    - Server-side errors should always include the name of the
        CGI program generating the error (scheduler or file upload handler)

    client/
        Makefile.in
        client_state.C
        client_types.C,h
        cs_files.C
        cs_scheduler.C
        file_xfer.C
        scheduler_op.C,h
        test_file_xfer.C
    doc/
        index.html
        project_startup.html (new)
        security.html
    lib/
        crypt.C,h
        parse.C,h
    sched/
        file_upload_handler.C
        handle_request.C,h
        main.C
        server_types.C,h
    test/
        init.inc
        prefs1.xml
        test_uc.php
    tools/
        add.C
        process_result_template.C

David A.    July 8, 2002
    - fix compile errors
    - fix bugs in FCGI version of scan_hex_data stuff

    client/
        client_types.C
    lib/
        crypt.C
        crypt_prog.C
    sched/
        file_upload_handler.C
    tools/
        Makefile.in
        add.C
        process_result_template.C