diff --git a/BOINC-apps-(introduction).md b/BOINC-apps-(introduction).md new file mode 100644 index 0000000..756b7b0 --- /dev/null +++ b/BOINC-apps-(introduction).md @@ -0,0 +1,149 @@ +Some concepts that are needed to deploy apps with BOINC. + +## Apps and app versions + +In BOINC, 'app' has a special meaning. +It's the abstraction of a program: +something that takes inputs and produces outputs. +Each app has a unique name (like 'autodock'). + +An 'app version' is a specific executable file +(or set of files) that performs this computation. +It's associated with a particular app, +and with a plaform (Windows/Intel, Mac/Intel, Linux/ARM, etc.). + +App versions have version numbers (like 3.2). +When the BOINC server sends a job to a client, +it uses the app version for that platform +with the largest version number. + +The [server cookbook](Create-a-BOINC-server-(cookbook)) +shows how to create apps and app versions. + +## Plan classes + +In some cases info beyond just platform +is needed to decide whether an app version can run +on a particular host. +For example: + +* The executable is compiled to use instructions (like SSE3) +that exist on some Intel-type processors but not others. +* The program uses a GPU and requires a particular +range of GPU models and video driver versions. +* The program requires that virtualization software +(like VirtualBox) be installed on the host. + +A set of such requirements is called a 'plan class'. +An app version can be associated with a plan class, +in which case it will be sent only to hosts +that meet the requirements. + +There are a number of pre-defined plan classes: +* mt: multithreaded programs that use from 2 to 16 cores. +* sse3: programs compiled to use SSE3 instructions. +* vbox_64: programs that use VirtualBox, 64-bit. +* cuda: programs that use NVIDIA GPU, compute capability 1.0+ + +You can define your own plan classes in XML. +Details are [here](AppPlan). + +## Files + +### File immutability + +We're concerned here with files +that are copied between client and server +(input and output files, app version files). +Such files are 'immutable'. +All replicas of a file with a given name, from a given project, +are assumed (and required) to be identical. +If a file is change, even by a single byte, +it becomes a new file and must be given a different name. +This can be done, e.g., by including a version number in the file name. + +### Server directory structure + +On a BOINC server, files are stored in 2-level +directory hierarchies, +one for uploads, one for downloads. +Each of these has 1024 subdirectories +(named '0' to '3ff'). +A file is placed in a subdirectory based +on the MD5 has of its filename. +This was done because projects can have large numbers (millions) +of files at a given time, +and some filesystems become extremely slow +when a single directory contains that many files. + +When a job is submitted, its input files are 'staged' +by moving or copying them to the appropriate place in +the download hierarchy. +When a client completes a job, it uploads the output files +to the appropriate place in the upload hierarchy. + +### Client directory structure + +Input and output files have, in addition to their +names (or 'physical names'), +a 'logical name', which is the name by which the application refers to them. +dir structure on client + +The BOINC client, in its data directory, has two subdirectories: +* **projects/** has a subdirectory for each project +that the client is attached to. +* **slots/** has a subdirectory for each currently active job. + +The slot directory contains "link files", with logical names, +that refer to the corresponding +physical file in the project directory. +Link files look like +``` +../../projects/project.url.edu/phys_filename +``` + +Thus, the client directory structure looks like this: + +directory structure + +In some cases (if indicated by a `````` tag the job or app version config file) +the client copies the file to the slot directory, +giving it the logical name. + +## Process structure + +The BOINC client communicates with running jobs +by message-passing through a shared memory segment. +The client can tell the app to suspend or resume, +or tell it to checkpoint. +The app can reports its fraction done and CPU time to the client, +or confirm that it's checkpointed. + +The app side of this communication is done by +a 'BOINC runtime library' - a C++ library that's +linked into the application. +The library also provides API functions for +converting logical names to physical names +(i.e. for parsing link files) +and for initialization and finalization. +Every BOINC app must use the library. +There are several ways to do this: + +## App packaging options + +There are several ways to 'package' an application for use with BOINC: + +* ""Native app**: add API calls to the program source code, +then compile it (for a given platform) and link it to the BOINC runtime library. +* Wrapper: use a BOINC-supplied program ('wrapper') that interfaces +between the BOINC client and an unmodified executable. +This removes the need to change your program's source code, +but you still need to build it for different platforms (e.g. Windows). +* VM apps: your app runs inside a virtual machine (typically Linux). +A BOINC-supplied program called vboxwrapper interfaces +between the BOINC client, VirtualBox, and the VM. + +Each approach has pros and cons. +The VM approach is most convenient - you don't have to +develop on Windows or Mac - but it has a performance overhead, +and (currently) you can't use GPUs. diff --git a/Boinc-apps-(introduction).md b/Boinc-apps-(introduction).md deleted file mode 100644 index 88a390f..0000000 --- a/Boinc-apps-(introduction).md +++ /dev/null @@ -1,7 +0,0 @@ - apps and app versions - plan classes - files; dir structure - process structure - BOINC API, communication - packaging options - native, wrapper, VM diff --git a/BOINC-CLI:-command-line-job-submission.md b/Command-line-job-submission.md similarity index 100% rename from BOINC-CLI:-command-line-job-submission.md rename to Command-line-job-submission.md diff --git a/Scientist-interface.md b/Scientist-interface.md index edb3963..87d33a5 100644 --- a/Scientist-interface.md +++ b/Scientist-interface.md @@ -149,7 +149,7 @@ There are several job-submission variants: * Use BC's generic file-management and job-submission web interfaces to submit jobs to Autodock (or other apps on BC). * Install the BOINC job-submission and boinc2docker packages. -Use the [command-line job submission system](https://github.com/BOINC/boinc/wiki/BOINC-CLI:-command-line-job-submission) +Use the [command-line job submission system](Command-line-job-submission) to submit jobs for arbitrary (Linux/x86 or Python) applications. In this case the user must select science area keywords. diff --git a/SoftwareDevelopment.md b/SoftwareDevelopment.md index 75cbc31..6e48617 100644 --- a/SoftwareDevelopment.md +++ b/SoftwareDevelopment.md @@ -10,7 +10,7 @@ * [Scientist interface](Scientist-interface). This is based on and replaces: * [The BOINC out of box experience](The-BOINC-out-of-box-experience) * [The BOINC test drive](The-BOINC-test-drive) -* [Command-line job submission](https://github.com/BOINC/boinc/wiki/BOINC-CLI:-command-line-job-submission) +* [Command-line job submission](Command-line-job-submission) * [Supporting Windows hosts with > 64 cores](WinMulticore) * [Maintaining upload statistics](UploadStatistics) * [Locality scheduling redesign](LocalityNew)