diff --git a/Docker-app-cookbook.md b/Docker-app-cookbook.md index 4ab85e4..0001eaa 100644 --- a/Docker-app-cookbook.md +++ b/Docker-app-cookbook.md @@ -1,32 +1,352 @@ -This cookbook shows how to create and deploy a Docker app in BOINC. -We assume you already have created a BOINC project. +This cookbook shows how to create and deploy Docker apps in BOINC. +We assume you already have +[created a BOINC project](Create-a-BOINC-server-(cookbook)). -We're going to use a Linux/x64 executable called ```worker```. -This takes, as command-line arguments, -the names of an input file and an output file. -``` -worker infile outfile -``` -It reads the input file, converts it to upper case, -and writes it to the output file. +For background, please read about [how Docker apps work](Docker-apps). +This explains the two ways of packaging Docker apps: -You can build ```worker``` in the BOINC source tree: +* Single-purpose app: there is one BOINC app per science app. +You need to create new BOINC app versions each time your +science app changes. +This means that scientists must have login access to +the BOINC server, and must learn BOINC's command-line tools. + +* Universal app: there is a single BOINC app +(and a set of per-platform app versions) +that handles all science apps. +Dockerfiles and science executables are included +in the input files for jobs. +This allows the creation of web-based job submission +systems that let scientists use (and evolve) +Docker apps without logging into the server +(or indeed knowing anything about BOINC). + +We're going to give cookbooks for both packaging methods. +In both cases we're going to make a Docker app for Windows; +the Linux and Mac OS cases are similar. + +Job for this app will be sent only to BOINC 9.0+ clients that have Docker. +On Windows this means that WSL must be enabled, +and the host must have a 'BOINC WSL distro' installed from the Windows Store. + +Both variants are going to use the same "science executable" - +an x64/Intel program that runs inside a Docker container. +This program is called `worker`; +it converts a file to upper case and optionally uses some CPU time. +``` +worker --nsecs 60 infile outfile +``` + +## Get "worker" + +You can download it: +``` +wget https://boinc.berkeley.edu/worker_3_x86_64-pc-linux-gnu +``` +or build it in the BOINC source tree: ``` cd samples/worker make ``` -or download it from -https://boinc.berkeley.edu/worker +and rename it to `worker_3_x86_64-pc-linux-gnu`. +(The '3' is a version number; +in general all files should have version numbers in case they change). +## Get docker_wrapper + +Download it: +``` +wget https://boinc.berkeley.edu/docker_wrapper_1__windows_x86_64.exe +``` +## Other files + +Make a file `Dockerfile_worker_1`: +``` +FROM debian +WORKDIR /app +CMD ./main.sh +``` + +Make a file `job_1.toml`: +``` +project_dir_mount = "/project" +``` + +Make a file `main_2.sh`: +``` +#! /bin/bash + +resolve () { + sed 's/..\/..\/projects\/[^\/]*\//\/project\//; s/<\/soft_link>//' $1 | tr -d '\r\n' +} + +$(resolve worker) --nsecs 60 $(resolve in) $(resolve out) +``` + +# Single-purpose app ## Create the app -config.xml: validator, assim +* Go to your project's admin web interface +* Click on Manage applications +* Add an application with name 'worker' and description 'Test app'. -## Create the app version +## Create input and output templates + +In `~/projects//templates/`, +create a file `worker_in`: +``` + + + + + + in + + + +``` +and `worker_out`: +``` + + + + + + 5000000 + + + + + + out + + + +``` +## Create an app version for Windows + +Using the above files, create the following directory structure: +``` +~/projects//apps/ + worker/ + 1.0/ + windows_x86_64__docker/ + Dockerfile_worker_1 + docker_wrapper_1.exe + job_1.toml + main_2.sh + worker_3_x86_64-pc-linux-gnu + version.xml +``` +where `version.xml` contains +``` + + + docker_wrapper_1.exe + + + + worker_3_x86_64-pc-linux-gnu + worker + + + main_2.sh + main.sh + + + + Dockerfile_worker_1 + Dockerfile + + + + job_1.toml + job.toml + + + + +``` + +In your project's main directory: +``` +bin/update_version --noconfirm +``` + +## Set up a validator and assimilator + +Edit your project's `config.xml` +and add the following to the `` section: +``` + + script_validator --app worker --init_script "validate_init.py" --compare_script "validate_compare.py" + + + script_assimilator -d 3 --app worker --script "sample_assimilate.py wu_name batch_id files" + +``` ## Test -Install Docker and BOINC +In your project's main directory: +``` +bin/stop +bin/start +``` +Create a file `infile` with some mixed-case text. +Run +``` +bin/submit_job worker infile +``` +It will print a job name. -Submit job +Go to a Windows computer that has Docker enabled (see above) +and has a BOINC 9.0+ client running and attached to your project. +Update the project. +It should fetch the job you just created, +run it (takes about 1 minute) and upload the output file. + +When the job is done, run +``` +bin/query_job +``` + +This should show that the job is completed, +and display its output file. + +# Universal app + +## Create the app + +* Go to your project's admin web interface +* Click on Manage applications +* Add an application with name 'buda' and description 'BOINC universal Docker app'. + +## Set up a validator and assimilator + +Edit your project's `config.xml` +and add the following to the `` section: +``` + + script_validator --app buda --init_script "validate_init.py" --compare_script "validate_compare.py" + + + script_assimilator -d 3 --app buda --script "sample_assimilate.py wu_name batch_id files" + +``` +## Create an app version for Windows + +Using the above files, create the following directory structure: +``` +projects//apps/ + buda/ + 1.0/ + windows_x86_64__docker/ + docker_wrapper_1.exe + version.xml +``` +where `version.xml` contains +``` + + + docker_wrapper_1.exe + + + + +``` + +Run +``` +bin/update_verions --noconfirm +``` + +## Create a BUDA directory + +Create a director `buda_dir` in the project and copy +files to it: +``` +buda_dir/ + Dockerfile_worker_1 + file_list + job_1.toml + main_2.sh + template_in + template_out + worker_3_x86_64-pc-linux-gnu +``` +where `file_list` contains +``` +Dockerfile_worker_1 +job_1.toml +main_2.sh +worker_3_x86_64-pc-linux-gnu +``` +and `template_in` contains +``` + + + + + + + + + + + + + + Dockerfile + + + + job.toml + + + + main.sh + + + + worker + + + in + + + +``` +and `template_out` contains +``` + + + + + + 5000000 + + + + + + out + + + +``` +## Test + +Run +``` +bin/submit_buda infile +``` + +Update the project on your Windows BOINC client +so that it fetches and runs the job. +Then enter +``` +bin/query_job +``` +to check the results.