mirror of https://github.com/BOINC/boinc.git
edits
parent
7f1ee1e921
commit
f96a81f131
|
@ -1,32 +1,352 @@
|
|||
This cookbook shows how to create and deploy a Docker app in BOINC.
|
||||
We assume you already have created a BOINC project.
|
||||
This cookbook shows how to create and deploy Docker apps in BOINC.
|
||||
We assume you already have
|
||||
[created a BOINC project](Create-a-BOINC-server-(cookbook)).
|
||||
|
||||
We're going to use a Linux/x64 executable called ```worker```.
|
||||
This takes, as command-line arguments,
|
||||
the names of an input file and an output file.
|
||||
```
|
||||
worker infile outfile
|
||||
```
|
||||
It reads the input file, converts it to upper case,
|
||||
and writes it to the output file.
|
||||
For background, please read about [how Docker apps work](Docker-apps).
|
||||
This explains the two ways of packaging Docker apps:
|
||||
|
||||
You can build ```worker``` in the BOINC source tree:
|
||||
* Single-purpose app: there is one BOINC app per science app.
|
||||
You need to create new BOINC app versions each time your
|
||||
science app changes.
|
||||
This means that scientists must have login access to
|
||||
the BOINC server, and must learn BOINC's command-line tools.
|
||||
|
||||
* Universal app: there is a single BOINC app
|
||||
(and a set of per-platform app versions)
|
||||
that handles all science apps.
|
||||
Dockerfiles and science executables are included
|
||||
in the input files for jobs.
|
||||
This allows the creation of web-based job submission
|
||||
systems that let scientists use (and evolve)
|
||||
Docker apps without logging into the server
|
||||
(or indeed knowing anything about BOINC).
|
||||
|
||||
We're going to give cookbooks for both packaging methods.
|
||||
In both cases we're going to make a Docker app for Windows;
|
||||
the Linux and Mac OS cases are similar.
|
||||
|
||||
Job for this app will be sent only to BOINC 9.0+ clients that have Docker.
|
||||
On Windows this means that WSL must be enabled,
|
||||
and the host must have a 'BOINC WSL distro' installed from the Windows Store.
|
||||
|
||||
Both variants are going to use the same "science executable" -
|
||||
an x64/Intel program that runs inside a Docker container.
|
||||
This program is called `worker`;
|
||||
it converts a file to upper case and optionally uses some CPU time.
|
||||
```
|
||||
worker --nsecs 60 infile outfile
|
||||
```
|
||||
|
||||
## Get "worker"
|
||||
|
||||
You can download it:
|
||||
```
|
||||
wget https://boinc.berkeley.edu/worker_3_x86_64-pc-linux-gnu
|
||||
```
|
||||
or build it in the BOINC source tree:
|
||||
```
|
||||
cd samples/worker
|
||||
make
|
||||
```
|
||||
or download it from
|
||||
https://boinc.berkeley.edu/worker
|
||||
and rename it to `worker_3_x86_64-pc-linux-gnu`.
|
||||
(The '3' is a version number;
|
||||
in general all files should have version numbers in case they change).
|
||||
|
||||
## Get docker_wrapper
|
||||
|
||||
Download it:
|
||||
```
|
||||
wget https://boinc.berkeley.edu/docker_wrapper_1__windows_x86_64.exe
|
||||
```
|
||||
## Other files
|
||||
|
||||
Make a file `Dockerfile_worker_1`:
|
||||
```
|
||||
FROM debian
|
||||
WORKDIR /app
|
||||
CMD ./main.sh
|
||||
```
|
||||
|
||||
Make a file `job_1.toml`:
|
||||
```
|
||||
project_dir_mount = "/project"
|
||||
```
|
||||
|
||||
Make a file `main_2.sh`:
|
||||
```
|
||||
#! /bin/bash
|
||||
|
||||
resolve () {
|
||||
sed 's/<soft_link>..\/..\/projects\/[^\/]*\//\/project\//; s/<\/soft_link>//' $1 | tr -d '\r\n'
|
||||
}
|
||||
|
||||
$(resolve worker) --nsecs 60 $(resolve in) $(resolve out)
|
||||
```
|
||||
|
||||
# Single-purpose app
|
||||
|
||||
## Create the app
|
||||
|
||||
config.xml: validator, assim
|
||||
* Go to your project's admin web interface
|
||||
* Click on Manage applications
|
||||
* Add an application with name 'worker' and description 'Test app'.
|
||||
|
||||
## Create the app version
|
||||
## Create input and output templates
|
||||
|
||||
In `~/projects/<proj_name>/templates/`,
|
||||
create a file `worker_in`:
|
||||
```
|
||||
<input_template>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<workunit>
|
||||
<file_ref>
|
||||
<open_name>in</open_name>
|
||||
</file_ref>
|
||||
</workunit>
|
||||
</input_template>
|
||||
```
|
||||
and `worker_out`:
|
||||
```
|
||||
<output_template>
|
||||
<file_info>
|
||||
<name><OUTFILE_0/></name>
|
||||
<generated_locally/>
|
||||
<upload_when_present/>
|
||||
<max_nbytes>5000000</max_nbytes>
|
||||
<url><UPLOAD_URL/></url>
|
||||
</file_info>
|
||||
<result>
|
||||
<file_ref>
|
||||
<file_name><OUTFILE_0/></file_name>
|
||||
<open_name>out</open_name>
|
||||
</file_ref>
|
||||
</result>
|
||||
</output_template>
|
||||
```
|
||||
## Create an app version for Windows
|
||||
|
||||
Using the above files, create the following directory structure:
|
||||
```
|
||||
~/projects/<proj_name>/apps/
|
||||
worker/
|
||||
1.0/
|
||||
windows_x86_64__docker/
|
||||
Dockerfile_worker_1
|
||||
docker_wrapper_1.exe
|
||||
job_1.toml
|
||||
main_2.sh
|
||||
worker_3_x86_64-pc-linux-gnu
|
||||
version.xml
|
||||
```
|
||||
where `version.xml` contains
|
||||
```
|
||||
<version>
|
||||
<file>
|
||||
<physical_name>docker_wrapper_1.exe</physical_name>
|
||||
<main_program/>
|
||||
</file>
|
||||
<file>
|
||||
<physical_name>worker_3_x86_64-pc-linux-gnu</physical_name>
|
||||
<logical_name>worker</logical_name>
|
||||
</file>
|
||||
<file>
|
||||
<physical_name>main_2.sh</physical_name>
|
||||
<logical_name>main.sh</logical_name>
|
||||
<copy_file/>
|
||||
</file>
|
||||
<file>
|
||||
<physical_name>Dockerfile_worker_1</physical_name>
|
||||
<logical_name>Dockerfile</logical_name>
|
||||
<copy_file/>
|
||||
</file>
|
||||
<file>
|
||||
<physical_name>job_1.toml</physical_name>
|
||||
<logical_name>job.toml</logical_name>
|
||||
<copy_file/>
|
||||
</file>
|
||||
<is_wrapper/>
|
||||
</version>
|
||||
```
|
||||
|
||||
In your project's main directory:
|
||||
```
|
||||
bin/update_version --noconfirm
|
||||
```
|
||||
|
||||
## Set up a validator and assimilator
|
||||
|
||||
Edit your project's `config.xml`
|
||||
and add the following to the `<daemons>` section:
|
||||
```
|
||||
<daemon>
|
||||
<cmd>script_validator --app worker --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
|
||||
</daemon>
|
||||
<daemon>
|
||||
<cmd>script_assimilator -d 3 --app worker --script "sample_assimilate.py wu_name batch_id files"</cmd>
|
||||
</daemon>
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
Install Docker and BOINC
|
||||
In your project's main directory:
|
||||
```
|
||||
bin/stop
|
||||
bin/start
|
||||
```
|
||||
Create a file `infile` with some mixed-case text.
|
||||
Run
|
||||
```
|
||||
bin/submit_job worker infile
|
||||
```
|
||||
It will print a job name.
|
||||
|
||||
Submit job
|
||||
Go to a Windows computer that has Docker enabled (see above)
|
||||
and has a BOINC 9.0+ client running and attached to your project.
|
||||
Update the project.
|
||||
It should fetch the job you just created,
|
||||
run it (takes about 1 minute) and upload the output file.
|
||||
|
||||
When the job is done, run
|
||||
```
|
||||
bin/query_job <job_name>
|
||||
```
|
||||
|
||||
This should show that the job is completed,
|
||||
and display its output file.
|
||||
|
||||
# Universal app
|
||||
|
||||
## Create the app
|
||||
|
||||
* Go to your project's admin web interface
|
||||
* Click on Manage applications
|
||||
* Add an application with name 'buda' and description 'BOINC universal Docker app'.
|
||||
|
||||
## Set up a validator and assimilator
|
||||
|
||||
Edit your project's `config.xml`
|
||||
and add the following to the `<daemons>` section:
|
||||
```
|
||||
<daemon>
|
||||
<cmd>script_validator --app buda --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
|
||||
</daemon>
|
||||
<daemon>
|
||||
<cmd>script_assimilator -d 3 --app buda --script "sample_assimilate.py wu_name batch_id files"</cmd>
|
||||
</daemon>
|
||||
```
|
||||
## Create an app version for Windows
|
||||
|
||||
Using the above files, create the following directory structure:
|
||||
```
|
||||
projects/<proj_name>/apps/
|
||||
buda/
|
||||
1.0/
|
||||
windows_x86_64__docker/
|
||||
docker_wrapper_1.exe
|
||||
version.xml
|
||||
```
|
||||
where `version.xml` contains
|
||||
```
|
||||
<version>
|
||||
<file>
|
||||
<physical_name>docker_wrapper_1.exe</physical_name>
|
||||
<main_program/>
|
||||
</file>
|
||||
<is_wrapper/>
|
||||
</version>
|
||||
```
|
||||
|
||||
Run
|
||||
```
|
||||
bin/update_verions --noconfirm
|
||||
```
|
||||
|
||||
## Create a BUDA directory
|
||||
|
||||
Create a director `buda_dir` in the project and copy
|
||||
files to it:
|
||||
```
|
||||
buda_dir/
|
||||
Dockerfile_worker_1
|
||||
file_list
|
||||
job_1.toml
|
||||
main_2.sh
|
||||
template_in
|
||||
template_out
|
||||
worker_3_x86_64-pc-linux-gnu
|
||||
```
|
||||
where `file_list` contains
|
||||
```
|
||||
Dockerfile_worker_1
|
||||
job_1.toml
|
||||
main_2.sh
|
||||
worker_3_x86_64-pc-linux-gnu
|
||||
```
|
||||
and `template_in` contains
|
||||
```
|
||||
<input_template>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<file_info>
|
||||
</file_info>
|
||||
<workunit>
|
||||
<file_ref>
|
||||
<open_name>Dockerfile</open_name>
|
||||
<copy_file/>
|
||||
</file_ref>
|
||||
<file_ref>
|
||||
<open_name>job.toml</open_name>
|
||||
<copy_file/>
|
||||
</file_ref>
|
||||
<file_ref>
|
||||
<open_name>main.sh</open_name>
|
||||
<copy_file/>
|
||||
</file_ref>
|
||||
<file_ref>
|
||||
<open_name>worker</open_name>
|
||||
</file_ref>
|
||||
<file_ref>
|
||||
<open_name>in</open_name>
|
||||
</file_ref>
|
||||
</workunit>
|
||||
</input_template>
|
||||
```
|
||||
and `template_out` contains
|
||||
```
|
||||
<output_template>
|
||||
<file_info>
|
||||
<name><OUTFILE_0/></name>
|
||||
<generated_locally/>
|
||||
<upload_when_present/>
|
||||
<max_nbytes>5000000</max_nbytes>
|
||||
<url><UPLOAD_URL/></url>
|
||||
</file_info>
|
||||
<result>
|
||||
<file_ref>
|
||||
<file_name><OUTFILE_0/></file_name>
|
||||
<open_name>out</open_name>
|
||||
</file_ref>
|
||||
</result>
|
||||
</output_template>
|
||||
```
|
||||
## Test
|
||||
|
||||
Run
|
||||
```
|
||||
bin/submit_buda infile
|
||||
```
|
||||
|
||||
Update the project on your Windows BOINC client
|
||||
so that it fetches and runs the job.
|
||||
Then enter
|
||||
```
|
||||
bin/query_job <job_name>
|
||||
```
|
||||
to check the results.
|
||||
|
|
Loading…
Reference in New Issue