edits

David Anderson 2024-11-13 12:37:16 -08:00
parent 7f1ee1e921
commit f96a81f131
1 changed files with 337 additions and 17 deletions

@ -1,32 +1,352 @@
This cookbook shows how to create and deploy a Docker app in BOINC.
We assume you already have created a BOINC project.
This cookbook shows how to create and deploy Docker apps in BOINC.
We assume you already have
[created a BOINC project](Create-a-BOINC-server-(cookbook)).
We're going to use a Linux/x64 executable called ```worker```.
This takes, as command-line arguments,
the names of an input file and an output file.
```
worker infile outfile
```
It reads the input file, converts it to upper case,
and writes it to the output file.
For background, please read about [how Docker apps work](Docker-apps).
This explains the two ways of packaging Docker apps:
You can build ```worker``` in the BOINC source tree:
* Single-purpose app: there is one BOINC app per science app.
You need to create new BOINC app versions each time your
science app changes.
This means that scientists must have login access to
the BOINC server, and must learn BOINC's command-line tools.
* Universal app: there is a single BOINC app
(and a set of per-platform app versions)
that handles all science apps.
Dockerfiles and science executables are included
in the input files for jobs.
This allows the creation of web-based job submission
systems that let scientists use (and evolve)
Docker apps without logging into the server
(or indeed knowing anything about BOINC).
We're going to give cookbooks for both packaging methods.
In both cases we're going to make a Docker app for Windows;
the Linux and Mac OS cases are similar.
Job for this app will be sent only to BOINC 9.0+ clients that have Docker.
On Windows this means that WSL must be enabled,
and the host must have a 'BOINC WSL distro' installed from the Windows Store.
Both variants are going to use the same "science executable" -
an x64/Intel program that runs inside a Docker container.
This program is called `worker`;
it converts a file to upper case and optionally uses some CPU time.
```
worker --nsecs 60 infile outfile
```
## Get "worker"
You can download it:
```
wget https://boinc.berkeley.edu/worker_3_x86_64-pc-linux-gnu
```
or build it in the BOINC source tree:
```
cd samples/worker
make
```
or download it from
https://boinc.berkeley.edu/worker
and rename it to `worker_3_x86_64-pc-linux-gnu`.
(The '3' is a version number;
in general all files should have version numbers in case they change).
## Get docker_wrapper
Download it:
```
wget https://boinc.berkeley.edu/docker_wrapper_1__windows_x86_64.exe
```
## Other files
Make a file `Dockerfile_worker_1`:
```
FROM debian
WORKDIR /app
CMD ./main.sh
```
Make a file `job_1.toml`:
```
project_dir_mount = "/project"
```
Make a file `main_2.sh`:
```
#! /bin/bash
resolve () {
sed 's/<soft_link>..\/..\/projects\/[^\/]*\//\/project\//; s/<\/soft_link>//' $1 | tr -d '\r\n'
}
$(resolve worker) --nsecs 60 $(resolve in) $(resolve out)
```
# Single-purpose app
## Create the app
config.xml: validator, assim
* Go to your project's admin web interface
* Click on Manage applications
* Add an application with name 'worker' and description 'Test app'.
## Create the app version
## Create input and output templates
In `~/projects/<proj_name>/templates/`,
create a file `worker_in`:
```
<input_template>
<file_info>
</file_info>
<workunit>
<file_ref>
<open_name>in</open_name>
</file_ref>
</workunit>
</input_template>
```
and `worker_out`:
```
<output_template>
<file_info>
<name><OUTFILE_0/></name>
<generated_locally/>
<upload_when_present/>
<max_nbytes>5000000</max_nbytes>
<url><UPLOAD_URL/></url>
</file_info>
<result>
<file_ref>
<file_name><OUTFILE_0/></file_name>
<open_name>out</open_name>
</file_ref>
</result>
</output_template>
```
## Create an app version for Windows
Using the above files, create the following directory structure:
```
~/projects/<proj_name>/apps/
worker/
1.0/
windows_x86_64__docker/
Dockerfile_worker_1
docker_wrapper_1.exe
job_1.toml
main_2.sh
worker_3_x86_64-pc-linux-gnu
version.xml
```
where `version.xml` contains
```
<version>
<file>
<physical_name>docker_wrapper_1.exe</physical_name>
<main_program/>
</file>
<file>
<physical_name>worker_3_x86_64-pc-linux-gnu</physical_name>
<logical_name>worker</logical_name>
</file>
<file>
<physical_name>main_2.sh</physical_name>
<logical_name>main.sh</logical_name>
<copy_file/>
</file>
<file>
<physical_name>Dockerfile_worker_1</physical_name>
<logical_name>Dockerfile</logical_name>
<copy_file/>
</file>
<file>
<physical_name>job_1.toml</physical_name>
<logical_name>job.toml</logical_name>
<copy_file/>
</file>
<is_wrapper/>
</version>
```
In your project's main directory:
```
bin/update_version --noconfirm
```
## Set up a validator and assimilator
Edit your project's `config.xml`
and add the following to the `<daemons>` section:
```
<daemon>
<cmd>script_validator --app worker --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
</daemon>
<daemon>
<cmd>script_assimilator -d 3 --app worker --script "sample_assimilate.py wu_name batch_id files"</cmd>
</daemon>
```
## Test
Install Docker and BOINC
In your project's main directory:
```
bin/stop
bin/start
```
Create a file `infile` with some mixed-case text.
Run
```
bin/submit_job worker infile
```
It will print a job name.
Submit job
Go to a Windows computer that has Docker enabled (see above)
and has a BOINC 9.0+ client running and attached to your project.
Update the project.
It should fetch the job you just created,
run it (takes about 1 minute) and upload the output file.
When the job is done, run
```
bin/query_job <job_name>
```
This should show that the job is completed,
and display its output file.
# Universal app
## Create the app
* Go to your project's admin web interface
* Click on Manage applications
* Add an application with name 'buda' and description 'BOINC universal Docker app'.
## Set up a validator and assimilator
Edit your project's `config.xml`
and add the following to the `<daemons>` section:
```
<daemon>
<cmd>script_validator --app buda --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
</daemon>
<daemon>
<cmd>script_assimilator -d 3 --app buda --script "sample_assimilate.py wu_name batch_id files"</cmd>
</daemon>
```
## Create an app version for Windows
Using the above files, create the following directory structure:
```
projects/<proj_name>/apps/
buda/
1.0/
windows_x86_64__docker/
docker_wrapper_1.exe
version.xml
```
where `version.xml` contains
```
<version>
<file>
<physical_name>docker_wrapper_1.exe</physical_name>
<main_program/>
</file>
<is_wrapper/>
</version>
```
Run
```
bin/update_verions --noconfirm
```
## Create a BUDA directory
Create a director `buda_dir` in the project and copy
files to it:
```
buda_dir/
Dockerfile_worker_1
file_list
job_1.toml
main_2.sh
template_in
template_out
worker_3_x86_64-pc-linux-gnu
```
where `file_list` contains
```
Dockerfile_worker_1
job_1.toml
main_2.sh
worker_3_x86_64-pc-linux-gnu
```
and `template_in` contains
```
<input_template>
<file_info>
</file_info>
<file_info>
</file_info>
<file_info>
</file_info>
<file_info>
</file_info>
<file_info>
</file_info>
<workunit>
<file_ref>
<open_name>Dockerfile</open_name>
<copy_file/>
</file_ref>
<file_ref>
<open_name>job.toml</open_name>
<copy_file/>
</file_ref>
<file_ref>
<open_name>main.sh</open_name>
<copy_file/>
</file_ref>
<file_ref>
<open_name>worker</open_name>
</file_ref>
<file_ref>
<open_name>in</open_name>
</file_ref>
</workunit>
</input_template>
```
and `template_out` contains
```
<output_template>
<file_info>
<name><OUTFILE_0/></name>
<generated_locally/>
<upload_when_present/>
<max_nbytes>5000000</max_nbytes>
<url><UPLOAD_URL/></url>
</file_info>
<result>
<file_ref>
<file_name><OUTFILE_0/></file_name>
<open_name>out</open_name>
</file_ref>
</result>
</output_template>
```
## Test
Run
```
bin/submit_buda infile
```
Update the project on your Windows BOINC client
so that it fetches and runs the job.
Then enter
```
bin/query_job <job_name>
```
to check the results.