variants are stored in <project>/buda_apps/<app>/<variant>.
This includes app files (copied from sandbox) and templates
(generated by handler)
- Add web interface for submitting BUDA jobs (not finished)
- Change implementation of user file sandbox
old: sandbox dir had 'link files' containing md5 and size;
actual file is in download hierarchy with sb_md5 name
new: sandbox dir has actual files.
parallel .md5/ dir has 'info files' (md5 size)
Files are not stored in download hierarchy.
New philosophy: names in the download hierarchy include
not only an MD5 (for uniqueness)
but also text describing the use of the file
(input file for a batch, part of a BUDA app, etc.).
This may allow duplicate files,
but it makes it possible to always clean up unused files.
- use readdir() instead of opendir()/scandir()
This lets you run arbitrary Docker applications using a single
BOINC app (and one app version per platform).
The Dockerfile and science executables are in the workunit.
The script tools/submit_buda lets you test this.
Basic tests were successful.
2) Unify the scripts for testing job submission
The 'test' assimilators (sample_assimilate.py and sample_assimilator.cpp)
do the same thing: they copy result files to
<proj_dir>/results/<batch_id>/<wu_name> (if 1 output file)
<proj_dir>/results/<batch_id>/<wu_name>_i (if >1 output file)
where <batch_id> is 0 if WU is not in a batch
... and they write error code to <wu_name>_error if the WU errored out
Scripts to submit jobs:
submit_job
submit_batch
submit_buda
Script to query jobs:
query_job
This works for either jobs or batches,
as long as the app uses one of the above assimilators
3) Add plan class 'docker' to plan_class_spec.xml.sample
This means that the tree rooted at WORKDIR is on the host filesystem,
and therefore survives system restarts;
otherwise it wouldn't.
This also simplifies file access from containers.
There's no need to copy input files from the slot dir to the container;
they're already in the container.
Similar with output files.
Remove these capabilities from the wrapper.
If a Docker app wants to access files in the project dir
(e.g. if they're huge files, shared with other jobs)
it can do so by mounting the project dir (using job.toml)
and then parsing the link files.
Merge the two test cases (copy and mount) into one
in both standalone and app_test.
The images and containers created by docker_wrapper have a particular form:
boinc__<proj>__<wuname> and boinc__<proj>__<resultname>.
docker_wrapper normally cleans up after itself.
But if it crashes these objects could hang around forever,
taking up disk space.
So we clean them up on client startup, as follows:
make lists of the names for active Docker tasks.
Then go through the list of Docker images and containers.
Remove any that
- were created by Docker wrapper (i.e. have boinc__ names)
- aren't in the active-task lists
On Unix, do this on the host itself if it has Docker installed.
On Win, do it for every WSL distro that has Docker installed.
David Anderson