- validate URL args that are used as filenames; prevent ../ stuff.
Do this by checking for '/'; is that sufficient?
- add 'delete app' function
- remove binary test file
old model:
no assimilator
output files live in upload hierarchy w/ physical names
WUs are marked as assimulated when batch is retired;
file_deleter deletes output files after that.
new model:
assimilator (e.g. sample_assimilate.py and sample_assimilator.cpp)
moves output files of canonical results to project/results/<batch_id>,
with names that include the batch name and the logical name.
WU is marked as assimilated; file_deleter deletes
output files of non-canonical results.
advantages of new model:
can see all output files of a batch on cmdline with ls
can zip output files of a batch without copying them
unified naming scheme for output files that encodes
the batch, the job (e.g. the BUDA job dir name)
and the logical name of the file.
------------
script assimilator: pass logical names to the script
Support both models. Choice of model is per app.
The project.inc file says which app uses which model.
For BUDA batches, description is '<app> (<variant>)'
Move job submission admin functions to their own page
Lay the groundwork for unifying output file handling
for remote job submission.
When you run a shell script on Unix, and it has Windows line endings (CRLF),
it fails with a misleading 'file not found' error message.
This can cause problems with BUDA apps, which can involve shell scripts,
and all files go through the user sandbox.
For example: if you put the script (with Unix endings) into Github
and check it out on a Win machine, all of a sudden it has Win endings!
If you upload it to your sandbox, it won't work.
So I added a sandbox feature where you can add a file
by pasting text into a web form.
Surprisingly, even this changed the LF to a CRLF!
I changed the form handler to convert CRLF to LF, and now it works.
How many man-years have been wasted on this line-ending BS?
I'm guessing the blame goes to Microsoft.
changes:
- creating a variant creates a JSON file, variant.json,
describing the dockerfile, app files, and in/out files.
Template files are now generated during job submission.
- no aliasing of files. If your main prog is foo,
your Dockerfile must end with CMD ./foo
- batch zip file must have shared input files
in a directory shared_input_files/.
All other directories are jobs.
variants are stored in <project>/buda_apps/<app>/<variant>.
This includes app files (copied from sandbox) and templates
(generated by handler)
- Add web interface for submitting BUDA jobs (not finished)
- Change implementation of user file sandbox
old: sandbox dir had 'link files' containing md5 and size;
actual file is in download hierarchy with sb_md5 name
new: sandbox dir has actual files.
parallel .md5/ dir has 'info files' (md5 size)
Files are not stored in download hierarchy.
New philosophy: names in the download hierarchy include
not only an MD5 (for uniqueness)
but also text describing the use of the file
(input file for a batch, part of a BUDA app, etc.).
This may allow duplicate files,
but it makes it possible to always clean up unused files.
- use readdir() instead of opendir()/scandir()
This lets you run arbitrary Docker applications using a single
BOINC app (and one app version per platform).
The Dockerfile and science executables are in the workunit.
The script tools/submit_buda lets you test this.
Basic tests were successful.
2) Unify the scripts for testing job submission
The 'test' assimilators (sample_assimilate.py and sample_assimilator.cpp)
do the same thing: they copy result files to
<proj_dir>/results/<batch_id>/<wu_name> (if 1 output file)
<proj_dir>/results/<batch_id>/<wu_name>_i (if >1 output file)
where <batch_id> is 0 if WU is not in a batch
... and they write error code to <wu_name>_error if the WU errored out
Scripts to submit jobs:
submit_job
submit_batch
submit_buda
Script to query jobs:
query_job
This works for either jobs or batches,
as long as the app uses one of the above assimilators
3) Add plan class 'docker' to plan_class_spec.xml.sample
Vitalii Koshura