Specifies cmdline args for all jobs in batch;
you can also specify per-job args, which come after.
create_work: add logic to support the above;
also fix bug involving pointers to stack vars (big no-no)
- Replace icons
- Remove unused dialog
- Remove SetAllUsers.dll
- Remove Print LA action (I don't believe anybody still uses it). If required - it could be always implemented.
- Replace ISSetupAllUsers 3rd-party function
- Remove unused icons
- Convert custom build target to custom build step to fix build of MSI file when only *.json file was changed.
- Remove SFHelper.dll
Signed-off-by: Vitalii Koshura <lestat.de.lionkur@gmail.com>
correctly show # of in progress, error, done, unsent jobs
fraction done is frac of success jobs
- Add 'verbose' checkbox for BUDA job submit.
docker_wrapper prints stuff to stderr (can view on result page)
- fix bugs in non-BUDA submit pages
- validate URL args that are used as filenames; prevent ../ stuff.
Do this by checking for '/'; is that sufficient?
- add 'delete app' function
- remove binary test file
old model:
no assimilator
output files live in upload hierarchy w/ physical names
WUs are marked as assimulated when batch is retired;
file_deleter deletes output files after that.
new model:
assimilator (e.g. sample_assimilate.py and sample_assimilator.cpp)
moves output files of canonical results to project/results/<batch_id>,
with names that include the batch name and the logical name.
WU is marked as assimilated; file_deleter deletes
output files of non-canonical results.
advantages of new model:
can see all output files of a batch on cmdline with ls
can zip output files of a batch without copying them
unified naming scheme for output files that encodes
the batch, the job (e.g. the BUDA job dir name)
and the logical name of the file.
------------
script assimilator: pass logical names to the script
Support both models. Choice of model is per app.
The project.inc file says which app uses which model.
For BUDA batches, description is '<app> (<variant>)'
Move job submission admin functions to their own page
Lay the groundwork for unifying output file handling
for remote job submission.
When you run a shell script on Unix, and it has Windows line endings (CRLF),
it fails with a misleading 'file not found' error message.
This can cause problems with BUDA apps, which can involve shell scripts,
and all files go through the user sandbox.
For example: if you put the script (with Unix endings) into Github
and check it out on a Win machine, all of a sudden it has Win endings!
If you upload it to your sandbox, it won't work.
So I added a sandbox feature where you can add a file
by pasting text into a web form.
Surprisingly, even this changed the LF to a CRLF!
I changed the form handler to convert CRLF to LF, and now it works.
How many man-years have been wasted on this line-ending BS?
I'm guessing the blame goes to Microsoft.
changes:
- creating a variant creates a JSON file, variant.json,
describing the dockerfile, app files, and in/out files.
Template files are now generated during job submission.
- no aliasing of files. If your main prog is foo,
your Dockerfile must end with CMD ./foo
- batch zip file must have shared input files
in a directory shared_input_files/.
All other directories are jobs.
variants are stored in <project>/buda_apps/<app>/<variant>.
This includes app files (copied from sandbox) and templates
(generated by handler)
- Add web interface for submitting BUDA jobs (not finished)
- Change implementation of user file sandbox
old: sandbox dir had 'link files' containing md5 and size;
actual file is in download hierarchy with sb_md5 name
new: sandbox dir has actual files.
parallel .md5/ dir has 'info files' (md5 size)
Files are not stored in download hierarchy.
New philosophy: names in the download hierarchy include
not only an MD5 (for uniqueness)
but also text describing the use of the file
(input file for a batch, part of a BUDA app, etc.).
This may allow duplicate files,
but it makes it possible to always clean up unused files.
- use readdir() instead of opendir()/scandir()
Vitalii Koshura