Specifies cmdline args for all jobs in batch;
you can also specify per-job args, which come after.
create_work: add logic to support the above;
also fix bug involving pointers to stack vars (big no-no)
correctly show # of in progress, error, done, unsent jobs
fraction done is frac of success jobs
- Add 'verbose' checkbox for BUDA job submit.
docker_wrapper prints stuff to stderr (can view on result page)
- fix bugs in non-BUDA submit pages
- validate URL args that are used as filenames; prevent ../ stuff.
Do this by checking for '/'; is that sufficient?
- add 'delete app' function
- remove binary test file
old model:
no assimilator
output files live in upload hierarchy w/ physical names
WUs are marked as assimulated when batch is retired;
file_deleter deletes output files after that.
new model:
assimilator (e.g. sample_assimilate.py and sample_assimilator.cpp)
moves output files of canonical results to project/results/<batch_id>,
with names that include the batch name and the logical name.
WU is marked as assimilated; file_deleter deletes
output files of non-canonical results.
advantages of new model:
can see all output files of a batch on cmdline with ls
can zip output files of a batch without copying them
unified naming scheme for output files that encodes
the batch, the job (e.g. the BUDA job dir name)
and the logical name of the file.
------------
script assimilator: pass logical names to the script
Support both models. Choice of model is per app.
The project.inc file says which app uses which model.
For BUDA batches, description is '<app> (<variant>)'
Move job submission admin functions to their own page
Lay the groundwork for unifying output file handling
for remote job submission.
When you run a shell script on Unix, and it has Windows line endings (CRLF),
it fails with a misleading 'file not found' error message.
This can cause problems with BUDA apps, which can involve shell scripts,
and all files go through the user sandbox.
For example: if you put the script (with Unix endings) into Github
and check it out on a Win machine, all of a sudden it has Win endings!
If you upload it to your sandbox, it won't work.
So I added a sandbox feature where you can add a file
by pasting text into a web form.
Surprisingly, even this changed the LF to a CRLF!
I changed the form handler to convert CRLF to LF, and now it works.
How many man-years have been wasted on this line-ending BS?
I'm guessing the blame goes to Microsoft.
changes:
- creating a variant creates a JSON file, variant.json,
describing the dockerfile, app files, and in/out files.
Template files are now generated during job submission.
- no aliasing of files. If your main prog is foo,
your Dockerfile must end with CMD ./foo
- batch zip file must have shared input files
in a directory shared_input_files/.
All other directories are jobs.
variants are stored in <project>/buda_apps/<app>/<variant>.
This includes app files (copied from sandbox) and templates
(generated by handler)
- Add web interface for submitting BUDA jobs (not finished)
- Change implementation of user file sandbox
old: sandbox dir had 'link files' containing md5 and size;
actual file is in download hierarchy with sb_md5 name
new: sandbox dir has actual files.
parallel .md5/ dir has 'info files' (md5 size)
Files are not stored in download hierarchy.
New philosophy: names in the download hierarchy include
not only an MD5 (for uniqueness)
but also text describing the use of the file
(input file for a batch, part of a BUDA app, etc.).
This may allow duplicate files,
but it makes it possible to always clean up unused files.
- use readdir() instead of opendir()/scandir()
Don't use hardwired constants in DB queries.
Note: we have 2 scripts that do about the same thing
ops/cancel_workunits.php
ops/cancel_wu_form.php
Both are linked to from ops page. Remove one of them.
- don't use tables when they're not needed.
In light mode, bootstrap puts noisy lines around them
- use constants PM_LH_PCT and FORUM_LH_PCT for width
of left (informational) column.
- show lists of posts (user posts, search results) in 2-column format
- use functions like row2() rather than <tr><td>...
- forums: show header (search box, PM links) only in top-level pages.
Not needed on e.g. form for posting a message
- change page titles to say what you're looking at (forum, thread)
- use sprintf() instead of long concatenations
- make terminology consistent:
you 'post' to a thread or you 'reply' to a message.
You don't reply to a thread.
On the main Forums page, show all subscriptions
(to both threads and forums) with checkboxes,
so you can unsubscribe to multiple items at once.
Modernize the forum code here and there.
- Get rid of <span>s - we don't do our own CSS anymore.
- Have functions return strings rather than printing text;
it makes them more general.
- Use table_row() etc. instead of printing <td> etc.
- Move code from top level to functions
- start_table() and end_table() always at same code level
- only admins (which actually includes ADMIN, DEV, and SCIENTIST)
can administer user permissions (not, e.g. moderators)
- Add a link to the user-permissions admin page on the home page of admins
(there were previously no links to it)
- modernize the code of the admin page; replace _mysql_query() stuff
There are two types of email address check:
1) syntax check
2) stopforumspam.com (SFS) check: enabled by USE_STOPFORUMSPAM in project.inc
- Separate these. Sometimes it's appropriate to use just syntax check.
When we do use SFS check, and the email is flagged,
we should tell the user what happened;
maybe they're incorrectly in the SFS DB.
- use PHP's filter_var() for the syntax check,
rather than our own funky regexp.
The scheduler is a CGI program, not a daemon; it doesn't have a PID file.
So the only way we can see if it's 'running'
is to check the 'stop_sched' trigger file.
Do this whether the scheduler is local or remote.
David Anderson