Waiting for subprocesses to finish

[jarmoza]

A QA command (i.e.archive, clear, run, collate) should finish before the next QA command begins. This is important because a command may be dependent on the output/state of the previous command.

The trick is that the main QA script or the QA module script needs to be able to block/wait until all of the system subprocesses it generates for the current command finish.

[jarmoza]

From https://stackoverflow.com/questions/100624/python-on-windows-how-to-wait-for-multiple-child-processes
(Modified for Python3 below.)

import queue, _thread, subprocess

def process_waiter(popen, description, que):
    try: popen.wait()
    finally: que.put( (description, popen.returncode) )

results= queue.queue()
process_count= 0

proc1= subprocess.Popen( ['python', 'mytest.py'] )
_thread.start_new_thread(process_waiter,
    (proc1, "1 finished", results))
process_count+= 1

proc2= subprocess.Popen( ['python', 'mytest.py'] )
_thread.start_new_thread(process_waiter,
    (proc2, "2 finished", results))
process_count+= 1

# etc

while process_count > 0:
    description, rc= results.get()
    print "job", description, "ended with rc =", rc
    process_count-= 1

[jarmoza]

Previously it seemed that this work could be delayed because the commands of each QA module could be called separately/manually. This must be implemented now however for the following reason: If the actual process being QA'd does not finish then it is possible the function producing the QA output for that process may not have access to all the necessary files it needs to produce the metrics we want to see. My suspicion is that this is the reason for some of the missing image (PIL.UnidentifiedImageError) errors we were seeing from the autocrop QA run last week - though definitrly not all as many PIL.UnidentifiedImageError were coming from autocrop.py itself.

There is also the current issue that the process being Q'Ad currently is called directly from qa_<qa_modulename>.py as a python3 call and not an sbatch call. Therefore the sbatch parameters being given to the qa_<qa_modulename>.sh are not being assigned to the right process. They should be given to the process being QA'd itself.

My planned re-implementation is for qa.py to call qa_<qa_modulename>.py directly instead of what it currently does, call a bash script qa_<qa_module>.sh. The previous code that was part of the main script in qa_<qa_modulename>.py will be folded into the QA_<QAModuleName> class's run call. This way 1) sbatch will be used to call the actual process being QA'd and 2) the class's run call will have direct access to those sbatch subprocesses generated and can wait till they all finish.

NOTE: QA commands like archive and clear rely on system calls that block until complete. Therefore any QA command that does not itself generate a subprocess should have an implicit wait till it is finished and should not have to use the new QAProcessWaiter queue to block while the OS completes calls like makedirs, rename, and unlink.

[jarmoza]

QA_Autocrop's run method will now issue an sbatch call to qa_autocrop.sh with the proper directives. But now qa_autocrop.sh will no longer be calling qa_autocrop.py and running its main script. Instead it will now call python3 autocrop.py (w/ appropriate arguments) directly. This way autocrop.py gets the appropriate sbatch resources and QA_Autocrop can keep track of when each qa_autocrop.sh finishes – meaning when each autocrop.py call finishes.

[jarmoza]

New classes for the queueing and waiting of processes, QAProcessWaiter and QAProcess, are now in qa_utilities.py.

One instance of QAProcessWaiter exists in the QA_Module base class.

Convenience methods call_command and wait have been added to QA_Module, as well as the method start_process and property process_is_in_queue in QAProcessWaiter.

[jarmoza]

After several implementation iterations, it appears that out of memory errors and large books are causing some hitches in a continuous 'one button' QA pipeline.

For instance, outputting of stats for autocrop jobs require their own process and cannot be included in the same slurm process as the autocropping call for that book.

Another issue that was encountered is that slurm job ID is not available when sbatch is being called from Python. This piece of information is required to call a dependent slurm job (job name specified via the -J flag cannot be used as an alternative to the output job id). One possibility is to move the relevant pieces of sbatch calls back to bash script from Python.

Piecemeal processing of book also seems to be one answer for this slurm out of memory issue as well as better curation of data sets for QA. In the meantime, running one QA command at a time (e.g. clear/archive, run, output_stats, collate) is the shortest workaround for this problem.

Moving this to blocked for now and will address this issue when there is time.

[jarmoza]

One solution to this problem may be the implementation for #19

By giving page limits to any QA module, we can also limit the amount of memory needed for each thread.

Even so, it may be more preferable to just drop a "done.txt" file in each book directory once the thread finishes (likely via a finally clause).

[jarmoza]

Next for this task is to meet with Kartik to go over some possibilities about how get sbatch dependencies to work via Python.

[jarmoza]

Meeting with Kartik today produced 2 potential possibilities:

1. Calling a new bash script that iterates that a) has a job array id and b) iterates over the book directories for running the QA task bash script (i.e. qa_line_extraction_final.sh).
2. Using Pyslurm

Doing 1 will require some means of relaying the job array id back to Python once subprocess makes the initial sbatch call for the new, overarching bash script. This is likely available via the environment variables that slurm uses to store these IDs.

Unfortunately for 2, after some trying it seems that Pyslurm is not compatible with the version of slurm on bridges.

Links:
https://slurm.schedmd.com/job_array.html
https://github.com/PySlurm/pyslurm

[jarmoza]

Blocked until new line extraction method, eynollah, is integrated into QA line extraction code.

[jarmoza]

In order for waits to be introduced between jobs for QA, this must happen in bash script as calls to sbatch from Python do not produce job IDs/vars necessary for Slurm array scheduling. Examples of this exist in the new run_workflow1_eynollah.sh for Eynollah line extraction.

The suggested implementation is a sort of intermediary script that makes the requisite sbatch call. Arguments for that intermediary bash script can be passed in from Python.