Some samples not uploaded to QAI

[donkirkby]

Several samples were not uploaded from the 23-Sep-2016.M01841 run, such as 73276A. The results are in Kive, so I'm not sure what went wrong. I noticed that the samples were rerun several times, so maybe there's a problem with the scratch files that get held when you rerun.

• Look for other samples that didn't get processed.
• Try to figure out why 23-Sep-2016.M01841 and 20-Sep-2016.M01841 runs only downloaded the last few samples. Can't figure it out for now. Wait to see if this happens again.
• Compress the couple of uncompressed samples I found.
• Try to figure out why 28-Jun-2013.M01841 run wasn't processed at all.
• Rename files in 28-Jun-2013.M01841. Chanson will redo the demultiplexing step after copying the sample sheet from 11-Jul-2013.M01841. (They use the same samples.)
• Check if scratch folders are deleted after a QAI upload error.

[donkirkby]

It looks like the problem happened before the outputs in the folder were collated. Both the cascade.csv and the g2p_summary.csv files had this list of samples:

• NEG-HCV_S65
• NEGMIDI-MidHCV_S68
• 73290A-HCV_S64
• 73290AMIDI-MidHCV_S67
• 73282A-HCV_S63
• 73282AMIDI-MidHCV_S66
• P73265A-HCV_S59
• P73265AMIDI-MidHCV_S62
• 73289A-HCV_S58
• 73289AMIDI-MidHCV_S61
• 73281A-HCV_S57
• 73281AMIDI-MidHCV_S60
• 73296A-HCV_S53
• 73296AMIDI-MidHCV_S56
• 73288A-HCV_S52
• 73288AMIDI-MidHCV_S55
• 73280A-HCV_S51
• 73280AMIDI-MidHCV_S54

There were several errors in the log around the time those jobs were running.

[donkirkby]

I found the same problem in the previous run: 20-Sep-2016.M01841. The two runs were processed by the 7.9 pipeline on the same day.

When looking for other instances of the same problem, I found issue #465 about mismatched MidHCV names. I also found a couple of samples where the FASTQ file wasn't found, because it wasn't gzipped: 35807A-PR-RT_S78 in 20-Aug-2013.M01841 and 35860A-INT_S60 in 06-Sep-2013.M01841.

[donkirkby]

The 28-Jun-2013.M01841 run went completely unprocessed by version 7.9, and I'm not sure why.

[donkirkby]

The 28-Jun-2013.M01841 run didn't get processed, because none of the file names have project codes. The sample sheet does have the project codes, so why don't the file names have them?

[donkirkby]

The problem happened again with the 14-Jun-2019.M01841 run. I think what happens is this:

1. MiCall watcher starts processing run folder X, uploading samples and launching Kive runs.
2. Some of the Kive runs complete, and the results are downloaded into scratch folders under run folder X.
3. Run folder Y completes, and an error occurs while uploading to QAI.
4. When the folders get scanned again, the scratch folders get deleted from under run folder X, and only the samples that complete later get downloaded from Kive and uploaded to QAI.

I haven't reproduced this, but that's my theory for what happened. Here's what I know in the case of the 14-Jun-2019.M01841 run:

1. I started reprocessing runs from 21 Jun 2019 back, and they ran overnight.
2. I hadn't configured the pipeline properly in QAI, so the QAI uploads all failed.
3. In the morning, I corrected the pipeline in QAI, and left MiCall watcher running.
4. MiCall watcher eventually tried to upload to QAI again, and "succeeded". The first two run folders had uploaded all the samples, but the 14-Jun-2019.M01841 run was missing several.
5. I deleted the doneprocessing file from that folder to force it to upload again, and left MiCall watcher running.
6. MiCall watcher eventually noticed, and uploaded again. This time it only uploaded NEG-V3-NEW-3-V3LOOP_S192, the last sample in the sample sheet.
7. I stopped MiCall watcher, deleted the doneprocessing file, and started MiCall watcher.
8. This time, it uploaded all the samples.