Remove working files after processing a run

[donkirkby]

The hard drive is getting full, so we should review what gets left in the data folders.

The prelim.csv and remap.csv look like the largest files, so consider deleting them from all but the ten most recent runs.

[ArtPoon]

Hard drive is full. Using a script for erase *.prelim.csv and *.remap.csv for all but the most recent 20 runs to free up space. Need to adapt driver script to drop intermediate files during processing for older runs. We cannot continue processing runs under 6.7 until this is in place.

[ArtPoon]

From log file:

2015-03-09 21:43:20.000230 - [INFO] Starting run /media/RAW_DATA/MiSeq/runs/150121_M01841_0099_000000000-AA2YY/
2015-03-10 00:14:38.112081 - [ERROR] Failed to post pipeline results - skipping run /media/RAW_DATA/MiSeq/runs/150121_M01841_0099_000000000-AA2YY/
2015-03-10 00:14:40.904460 - [INFO] Starting run /media/RAW_DATA/MiSeq/runs/150119_M01841_0098_000000000-AA8AV/
2015-03-10 00:16:24.461617 - [ERROR] Quality could not be downloaded. - skipping run /media/RAW_DATA/MiSeq/runs/150119_M01841_0098_000000000-AA8AV/
2015-03-10 00:16:25.159746 - [INFO] Starting run /media/RAW_DATA/MiSeq/runs/140903_M01841_0074_000000000-A8EGW/
2015-03-10 00:17:24.631437 - [ERROR] Quality could not be downloaded. - skipping run /media/RAW_DATA/MiSeq/runs/140903_M01841_0074_000000000-A8EGW/

Pipeline has since started up on latest run 150309_M01841_0111_000000000-ACDP3.
/data has 2T available. Not sure if the failed runs (due to HD space) will get re-queued.

[ArtPoon]

75 runs have been flagged with errorprocessing.

• 3 runs "failed to post pipeline results"
• 8 runs "Parsing sample sheet failed" (these are very early runs, see issue Old sample sheets can't be parsed #174)
• 64 runs "Quality could not be downloaded"; probably due to full HD issue

[ArtPoon]

After testing development version to make sure intermediate files are removed correctly, release as 6.7.1 and push to production.

[ArtPoon]

Fixed by commit c9b7710690f7c24472c6a0ad67b3f61525c1f654