Combine midi samples in MiSeq monitor

[donkirkby]

In order to generate HCV reports, we have to combine HCV sample pairs: the main sample and the midi sample. That means that we will have three pipelines:

1. filter_quality that runs once for the whole run
2. main pipeline that generates amino.csv and coverage maps
3. resistance pipeline that combines the two samples and generates resistance reports

Since this means big changes for the MiSeq monitor configuration, I thought I should change it to run as a service, and also to read the configuration settings from environment variables instead of settings.py.

In my research, I found the python-daemon package for launching a daemon process from Python. However, it looks like that's not really necessary, because systemd services can wrap a plain executable.

• Define filter_quality pipeline in local Kive.
• Define main pipeline in local Kive.
• Define resistance pipeline in local Kive.
• Change monitor to run three pipelines, with pipeline ids loaded from environment variables.
• Upload to QAI.
• External datasets.
• New folder appears.
• Kive retries and session refresh. Mark failed runs.
• Use preexisting runs, if any.
• Restart run when cancelled by user.
• Convert logging settings to an optional file, with default useful for developers. (json, yaml, or Python?)
• Set high priority on latest run. Not exposed through the API wrapper, yet.
• Convert other settings to environment variables with useful defaults.
• Configure as a systemd service, and document configuration.
• Find pipeline by name if environment variable is not set (for developers).

[jamesnakagawa]

Will this mean that MidHCV will be combined under the HCV project in lab_miseq_review_decisions?

[donkirkby]

This issue only covers the changes to MiSeq Monitor that combine the two samples together when making the resistance calls. Issue #427 covers what you're asking about, and I'll add some notes there.