Pre-initial dependence problem in warm starts.

[hjoliver]

[scheduling]
    initial cycle point = 2015
    [[dependencies]]
        [[[R1]]]
            graph = start
        [[[P1Y]]]
            graph = start[^] => foo

After a cold start, every instance of foo depends on start[2015]. After a warm start at a later cycle, that dependence gets ignored forever because it is wiped out by our pre-initial-dependence handling.

[hjoliver]

A warm start is technically a restart without reference to a prior state. Hopefully the operator knows something about the prior state, but cylc does not, so cylc should not assume anything about it. Currently cylc assumes that all pre-initial dependence is satisfied and can therefore be ignored, which this issue shows is not always the case. The correct way to handle this is to retain dependence on pre-initial tasks, but insert them into the suite at start-up in the held state and let the operator determine their fate according to his knowledge of prior state (probably just set them all 'succeeded' - and we could have an option to automatically do that - but we can't assume that that's always the right thing to do; e.g. an install task might need to be re-triggered, or there might be non-success triggers involved).

[hjoliver]

Interestingly the gcylc graph view doesn't behave properly either, for a warm start with start[^] => foo - a start base node appears, but for the wrong cycle point, and its cycle point changes as the suite evolves.

[hjoliver]

NOTE I have moved comments on pre-initial dependence in cold-starts to #1544 and edited them out of this issue.

[hjoliver]

Another unfortunate side-effect of our pre-initial handling in warm starts:

[scheduling]
    initial cycle point = 2015
    [[dependencies]]
        [[[R1]]]
            graph = """INSTALL
                   install_1 => install_2 & install_3"""
        [[[P1Y]]]
            graph = INSTALL[^]:succeed-all => foo
[runtime]
   [[INSTALL]]
   [[[install_1, install_2, install_3]]]
        inherit = INSTALL

(i.e. an initial R1 family with internal dependencies).

Some time after a warm start I need to rebuild a model used by the suite. I should be able to insert INSTALL again and trigger it, BUT our pre-initial handling at warm start also wipes out the internal dependencies among the INSTALL family members. So the suite has to be held before the family insert, then individual members have to be triggered manually in the correct order.

[arjclark]

As a thought, if the warm start time is recorded somewhere (see e.g. #1587) and kept separate from the initial cycle point, I believe it would be a relatively easy change to the existing code to make only pre-initial cycle dependencies get pruned and keep pre-warm but post-initial ones around on performing a manual insert.

Conversely, after a warm-start has been carried out, any tasks that exist before the warm start point in the pool must have been manually inserted in some way, so shouldn't have dependencies fixed. Should be similarly easy to fix, and probably the better way to go, as it would allow an inserted task to cycle forward, maintaining dependencies, up to the warm cycle point.

[hjoliver]

[meeting]

• on reflection, the first comment above isn't really a problem unless start hasn't finished when you do a warm start
  • it's not unreasonable to assume that a warm start implies all previous cycle points have finished
• we can solve the family insertion problem above pretty easily by only wiping out pre-start dependencies that cross the start point.

Therefore, this issue more or less evaporates 😐. Supersede with #1905 for the family (or workflow) insert problem.