Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relitigate ignoring of pre-initial dependence. #1544

Closed
hjoliver opened this issue Jul 22, 2015 · 3 comments
Closed

Relitigate ignoring of pre-initial dependence. #1544

hjoliver opened this issue Jul 22, 2015 · 3 comments

Comments

@hjoliver
Copy link
Member

(This arose as a side-issue of #1540 which argues against pre-initial-ignore specifically in warm starts).

Ignoring pre-initial dependence was invented as a workaround for a deficiency of cylc-5 that no longer exists. E.g. this suite would stall at start-up if we did not ignore foo[2014] (and if we didn't use old-style cold-start tasks and a conditional trigger):

[scheduling]
    initial cycle point = 2015
    [[dependencies]]
        [[[R1]]] # first cycle point
            graph = start => foo
        [[[P1Y]]]  # every cycle point (including the first)
            graph = foo[-P1Y] => foo

Since cylc-6 though, we have the ability to handle any start-up scenario correctly without ignoring pre-initial dependence:

[scheduling]
    initial cycle point = 2015
    [[dependencies]]
        [[[R1]]] # first cycle point
            graph = start => foo
        [[[R/+P1Y/P1Y]]]  # second and subsequent points
            graph = foo[-P1Y] => foo

This second version is only harder to understand than the first if you don't understand our graph syntax. Arguably it is easier to understand because (a) there is no overlap between the R1 and general R sections; and (b) the graph syntax alone is sufficient to describe exactly what happens at start-up. In the first case, something mysterious has to happen to allow the suite to start up (i.e. pre-initial dependence is automatically ignored).

In my opinion we should deprecate this behaviour. It is now unnecessary, it is quite complicated code, and as a workaround it is liable to have unforeseen consequences in edge cases (e.g. #1540 for its application to warm starts).

@hjoliver
Copy link
Member Author

options

  1. Simply stop ignoring pre-initial dependence. Suites will stall if they have any inter-cycle dependence that isn't handled properly at start-up (in accordance with what the suite definition says!).
  2. As for 1. plus actually insert pre-initial tasks into the pool, in the 'held' state (c.f. warm start as proposed in Pre-initial dependence problem in warm starts. #1540)

In option 2 the reason for a stalled suite will be more obvious: some tasks depend on pre-initial tasks, which being 'held' are easier to see than the stalled dependent tasks. To proceed, users can:

  • (ideally) fix their suite graph structure as in the second example suite above, or
  • (manually) as a temporary measure, reset the pre-initial tasks to 'succeeded' (or whatever), or
  • (optionally) start the suite with pre-initial tasks already succeeded (or whatever) as a proxy for the current pre-initial-ignore behaviour - like my proposed warm start option in Pre-initial dependence problem in warm starts. #1540)

Perhaps we could retain the current pre-initial behaviour as an option, rather than deprecate it. I would prefer not to retain it as the default though, because as argued above it is really a workaround that is not universally valid, not a correct way of handling start-up.

@hjoliver hjoliver added this to the soon milestone Jul 22, 2015
@hjoliver
Copy link
Member Author

Finally, as an alternative to setting pre-initial tasks to 'held' in option 2 above, we could have a new task state for that purpose (to avoid obscuring pre-initial tasks if the suite is started with --hold).

@hjoliver hjoliver changed the title Relitigate ignoring of pre-initial tasks/dependencies. Relitigate ignoring of pre-initial dependence. Jul 22, 2015
@hjoliver hjoliver modified the milestones: later, soon Jul 22, 2015
@hjoliver
Copy link
Member Author

[meeting] with the pre-start warm start issue addressed (#1905) this can be closed as no further problems have been identified, pre-start ignore undoubtedly does make the start-up graph easier for users, and there's nothing to stop users doing this "properly" if they want to 😬

@matthewrmshin matthewrmshin removed this from the later milestone Jun 22, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants