Datastore fix for removed taskdef.

[hjoliver]

Close #5057

Fix datastore bug on restart after removing a task from the graph.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg and conda-environment.yml.
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

[oliver-sanders]

Bumped onto the 8.0.x branch, @hjoliver can you rebase this one.

[datamel]

I've checked this out and run the bug recreation steps from the issue. The traceback appearing on master is fixed, and looking at the gui, the removed task is not present, which is super.

I have a small query about the logs after restart, removed task b is appearing in the restart logs as failed. Is this by design? Is this confusing for users if their removed task appears in the logs (perhaps apart from maybe being warned that it was once there but now removed). I'm probably missing something!

[dwsutherland]

Reproduced, and fix confirmed working.

I suppose we have to filter out DB loads somewhere.

[hjoliver]

I have a small query about the logs after restart, removed task b is appearing in the restart logs as failed.

Good spotting. It's not exactly wrong - if the removed task was still running at shutdown, the scheduler should poll it and log its status and progress (but there would be no further instances of it). However, I'm not sure why it is being polled here, as it failed before shutdown. Investigating ... on second look at the log, both tasks are getting polled, so that's fine. We're only polling failed tasks in the task pool (i.e., incomplete tasks) not all failed tasks.

[hjoliver]

@datamel - I've rebased to remove all the spurious commits that appeared after changing the PR base branch (not sure why it did that)... and added a new test.

I'll cherry-pick to master once this is approved.

[oliver-sanders]

I'll cherry-pick to master once this is approved.

FYI: we've been merging 8.0.x into master (you can create a PR from the GH web app).

[hjoliver]

@datamel I've reformulated the functional test to not rely on timing.

[datamel]

All working smoothly for me. Thanks!