Manual flow retries

[anticorrelator]

closes #7127

Implements the ability to manually retry flow runs in terminal states by proposing an AwaitingRetry scheduled state

This PR also includes a refactor of flow retry logic

• we no longer check for failed task runs and marking them rerunnable on retry/restart
• instead, we track whether or not a flow is retrying
• while attempting to retry a flow, the engine attempts to orchestrate any tasks it discovers

This should significantly reduce the overhead of retrying flows with many task runs

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• This pull request includes tests or only affects documentation.
• This pull request includes a label categorizing the change e.g. fix, feature, enhancement

[netlify]

✅ Deploy Preview for prefect-orion ready!

Name	Link
🔨 Latest commit	83d98cc
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-orion/deploys/63571e23141967000975ac76
😎 Deploy Preview	https://deploy-preview-7152--prefect-orion.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[billpalombi]

This design is consistent with what we discussed yesterday. It's somewhat more complex than the "soft" restart, but I appreciate the protection against potential failure modes.

[cicdw]

A few questions for ya

[zanieb]

Perhaps we can separate the changes required for a hard-restart from the changes required to expose the existing retry mechanism? If they're in two pull requests we can first review exposure of the retry/restart mechanism via an API route then , separately and without blocking the feature we need to deliver, we can have a more extended discussion about adding an option to that route to perform a hard restart.

[cicdw]

A few questions

[jlowin]

This still seems way overcomplicated to me, in particular the need to separately track "retries", "restarts" and "run counts" suggests that the core motivation of running the flow again is being dominated by some degree of edge cases.

[jlowin]

🎉 I really like how clean this feels now, nice work @anticorrelator!