[Fleet] Improve recoverability and stability of package installation

[kpollich]

Meta issue tracking the work for recoverability and stability of package installation

Currently, it's hard to recover a failed package installation as our only recourse is typically to reinstall the package. There's no granular recovery steps we can take, and we often lack visibility into which particular steps failed. It'd be ideal if we could build a more "state machine" like implementation for packages with specific recovery steps for each state transition along the way.

Ref #166857
Ref #166798

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[criamico]

Adding some considerations as discussed with @kpollich

We could start by looking at the specific steps that are covered by the installation process and documenting it. We have a complex state machine and we go through those steps (and a lot of side effect) every time an integration is installed, but we don't really have it documented anywhere and whole install process is a little opaque.

This brings me to the second point: whenever an integration goes to a bad state (like failed_install) we don't really have a way to restart from the failed step, but we need to force doing it all over. As highlighted in this comment, we could even implement retries on those steps, but currently we don't even have granularity on the steps. It's just a single endpoint and what we ask users to do is usually this:

# Force uninstall
DELETE kbn:api/fleet/epm/packages/<integration>/<version>
{
  "force": true
}

# Force reinstall
POST kbn:api/fleet/epm/packages/<integration>/<version>
{
  "force": true
}

Third consideration is that we could maybe reuse the new input template endpoint to simplify the installation process. The endpoint only returns the inputs, but we could easily reuse part of the logic to return the rest of the integration info and simplify the whole install flow. We could easily add an endpoint under the same namespace that returns the rest of the integrations info and not only the inputs.

[criamico]

Adding some comments per discussion with @nchaulet:

• We should identify what are the different components of a package installation and propose some endpoints to reinstall only those parts, for example for kibana assets something like: POST kbn:/api/fleet/epm/packages<pkg>/<version</kibana_assets>
• Another thing could be to restart a failed install from where it got stopped the previous time instead of restarting from scratch every single time. We already store some information about the failure so we could expand on that. This could be an option to set up explicitly with a flag, since in some cases is better to restart an install from scratch.

[criamico]

@kpollich @nchaulet I converted this ticket to a "meta" one and wrote some more tickets based on our discussion. Feel free to comment/update as needed.

[criamico]

@kpollich I split the items in phase 1 and added some further details in the descriptions as we discussed recently.

[nimarezainia]

@kpollich what should we do with the remaining issues here? should we split them into another meta to be dealt with later?

[kpollich]

@kpollich what should we do with the remaining issues here? should we split them into another meta to be dealt with later?

I don't think there's any reason to split them out into a new issue. This has sort of just become a tracking issue for tech debt to fill out quality sprints which I think is fine.

[nimarezainia]

@kpollich what should we do with the remaining issues here? should we split them into another meta to be dealt with later?

I don't think there's any reason to split them out into a new issue. This has sort of just become a tracking issue for tech debt to fill out quality sprints which I think is fine.

ok I have moved it out to Q1 for now