[Fleet] Request diagnostics

[juliaElastic]

Summary

Closes #141074

Request diagnostics action

Added new action for single agent (Agent details page and Agent list row actions) to request diagnostics.
When clicking on the action, an API request is made that creates a REQUEST_DIAGNOSTICS type action in .fleet-actions index.

Diagnostics uploads display

When the action is submitted, the user is navigated to the new Agent Details / Diagnostics tab, which shows the list of pending and completed diagnostics file uploads. The information is coming from the /action_status (for action status) as well as the /uploads endpoint (for file name and path)
By clicking on a diagnostics link, the file should be downloaded in zip.

Failed uploads display:

Expired status was not specified in the design separately, it will be shown like the failed status (with warning icon).

Mock data (blocker)

Currently returning mock data in the /uploads API, because of a blocker in Kibana File Service, see here.

Bulk action

Added bulk action too:

Shows up in agent activity:

The Fleet Server / Agent changes are not there yet, though FS delivers the action, and Agents ack it (looks like default behavior for unkown actions as well)

Confirmation modal

Added a confirmation modal when clicking on action button everywhere, except for the Request diagnostics button on the Diagnostics page.
Open question:

• Do we want to display the confirmation window on the Diagnostics page button too?

Download

Generated file path to download in this format: /api/fleet/agents/files/{fileId}/{fileName}

Decided not to try to use files plugin's API because it doesn't have the Fleet authorization around it.

Screen recording demonstrating the download of an agent diagnostics zip file, that I uploaded using the Fleet Server upload API (using Dan's pr locally)

file_download.mov

Notification

Added toast message to show up when a diagnostics becomes ready, when we are on the Diagnostics tab.

diag_notif.mov

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

@juliaElastic super excited to see so much progress on this already. Do we need to wait for the Fleet Server and Agent dependencies to be merged before we merge this?

[juliaElastic]

@joshdover yes, we need the backend implementation of the action, otherwise it would be an action that never completes.
Also we need a real implementation of the uploads api which is blocked currently.

[juliaElastic]

@elasticmachine merge upstream

[juliaElastic]

ResponseOps changes look good!

We're chatting about this PR right now, since we don't think we've reviewed your usage of task manager - but can now see it when new task types get added (via the check in check_registered_task_types.ts. Thinking it would be good to get an overview of how you're using it, we may have some thoughts/hints/tips.

@pmuellr See the reasoning here: #138870
To summarize, we have Fleet bulk actions that are being run in async mode for more than 10k agents. The Kibana Task Manager is used to catch errors and trigger retry of actions in a Task, and also to check completion in a Task after 5m.
The same logic is used for all types of actions (upgrade, unenroll, reassign, update tags, request diagnostics).

[juliaElastic]

Added feature flag to hide Request diagnostics action by default at 3 places: Agent list single action, bulk action, Agent details action.
For local testing, set this boolean to true.

Feature flag off:

Toggle on:

[kpollich]

Thanks for adding this!

[juliaElastic]

taking the file_id from action results, this part is not yet implemented, asked here: elastic/elastic-agent#1631 (comment)

[juliaElastic]

This callout text might have to be updated if we agreed on a different max age then 30 days.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 89888ae

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	733	735	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	898	912	+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	861.4KB	868.8KB	+7.4KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	115.5KB	116.9KB	+1.4KB

Unknown metric groups

API count

id	before	after	diff
fleet	1001	1015	+14

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	108	113	+5
securitySolution	440	446	+6
total			+19

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	109	115	+6
securitySolution	517	523	+6
total			+20

History

• 💔 Build #86339 failed 3cfaf2e
• 💔 Build #86214 failed b3123fa
• 💔 Build #86142 failed c0755cf
• 💔 Build #86131 failed a486196
• 💛 Build #85818 was flaky b164e02

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[criamico]

I tested the branch locally, I'm not sure if the upload part can be tested yet but up to the point works as per requirements.
In my local I see that the UI gets stuck in loading state when requesting the diagnostic, so it's difficult to test the rest of the flow.

I just have a question. From the agent activity is possible to see that the diagnostics where requested, but it's hard to know for which agent. It's totally fine in case of bulk requests, but would it make sense to consider adding the agent id for single requests?

[criamico]

Code LGTM 🚢

[criamico]

Could you add the new endpoints to the docs? It can be done in a separate PR as well

[juliaElastic]

I tested the branch locally, I'm not sure if the upload part can be tested yet but up to the point works as per requirements. In my local I see that the UI gets stuck in loading state when requesting the diagnostic, so it's difficult to test the rest of the flow.

Thanks for the review!
I can document the steps how I tested e2e locally (checking out the fleet-server pr for the upload API, upload a test file with a script, adding some code to kibana that immediately saves the action result with the file id when taking the action).

I just have a question. From the agent activity is possible to see that the diagnostics where requested, but it's hard to know for which agent. It's totally fine in case of bulk requests, but would it make sense to consider adding the agent id for single requests?

We have an enhancement to add more insight into the agents actioned: #141206

I think in Request Diagnostics case this shouldn't be an issue, because when taking the action on a single agent, the UI should navigate to Agent Details / Diagnostics tab.

[juliaElastic]

Here are the steps how I tested e2e locally:

• check out the fleet-server pr for the upload API: File Upload Feature fleet-server#1902
• start Fleet Server locally

make local
./bin/fleet-server --config fleet-server.dev.yml 2>&1

• upload a test file with a script: extract this zip to your fleet-server directory, run go run ./upload.go elastic-agent-diagnostics-2022-10-04T09-54-34Z-00.zip
  upload_test.zip

• add some code to kibana that immediately saves the action result with the file id when taking the action:
  replace requestDiagnostics function in x-pack/plugins/fleet/server/services/agents/request_diagnostics.ts:

Show code

export async function requestDiagnostics(
  esClient: ElasticsearchClient,
  agentId: string
): Promise<{ actionId: string }> {
  const id = new Date().toISOString();
  const response = await createAgentAction(esClient, {
    agents: [agentId],
    created_at: new Date().toISOString(),
    type: 'REQUEST_DIAGNOSTICS',
    id,
  });
  await bulkCreateAgentActionResults(
    esClient,
    [agentId].map((agent) => ({
      agentId: agent,
      actionId: id,
      data: {file_id: 'a59cdf67-65d9-4070-90ee-6b7c488215aa.b324764f-3631-40a5-8ff1-756fbbfbc3ac'} // replace with the actual file id, you can check this in console `GET .fleet-agent-files/_search`
    }))
  );
  return { actionId: response.id };
}

• add data parameter to bulkCreateAgentActionResults in actions.ts:

Show code

export async function bulkCreateAgentActionResults(
  esClient: ElasticsearchClient,
  results: Array<{
    actionId: string;
    agentId: string;
    error?: string;
    data?: any;
  }>
): Promise<void> {
  if (results.length === 0) {
    return;
  }

  const bulkBody = results.flatMap((result) => {
    const body = {
      '@timestamp': new Date().toISOString(),
      action_id: result.actionId,
      agent_id: result.agentId,
      error: result.error,
      data: result.data,
    };

    return [
      {
        create: {
          _id: uuid.v4(),
        },
      },
      body,
    ];
  });

  await esClient.bulk({
    index: AGENT_ACTIONS_RESULTS_INDEX,
    body: bulkBody,
    refresh: 'wait_for',
  });
}

• Go to Agent Details UI and trigger Request Diagnostics action. The file should immediately be available to download, and the uploaded zip file should be the same as you uploaded.

[juliaElastic]

NOTE: this feature is turned off in 8.6 as the backend is not ready yet. The feature is planned to be enabled in 8.7.

[amolnater-qasource]

Hi Team,

We have created 11 testcases for this feature under our Fleet test suite at link:

• Fleet>Agent Diagnostics

Please let us know if any other scenario is missing from our end.
Thanks