[Fleet] Agent activity flyout enhancements

[jillguyonnet]

Summary

Closes #153232

Enhancements addressed in this PR:

1. Add a "Review errors" badge that is rendered when new agent activity with errors are loaded. When clicked, the badge opens the Agent activity flyout and is de-rendered.
2. Enhance the backend logic behind GET /agents/action_status.
   • Introduce unified pagination for agent actions as well as agent policy changes.
   • Guarantee ordering by descending timestamp across both types (agent and agent policy) actions.
3. Render "Show more" button in Agent activity flyout at the bottom of actions list that loads 10 more actions.
   • The button is disabled after clicking it once does not yield more actions. This behaviour could be the object of a future enhancement.
4. Render "Go to date" button with date picker that sets a date filter (of 1 day). Conditionally render a "Clear selected date" button to reset the date filter.
5. Enhance the backend logic behind POST /agents to allow actionIds to either refer to agent or agent policy actions ids. This allow the "View agents" button in the Agent activity flyout to work with agent policy change actions.
   • It should be noted that this button will display the agents currently assigned to the policy, which may differ from the agents that were assigned to it at the time of the change. A tooltip was added to make this clear to the user.
6. Hide completion date if set to 1 Jan.
   • Added a comment for later investigation.

Steps for testing

1. Create a couple of agent policies and enroll a few agents on both policies.
2. Perform some upgrades, tag assignments and policy reassignment on the agents. Make some changes to the agent policies. You should aim to have more than 20 changes in total.
3. Open the Agent activity flyout. Your actions should be listed and the "Show more" and "Go to date" buttons should be rendered at the bottom of the list.
4. Click "Show more" until all actions are loaded (10 at a time). The button should become disabled after there are no more actions to load.
5. Test the date filtering: if you select a date (e.g. today) with actions, they should be rendered; if you select a date with no actions, the empty state should be rendered. Test that the "Clear selected date" button gets rendered in both cases and correctly resets the date filter.
6. Test that the "View agents" button correctly filters agents for agent policy changes.
7. Generate some action errors on purpose: one way to do this is to try to upgrade an agent to a nonexistent version number by typing it manually. On the following agent data refresh (max 30 seconds), the "Review errors" badge should appear and hovering on it should read "There is 1 new agent activity error. Click to view". If other errors are generated before the flyout is open, the count should increase accordingly. Clicking on the badge should open the flyout and clear the badge (reset the count of new activity errors). NB: if another page is loaded before the Agent activity flyout is open, the badge will be cleared.

Screenshots and screen recordings

Agent activity flyout with no date filter selected and remaining actions to load:

Agent activity flyout with a date filter selected and all actions loaded:

Agent activity flyout with a date filter selected and no actions for that date:

Tooltip on the "View agents" button for policy changes:

"Review errors" badge with 1 new error:

"Review errors" badge tooltip details (1 and 2 errors):

"Show more" and date filtering functionalities:

Screen.Recording.2024-04-05.at.16.38.28.mov

Tooltip on the "View agents" button for policy changes:

Screen.Recording.2024-04-05.at.17.02.19.mov

"Review errors" badge:

Screen.Recording.2024-04-10.at.12.42.39.mov

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[jillguyonnet]

Did a partial push with the WIP "show more" function functionality to discuss some open implementation questions - @juliaElastic @criamico if you could have a look, you probably have the most context 🙏

Question 1: backend pagination behaviour

The current backend implementation for retrieving action status items from /agents/action_status is a little complex behind the scenes. Essentially, what it does is:

1. Fetch agent actions from .fleet-actions. This currently supports pagination.
2. Transform agent actions into action results.
3. Fetch agent policies from .fleet-policies. This currently does not support pagination: it fetches the latest 10 actions, which are typically processed into less action results (more on that below).
4. Concatenate the agent and agent policy action results. This is what is ultimately returned.

As discussed in the issue, it would make sense for agent policy actions to support pagination. The question is what should we expect from the parameters, since we concatenate two results. For example, with /agents/action_status?page=0&perPage=20, should we query:

• 20 agent actions + 20 agent policy actions
• 10 agent actions + 10 agent policy actions

I temporarily went with the first option in this commit.

Note that because of the way agent policy actions are processed (see below), they seem to be roughly halved, so with /agents/action_status?page=0&perPage=20 we'd actually return something like:

• 20 agent actions results + ~10 agent policy action results ~= 30 action results
• 10 agent actions results + ~5 agent policy action results ~= 15 action results

Question 2: agent policy actions processing

My original intention was to introduce UI logic to disable the "Show more" button if we don't have any more actions to load. I found it doable with agent actions, since we can just get the number of total hits from the ES query and then know whether there are more we can fetch, but it's not as easy with agent policy actions. The reason is that they are processed in the following way:

1. Fetch the agent policies from .fleet-policies. For example, we get 20 and there are 34 in total.
2. Transform the agent policies so that action results are identified by ${hit.policy_id}:${hit.revision_idx}. This combination, however, is not unique: for example, I'm seeing that a lot of agent policy changes result in 2 or 3 items in .fleet-policies with the same policy_id and revision_idx combination, so that the 20 policies we got before can result in e.g. 10. I'm not clear on the reason behind that, as I didn't spot any obvious differences between the items.
3. These are then processed into agent policy change actions.

I decided to scrap the UI logic to disable the "Show more" button for now, since I can't see a way to predict how many policy change actions we have in total. Any thoughts on that?

Question 3: UI loading implementation

I'm not sure about this WIP implementation, maybe you can help: I originally intended to only load actions as needed, e.g.:

1. Load first 20 actions on first render
2. Load the next 10 when clicking "Show more" and add those to the component's state

But I ran into issues because the component re-renders often and includes refresh logic, so it fetches the actions often. I found it difficult to fit this with pagination, so I eventually resorted to a simpler approach where page is always 0 and perPage changes. I don't like it however, as it could result in very large queries and it doesn't look great performance-wise. Can you advise?

[juliaElastic]

Thanks for the detailed explanation!

The question is what should we expect from the parameters, since we concatenate two results.

Good point, I would go with your suggestion, I think there is no harm to show more actions rather than less, so it requires less clicking of Show more.
Ideally it would be best to always load the same number of action results at a time (e.g. 20), but that would get complicated to keep track of the pagination after transforming the results, so we might not want to do that.

I'm seeing that a lot of agent policy changes result in 2 or 3 items in .fleet-policies with the same policy_id and revision_idx combination, so that the 20 policies we got before can result in e.g. 10. I'm not clear on the reason behind that, as I didn't spot any obvious differences between the items.

The reason of multiple policy docs with the same revision is that kibana creates one doc per revision with coordinator_idx:0, and then fleet-server picks up the change and creates a doc with coordinator_idx:1 (same id and revision). I think we should add a filter on coordinator_idx:0, so that would make the result count more predictable (the coordinator_idx:1 docs are not relevant for kibana). I should have considered this in the original implementation.

But I ran into issues because the component re-renders often and includes refresh logic, so it fetches the actions often. I found it difficult to fit this with pagination, so I eventually resorted to a simpler approach where page is always 0 and perPage changes. I don't like it however, as it could result in very large queries and it doesn't look great performance-wise. Can you advise?

I see your point, I'll give this some thought.

[criamico]

Thanks @jillguyonnet for the thoroughly investigation!

Regarding the pagination, you mention that:

with /agents/action_status?page=0&perPage=20 we'd actually return something like:
20 agent actions results + ~10 agent policy action results ~= 30 action results

It's kinda odd to request 20 items and get 30, but if there's no easy way to fix the agent policy actions it should be fine; I agree with @juliaElastic that is better to get more than fewer.

[jillguyonnet]

Thank you @juliaElastic and @criamico for your input!

I agree that it would be ideal if we got 20 (combined) action results if perPage is 20. I think it would be doable if we could exactly predict the number of agent policy actions results, but I'm not convinced of that yet.

On that, thanks for the explanation about the multiple policy docs, it makes sense to add a filter for coordinator_idx:0 👍 In my setup I see one policy change with 3 docs, 2 of which have coordinator_idx:1. I'm not sure why that is, but it's making me doubt that we can predict how many agent policy actions results we'll get.

[juliaElastic]

Related to the 3. question, I think it's unlikely that the users would have more than 1k actions and that they would click on Show more that many times to really cause an issue with loading from 0 up to the total actions.

The alternative would be to do multiple API calls on every refresh to query each page separately, and the results might get inconsistent (e.g. a new action appears on the first page, and the last action on the first page moves to the second page).

Another idea to reload the in progress actions only, because completed ones are not going to be updated on the backend. This would add some complexity though to reorder on UI and merge with the completed actions that are not reloaded.

I would go with the first approach to start with, and revisit later if the performance becomes an issue.

[juliaElastic]

Great to see the refactor to smaller components!

[juliaElastic]

what is the purpose of this additional useEffect?

[jillguyonnet]

To fetch the actions when we want more of them (after clicking "Show more"). I think it wasn't working without it, but I can check that, as I've done a lot of back and forth with the UI code.

[juliaElastic]

I would think it should work without, because nActions is a dependency of loadActions

[jillguyonnet]

OK, thanks, I'll check it. Definitely don't want more useEffect than we need 🙂

[juliaElastic]

Should we add a comment on why the second useEffect is needed? I tried without it and didn't work, I don't have a better idea now than to leave it in.

[jillguyonnet]

Oh yeah, sorry for not commenting back on this question. I wasn't successful either, I will add a comment 👍

[jillguyonnet]

By the way, something I've observed while working on this "Show more" functionality: something potentially very confusing for the user is that newly loaded actions are not necessarily older because of that concatenation between agent and agent policy actions. For instance, say that on first query we have:

• 20 agent actions between 2024-01-15 and 2024-01-17
• 8 agent policy actions between 2024-01-10 and 2024-01-16

In both cases we are getting the 20 most recent items and we got all the agent policy actions (there are no more). When we query more, we get an additional 20 agent actions between 2024-01-12 and 2024-01-15, which are more recent than the oldest (agent policy) actions we already have. In the UI, this means they'll get listed above and not below and the user might be wondering if the button works.

I'm wondering if that already caused confusion without the pagination, as the flyout appeared to drop recent elements and keep older ones.

I'm not sure how easy it would be to fix this - one idea could be some logic where we introduce a shared time window and fetch both types of actions until we reach the pagination limit, which definitely sounds more complex than the current implementation. I'll give this more thought, but please share if you have any suggestions or concerns.

[juliaElastic]

I'm wondering if that already caused confusion without the pagination, as the flyout appeared to drop recent elements and keep older ones.

Good catch, I think this definitely caused confusion before.

I'm not sure how easy it would be to fix this - one idea could be some logic where we introduce a shared time window and fetch both types of actions until we reach the pagination limit, which definitely sounds more complex than the current implementation. I'll give this more thought, but please share if you have any suggestions or concerns.

Good idea, we could do something like query one day at a time in a loop until we reach N results, though it might end up a lot of queries.
Or, perhaps read the first 100 (or double the desired N results) hits from actions, to reduce the likelihood of getting older policy changes.

[jillguyonnet]

Good idea, we could do something like query one day at a time in a loop until we reach N results, though it might end up a lot of queries.
Or, perhaps read the first 100 (or double the desired N results) hits from actions, to reduce the likelihood of getting older policy changes.

I like those suggestions! Maybe it would be worth exploring that, I'm concerned that the current implementation might be too confusing (this may have to do with the fact that I spent an hour trying to debug an inexistent issue because I couldn't see that my new actions were higher up and it looked like nothing had happened 😬).

[jillguyonnet]

I've had a think about the API pagination. I think the following approach would ensure we preserve combined sorting by timestamp (between the two kinds of actions):

• For page=0 and perPage=N: fetch the first N results from agent and agent policies actions (max 2*N results), sort these and take the first N.
• For page=1 and perPage=N: fetch the first 2*N results from agent and agent policies actions (max 4*N results)), sort these and take results N to 2*N.
• Etc.

So the query becomes bigger as page increases, but at least for the first pages it should work with only one query.

[jillguyonnet]

Something I think would be good to fix: I've noticed that the items from .fleet-actions-results sometimes have '@timestamp': '0001-01-01T00:00:00Z' (which for some reason formats to Jan 01, 1 12:34 AM in the UI). I'm not sure why (seems to happen with agent upgrades, but there's also an example of agent policy applied to inactive agents), I'm guessing some default timestamp is applied. I'd suggest not putting the date in those cases (just say "Completed").

[criamico]

About the UI loading implementation, where

the component re-renders often and includes refresh logic, so it fetches the actions often

We should probably verify if the agent activity flyout has a timeout that refetches the actions every x seconds, but if that's the case then having the show more button in that panel doesn't play well with it, because of the behaviour you noted.
I think that pagination (or lazy loading) functionality is better suited for a static panel.

This is just an idea, but maybe we could show a button that opens a new static view in the flyout where we show the summary of the selected actions and the show more could live there. This way it would have more space available to fit a long list of actions. However I'm not sure if this pattern exists elsewhere in Kibana, so it would be good to verify it first.

[jillguyonnet]

I think that pagination (or lazy loading) functionality is better suited for a static panel.

Agreed, either a dynamic list (e.g. 20 most recent actions, periodically refreshed) or a static list (no refresh, load more functionality) would make sense to me. Your suggestion might be a good improvement.

I also think it might be worth understanding why the component is rendered so many times in the first place.

[juliaElastic]

interesting, it sounds like a bug, maybe we could add a comment where this value is coming from, if we want to investigate later

[jillguyonnet]

@juliaElastic @criamico @kpollich Can I ask for some input on points 1 and 5:

1. Review errors button in agent list

It's worth noting that at the moment actions data is only fetched when the flyout is open. I was checking how we might implement this functionality in a minimal way, but I don't think we can get around querying new actions from the page component, either periodically or after changes are made (the latter way might not even work, because it might take some time for an error to happen, e.g. in the case of a failed agent upgrade). Can you see any cause for concern about this?

5. View agents button needs to be updated to work with new Agent Policy Change section

Longer question (apologies), which starts with my understanding of how policy change actions are generated on the fly in this function. Given a policy change action, say:

1 agent applied policy change
Agent policy 1 changed to revision 2 at Mar 19, 2024 10:33 AM.

I would expect that this means that 1 agent was affected by the policy change at that time. However, say I assign another agent to this policy a few days later and then make a change to the policy. Then the above change action becomes:

2 agents applied policy change
Agent policy 1 changed to revision 3 at Mar 19, 2024 11:56 AM.

Which doesn't make sense to me, as this change was really applied to 1 agent, so it looks like a bug.

I've dug into the code and I think I know why that is: I think that older actions (revision less than current) have an incorrect number of assigned agents, which is the number of assigned agents on the current revision. Details:

1. Fetch agent policies with revision > 1.
2. Transform this data into an object with keys given by policy_id:revision_idx. So far change history is preserved.
3. Then, in order to know how many agents were affected, fetch an aggregation from .fleet-agents that gives us, for each policy id, the number of agents assigned to this policy and how many agents per revision (agents_per_rev). This is where I get confused: from what I can see, we'd want the agents_per_rev aggregation to have "historical" data, e.g. all revisions > 1 leading to the latest; however, from what I can see, it only gets the latest revision (which would make sense, as AFAIK the agents data doesn't hold that historical data?).
4. This gets transformed into a map: looking at the code, it would seem again that we expect the map entry associated to an agent policy id to contain an array with the historical agents per revision data. I've printed this and always see a size-1 array with the latest revision, which is coherent with the previous point but, I think, not what we want.
5. The rest of the function processes this to set the number of assigned agents to the policy.

I came across this as I was investigating how the "View Agents" button might work for policy changes, but unless I'm mistaken this would involve fixing this first and making sure we so have access to this historical data. Perhaps we would need is to generate and store policy change actions that record which agents were affected (this is what is done with agent actions). I've hidden the "View Agents" button for policy changes as a temporary measure.

[juliaElastic]

1. Review errors button in agent list

I'm not against including the action status poll without opening the flyout, but I'm wondering if this small feature is worth adding it.
Perhaps we could make a more efficient API call to determine if there are new error results, something like: query .fleet-actions-results where error field exists in the last 30s (or the time since the last poll).

5. View agents button needs to be updated to work with new Agent Policy Change section

This is what I had in mind when implementing the policy change agent activity:
.fleet-agents contains the current state of agents (no historical data), which includes the policy revision the agent is currently at (because agents need to check in and ack the policy change in order to update their policy revision)

So when .fleet-policies has a revision:3 for example, the query is checking how many agents on this policy with revision <= 3. Agents which are on less than revision 3 are not yet completed the policy change (in progress), agents on revision 3 completed.

The logic also considers older revisions, e.g. revision:2 would count agents the same way + if there are any agents on newer revision than 2, it would count those as completed revision:2 policy change (supposing that if it received revision 3, in the past it had to receive revision 2).

This logic is not perfect, like you said, it is sometimes incorrect when you reassign agents to another policy e.g. if you do a policy change and then reassign, the previous policy change would say 0 agents, because there are no longer any agents with that policy, and there is no way to know which agents were assigned in the past.

The way to fix this would be to add agent docs for policy change to keep track, though that would be some overhead and potential changes needed in fleet-server to ignore them. I was thinking about this before, but wasn't sure if we need this much complexity for Agent activity. The current logic is best effort.

[jillguyonnet]

Thank you @juliaElastic for your quick feedback!

Review errors button in agent list

Yes, I wasn't sure either if this small feature would be worth the extra API calls. Your suggestions is interesting, though, I'll give it a try.

View agents button needs to be updated to work with new Agent Policy Change section

Thanks for confirming 🙏 I agree with you that the changes to introduce proper policy change history seem too complex for the purpose of this issue. For the UI, I suppose we have a choice then: either hide the "View Agents" button for policy changes, or show agents currently assigned to the policy (which sort of matches what the API is doing, but may confuse the user if more agents are assigned).

[juliaElastic]

For the UI, I suppose we have a choice then: either hide the "View Agents" button for policy changes, or show agents currently assigned to the policy (which sort of matches what the API is doing, but may confuse the user if more agents are assigned).

I would go with showing View Agents with filtering on the policy, though we should probably document/add some explanation on how it works.

[jillguyonnet]

I would go with showing View Agents with filtering on the policy, though we should probably document/add some explanation on how it works.

How about adding a tooltip that says View agents currently assigned to this policy? Either directly on the button or on an i icon next to it.

[juliaElastic]

I noticed this error without any agents, not sure if existing issue. Can we catch the 404 error when querying action statuses?

[juliaElastic]

Tested the latest changes, looks great overall!

One thing that might be a bug, when I click on View agents for a policy change that doesn't have any assigned agents, the agent list is not filtered down. Should we make the View agents button disabled if there are no assigned agents?

[jillguyonnet]

I noticed this error without any agents, not sure if existing issue. Can we catch the 404 error when querying action statuses?

@juliaElastic How exactly did you set this up? In the absence of a fleet server, I only see the Add a Fleet Server state and no Agent activity button.

[juliaElastic]

I noticed this error without any agents, not sure if existing issue. Can we catch the 404 error when querying action statuses?

@juliaElastic How exactly did you set this up? In the absence of a fleet server, I only see the Add a Fleet Server state and no Agent activity button.

I had xpack.fleet.internal.fleetServerStandalone: true in kibana.dev.yml, I usually use it to test some things without having to enroll fleet-server / or when running fleet-server as standalone.

[juliaElastic]

LGTM, thanks for addressing the review comments!

[jillguyonnet]

Thank you for your thorough testing and feedback @juliaElastic! The CI failures look unrelated now, mostly timeouts.

QQ - were you able to successfully test the "Review errors" badge? I found it a little tricky to get errors on purpose.

[jillguyonnet]

@elasticmachine merge upstream

[juliaElastic]

QQ - were you able to successfully test the "Review errors" badge? I found it a little tricky to get errors on purpose.

I haven't tested that, I can try to test locally. If hard to reproduce an error, we could try inserting errors manually to ES.

[juliaElastic]

I tried to test it, but didn't see the badge showing up.

Steps I took:

• performed an upgrade action on an offline agent
• took the action id and agent id from .fleet-actions
• inserted a doc to .fleet-actions-results with an error as fleet-superuser
• the error is visible in Agent activity

  curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \
  -H'x-elastic-product-origin:fleet' \
  http://localhost:9200/.fleet-actions-results/_doc -d '
  {
    "@timestamp": "2024-04-10T09:11:24.773Z",
    "action_id": "55275c23-0006-4c2b-baf2-d6696051cb98",
    "agent_id":"1dfdd655-6530-48bb-9b1a-94250cdf683c",
    "error": "upgrade version invalid"
  }'

[jillguyonnet]

I tried to test it, but didn't see the badge showing up.

Thanks for that. I'm not sure from the steps you outlined, but maybe it's a limitation of the current implementation, as in the new errors will only be picked up when the Agents page is open and there are new errors that came in in the last 30 seconds. If you load the page and the errors are older, the badge won't show up. I'll write a description in the followup, as this is something we might want to revisit in the future but may require some substantial changes in order to improve.

In the mean time, I found a way to consistently get an error by trying to upgrade to an inexistent version number (8.12.7 in this case). The badge shows up, but the number of errors in the tooltip is incorrect, which I see is due to a logic error in the custom hook. I will fix that now.

[jillguyonnet]

@juliaElastic I pushed a fix along with a screen recording and updated steps in the PR description. Let me know if that looks OK to you 🙏

[juliaElastic]

Is it necessary to refresh every 10s? I would keep the 30s to keep the API requests down.

[jillguyonnet]

Oops, I changed that for testing only! Sorry, reverting.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: cbc3a7e

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	984	992	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.3MB	1.3MB	+4.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	158.7KB	158.7KB	+12.0B

History

• 💔 Build #202527 failed 4eabb3d
• 💔 Build #202489 failed 4ae1da3
• 💚 Build #202427 succeeded 52fdef7
• 💔 Build #202243 failed c72fc60
• 💔 Build #202215 failed ca15300
• 💛 Build #202041 was flaky 1e7190d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jillguyonnet

[juliaElastic]

🚀

[kpollich]

Amazing work here @jillguyonnet! I took a look through the code and things look very well factored overall. Breaking down the activity flyout into smaller components definitely has a noticeable impact on the readability of the code here - made it very easy to review. 🚀