Add filter to alerting flyout in logs UI

[mukeshelastic]

Describe the feature:
Current alerting mock up as described here allows users to define an alert but you can not specify a filter condition to restrict the alert condition evaluation. Cases when you'd like to evaluate alert condition when certain conditions are met - only specific index, specific hosts or service names etc.

Filter should look similar to alerting flyout on metrics UI

[jasonrhodes]

Things to talk about: do we need this in logs if we can just set it as part of the condition? Can metrics fake this with a condition as well?

[Kerry350]

Things to talk about: do we need this in logs if we can just set it as part of the condition? Can metrics fake this with a condition as well?

I wanted to come back to this due to a recent conversation with a solutions architect.

With regards to just adding a filter, we made the right decision to shelf this as conditions can mimic all of that functionality. It's not needed (for logs at least).

However, an idea I'd suggested in a recent(ish) meeting was "reusable filters", filters that could be shared between Observability alert types. Again, we (correctly) shelved the idea due to there being no concrete user story behind this.

After speaking with an SA though, it was clear there's a real use case here:

Reusable filters would really make some happiness...
give an example of reusable filters?
I will need to get back... but the simple one that I see is based on tags / meta data to segment in large orgs.
tag.site : "sandiego" AND tag.admin_contact : "UNIX_ADMIN"
So Alerts get group / segmented by Data Center and Resolver Group
[LARGE CUSTOMER] will use filters like that an potentially 100s of alerts

Do we think this is worth exploring? If we still think this has no legs that's also fine, just wanted to share the new information 😄

[jasonrhodes]

Interesting! So filters would be more than just free text but also have a crud aspect for saved filter items?

[Kerry350]

Interesting! So filters would be more than just free text but also have a crud aspect for saved filter items?

Yeah, in my head I imagine this working a little like "Saved views" in Metrics. You could either enter things in the input to make a new filter, or apply a pre-saved one (even cooler if at the alerting top level you could set some sort of default). Those filters would most likely live within a saved object.

I don't know the technicalities around sharing saved objects across plugins, so I'm sure this would need R&D if we wanted to pursue it.

[weltenwort]

Kibana has the concept of saved queries by now (https://github.com/elastic/kibana/tree/16a51a18f82a8d289e56563d4ff452d11882a193/src/plugins/data/public/query/saved_query). Maybe we could integrate with that?

[weltenwort]

related: #53123

[Kerry350]

@weltenwort Oh, that's nice. It looks like this would work for the use case, and if so +1 to using that service.

[jasonrhodes]

Things to talk about: do we need this in logs if we can just set it as part of the condition? Can metrics fake this with a condition as well?

I wanted to come back to this due to a recent conversation with a solutions architect.

With regards to just adding a filter, we made the right decision to shelf this as conditions can mimic all of that functionality. It's not needed (for logs at least).

However, an idea I'd suggested in a recent(ish) meeting was "reusable filters", filters that could be shared between Observability alert types. Again, we (correctly) shelved the idea due to there being no concrete user story behind this.

After speaking with an SA though, it was clear there's a real use case here:

Reusable filters would really make some happiness...
give an example of reusable filters?
I will need to get back... but the simple one that I see is based on tags / meta data to segment in large orgs.
tag.site : "sandiego" AND tag.admin_contact : "UNIX_ADMIN"
So Alerts get group / segmented by Data Center and Resolver Group
[LARGE CUSTOMER] will use filters like that an potentially 100s of alerts

Do we think this is worth exploring? If we still think this has no legs that's also fine, just wanted to share the new information 😄

@Kerry350 can you write up a new ticket describing what you'd like to see for this and how it could leverage saved queries? Then just add it to the Logs project board (no column) and it'll appear in the triage list for us to think through how to prioritize. Thanks!!!