[ESQL] ability to migrate saved searches to ESQL data views

[drewdaemon]

Describe the feature:
Many customers have legacy visualizations built on saved searches. This is helpful to them because they can

• make query changes in one place and have it reflected in all visualizations (example)
• keep saved search dashboard panels in sync with the visualizations that accompany them (example)

Saved queries seem like they should address this need, but they do not because they are not by-reference. This means that when you use a saved query on a piece of content and then make changes to the query, those changes are not reflected in other content using the same saved query.

ESQL data views will solve this case for Lens. When an ESQL query is updated, all Lens visualizations built on that query will update.

It would be really nice if we could define a migration from saved searches to ESQL query data views. Perhaps this could be part of the "Open in Lens" routine.

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[stratoula]

Atm legacy visualizations that are made with saved searches are migrated to Lens with dataviews/filters. So the migration is done for them correctly.

ESQL dataviews is currently a discussion and is very vague how these will work. When they are part of ES/Kibana then we can start discussing about how we can implement them in Lens. But even then we might need to to change our architecture in Lens before we can accommodate them.

I am closing this for now, not because we dont want to do it, but because it is currently blocked by many other tasks and discussions so I think it is better to have the dataviews first and then we can define our strategy towards Lens and the saved searches. This is why I havent created an issue for that so far

[MakoWish]

Atm legacy visualizations that are made with saved searches are migrated to Lens with dataviews/filters. So the migration is done for them correctly.

Unless I am missing something, just copying filters to the new Lens replacement does not have the same effect. You lose that centrally-managed "saved search" that will update all visualizations based on that saved search. I understand that converting a legacy visualization based on a saved search to a Lens visualization will copy of the same query/filters, but what if we later want to update the query/filters. We would have to update every visualization.

...I think it is better to have the dataviews first and then we can define our strategy towards Lens and the saved searches.

There are two issues with this:

1. When building an Integration package, you cannot define a Data View in place of where a Saved Search would previously have been used.
2. Relying on Data Views to filter a data source subjects you to Data View sprawl. If you have one index that you want filtered many different ways for different dashboards (very common for us), you would need to create a new Data View for each of those dashboards. This not only creates a mess, it is confusing for users.

Just my $0.02.

[drewdaemon]

@MakoWish thanks for the feedback. This is valuable.

You're right that you lose that central editing feature when converting saved-search-based visualization to Lens. I think the most immediate solution is to allow saved queries to work in the same way (issue).

Like @stratoula says, there are a lot of questions around what ESQL data views will look like, how they will be configured, how they will be used, where in the UI they will show up, etc. They may look quite different from the data views of today.

Any strategy we pursue on these will be coordinated across the integrations contributors.