[Lens] Use accordion menus in field list to fix the "where are my fields" problem

[wylieconlon]

There are multiple issues about the confusing first-time experience when using Lens, especially related to our decision to show a subset of fields that we know have data in the time range. To solve these issues, I propose the following logic:

• Show the complete list of fields in the index pattern
• Keep the toggle in the "filter by type" popover to enable filtering for fields with data
• Keep the current styling where fields without data are shown with a gray background to reduce contrast
• When we detect that there are over > 1,000 fields, add a really obviously placed button in the field list that will add the "fields with data" filter.

I am not worried about the performance implications of showing 1000s of fields because we already use lazy-loading in the various components that are affected.

Related issues:

Confusing empty state #58276
Sampling method is not intuitive #58330

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[AlonaNadler]

I still think that showing only fields with data is the right way to go.
We have a few potential issues. The sample might not be large enough. The switch to change the default is too hidden.

Before we change the default behavior, which reasonate if you think about beats which most fields are empty.
What do you think about moving the switch to a visible place? Quick mock:

cc: @cchaos

[wylieconlon]

Changing the default solves a different problem than moving the button: Lens currently looks empty on first page load. You've previously asked for solutions to that problem, is it no longer important?

[AlonaNadler]

Lens currently looks empty on first page load

How will changing the default help with the empty look?

[wylieconlon]

If the default time range in Kibana doesn't contain any data, then users see a blank list of fields on the left hand side. Changing the default will let them interact with the list of fields in this case.

[AlonaNadler]

Is there a way to only show all fields (with and without data) when there is no data in the initial time range?

[cchaos]

No data = No chart

If there is no data in any of the fields because of the selected time range, just simply showing all fields without data does not help the user actually create a visualization. The visualization itself (once configured) will still not show a chart because no data exists. Therefore, we need to more persistently push them to adjust their time filter.

We already adjusted the original message from having a direct "Show all field's" button in the empty state, to directing them to either adjust the time or show all fields. If the "Show all field's" button wasn't working, then moving the "Show fields without data" toggle outside of the popover won't either as they're the same concept.

They need to adjust their time range

The problem with showing all fields by default is that they'll never see the "You don't have any data because of your selected time range" message. So they'll continued to configure a chart that will never show results.

I finally got around to creating a mock as to how we can accomplish pushing them towards the time filter using EuiTour. The implementation part might be a bit tricky being that the time picker is a global component and not specific to Lens.

Figma proto

[cchaos]

Another thing that will go a long way, is remembering the previous time range on refresh.

[wylieconlon]

@cchaos @AlonaNadler I'm deliberately trying to separate the issues with the time picker and the list of fields: the most common question I hear about Lens is "where are the fields I'm expecting to see?" I 100% agree that the time picker is important, but I would really like to focus on whether this proposal makes sense on its own.

[cchaos]

I'll add the mock to #58276 as well, but I my point is that showing fields without data does not help them create a chart. It all comes back to time.

[wylieconlon]

It also doesn't help to create a chart if you know what you're trying to visualize, but can't find it in Lens because we are hiding it. The default we currently use, where we completely hide fields, has been confusing users since release. Instead, we should use the existing style for this: lower contrast on fields we think don't have data.

[flash1293]

The tour component guiding the user to change the time range definitely makes sense and we should add it, but I agree with @wylieconlon in this case, because there are other scenarios that could hide fields in a confusing way which are not solved this way:

• There is data in the time range, but not for the fields the user wants to visualize (and knows they exist)
• By chance the sampling excludes all documents that would contain a certain field, even for the given time range
• The configured filter excludes documents which would contain a certain field

Another option would be to move the fields without data to a separate group below the fields containing data (maybe even collapsed by default). IMHO showing all fields initially is the safer default - if the user has a ton of fields, they can still hide it with the toggle. We could persist the users choice per index pattern in local storage and make it configurable as well via advanced settings.

In summary, my suggestions:

• Implement tour component to hint at time range if there is no data at all in the current time range
• Show fields without data in the field list by default (exact design tbd)
  • Make this configurable via a flag in advanced settings
• Persist the "Show fields without data" preference for the current user per index pattern (if set overwrites the advanced setting)

[cchaos]

Fair points @wylieconlon & @flash1293 , I was only thinking of the case when no fields have data at all.

I remember the idea of setting the fields without data apart from those with data was floated around a long time ago and could still be a valid idea, but can't remember why we initially decided against it. I'm also always for saving user initiated settings, so yes please! Though I think it's odd to do it by index pattern and the Lens app should just remember the last state of the options.

I'll work on some mocks for the grouping (just as an idea), and making the filter button more prominent.

Also to note, we're working on an effort to align the fields lists of Discover and Lens (and possibly then other apps as well) so we may want to consider how this could benefit/effect Discover and vice versa.

[flash1293]

Though I think it's odd to do it by index pattern and the Lens app should just remember the last state of the options

No strong opinion, I thought it would be nice to have different preferences for different index patterns because some can be very large with consistent existing fields (e.g. beat indices) while others could be sparse and just contain a handful of fields. Saving separate preferences would avoid having the user toggling the switch every time they are switching between those index patterns. But it might be too much magic again, so I'm fine with a single boolean as well - better than the current state.

[wylieconlon]

Renamed to indicate the proposed design and implementation of this

[mbondyra]

I'll work on some mocks for the grouping (just as an idea), and making the filter button more prominent.

@cchaos just a headsup – any updates on design for this?

[mbondyra]

Designs:

[cchaos]

^^ Fixed the mocks above