[Lens] Implement rare terms #121500

flash1293 · 2021-12-17T10:15:53Z

This PR adds support for rare terms to aggconfigs and Lens.

Be aware of the following things:

No sorting
No "size" parameter (only max doc count like supported by API)
No other bucket, no missing values
No multi fields

If a precision warning is shown and the current sorting is ascending count, there's a call to action button to turn it into rare terms:

Title changes automatically to "Rare values of"

Sorting can be enabled from the flyout by picking "Rarity" in the ranking options:

If it's picked, the "add field" button, the "size" input, order direction and existing advanced options are greyed out because they are not applicable. A new input is shown for the max doc count. Max doc count is defaulting to 1 which is also the min value. The maximum value is 100 (both values set by Elasticsearch in the same way)

flash1293 · 2021-12-17T10:16:43Z

cc @MichaelMarcialis tests are failing, but I think we can start thinking about the open UI questions already

flash1293 · 2021-12-17T14:16:58Z

@ghudgins @MichaelMarcialis Just discussed this in the dev sync and @dej611 suggested to make this a new function instead of hacking it in as a rank option (there would be "Rare values" and "Top values"). In the rare values config UI there's just max doc count, no disabled features.

ghudgins · 2021-12-17T14:28:01Z

i'm speaking purely user interface here but "top values" implies the top of some list ordered by something. I think rare terms fits better in the top values function for this reason....if we have "rare values" then we could apply this logic and have "alphabetical values" and it gets too verbose. if there's a good technical reason to keep it separate then we can adjust but using Lens as a translation for these technical details was something I was hoping we could work in. just my opinion though... @MichaelMarcialis?

flash1293 · 2021-12-17T14:34:43Z

No technical reason to keep them apart, it doesn't matter much either way - our operations architecture is flexible enough to handle both ways well

dej611 · 2021-12-17T14:53:59Z

i'm speaking purely user interface here but "top values" implies the top of some list ordered by something. I think rare terms fits better in the top values function for this reason....if we have "rare values" then we could apply this logic and have "alphabetical values" and it gets too verbose. if there's a good technical reason to keep it separate then we can adjust but using Lens as a translation for these technical details was something I was hoping we could work in. just my opinion though... @MichaelMarcialis?

I think I agree with you for the general case. I have no strong opinion here, but just thought it may be useful to discuss this peculiar case.
The particular case of Rare terms vs other ranking logics here is mostly a UI/UX problem, where the user has everything disabled or "reduced" just picking a Rank by option: while for a regular single term case this means losing few bits like the Other and Missing categories, in case of multi terms the Rank by option "Rarity" gives the illusion to work but effectively will remove the full list of terms already configured. That was the lead for the proposal of a specific function.

flash1293 · 2021-12-17T15:08:08Z

I don't have a strong opinion here - given that this is more of an expert feature anyway (an everyday user won't look at the rare terms) I'm leaning slightly towards keeping it as a special sorting option so it's not bloating the list of operations with "weird" things.

MichaelMarcialis · 2021-12-17T19:44:53Z

I don't have a particularly strong opinion here regarding whether this should be a standalone quick function versus a sorting option within the existing top values function. My only concern with adding it as a sorting option to the top values function is discoverability. However, if we're comfortable making the feature a bit less discoverable, then it's a non-issue (which I assume is indeed the case, given Joe's description of it being a more expert feature).

elasticmachine · 2022-01-10T17:55:53Z

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

elasticmachine · 2022-01-10T17:55:54Z

Pinging @elastic/kibana-app-services (Team:AppServicesSv)

flash1293 · 2022-01-10T17:57:11Z

Not sure why docs build is failing - the build contains this line:

17:18:34 runbld>>> There were errors while communicating with Elasticsearch.   There may be data missing in the Build Stats cluster.   Infra has been notified.

flash1293 · 2022-01-10T18:08:41Z

@elasticmachine merge upstream

Dosant

rare terms agg addition lgtm

flash1293 · 2022-01-17T09:03:55Z

@elasticmachine merge upstream

dej611

Tested locally and it works.
Left some minor comments, and found some edge case problem with the labelling.

When the label is cleared the placeholder is not handling the Rarity flag properly:

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/index.tsx

dej611 · 2022-01-17T13:56:40Z

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/index.tsx

+            label={i18n.translate('xpack.lens.indexPattern.terms.maxDocCount', {
+              defaultMessage: 'Max doc count per term',
+            })}
+            maxValue={100}


Where this value come from?

It's an Elasticsearch limit: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-rare-terms-aggregation.html#search-aggregations-bucket-rare-terms-aggregation-max-doc-count

Moved it into a constant, magic numbers are bad

flash1293 · 2022-01-18T12:10:03Z

Good catch about the label thing - the problem is that the label input is storing the "old" value internally but it doesn't refresh if the order by is changed. Fixed it by re-rendering the label input in case the default label changes and the user hasn't changed it yet. This is fixing the same issue for the case if the user is changing the field name, then clearing the label input. @dej611 could you review again?

kibana-ci · 2022-01-18T13:27:56Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	518	520	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`data`	2758	2766	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	1.0MB	1.0MB	+2.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`core`	291.1KB	291.2KB	+72.0B
`data`	446.5KB	448.8KB	+2.3KB
`visTypeHeatmap`	10.3KB	10.3KB	+42.0B
`visTypeMetric`	8.7KB	8.7KB	+14.0B
`visTypePie`	7.9KB	7.9KB	+28.0B
`visTypeTable`	15.0KB	15.1KB	+28.0B
`visTypeVislib`	18.7KB	18.7KB	+28.0B
`visTypeXy`	41.1KB	41.3KB	+168.0B
total			+2.7KB

Unknown metric groups

API count

id	before	after	diff
`data`	3355	3363	+8

References to deprecated APIs

id	before	after	diff
`data`	477	485	+8

History

💚 Build #17729 succeeded 2964b36
💚 Build #17561 succeeded e861980
💚 Build #16513 succeeded 8676edc
💚 Build #16462 succeeded 6cb43df
💔 Build #16418 failed 34260ea
💔 Build #16403 failed 27b9eba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dej611

Tested again and it worked fine 👍

flash1293 added 2 commits December 17, 2021 11:06

implement feature

9f48c27

fix default rare terms value

a5838cd

flash1293 added release_note:enhancement Team:Visualizations Visualization editors, elastic-charts and infrastructure Team:AppServicesSv backport:skip This commit does not require backporting v8.1.0 labels Dec 17, 2021

flash1293 added 6 commits January 10, 2022 11:44

Merge remote-tracking branch 'upstream/main' into rare-terms

ef3d3c4

fix tests

952059f

add tests

27b9eba

add docs

34260ea

fix expression test

f180a6d

fix types

6cb43df

flash1293 marked this pull request as ready for review January 10, 2022 17:55

flash1293 requested review from a team as code owners January 10, 2022 17:55

Merge branch 'main' into rare-terms

8676edc

pgayvallet approved these changes Jan 11, 2022

View reviewed changes

Dosant approved these changes Jan 11, 2022

View reviewed changes

flash1293 added 3 commits January 14, 2022 16:36

Merge remote-tracking branch 'upstream/main' into rare-terms

57a89d6

Merge branch 'rare-terms' of github.com:flash1293/kibana into rare-terms

734bca1

conflict

e861980

Merge branch 'main' into rare-terms

2964b36

dej611 reviewed Jan 17, 2022

View reviewed changes

flash1293 added 2 commits January 18, 2022 12:52

Merge remote-tracking branch 'upstream/main' into rare-terms

e40a33a

review comments

26edbf3

flash1293 added the Feature:Lens label Jan 18, 2022

dej611 approved these changes Jan 18, 2022

View reviewed changes

flash1293 merged commit 38de584 into elastic:main Jan 18, 2022

ghudgins mentioned this pull request Jan 20, 2022

[Lens] add rare terms support #59430

Closed

ogupte pushed a commit to ogupte/kibana that referenced this pull request Jan 28, 2022

[Lens] Implement rare terms (elastic#121500)

787b733

This was referenced Jan 31, 2022

Add support for RareTerms aggregation #26340

Closed

[data.search.aggs] Support rare terms agg in AggConfigs #67393

Closed

drewdaemon mentioned this pull request Jan 27, 2023

[Lens] Don't block render on missing field #149262

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Implement rare terms #121500

[Lens] Implement rare terms #121500

flash1293 commented Dec 17, 2021 •

edited

Loading

flash1293 commented Dec 17, 2021 •

edited

Loading

flash1293 commented Dec 17, 2021

ghudgins commented Dec 17, 2021

flash1293 commented Dec 17, 2021 •

edited

Loading

dej611 commented Dec 17, 2021

flash1293 commented Dec 17, 2021 •

edited

Loading

MichaelMarcialis commented Dec 17, 2021

elasticmachine commented Jan 10, 2022

elasticmachine commented Jan 10, 2022

flash1293 commented Jan 10, 2022

flash1293 commented Jan 10, 2022

Dosant left a comment

flash1293 commented Jan 17, 2022

dej611 left a comment

dej611 Jan 17, 2022

flash1293 Jan 17, 2022

flash1293 Jan 18, 2022

flash1293 commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

API count

References to deprecated APIs

dej611 left a comment

[Lens] Implement rare terms #121500

[Lens] Implement rare terms #121500

Conversation

flash1293 commented Dec 17, 2021 • edited Loading

flash1293 commented Dec 17, 2021 • edited Loading

flash1293 commented Dec 17, 2021

ghudgins commented Dec 17, 2021

flash1293 commented Dec 17, 2021 • edited Loading

dej611 commented Dec 17, 2021

flash1293 commented Dec 17, 2021 • edited Loading

MichaelMarcialis commented Dec 17, 2021

elasticmachine commented Jan 10, 2022

elasticmachine commented Jan 10, 2022

flash1293 commented Jan 10, 2022

flash1293 commented Jan 10, 2022

Dosant left a comment

Choose a reason for hiding this comment

flash1293 commented Jan 17, 2022

dej611 left a comment

Choose a reason for hiding this comment

dej611 Jan 17, 2022

Choose a reason for hiding this comment

flash1293 Jan 17, 2022

Choose a reason for hiding this comment

flash1293 Jan 18, 2022

Choose a reason for hiding this comment

flash1293 commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

References to deprecated APIs

History

dej611 left a comment

Choose a reason for hiding this comment

flash1293 commented Dec 17, 2021 •

edited

Loading

flash1293 commented Dec 17, 2021 •

edited

Loading

flash1293 commented Dec 17, 2021 •

edited

Loading

flash1293 commented Dec 17, 2021 •

edited

Loading