[Index Patterns] Use deprecation api for scripted fields #100781

Dosant · 2021-05-27T13:46:36Z

Summary

Close #97947
Scripted fields are deprecated. Runtime fields should be used instead. This pr adds information about this to the upgrade assistant. It hasn't been decided when scripted fields will be removed yet, hence keeping this as warning.

Release notes

Add scripted fields depreciation information to upgrade assistant

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

How to test

I viewed upgrade assistant UI by tweaking the condition in the code

kibana/x-pack/plugins/upgrade_assistant/public/application/app.tsx

Lines 26 to 28 in 186b936

    
           if (isReadOnlyMode) { 
        
             return <ComingSoonPrompt />; 
        
           }

. Current functional test doesn't test the UI #100781 (comment), but it calls upgrade assistant API and checks that there is a part about scripted fields

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

Dosant · 2021-05-27T14:05:16Z

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

+      if (indexPatternsWithScriptedFields.length > 0) {
+        // no need to look up all index patterns since we've found at least one that has scripted fields
+        await finder.close();
+        break;


I intentionally don't fetch all index patterns to make this function cheaper. This is reflected in the wording of the message below: You have **at least** <n> index patterns...

I think we should show a full list of index patterns that need updating as some setups may have a large number of index patterns and only a few might have scripted fields. We need a way to deliminate the index patterns as they can have commas in them. Could we link directly to the index patterns?

Could we link directly to the index patterns?

Nope, as I understand we can only use plain text.

At this moment this pr shows at max 3 index pattern titles as a hint.

Do you think we should show more out from the page? (e.g. in the current implementation we can remove that .slice(0,3) where I prepare the message

Or do you think we should fetch all the index patterns and show the whole list no matter how long it is?

In the current implementation user can:

Take a look at a message, see up to 3 index pattern titles that need to be fixed

Fix them

If there are more to fix, see another 3 index pattern titles and fix them

Not ideal, but not sure we want to fetch all the index patterns?

I updated so in the main message we show just up to 3 index pattern titles, but then when you open dialog with details we list all of them!

Dosant · 2021-05-27T14:06:35Z

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

+          correctiveActions: {
+            manualSteps: [
+              'Navigate to Stack Management > Kibana > Index Patterns.',
+              `Update index patterns that have scripted fields (${indexPatternsTitlesHelp}...) to use runtime fields instead. In most cases, to migrate existing scripts, changing "return <value>;" to "emit(<value>);" would be enough.`,


Might need some help with wording

There's going to be a blog post about this shortly, lets update this text once its available

elasticmachine · 2021-05-27T15:02:17Z

Pinging @elastic/kibana-app-services (Team:AppServices)

src/plugins/data/server/index_patterns/deprications/index.ts

mattkime · 2021-05-27T16:33:20Z

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

+          correctiveActions: {
+            manualSteps: [
+              'Navigate to Stack Management > Kibana > Index Patterns.',
+              `Update index patterns that have scripted fields (${indexPatternsTitlesHelp}...) to use runtime fields instead. In most cases, to migrate existing scripts, changing "return <value>;" to "emit(<value>);" would be enough.`,


There's going to be a blog post about this shortly, lets update this text once its available

mattkime · 2021-05-27T16:39:37Z

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

+      if (indexPatternsWithScriptedFields.length > 0) {
+        // no need to look up all index patterns since we've found at least one that has scripted fields
+        await finder.close();
+        break;


I think we should show a full list of index patterns that need updating as some setups may have a large number of index patterns and only a few might have scripted fields. We need a way to deliminate the index patterns as they can have commas in them. Could we link directly to the index patterns?

mattkime · 2021-05-27T16:42:53Z

'data' is using a deprecated feature is an odd statement to show the user. Are we able to change this? Did core create this API? Would be nice if this could be improved.

pgayvallet · 2021-05-27T18:48:18Z

src/plugins/data/server/index_patterns/deprications/scripted_fields.ts

+    const finder = context.savedObjectsClient.createPointInTimeFinder<IndexPatternAttributesWithFields>(
+      {
+        type: 'index-pattern',
+        perPage: 100,
+        fields: ['title', 'fields'],
+      }
+    );


@Bamieh do you think we should introduce some kind of caching in the deprecation service? atm everytime the /api/deprecations route is called, all the deprecation providers are executed, and I wonder if this will be alright when we'll have a lot that are performing ES requests like this one. wdyt?

If we do, not sure how it will work exactly? It is likely that with the cashing in place the use case I expressed in e2e testtest/api_integration/apis/index_patterns/deprecations/scripted_fields.ts might break?

Also @pgayvallet, I think having so many index patterns is more of an edge case and inside the app we simply currently do

kibana/src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts

Lines 96 to 100 in 51616e1

const so = await this.savedObjectsClient.find<IndexPatternSavedObjectAttrs>({

type: 'index-pattern',

fields: ['title'],

perPage: 10000,

});

@Dosant this should not impact this PR, I was more wondering about long-term performances.

Initially i had caching in place for the service but it didnt make sense to keep it. The API is called only at the user's request when they open the UA or explicitly via the API. Generic caching wont work since we do not want to return already resolved deprecations from the api. We always want the latest state of the deprecations.

For performance it might make sense to introduce pagination rather than caching. The service would paginate the kibana deprecations so we dont have to loop over all deprecations every time. Since the deprecations are registered at setup in an array we can guarantee the order and create basic pagination from the public contract.

I agree with @pgayvallet this doesnt impact the PR. I'll open a separate issue to continue the discussion there.

Dosant · 2021-05-28T12:56:54Z

@mattkime: 'data' is using a deprecated feature is an odd statement to show the user. Are we able to change this? Did core create this API? Would be nice if this could be improved.

I think we can't change this atm

…cation-api-for-index-pattern # Conflicts: # test/api_integration/apis/index_patterns/index.js

mattkime · 2021-05-28T16:53:17Z

I'm talking to @alisonelizabeth to see how this could be tested by QA. The UI for the deprecation service might not be available until 7.15. Considering how to proceed.

Looks good and works well, waiting to hear from Alison about how we want this to go through QA.

Bamieh · 2021-06-01T11:25:26Z

'data' is using a deprecated feature is an odd statement to show the user. Are we able to change this? Did core create this API? Would be nice if this could be improved.

@mattkime we have this issue to improve this #99625

Looks good and works well, waiting to hear from Alison about how we want this to go through QA.

I'm trying to address this as well (context: #99043 (comment))

The service provided by core is testable although we do need UA to provide some tools to allow teams to run functional tests for their deprecations in the UA UI

mattkime · 2021-06-07T14:38:44Z

@elasticmachine merge upstream

…cation-api-for-index-pattern # Conflicts: # test/api_integration/apis/index_patterns/index.js

…m:Dosant/kibana into dev/use-deprecation-api-for-index-pattern

mattkime · 2021-06-29T02:18:54Z

@elasticmachine merge upstream

mattkime · 2021-06-29T02:22:11Z

@Dosant Do you know where we are on getting this merged? I know we were waiting for a flag to be able to test this without recompiling.

mattkime · 2021-07-05T04:38:47Z

@elasticmachine merge upstream

Dosant · 2021-07-07T10:58:15Z

@elasticmachine merge upstream

kibanamachine · 2021-07-07T13:02:46Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #136248 succeeded a56b102
💚 Build #134925 succeeded 8edffc1
💚 Build #130879 succeeded 7b55f1e
💚 Build #129520 succeeded 0737efb
💚 Build #127901 succeeded 7339216

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

)

kibanamachine · 2021-07-07T13:32:26Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…104654) Co-authored-by: Anton Dosov <anton.dosov@elastic.co>

use deprecation api for scrited fields

c66111d

Dosant added Feature:Data Views Data Views code and UI - index patterns before 8.0 Feature:Scripted Fields Scripted fields features release_note:deprecation Team:AppServices v7.14.0 v8.0.0 labels May 27, 2021

Dosant commented May 27, 2021

View reviewed changes

Dosant marked this pull request as ready for review May 27, 2021 15:02

Dosant requested a review from a team as a code owner May 27, 2021 15:02

Dosant requested review from mattkime and a team May 27, 2021 15:02

mattkime requested changes May 27, 2021

View reviewed changes

src/plugins/data/server/index_patterns/deprications/index.ts Outdated Show resolved Hide resolved

mattkime requested changes May 27, 2021

View reviewed changes

pgayvallet reviewed May 27, 2021

View reviewed changes

Dosant added 2 commits May 28, 2021 15:08

Merge branch 'master' of github.com:elastic/kibana into dev/use-depre…

88f4db9

…cation-api-for-index-pattern # Conflicts: # test/api_integration/apis/index_patterns/index.js

update

7339216

Dosant requested a review from mattkime May 28, 2021 14:04

pgayvallet approved these changes Jun 1, 2021

View reviewed changes

kibanamachine and others added 4 commits June 7, 2021 10:38

Merge branch 'master' into dev/use-deprecation-api-for-index-pattern

0737efb

Merge branch 'master' of github.com:elastic/kibana into dev/use-depre…

767ee80

…cation-api-for-index-pattern # Conflicts: # test/api_integration/apis/index_patterns/index.js

fix tests imports

442121a

Merge branch 'dev/use-deprecation-api-for-index-pattern' of github.co…

7b55f1e

…m:Dosant/kibana into dev/use-deprecation-api-for-index-pattern

Merge branch 'master' into dev/use-deprecation-api-for-index-pattern

8edffc1

spalger added v7.15.0 and removed v7.14.0 labels Jun 30, 2021

Merge branch 'master' into dev/use-deprecation-api-for-index-pattern

a56b102

mattkime approved these changes Jul 6, 2021

View reviewed changes

Merge branch 'master' into dev/use-deprecation-api-for-index-pattern

d3d1509

Dosant added the auto-backport Deprecated - use backport:version if exact versions are needed label Jul 7, 2021

Dosant merged commit 44d4d23 into elastic:master Jul 7, 2021

kibanamachine mentioned this pull request Jul 7, 2021

[7.x] [Index Patterns] Use deprecation api for scripted fields (#100781) #104654

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 7, 2021

[Index Patterns] Use deprecation api for scripted fields (elastic#100781

64428b3

)

kibanamachine added a commit that referenced this pull request Jul 7, 2021

[Index Patterns] Use deprecation api for scripted fields (#100781) (#…

2180bbf

…104654) Co-authored-by: Anton Dosov <anton.dosov@elastic.co>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Index Patterns] Use deprecation api for scripted fields #100781

[Index Patterns] Use deprecation api for scripted fields #100781

Dosant commented May 27, 2021 •

edited

Loading

Dosant May 27, 2021

mattkime May 27, 2021

Dosant May 28, 2021

Dosant May 28, 2021 •

edited

Loading

Dosant May 27, 2021

mattkime May 27, 2021

elasticmachine commented May 27, 2021

mattkime May 27, 2021

mattkime May 27, 2021

mattkime commented May 27, 2021

pgayvallet May 27, 2021

Dosant May 28, 2021 •

edited

Loading

Dosant May 28, 2021 •

edited

Loading

pgayvallet May 28, 2021

Bamieh Jun 1, 2021 •

edited

Loading

Dosant commented May 28, 2021

mattkime commented May 28, 2021 •

edited

Loading

Bamieh commented Jun 1, 2021 •

edited

Loading

mattkime commented Jun 7, 2021

mattkime commented Jun 29, 2021

mattkime commented Jun 29, 2021

mattkime commented Jul 5, 2021

Dosant commented Jul 7, 2021

kibanamachine commented Jul 7, 2021

kibanamachine commented Jul 7, 2021

	const so = await this.savedObjectsClient.find<IndexPatternSavedObjectAttrs>({
	type: 'index-pattern',
	fields: ['title'],
	perPage: 10000,
	});

[Index Patterns] Use deprecation api for scripted fields #100781

[Index Patterns] Use deprecation api for scripted fields #100781

Conversation

Dosant commented May 27, 2021 • edited Loading

Summary

Release notes

Checklist

For maintainers

How to test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dosant May 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkime commented May 27, 2021

Choose a reason for hiding this comment

Dosant May 28, 2021 • edited Loading

Choose a reason for hiding this comment

Dosant May 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bamieh Jun 1, 2021 • edited Loading

Choose a reason for hiding this comment

Dosant commented May 28, 2021

mattkime commented May 28, 2021 • edited Loading

Bamieh commented Jun 1, 2021 • edited Loading

mattkime commented Jun 7, 2021

mattkime commented Jun 29, 2021

mattkime commented Jun 29, 2021

mattkime commented Jul 5, 2021

Dosant commented Jul 7, 2021

kibanamachine commented Jul 7, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Jul 7, 2021

💚 Backport successful

Dosant commented May 27, 2021 •

edited

Loading

Dosant May 28, 2021 •

edited

Loading

Dosant May 28, 2021 •

edited

Loading

Dosant May 28, 2021 •

edited

Loading

Bamieh Jun 1, 2021 •

edited

Loading

mattkime commented May 28, 2021 •

edited

Loading

Bamieh commented Jun 1, 2021 •

edited

Loading