[Mappings Editor] Disable _source field in serverless #181712

ElenaStoeva · 2024-04-25T13:12:16Z

Summary

This PR disables the _source field in the Mappings editor's advanced options when in serverless or when the _source.mode property is set.

How to test:

Start a serverless project.
Go to Index Management and start creating an index template or a component template.
In the Mappings step, go to the "Advanced options" tab.
Verify that the _source field is not displayed and that creating a template doesn't add this property to the Es request.
Verify that, in stateful Kibana, the _source field still exists and works as expected.
In both stateful and serverless, verify that when you create an index template with the _source.mode property set (example Es request below), the whole _source field section is hidden.

PUT _index_template/my_template
{
  "index_patterns": ["foo*"],
  "template": {
    "mappings": {
      "_source": {
        "mode": "synthetic"
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z yyyy"
        }
      }
    }
  }
}

ElenaStoeva · 2024-04-25T13:12:35Z

/ci

ElenaStoeva · 2024-04-26T11:40:25Z

/ci

ElenaStoeva · 2024-04-26T12:10:46Z

x-pack/plugins/index_management/server/config.ts

@@ -52,6 +52,11 @@ const schemaLatest = schema.object(
      // We take this approach in order to have a central place (serverless.yml) for serverless config across Kibana
      serverless: schema.boolean({ defaultValue: true }),
    }),
+    enableMappingsSourceField: offeringBasedSchema({


I was thinking about renaming the config to enableMappingsSourceFieldConfiguration but this might be too long. I'm open to any suggestions for the name of the config.

What about enableMappingsSourceFieldConfig or enableMappingsSourceFieldSection? A little shorter, but not much 😅 . I'm also OK to leave it how you have it.

Thanks for the suggestions! I like enableMappingsSourceFieldSection since the component for the _source field configuration is called SourceFieldSection. Added this change with 813297e.

jloleysens

Core changes LGTM, did not test locally

…-fix'

elasticmachine · 2024-04-26T13:20:00Z

Pinging @elastic/kibana-management (Team:Kibana Management)

test/plugin_functional/test_suites/core_plugins/rendering.ts

Wanted to request a change

jeramysoucy

Kibana security changes LGTM, just one nit request.

jeramysoucy · 2024-04-26T14:57:22Z

test/plugin_functional/test_suites/core_plugins/rendering.ts

@@ -276,6 +276,7 @@ export default function ({ getService }: PluginFunctionalProviderContext) {
        'xpack.index_management.enableIndexStats (any)',
        'xpack.index_management.editableIndexSettings (any)',
        'xpack.index_management.enableDataStreamsStorageColumn (any)',
+        'xpack.index_management.enableMappingsSourceField (any)',


Nit: could we just move this down below the comment with the other conditional flags, or comment as 'any' due to offering based schema?

Sure, I moved it below the comment as well as the rest of the index management offering-based configs for consistency.

alisonelizabeth

Thanks for working on this! Tested on serverless ES and stateful and _source configuration was hidden/shown as expected.

One thing I noticed -

"Synthetic" mode is still supported via the API, but we haven't build the UI yet (follow up: #181609). If I create a template via the API with this configuration, the mappings editor doesn't handle it gracefully. Can you look into how difficult it would be to pass through the value?

Example request:

PUT _index_template/my_template
{
  "index_patterns": ["foo*"],
  "template": {
    "mappings": {
      "_source": {
        "mode": "synthetic"
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z yyyy"
        }
      }
    }
  }
}

Result in the UI:

alisonelizabeth · 2024-04-26T17:53:17Z

x-pack/plugins/index_management/server/config.ts

@@ -52,6 +52,11 @@ const schemaLatest = schema.object(
      // We take this approach in order to have a central place (serverless.yml) for serverless config across Kibana
      serverless: schema.boolean({ defaultValue: true }),
    }),
+    enableMappingsSourceField: offeringBasedSchema({


What about enableMappingsSourceFieldConfig or enableMappingsSourceFieldSection? A little shorter, but not much 😅 . I'm also OK to leave it how you have it.

ElenaStoeva · 2024-04-29T10:40:29Z

Thanks for the review @alisonelizabeth!

"Synthetic" mode is still supported via the API, but we haven't build the UI yet (follow up: #181609). If I create a template via the API with this configuration, the mappings editor doesn't handle it gracefully. Can you look into how difficult it would be to pass through the value?

That's a good call, thanks for bringing it up! I added the mode setting for the _source field to the mappings configuration schema so the UI shouldn't display the warning now. Let me know if you see any other issues.

alisonelizabeth

Thanks @ElenaStoeva!

I noticed one more thing, but it may be related to an existing issue: #106006. If I create a template via the API with the synthetic mode, then click on the "Advanced" tab, then go to save, the mode is dropped from the request. Can you take a look and see how feasible it would be to prevent the field from getting dropped?

jeramysoucy

Kibana security changes LGTM

ElenaStoeva · 2024-04-29T19:11:00Z

I noticed one more thing, but it may be related to an existing issue: #106006. If I create a template via the API with the synthetic mode, then click on the "Advanced" tab, then go to save, the mode is dropped from the request. Can you take a look and see how feasible it would be to prevent the field from getting dropped?

Nice catch! I looked into this but couldn't find an easy fix - it seems the form overwrites the request with the current values in the form, and since there is no form data for the mode field, it gets dropped. It might be easiest if we just add a simple field for the mode property (e.g. a dropdown with the different options). Wdyt?

I can still investigate some more tomorrow to check again if there is any way to fix this without adding a field, I could have missed something.

ElenaStoeva · 2024-04-30T10:30:09Z

Hi @alisonelizabeth, I found that the mode property cannot be set if enabled or non-empty includes/excludes properties are set. For example, the following Es request would fail:

PUT _index_template/my_template
{
  "index_patterns": ["foo*"],
  "template": {
    "mappings": {
      "_source": {
        "enabled": false,
        "mode": "synthetic"
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z yyyy"
        }
      }
    }
  }
}

Therefore, I think it wouldn't make sense to display any of the existing source fields in the UI if the mode field is set, so I decided, as a short-term solution, to hide the source fields if the mode is set. Now the mode field is no longer dropped as I added it to the serialization function. The fix is in d7359e4.

As a follow-up work, we can improve the UI to have a toggle between the two options (setting a mode or setting the existing fields).

kibana-ci · 2024-04-30T16:53:59Z

💚 Build Succeeded

Buildkite Build
Commit: b9c1a1a

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexManagement`	656.2KB	656.6KB	+426.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`indexManagement`	42.3KB	42.4KB	+79.0B

History

💔 Build #206869 failed 5d775db
💚 Build #206565 succeeded 813297e
💔 Build #206325 failed e18e1f5
💔 Build #206299 failed 76ec03e
💔 Build #205982 failed 72d4c64

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ElenaStoeva

alisonelizabeth

Thanks @ElenaStoeva! Latest lgtm. Great work 🎉

* master: (1654 commits) Bump ejs from 3.1.9 to 3.1.10 Don't render exceptions flyout if data is loading (elastic#181588) Enable value list modal (elastic#181593) skip flaky suite (elastic#181777) skip failing test suite (elastic#182263) [Mappings Editor] Disable _source field in serverless (elastic#181712) [data.search] Fix unhandled promise rejections (elastic#181785) [Fleet] Fix logic for detecting first time Elastic Agent users (elastic#182214) [ML] Decouple data_visualizer from MapEmbeddable (elastic#181928) [ES|QL] Sorting accepts expressions (elastic#181916) [ML] Single Metric Viewer: ensures chart displays correctly when opening from a job annotation (elastic#182176) Adding optional Description field to Roles APIs (elastic#182039) Upgrade Markdown-it to 14.1.0 (elastic#182244) Bump xml-crypto from 5.0.0 to 6.0.0 [DOCS] Fix docs and screenshots for rule creation changes (elastic#181925) Update dependency elastic-apm-node to ^4.5.3 (main) (elastic#182236) [Obs AI Assistant] register alert details context in observability plugin (elastic#181501) Add `@typescript-eslint/no-floating-promises` (elastic#181456) [Playground] Propagate Error message into FE (elastic#182201) [ES|QL] Rename the setting to a more generic one and move to the general section (elastic#182074) ...

Closes elastic#181555 ## Summary This PR disables the `_source` field in the Mappings editor's advanced options when in serverless. **How to test:** 1. Start a serverless project. 2. Go to Index Management and start creating an index template or a component template. 3. In the Mappings step, go to the "Advanced options" tab. 4. Verify that the `_source` field is not displayed and that creating a template doesn't add this property to the Es request. 5. Verify that, in stateful Kibana, the `_source` field still exists and works as expected.  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

[Mappings Editor] Disable _source field in serverless

72d4c64

ElenaStoeva added release_note:skip Skip the PR/issue when compiling release notes Feature:Mappings Editor Index mappings editor UI labels Apr 25, 2024

ElenaStoeva self-assigned this Apr 25, 2024

Update and add tests

e019e0a

ElenaStoeva force-pushed the im/serverless/disable-mappings-source-field branch from 76ec03e to e019e0a Compare April 26, 2024 11:55

ElenaStoeva marked this pull request as ready for review April 26, 2024 12:01

ElenaStoeva requested review from a team as code owners April 26, 2024 12:01

ElenaStoeva commented Apr 26, 2024

View reviewed changes

Merge branch 'main' into im/serverless/disable-mappings-source-field

04dacec

ElenaStoeva force-pushed the im/serverless/disable-mappings-source-field branch from b178de6 to 04dacec Compare April 26, 2024 12:23

ElenaStoeva requested a review from a team as a code owner April 26, 2024 12:23

jloleysens approved these changes Apr 26, 2024

View reviewed changes

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

e18e1f5

…-fix'

ElenaStoeva added the Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more label Apr 26, 2024

jeramysoucy requested changes Apr 26, 2024

View reviewed changes

test/plugin_functional/test_suites/core_plugins/rendering.ts Outdated Show resolved Hide resolved

jeramysoucy previously approved these changes Apr 26, 2024

View reviewed changes

jeramysoucy requested changes Apr 26, 2024

View reviewed changes

alisonelizabeth reviewed Apr 26, 2024

View reviewed changes

ElenaStoeva and others added 3 commits April 29, 2024 10:20

Address feedback and fix failing test

8818654

Merge branch 'main' into im/serverless/disable-mappings-source-field

06d98ba

Rename config name

813297e

ElenaStoeva requested review from jeramysoucy and alisonelizabeth April 29, 2024 15:24

alisonelizabeth reviewed Apr 29, 2024

View reviewed changes

jeramysoucy approved these changes Apr 29, 2024

View reviewed changes

Fix issue with source field mode

d7359e4

Merge branch 'main' into im/serverless/disable-mappings-source-field

5d775db

ElenaStoeva force-pushed the im/serverless/disable-mappings-source-field branch from 1ac4492 to 5d775db Compare April 30, 2024 10:37

ElenaStoeva and others added 4 commits April 30, 2024 14:52

Merge branch 'main' into im/serverless/disable-mappings-source-field

ef8ebef

Fix type errors and remove matchMedia fix

e120b9a

Merge branch 'main' into im/serverless/disable-mappings-source-field

32946b9

Use useCallback to optimize form serializer

b9c1a1a

alisonelizabeth requested review from yuliacech and removed request for yuliacech May 1, 2024 01:03

alisonelizabeth approved these changes May 1, 2024

View reviewed changes

ElenaStoeva merged commit 8e25828 into elastic:main May 1, 2024
17 checks passed

kibanamachine added v8.15.0 backport:skip This commit does not require backporting labels May 1, 2024

ElenaStoeva mentioned this pull request May 2, 2024

[Mappings editor] Support synthetic source configuration #181609

Open

ElenaStoeva deleted the im/serverless/disable-mappings-source-field branch June 24, 2024 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mappings Editor] Disable _source field in serverless #181712

[Mappings Editor] Disable _source field in serverless #181712

ElenaStoeva commented Apr 25, 2024 •

edited

Loading

ElenaStoeva commented Apr 25, 2024

ElenaStoeva commented Apr 26, 2024

ElenaStoeva Apr 26, 2024

alisonelizabeth Apr 26, 2024

ElenaStoeva Apr 29, 2024

jloleysens left a comment

elasticmachine commented Apr 26, 2024

jeramysoucy left a comment

jeramysoucy Apr 26, 2024 •

edited

Loading

ElenaStoeva Apr 29, 2024

alisonelizabeth left a comment

alisonelizabeth Apr 26, 2024

ElenaStoeva commented Apr 29, 2024

alisonelizabeth left a comment

jeramysoucy left a comment

ElenaStoeva commented Apr 29, 2024

ElenaStoeva commented Apr 30, 2024

kibana-ci commented Apr 30, 2024

alisonelizabeth left a comment

[Mappings Editor] Disable _source field in serverless #181712

[Mappings Editor] Disable _source field in serverless #181712

Conversation

ElenaStoeva commented Apr 25, 2024 • edited Loading

Summary

ElenaStoeva commented Apr 25, 2024

ElenaStoeva commented Apr 26, 2024

ElenaStoeva Apr 26, 2024

Choose a reason for hiding this comment

alisonelizabeth Apr 26, 2024

Choose a reason for hiding this comment

ElenaStoeva Apr 29, 2024

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 26, 2024

jeramysoucy left a comment

Choose a reason for hiding this comment

jeramysoucy Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

ElenaStoeva Apr 29, 2024

Choose a reason for hiding this comment

alisonelizabeth left a comment

Choose a reason for hiding this comment

alisonelizabeth Apr 26, 2024

Choose a reason for hiding this comment

ElenaStoeva commented Apr 29, 2024

alisonelizabeth left a comment

Choose a reason for hiding this comment

jeramysoucy left a comment

Choose a reason for hiding this comment

ElenaStoeva commented Apr 29, 2024

ElenaStoeva commented Apr 30, 2024

kibana-ci commented Apr 30, 2024

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

alisonelizabeth left a comment

Choose a reason for hiding this comment

ElenaStoeva commented Apr 25, 2024 •

edited

Loading

jeramysoucy Apr 26, 2024 •

edited

Loading