Index settings and scripted fields lost or clobbered by multiple concurrent users

[william-uber]

Kibana
Version 4.5.0
Build 9889
Commit SHA ff5cfc5

Elasticsearch

"version": {
    "number": "2.3.2",
    "build_hash": "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp": "2016-04-21T16:03:47Z",
    "build_snapshot": false,
    "lucene_version": "5.5.0"
}

Summary:
When multiple users in an organization attempt to define scripted fields on the same index pattern at the same time, the last writer overwrites any changes made by the other users with no warning.

Steps to reproduce:

1. Open Kibana in a new tab (Tab 1).
2. Tab 1: Select an index pattern and navigate to the Scripted Fields tab.
3. Open Kibana in a new tab (Tab 2).
4. Tab 2: Select the same index pattern and navigate to the Scripted Fields tab.
5. Tab 1: Add a scripted field.
   • Name: Field A
   • Script: 1
   • Create Field
6. Tab 1: Observe that Field A has been created.
7. Tab 2: Add a scripted field.
   • Name: Field B
   • Script: 2
   • Create Field
8. Tab 2: Observe that Field B has been created and Field A is nowhere to be found, with no warnings.
9. Tab 1: Refresh. Observe that Field A has disappeared with no warnings.

Expected behavior:
Ideally, step 8 above should result in Field B being added below Field A in Tab 2. Step 9 should result in Field B appearing after Field A in Tab 1.
Less ideally, step 8 above should fail with an error message along the lines of a concurrent modification exception. Step 9 should result in Field A remaining.

Ideas:

• Persist scripted fields as child documents of index patterns. Idea is to operate on each scripted field atomically independent of other scripted fields or index patterns.
• Detect conflicts and automatically merge scripted fields in an index pattern document. If automatic merge is not possible (e.g. the same scripted field is changed in different ways), then fail with a concurrent modification exception.
• Fall back to Elasticsearch's Optimistic Concurrency Control and fail with the version_conflict_engine_exception whenever concurrent modifications to an index pattern occur.

Any of these would be better than the currently observed behavior of silent data loss. It causes random, unpredictable disappearances and rollbacks of scripted fields. Dashboards and visualizations relying on scripted fields may work for a period of time and then randomly error out when scripted fields disappear.

[epixa]

Thanks for the in depth report, and sorry for the really long delay in getting a response to this.

This is definitely a big problem throughout Kibana, and I suspect it's not entirely limited to scripted fields and index settings.

I think the overall best approach is to widely adopt the Optimistic Concurrency Control approach in ES whenever we're updating documents, and then on a case by case basis for specific features in Kibana, we can come up with other options that may help ensure that people experience version conflicts as infrequently as possible. Your suggestion of storing scripted fields as separate docs might be a viable example of the latter.

[alexfrancoeur]

I discussed this issue with the management team and the general feeling is that this concurrency control should be across all Kibana objects and not just scripted fields. I've added a platform label to this issue. @epixa would you prefer I open a separate issue to track concurrency control across all saved objects?

Linking to #9675 as well.

[bmcconaghy]

Is this still considered a platform issue, or should we be trying to solve this for scripted fields specifically? @epixa @alexfrancoeur

[Bargs]

FYI there was another, more specific issue filed which is really just a symptom of this root problem. Didn't know this issue existed at the time.

#15738

[epixa]

@bmcconaghy Don't let the area label block you. If you can think of a reasonable way to handle this for some specific use case, then don't hesitate to give it a go.

[alexfrancoeur]

@Bargs this is the same user I believe. @bmcconaghy I think this issue and the one Matt had mentioned are fairly self explanatory, but I can provide some additional background if needed. When speaking with @epixa last year, we decided not to pursue concurrency control for just index patterns and instead address this issue globally throughout Kibana. This is obviously a much larger effort and likely the reason behind the Platform label. That being said, this is actually a large pain point for implementations such as the one described in this issue. If you have some ideas on how to approach this, as Court said - give it a go 😄

[epixa]

I should clarify, the reason why we considered this a broader change rather than something we could apply to a specific circumstance is that all of this stuff is powered by our generic saved object API, which doesn't have specific logic for individual types of object. I'm still not clear how someone could solve this problem for only one specific use case, but if anyone does have a good idea around this, I don't want our decision from back in the day to stop progress there.

[bmcconaghy]

I really don't have a good idea for this. My question was more asking "should I be pursuing this or wait for a more global platform solution?"

[epixa]

@bmcconaghy Ah, I think only you can answer that one. As far as I know, this is not on the short term roadmap for platform.

[Bargs]

Should be fixed by #18937