Core should expose a dedicated Elasticsearch client for interacting with Kibana system indices

[rudolf]

Background:

The background context and motivation is outlined in #81536

Implementation

To allow plugins to continue to access Kibana's system indices all requests to Elasticsearch will have to use a path prefixed by _kibana/. To reduce the amount of effort to adopt this and maintain the ElasticsearchJS client's typescript types, Core should expose a special system indices Elasticsearch client that will automatically prefix all paths.

Investigate deprecating / removing callAsInternalUser

In most cases the new system index client should replace using the callAsInternalUser API. However there are some places where plugins rely on the permission of the kibana system user while access non-system index API's.

Interacting with kibana indices that aren't system indices

The following is a list of kibana indices that aren't system indices which probably use the kibana system user callAsInternalUser in some way:

• Alerting's event log - .kibana-event-log-*
• Monitoring - .monitoring-*
• Detection engine signals - .siem-signals-*
• Security solution lists - .lists and .values

Accessing ES API's that aren't related to an index / system index

The following places use the internal user to call non-document / non-index API's. We could continue to use the system ElasticsearchJS client for this, but add a remove list to not prefix these API paths.

• Alerting checks if Elasticsearch security is enabled

  kibana/x-pack/plugins/alerts/server/routes/health.ts

  Lines 55 to 58 in ce45dad

  	.callAsInternalUser('transport.request', {
  	method: 'GET',
  	path: '/_xpack/usage',
  	});

• Fetching licensing information from ES

  kibana/x-pack/plugins/licensing/server/plugin.ts

  Lines 183 to 186 in 5f781dc

  	const response = await clusterClient.callAsInternalUser('transport.request', {
  	method: 'GET',
  	path: '/_xpack',
  	});

• Security plugin calls several "shield" / security API's on Elasticsearch such as

  kibana/x-pack/plugins/security/server/authentication/api_keys.ts

  Line 144 in 9356966

  await this.clusterClient.callAsInternalUser('shield.invalidateAPIKey', {

Other places to audit callAsInternalUser usage

• Usage collectors might use the internal user to analyze data in some indices that aren't system indices:

  kibana/src/plugins/usage_collection/server/usage_collection.mock.ts

  Lines 56 to 57 in 9afd63f

  	callCluster: elasticsearchServiceMock.createLegacyClusterClient().callAsInternalUser,
  	esClient: elasticsearchServiceMock.createClusterClient().asInternalUser,

[mshustov]

If we want System indices' elasticsearch client to access a limited subset of elasticsearch API, we need to coordinate this effort with #81237

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

Note that ideally, we would only be providing these dedicated clients using the new client API, meaning that all usages of the legacy client to call system indices should be migrated before, or during, the process of implementing that feature.

[mshustov]

I checked .reporting-${date}, .kibana_task_manager_${integer}, .kibana_security_session_${integer}, .kibana_security_session_${integer} plugins use a different approach to store & retrieve data.
Only the task_manager leverages SavedObjects client to access data, but SOC doesn't cover all the interactions. It means that all the plugins communicate with Elasticsearch via ES client.
I suppose that in the long term, we'd like to provide SO as a single option to work with Kibana internal data, plus it'd allow plugins to leverage all the SO functionality provided: consistent DX, migrations out-of-the-box, audit logging, etc.

To make it happen, we need to understand why plugins still have to rely on the raw ES client usage rather than SO. The main reasons seem to be:

• SO lacks filtering capabilities for delete/update operations.
  delete query in Session Management:
  https://github.com/elastic/kibana/blob/master/x-pack/plugins/security/server/session_management/session_index.test.ts#L491-L579
  update query in Task manager store:

  kibana/x-pack/plugins/task_manager/server/task_store.ts

  Lines 285 to 292 in 5dfa45d

  	query: matchesClauses(
  	mustBeAllOf(
  	claimTasksById && claimTasksById.length
  	? asPinnedQuery(claimTasksById, queryForScheduledTasks)
  	: queryForScheduledTasks
  	),
  	filterDownBy(InactiveTasks)
  	),

  
  a search query in Task manager store:
  

  kibana/x-pack/plugins/task_manager/server/task_store.ts

  Lines 512 to 515 in 5dfa45d

  	body: ensureAggregationOnlyReturnsTaskObjects({
  	query,
  	aggs,
  	size,

  
  @rudolf is this something we'd like to support in the long term? Is there an issue?

• SO doesn't support more than 10k+ SavedObjects limit greater than ES max result window #22636, Add support to SavedObjects.find for fetching more than 10k objects #77961, [SIEM] Meta issue for saved object needs for large lists  #64715

• performance overhead. SOC and all the wrappers add a performance penalty. It might be critical for such hot-paths as security_session called on every incoming request.

Are there other reasons to prefer the ES client rather than the SO client? For example, SO might not be the best option when working with cross-stack indices (.apm-agent-configuration). @elastic/kibana-security @elastic/kibana-reporting-services @elastic/kibana-alerting-services

That looks like a lot to do, and likely we aren't going to work on this any time soon, but it might be preferable to have a long-term vision.

Regarding the current issue:

• The core form will provide SystemIndices ES client as a part of the Elasticsearch service contract.
• SavedObjects service should migrate to SystemIndices ES client
• SystemIndices ES client is the new @elastic/elasticsearch client, plugins need to migrate to the new ES client before or during migration to SystemIndices ES client
• Talked to @delvedor about migration to /_kibana prefix for all SystemIndices ES client requires. The best option seems to be to use a custom Transport implementation:

class MyTransport extends Transport {
  request (params, options) {
    params.path = '/_kibana' + params.path
    return super.request(params, options)
  }
}

This option allows us to reuse the connection pool:

const child = parent.child({ Transport: MyTransport })

[pmuellr]

Interacting with kibana indices that aren't system indices

The following is a list of kibana indices that aren't system indices which probably use the kibana system user callAsInternalUser in some way:

• Alerting's event log - .kibana-event-log-*

That is correct. We named this so that we could take advantage of the fact that this index should have the same kind of security in place as .kibana; normally users won't have direct access to this index, but we also don't really want to make this a system index just yet, as it has some value with folks willing to add a role to give them access, kinda thing. We also want the customers to be able to customize the ILM policy here, so not sure how that would work if it was a system index.

It's also not SO-shaped - it will eventually become an append-only datastream that should allow arbitrary queries across it, so if it were to be a system index, we'd need those system index API shapes available for it - maybe that's already accounted for, not sure.

[kobelb]

Are there other reasons to prefer the ES client rather than the SO client? For example, SO might not be the best option when working with cross-stack indices

I originally started #80912 to explore these reasons as well.

[rudolf]

@rudolf is this something we'd like to support in the long term? Is there an issue?

I think this could be valuable for other plugins too and don't see an obvious reason we wouldn't be able to add it. I'm not aware of any existing issues, I guess this has just never been brought up.

[mshustov]

It looks like we have to provide SystemIndices Elasticsearch client for some time until plugins migrate to SO service.
Are we okay with the plan proposed in #82716 (comment)?

The core form will provide SystemIndices ES client as a part of the Elasticsearch service contract.
SavedObjects service should migrate to SystemIndices ES client
SystemIndices ES client is the new @elastic/elasticsearch client, plugins need to migrate to the new ES client before or during migration to SystemIndices ES client
Talked to @delvedor about migration to /_kibana prefix for all SystemIndices ES client requires. The best option seems to be to use a custom Transport implementation:

We can limit SystemIndices client to a sub-set of ES client API as done for Repository client #81237 (comment)
It would help up to make sure all the use-cases can be migrated to SO API later.

[rudolf]

👍

Sounds like a good plan.

[kobelb]

@restrry I like the plan as well.

Do you think it's feasible for us to allow the SystemIndices ES Client to work with other system's system-indices as well? I believe the current plan is for the upcoming standalone Fleet Server and the Fleet application in Kibana to both communicate with the Fleet system-indices. @ruflin can you confirm this?

[ruflin]

@kobelb Correct: elastic/elasticsearch#64971

[mshustov]

@kobelb Kibana SystemIndices ES client === a sub-set of ES client API + Kibana specific rules (_kibana url prefix + #82716)
Considering that:

• we might want to migrate to SO API to store Kibana-specific data in the future Reasons for not using saved objects for storing kibana data #80912
• Fleet might need custom rules (for example, logstash plugin uses /_logstash url prefix not even listed in ES public API Update logstash pipeline management to use system index APIs #80405)
• Fleet might use a bit different sub-set of ES REST API.

I'd suggest using an ES client already provided by the core. Fleet plugin can wrap it in a custom client to declare required API sub-set explicitly #81237 (comment).

[kobelb]

@restrry that's a really good point that the Fleet ES Rest APIs are potentially deviating from the approach of essentially exposing standard ES APIs behind the _kibana prefix... If so, my original request doesn't make a bit of sense.

[joshdover]

One thing I want to note is that we should verify that this client with the custom transport works well with the upcoming APM instrumentation of @elastic/elasticsearch: elastic/apm-agent-nodejs#1877

[kobelb]

Per recent changes to #81536 and the introduction of externally managed system indices this is no longer necessary. However, if you all would like to further protect the types of operations plugins can perform using the exposed Elasticsearch client, an approach similar to what is outlined here seems reasonable.

[pgayvallet]

@elastic/kibana-core wdyt? do we want to keep this open after the implementation changes of #81536?

[pgayvallet]

Given the new implementation regarding system indices in #81536, we don't think exposing a dedicated client to interact with system indices if worth the cost.

The (currently) only identified change we need to do is to make sure requests performed from the dev console do not have this header, to avoid end users querying system indices from the UI. I created #90123 to discuss it.