Prevent migration of system indices that match templates

[grcevski]

Legacy and composable templates may apply to system indices. We don't want this behavior during feature migration. The destination index should get its settings, mappings, and aliases from its system index descriptor. A user who gets this warning should modify their templates so that they do not match system indices, at least during the migration.

This PR is an 8.x version of #87262, where we add an additional flag to all SystemIndexDescriptors that allows certain SystemIndexDescriptors to use index templates. At the moment this is only applicable to Kibana (see elastic/kibana#134897).

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Hi @grcevski, I've created a changelog YAML for you.

[grcevski]

This is interesting, in 8 we weren't properly determining the error status of the upgrade because the index reported in the exception has the upgraded suffix. My tests hit this.

[gwbrown]

Whoops, that one's my bad! Good catch.

[grcevski]

@rjernst I added the flag as we discussed and I tagged @gwbrown to review as he was reviewing the initial 7.17 version of the PR. I'll backport the flag back to 7.17.x and the migration logic/tests once we merge this here.

[gwbrown]

Left some minor notes, otherwise LGTM!

[gwbrown]

We should reference issues #86801 and #87827 here.

[gwbrown]

We should check that the upgrade is actually started (minimally, that accepted is true) here, otherwise I think testBailOnMigrateWithTemplatesV1 could pass if we get into a situation where we don't think the index needs to be migrated.

[gwbrown]

Same thing about checking the response.

[rjernst]

LGTM. My only question is whether we could be a little more strict. Does Kibana use v1 or v2 templates? Could we completely block the other one (ie not allow that unused template type completely)?

[grcevski]

LGTM. My only question is whether we could be a little more strict. Does Kibana use v1 or v2 templates? Could we completely block the other one (ie not allow that unused template type completely)?

Good idea, let me see what they use. They mentioned they are using the new type, but I'd have to confirm.

[grcevski]

OK, I don't think I can restrict it further. At some point they had a legacy index template, which they explicitly remove now and replace it with the new version. However a customer upgrading from an old version can potentially have the legacy one at the time of Elasticsearch upgrade because they haven't upgraded Kibana yet. From their code:

    const sessionIndexTemplateName = `${this.options.kibanaIndexName}_security_session_index_template_${SESSION_INDEX_TEMPLATE_VERSION}`;
    return (this.indexInitialization = new Promise<void>(async (resolve, reject) => {
      try {
        // Check if legacy index template exists, and remove it if it does.
        let legacyIndexTemplateExists = false;
        try {
          legacyIndexTemplateExists = await this.options.elasticsearchClient.indices.existsTemplate(
            {
              name: sessionIndexTemplateName,
            }
          );
        } catch (err) {
          this.options.logger.error(
            `Failed to check if session legacy index template exists: ${err.message}`
          );
          return reject(err);
        }

        if (legacyIndexTemplateExists) {
          try {
            await this.options.elasticsearchClient.indices.deleteTemplate({
              name: sessionIndexTemplateName,
            });
            this.options.logger.debug('Successfully deleted session legacy index template.');
          } catch (err) {
            this.options.logger.error(
              `Failed to delete session legacy index template: ${err.message}`
            );
            return reject(err);
          }
        }

        // Check if required index template exists.
        let indexTemplateExists = false;
        try {
          indexTemplateExists = await this.options.elasticsearchClient.indices.existsIndexTemplate({
            name: sessionIndexTemplateName,
          });
        } catch (err) {
          this.options.logger.error(
            `Failed to check if session index template exists: ${err.message}`
          );
          return reject(err);
        }