Upgrade to 8.0: Reindexing .security index fails if security-index-template-v6 is still present

[JohannesMahne]

Elasticsearch Version

7.17

Installed Plugins

No response

Java Version

bundled

OS Version

Any

Problem Description

Symptoms

This problem can occur during the running of the 7.17 Upgrade Assistant Migrate system indices action:

• If System indices migration failed. An error occurred while migrating system indices for security: exception error is reported.
• The cluster logs will report "reason": "mapping set to strict, dynamic introduction of [allow_restricted_indices] within [indices] is not allowed", and other similar messages for the new .security-7-reindexed-for-8 index.
• The same error will also be reported in the output of GET /_migration/system_features.
• There will also be a legacy index template (e.g. security-index-template-v6), that matches the .security-* index pattern.

Fix/Workaround

The following should be done to fix the migration issue:

1. Delete the index template security-index-template-v6.
2. Delete partially reindexed .security-7-reindexed-for-8 index.
3. Rerun the Migrate system indices action form the Upgrade Assistant.

Steps to Reproduce

The required reindexing step of the .security-7 index may fail, if older index templates matching the .security-* index pattern are still present. This may occur on clusters that were originally created on version 5.x.

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[albertzaharovits]

I was able to reproduce this. I have tried migrating a Security index created in version 6.

If there is a template (I've only tested a legacy templates, but I think the composite templates behave the same), and the .security* index needs to be migrated, the settings and mappings from the template will be merged in the resulting .security-6-reindexed-for-8. This is a problem because most likely Security will become unusable at this point (hence the cluster will be inaccessible).

I don't think we should apply templates when we migrate the .security index.
We don't apply them when the .security index is created automatically, and we also do not apply them if the user tries to manually create it.
Can I please have someone's take on this from @elastic/es-core-infra @elastic/es-data-management ?

[williamrandolph]

The unexpected effects of templates on important internal indices were a big motivation for creating system indices in the first place. Unfortunately, we didn't manage to get to #42508 before the 8.0 release, so now anything we do needs to be backwards-compatible. But it's a big problem and I need to work on a solution for it.

For this specific issue, we could add a pre-migration hook to the Security plugin class. Watcher and MachineLearning have implemented pre-migration hooks, if you want to see an example. In Security, I imagine it would look something like this:

    @Override
    public void prepareForIndicesMigration(ClusterService clusterService, Client client, ActionListener<Map<String, Object>> listener) {
        Client originClient = new OriginSettingClient(client, ClientHelper.SECURITY_ORIGIN);
        originClient.execute(
            DeleteIndexTemplateAction.INSTANCE,
            new DeleteIndexTemplateRequest("security-index-template-v6"),
            ActionListener.wrap(response -> listener.onResponse(Map.of()), listener::onFailure)
        );
    }

I'm not sure if there's something better to do with the listener; the map is used for passing state to a post-upgrade hook.

[williamrandolph]

I tried to reproduce this with a 6.8-to-7.17 upgrade. There was no security-index-template-v6, so either I don't have the right sequence of versions for the upgrade, or this was a user-created template. I ended up reproducing by posting my own v1 ("legacy") template.

The migration issues a create index request, and the code in the MetadataCreateIndexService class is what merges the template settings and mappings with the ones defined by the system index.

The core-infra team is going to sync up next week and we'll report back with a plan of action. This is a symptom of an old, old problem (see #42508) and we need a plan for getting past it.

[williamrandolph]

We talked this over today and decided to try two things:

1. For 7.x, we want to add a check to the system feature migration that aborts the migration if a global template would apply to the new index that the migration creates. This is PR'd in Don't migrate system indices that match user templates #87262
2. We are also looking considering it a bug that user templates can affect system indices in 8.x. A potential fix for that is here: System indices ignore all user templates #87260

If we can get these two fixes merged, we'll prevent future 7.x-to-8.x upgrades from having this problem, and avoid having the issue keep biting us in the 8.x line of releases.