[Stack Monitoring] Use doc ID to deduplicate shard allocations

[miltonhultgren]

Summary

This PR depends on Metricbeat changes introduced in this PR elastic/beats#33457 or the Internal collection changes introduced in elastic/elasticsearch#91153.

The above PR fixes a bug that makes Metricbeat properly report on all the shards in an index, but our shard legend had the same kind of bug, it only displayed the first replica because it didn't account for the replica "id" in the function that tries to deduplicate the results.

Since the above PR ensures we report each shard with a unique document ID, we change the Kibana code to use the document ID instead of trying to regenerate a unique ID.

How to test

Get the changes: git fetch git@github.com:miltonhultgren/kibana.git sm-shard-allocations:sm-shard-allocations && git switch sm-shard-allocations

Start Elasticsearch, create an index like this:

PUT /some-index
{
  "settings": {
    "index": {
      "number_of_shards": 3,  
      "number_of_replicas": 3
    }
  }
}

Run Metricbeat with xpack.enabled: true.

Visit the Index details page for some-index in the Stack Monitoring app and verify that on a single node cluster you have 12 shards, 9 of which are unassigned.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[crespocarlos]

Interesting! I see the boxes and the numbers, but I still don't know how I would use this information. Do we match these numbers with something else to make more sense of this information?

[miltonhultgren]

@crespocarlos

So the number (0, 1, 2) means the shard number. So you'll see 4 0s which is:
0 (primary 0 for index some-index)
0 (replica 1 of primary 0)
0 (replica 2 of primary 0)
0 (replica 3 of primary 0)

And so on for the other 2 primaries and their 6 replicas.
The fact that they are all unassigned replicas means they all get the color yellow but if they were primaries they would be orange.

So in your screenshot, the 3 primaries for shard 0, 1 and 2 are allocated to my machine. While the 9 replicas (3 for primary 0, 3 for primary 1, 3 for primary 2) are unassigned. If you add more nodes it becomes a bit more clear.

Here you can see 1 replica for each shard (0, 1 and 2) are unassigned, while the 3 primaries are on main-1 while the other replicas are spread across main-0 and main-2.

[crespocarlos]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 45d374d

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	108	113	+5
securitySolution	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	109	115	+6
securitySolution	518	524	+6
total			+20

History

• 💔 Build #88311 failed 01ed1db
• 💚 Build #87366 succeeded b27e252
• 💚 Build #86960 succeeded 3cdb371
• 💔 Build #86942 failed 51335c6
• 💔 Build #82543 failed 21ea2ba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @miltonhultgren

[crespocarlos]

is this s in s${shard.shard} intentional?

[miltonhultgren]

Yes, it's to mirror the JSON data which looks like shard: 0, and to match with p or r1, "shard 0, primary", "shard 1, replica 2" etc.

[crespocarlos]

LGTM!

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.6

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation