Plan for improving etcd continuous integration

[jmhbnz]

What would you like to be added?

This is an umbrella issue to plan a short series of work that will complete a set of improvements to etcd continuous integration, with some particular focus on our arm64 self hosted infrastructure, along with some broader fixes.

This issue was motivated by recent discussion in #15927.

Below are the tasks identified, these will be created as individual issues:

• Run robustness tests on arm64 #15886
• Run arm64 tests in containers on self hosted runners #15927
• Etcd nightly arm64 e2e common tests state they have been cached #15986
• Introduce linting for actions workflow files #15955
• Add clean of golang test cache to remaining etcd ci test suites #15995
• Document access management for etcd arm64 ci infrastructure #15952
• Document process for requesting a new etcd arm64 ci machine #15953
• Document steps to set up a new etcd arm64 github runner #15954
• Run arm64 integration and e2e workflows against a supported release branch #15832
• Move .golangci.yaml out of top level directory to reduce clutter #15987
• Consider migrating all etcd unit, integration, e2e and robustness tests to containers #15956
• Streamline test commands between branches #16176
• Enable race detection for all amd64 and arm64 tests #16315
• Reduce metrics log verbosity of TestConnectionMultiplexing #16422
• Cache LazyFS binary in CI #16326
• Upgrade to debian 12 bookworm #16462

Potential additional tasks:

• Consider cleanup process for maintaining etcd arm64 github runner docker installations (need to verify if this will actually be a problem or not once Run arm64 tests in containers on self hosted runners #15927 merges.)
• Consider scaling up arm64 runner capacity by installing additional runner instances on each machine (this would subsequently enable running arm64 tests on a per pull request basis instead of only nightly.)
• Consider creating some form of documentation for etcd ci workflows, there are a lot of files under main now which can be hard to decipher.

Why is this needed?

Once the above work items are complete in relation to arm64 infrastructure we should be able to progress with tier 1 support for arm64, and in general be in a better position when it comes to our continuous integration.

[BobyMCbobs]

Had a chat with @jmhbnz, a future improvement may be to move from a Docker-based containerised arm64 job to running these within a cluster with autoscaling on Equinix Metal.
Something like Cluster-API + node autoscaler + actions-runner-controller.

This can be implemented after the current plan has made it through.

[jmhbnz]

@serathius, @ahrtr can I propose we pin this issue? It's about to drop off page 1. I suspect there will be a few more tasks to come under this umbrella and I've found it useful to group ci/testing things under this.

[ahrtr]

can I propose we pin this issue?

Done.

[dneary]

Does adding actuated support close the "add runners" task?

[jmhbnz]

Does adding actuated support close the "add runners" task?

It does! Updated, thanks :)

[jmhbnz]

Closing - the primary set of improvements originally scoped have been completed.