Add failover test cases

[ahrtr]

Fix 13707

Add failover test to cover the following two scenarios,

• client failover when the first server listed goes unavailable after the client has established a connection
• client failover when the first server listed is unavailable when the client is first created

cc @ptabor @serathius @spzala

[codecov-commenter]

Codecov Report

Merging #13737 (3fc787b) into main (a7106dc) will decrease coverage by 0.14%.
The diff coverage is n/a.

❗ Current head 3fc787b differs from pull request most recent head 833b0e9. Consider uploading reports for the commit 833b0e9 to get more accurate results

@@            Coverage Diff             @@
##             main   #13737      +/-   ##
==========================================
- Coverage   72.78%   72.63%   -0.15%     
==========================================
  Files         467      467              
  Lines       38287    38287              
==========================================
- Hits        27867    27811      -56     
- Misses       8617     8669      +52     
- Partials     1803     1807       +4     

Flag	Coverage Δ
all	72.63% <ø> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/proxy/grpcproxy/register.go	69.76% <0.00%> (-9.31%)	⬇️
client/v3/namespace/watch.go	87.87% <0.00%> (-6.07%)	⬇️
server/storage/mvcc/watchable_store.go	84.42% <0.00%> (-5.44%)	⬇️
server/etcdserver/api/v3rpc/lease.go	77.21% <0.00%> (-5.07%)	⬇️
raft/rafttest/node.go	95.00% <0.00%> (-5.00%)	⬇️
server/storage/schema/cindex.go	95.34% <0.00%> (-4.66%)	⬇️
client/v3/leasing/cache.go	87.77% <0.00%> (-3.89%)	⬇️
api/etcdserverpb/raft_internal_stringer.go	76.78% <0.00%> (-3.58%)	⬇️
server/etcdserver/api/v3rpc/util.go	70.96% <0.00%> (-3.23%)	⬇️
server/etcdserver/api/v3rpc/member.go	93.54% <0.00%> (-3.23%)	⬇️
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7106dc...833b0e9. Read the comment docs.

[ptabor]

LGTM. Thank you.

[ahrtr]

Resolved all the comments. We can ignore the semaphoreci failure, which was caused by the docker hub rate limitation . cc @serathius @ptabor @spzala

[ahrtr]

All test are green now after retriggering the pipeline.

[liggitt]

does this TLS config match what is expected in production w.r.t. validity of serving certificates and distinct hostnames (or is it using localhost / identical loopback IPs for all members)? I have memories of the etcd client not handling TLS validation on failover correctly (e.g. #10911) and wanted to make sure we're actually exercising the scenario

[ahrtr]

It's a good point, thanks for pointing out this.

It's using localhost & 127.0.0.1 (see below). The e2e & integration tests are supposed to be automatically executed in pipeline, so all etcd members are always running on the same host?

Probably we need to think about how to run & test etcd in an environment which is close to production? cc @ptabor @serathius @spzala

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                F4:B1:45:B9:C4:18:A5:74:3B:5A:C9:21:10:60:4F:6F:5F:EF:12:64
            X509v3 Authority Key Identifier: 
                keyid:29:26:2B:F4:ED:D9:18:F5:60:74:CE:00:7A:C0:E6:FF:1C:C8:BC:A0

            X509v3 Subject Alternative Name: 
                DNS:localhost, IP Address:127.0.0.1