server/v3/storage/mvcc TestHashKVWhenCompacting is flaking

[Rajalakshmi-Girish]

Which github workflows are flaking?

Job in our internal CI are flaking

Which tests are flaking?

server/v3/storage/mvcc TestHashKVWhenCompacting

Reason for failure (if possible)

The test is failing at https://github.com/etcd-io/etcd/blob/main/server/storage/mvcc/kvstore_test.go#L629
Seems like a test timeout issue.

This fails in ppc64le architecture.

[root@raji-workspace etcd]# go test -v go.etcd.io/etcd/server/v3/storage/mvcc -c -count=1
[root@raji-workspace etcd]#
[root@raji-workspace etcd]# /root/go/bin/stress ./mvcc.test -test.run TestHashKVWhenCompacting
5s: 0 runs so far, 0 failures
10s: 8 runs so far, 0 failures
15s: 8 runs so far, 0 failures

/tmp/go-stress-20230718T085107-1901309625
--- FAIL: TestHashKVWhenCompacting (10.38s)
    logger.go:130: 2023-07-18T08:51:15.864Z     INFO    kvstore restored        {"current-rev": 1}
    logger.go:130: 2023-07-18T08:51:16.040Z     INFO    compact tree index      {"revision": 9900}
    logger.go:130: 2023-07-18T08:51:17.078Z     INFO    finished scheduled compaction   {"compact-revision": 9900, "took": "1.038612654s", "hash": 2982592828}
    logger.go:130: 2023-07-18T08:51:17.079Z     INFO    storing new hash        {"hash": 2982592828, "revision": 9900, "compact-revision": -1}
    logger.go:130: 2023-07-18T08:51:17.193Z     INFO    compact tree index      {"revision": 9901}
    logger.go:130: 2023-07-18T08:51:17.193Z     INFO    finished scheduled compaction   {"compact-revision": 9901, "took": "439.747µs", "hash": 4092902393}
    logger.go:130: 2023-07-18T08:51:17.193Z     INFO    storing new hash        {"hash": 4092902393, "revision": 9901, "compact-revision": 9900}
    logger.go:130: 2023-07-18T08:51:17.308Z     INFO    compact tree index      {"revision": 9902}
    logger.go:130: 2023-07-18T08:51:17.308Z     INFO    finished scheduled compaction   {"compact-revision": 9902, "took": "451.774µs", "hash": 3442946395}
    logger.go:130: 2023-07-18T08:51:17.308Z     INFO    storing new hash        {"hash": 3442946395, "revision": 9902, "compact-revision": 9901}
    logger.go:130: 2023-07-18T08:51:17.424Z     INFO    compact tree index      {"revision": 9903}
    logger.go:130: 2023-07-18T08:51:17.424Z     INFO    finished scheduled compaction   {"compact-revision": 9903, "took": "624.053µs", "hash": 3934476647}
    logger.go:130: 2023-07-18T08:51:17.424Z     INFO    storing new hash        {"hash": 3934476647, "revision": 9903, "compact-revision": 9902}
    logger.go:130: 2023-07-18T08:51:17.642Z     INFO    compact tree index      {"revision": 9904}
    logger.go:130: 2023-07-18T08:51:17.643Z     INFO    finished scheduled compaction   {"compact-revision": 9904, "took": "415.929µs", "hash": 1979317650}
    logger.go:130: 2023-07-18T08:51:17.643Z     INFO    storing new hash        {"hash": 1979317650, "revision": 9904, "compact-revision": 9903}
    logger.go:130: 2023-07-18T08:51:17.740Z     INFO    compact tree index      {"revision": 9905}

…

/tmp/go-stress-20230718T085107-4175905429
panic: schedule: schedule to stopped scheduler

goroutine 115 [running]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).Schedule(0xc0004447e0, {0x90bc58?, 0xc0004b68a0})
        /root/etcd/pkg/schedule/schedule.go:111 +0x1e0
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).compact(0xc00035aea0, 0xc0000a9f50, 0x270f, 0x270e, 0x1)
        /root/etcd/server/storage/mvcc/kvstore.go:261 +0x144
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).Compact(0xc00035aea0, 0xc0000a9f50, 0xdc9200?)
        /root/etcd/server/storage/mvcc/kvstore.go:287 +0x234
go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting.func3()
        /root/etcd/server/storage/mvcc/kvstore_test.go:614 +0xfc
created by go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting
        /root/etcd/server/storage/mvcc/kvstore_test.go:611 +0x3ec


ERROR: exit status 2

20s: 16 runs so far, 2 failures (12.50%)
25s: 16 runs so far, 2 failures (12.50%)
30s: 24 runs so far, 2 failures (8.33%)
35s: 24 runs so far, 2 failures (8.33%)
^C
[root@raji-workspace etcd]#

Anything else we need to know?

Attached are the logs traces:
go-stress-20230718T085107-4175905429.txt
go-stress-20230718T085107-1901309625.txt

[Rajalakshmi-Girish]

@fuweid any pointers on this, please?

[fuweid]

@Rajalakshmi-Girish I run it in my local (amd64) and I didn't reproduce it. Let me try to check the code and update later.

[jmhbnz]

Hey @Rajalakshmi-Girish - You mentioned this flake is only occurring on ppc64le. Please note this is currently a tier 3 supported platform meaning the project generally only invest time in ensuring the build completes successfully, refer to: https://etcd.io/docs/v3.5/op-guide/supported-platform

I'll leave this issue open in case a community member has time to investigate further, though this issue may end up closing as stale.

[Rajalakshmi-Girish]

@kishen-v @Karthik-K-N ^^