Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

raft/rafttest: TestBasicProgress failed #122049

Closed
cockroach-teamcity opened this issue Apr 9, 2024 · 4 comments
Closed

raft/rafttest: TestBasicProgress failed #122049

cockroach-teamcity opened this issue Apr 9, 2024 · 4 comments
Assignees
@pav-kv
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-stale
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 9, 2024 •
edited by cockroach-jira-scripts
Loading

raft/rafttest.TestBasicProgress failed with artifacts on master @ 64341d7802a1ba8b3fbd6a59e5afd58b693dc569:

raft2024/04/09 20:16:49 INFO: 1 has received 2 MsgVoteResp votes and 2 vote rejections
raft2024/04/09 20:16:49 INFO: 1 received MsgVoteResp from 5 at term 2
raft2024/04/09 20:16:49 INFO: 1 has received 3 MsgVoteResp votes and 2 vote rejections
raft2024/04/09 20:16:49 INFO: 1 became leader at term 2
raft2024/04/09 20:16:49 INFO: raft.node: 1 elected leader 1 at term 2
raft2024/04/09 20:16:49 INFO: 4 [logterm: 1, index: 5] sent MsgVote request to 3 at term 3
raft2024/04/09 20:16:49 INFO: 4 [logterm: 1, index: 5] sent MsgVote request to 5 at term 3
raft2024/04/09 20:16:49 INFO: 4 received MsgVoteResp from 4 at term 3
raft2024/04/09 20:16:49 INFO: 4 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/04/09 20:16:49 INFO: 5 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/04/09 20:16:49 INFO: 5 became follower at term 3
raft2024/04/09 20:16:49 INFO: 5 [logterm: 1, index: 5, vote: 0] cast MsgVote for 4 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 3 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/04/09 20:16:49 INFO: 3 became follower at term 3
raft2024/04/09 20:16:49 INFO: 3 [logterm: 1, index: 5, vote: 0] cast MsgVote for 4 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 2 received MsgVoteResp from 2 at term 3
raft2024/04/09 20:16:49 INFO: 2 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/04/09 20:16:49 INFO: 2 [logterm: 1, index: 5, vote: 2] rejected MsgVote from 4 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 4 received MsgVoteResp from 5 at term 3
raft2024/04/09 20:16:49 INFO: 4 has received 2 MsgVoteResp votes and 0 vote rejections
raft2024/04/09 20:16:49 INFO: 4 received MsgVoteResp from 3 at term 3
raft2024/04/09 20:16:49 INFO: 4 has received 3 MsgVoteResp votes and 0 vote rejections
raft2024/04/09 20:16:49 INFO: 4 became leader at term 3
raft2024/04/09 20:16:49 INFO: raft.node: 4 elected leader 4 at term 3
raft2024/04/09 20:16:49 INFO: 3 [logterm: 1, index: 5, vote: 4] rejected MsgVote from 2 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: raft.node: 5 elected leader 4 at term 3
raft2024/04/09 20:16:49 INFO: raft.node: 3 elected leader 4 at term 3
raft2024/04/09 20:16:49 INFO: 1 [term: 2] received a MsgApp message with higher term from 4 [term: 3]
raft2024/04/09 20:16:49 INFO: 2 [term: 3] ignored a MsgApp message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 1 became follower at term 3
raft2024/04/09 20:16:49 INFO: found conflict at index 6 [existing term: 2, conflicting term: 3]
raft2024/04/09 20:16:49 INFO: replace the unstable entries from index 6
raft2024/04/09 20:16:49 INFO: raft.node: 1 changed leader from 1 to 4 at term 3
raft2024/04/09 20:16:49 INFO: 1 [logterm: 3, index: 6, vote: 0] rejected MsgVote from 2 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 1 [logterm: 3, index: 6, vote: 0] rejected MsgVote from 4 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 5 [logterm: 3, index: 6, vote: 4] rejected MsgVote from 2 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 2 [term: 3] ignored a MsgHeartbeat message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 4 [logterm: 3, index: 6, vote: 4] rejected MsgVote from 2 [logterm: 1, index: 5] at term 3
raft2024/04/09 20:16:49 INFO: 3 [term: 3] ignored a MsgApp message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 4 [term: 3] ignored a MsgApp message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 2 received MsgVoteResp rejection from 3 at term 3
raft2024/04/09 20:16:49 INFO: 2 has received 1 MsgVoteResp votes and 1 vote rejections
raft2024/04/09 20:16:49 INFO: 2 became follower at term 3
raft2024/04/09 20:16:49 INFO: raft.node: 2 elected leader 4 at term 3
raft2024/04/09 20:16:49 INFO: 5 [term: 3] ignored a MsgHeartbeat message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 3 [term: 3] ignored a MsgHeartbeat message with lower term from 1 [term: 2]
raft2024/04/09 20:16:49 INFO: 4 [term: 3] ignored a MsgHeartbeat message with lower term from 1 [term: 2]
    node_test.go:47: commits failed to converge!
--- FAIL: TestBasicProgress (5.26s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-37676
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv-replication labels Apr 9, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 9, 2024
@kvoli
Copy link
Collaborator

kvoli commented Apr 9, 2024

Duplicate of #121745

@kvoli kvoli marked this as a duplicate of #121745 Apr 9, 2024
@kvoli kvoli added X-duplicate Closed as a duplicate of another issue. P-1 Issues/test failures with a fix SLA of 1 month labels Apr 9, 2024
@arulajmani arulajmani added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 10, 2024
@arulajmani
Copy link
Collaborator

Removing the release-blocker label and adding a GA-blocker to mirror #121745.

@pav-kv pav-kv added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure and removed GA-blocker P-1 Issues/test failures with a fix SLA of 1 month labels Apr 10, 2024
@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestBasicProgress failed with artifacts on master @ 86f20a299e0ca1bbf8ba77e624e86d92e3d3baa3:

raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored entry appends from a MsgStorageAppendResp message with lower term [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 4 [term: 3] ignored a MsgAppResp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 2 [term: 3] ignored a MsgHeartbeat message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgHeartbeat message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 3 [term: 3] ignored a MsgHeartbeat message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 1 [term: 3] ignored a MsgHeartbeat message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
raft2024/04/12 17:37:45 INFO: 5 [term: 3] ignored a MsgApp message with lower term from 4 [term: 2]
    node_test.go:47: commits failed to converge!
2024/04/12 17:37:50 raft.1: stop
--- FAIL: TestBasicProgress (5.19s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@nvanbenschoten nvanbenschoten added the P-3 Issues/test failures with no fix SLA label May 13, 2024
@exalate-issue-sync exalate-issue-sync bot added T-kv KV Team and removed T-kv-replication labels Jun 28, 2024
@github-actions GitHub Actions
Copy link

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

@github-actions github-actions bot closed this as completed Aug 6, 2024
@pav-kv pav-kv reopened this Aug 6, 2024
@github-actions github-actions bot added X-stale and removed X-stale labels Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pav-kv pav-kv

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-stale
Projects
No open projects
KV
Status: Incoming
Milestone
24.1
Development

No branches or pull requests
5 participants
@pav-kv @nvanbenschoten @arulajmani @cockroach-teamcity @kvoli