Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: prevent unbounded raft log growth without quorum #27774

Merged
merged 2 commits into from
Jul 23, 2018
Merged

storage: prevent unbounded raft log growth without quorum #27774

merged 2 commits into from
Jul 23, 2018
+241 −30

Commits on Jul 23, 2018

  1. storage: prevent unbounded raft log growth without quorum 

    Fixes cockroachdb#27772.

This change adds safeguards to prevent cases where a raft log
would grow without bound during loss of quorum scenarios. It
also adds a new test that demonstrates that the raft log does
not grow without bound in these cases.

There are two cases that need to be handled to prevent the
unbounded raft log growth observed in cockroachdb#27772.
1. When the leader proposes a command and cannot establish a
   quorum. In this case, we know the leader has the entry in
   its log, so there's no need to refresh it with `reasonTicks`.
   To avoid this, we no longer use `refreshTicks` as a leader.
2. When a follower proposes a command that is forwarded to the
   leader who cannot establish a quorum. In this case, the
   follower can't be sure (currently) that the leader got the
   proposal, so it needs to refresh using `reasonTicks`. However,
   the leader now detects duplicate forwarded proposals and
   avoids appending redundant entries to its log. It does so
   by maintaining a set of in-flight forwarded proposals that
   it has received during its term as leader. This set is reset
   after every leadership change.

Both of these cases are tested against in the new
TestLogGrowthWhenRefreshingPendingCommands. Without both of
the safeguards introduced in this commit, the test fails.

Release note (bug fix): Prevent unbounded growth of the raft log
caused by a loss of quorum.
    @nvanbenschoten
    nvanbenschoten committed Jul 23, 2018
    Configuration menu
    Copy the full SHA
    03b116f View commit details
    Browse the repository at this point in the history

  2. storage: s/replica.mu.proposals/replica.mu.localProposals/ 

    Release note: None
    @nvanbenschoten
    nvanbenschoten committed Jul 23, 2018
    Configuration menu
    Copy the full SHA
    192a828 View commit details
    Browse the repository at this point in the history