stability: keep a cache of pushed transactions

[petermattis]

Keep a cache of pushed transactions on a store to avoid repushing further intents after a txn has already been aborted or its timestamp moved forward.

[yananzhi]

@petermattis Could you assign this to me? I Would like to do some contribute.

[petermattis]

@yananzhi Github is not allowing me to assign an issue to someone who is not in the cockroachdb org. Regardless of assignment, feel free to work on this. No one else is and your comment on this issue will be enough to indicate that you are doing so.

[yananzhi]

@petermattis thanks

[tbg]

closing this for now. We have a related idea in #2632 and all of this is post-1.0 anyway.

[tbg]

Now that we have SQL and dropping tables issues DeleteRange on large swaths of keyspace, this is becoming relevant again. In the above scenario, the following could happen:

• Transactional DeleteRange hits a whole range and creates (say) 10k intents
• Later, it has to abort - the intents remain
• The new incarnation of the transaction tries to perform the same DeleteRange again.
• On the first key, it sees the intent, pushes the transaction (trivially successful since that txn is aborted), resolves the intent, performs the deletion (again).
• On the second key, <everything from above>
• ...

That turns a single DeleteRange MVCC operation into 10k RPCs, which clearly has no chance of going well.

It's time to augment the intentResolver so that after a successful push, it remembers the pushee (or rather, its TxnID and Epoch are mapped to the resulting TxnMeta). A subsequent push can then synthesize a corresponding []roachpb.Intent.
[nb: The geek in me wants to use a Cuckoo Filter for this task, but probably LRU is the way to go.]

This itself isn't enough: it saves the 10k pushes, but we will still have to resolve 10k intents and have the DeleteRange bounce back and forth 10k times, which is guaranteed to be a major drag on Raft.

An elegant solution comes up with leader-proposed Raft: since we see the intents before proposing to Raft, we can have more complex logic which performs a single push for the first intent, and then "directly" overwrites the remaining "identical" intents it encounters for the remainder of the DeleteRange. If we took that to the extreme, it could mean supplying more logic down to the lower MVCC levels (about which intents cause write intent errors), but I think we should hold off on that unless wide deletions are a regular thing that needs to be very fast.

Even before leader-proposed Raft, we could in principle do the same thing: once a DeleteRange runs into an intent, we execute it before Raft (without committing the result, just to discover more intents) and supply a batch-resolution to Raft. But that's fairly complex when you take into account that it needs to go away anyway.

My hunch is that this #7499 should be addressed first since there is a fairly clear path there to avoid these deletions in the first place (though users may still perform table deletions which have much of the same problems).

[spencerkimball]

Closing as this is a solution hatched before we've observed the problem. I for one am hoping that instead of range-wide DeleteRange calls, we smartly move dropped tables into a trash system database and then when dropping from the system database, we use ClearRange. The larger point is I want to close these conceptual solution issues that aren't anchored to a pressing concern based on real-world customer needs / issues.