op-node: handle sequencer errors with derivation-levels, reset when L1 origins are inconsistent

[protolambda]

Description

On Goerli we have hit an interesting sequencer edge-case:

• The sequencer conf-depth is lower than the verifier conf-depth
• The L1 reorgs with a reorg-depth in-between these
• The verifier only ever sees the good chain
• The sequencer builds an unsafe L2 block with a L1 origin that doesn't match the canonical chain
• The verifier code-path drops the batch because of the bad L1 origin (epoch) value
• The verifier code-path patiently waits for the sequencer to provide an alternative L2 batch with good origin, because the sequencing window has not elapsed yet.
• The sequencer code-path doesn't undo the unsafe L2 block with bad L1 origin, nor does the verifier code-path

This situation locks up the chain until the sequencing window elapses. To prevent this, we need the sequencer to build a good L2 block, without having to restart it manually to get it back onto an unsafe L2 head with good L1 origin (the work-around we've executed two times now).

Changes:

• Removed unused idleDerivation; the sequencing and derivation codepaths are already independent, and handle conflicts through through the shared engine-control
• Added error-handling cases to the RunNextSequencerAction to handle when starting/sealing fails due to crit/reset/temp/other error.
• Changed the L1-origin consistency check in the sequencer to a reset-worthy error
• Make RunNextSequencerAction bubble up the error if it's critical, it's up to the driver to handle that (in this case log + shutdown driver, since we prefer safety over liveness).

Tests

• Added an action-test to run through this edge-case as regression test.
• Updated the sequencer chaos test to simulate these type of errors that require a Reset() call to recover from.

Invariants

Ensure that the selected L1 origin of every new sequenced L2 block is consistent (i.e. equal or parent-hash of new block matches current) with the previous L1 origin, and execute a reset to make things consistent otherwise.

Additional context

Follow-up to Goerli sequencing issue.

Metadata

Fix CLI-3404

[changeset-bot]

⚠️ No Changeset found

Latest commit: 36dca77

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codecov]

Codecov Report

Merging #4930 (36dca77) into develop (a340d59) will decrease coverage by 4.66%.
The diff coverage is 34.42%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4930      +/-   ##
===========================================
- Coverage    40.92%   36.26%   -4.66%     
===========================================
  Files          324      183     -141     
  Lines        19648    15575    -4073     
  Branches       770        0     -770     
===========================================
- Hits          8041     5649    -2392     
+ Misses       11007     9351    -1656     
+ Partials       600      575      -25     

Flag	Coverage Δ
bedrock-go-tests	36.26% <34.42%> (+<0.01%)	⬆️
contracts-bedrock-tests	?
contracts-tests	?
core-utils-tests	?
dtl-tests	?
fault-detector-tests	?
sdk-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
op-node/rollup/derive/engine_queue.go	21.55% <0.00%> (-0.11%)	⬇️
op-node/rollup/derive/pipeline.go	0.00% <ø> (ø)
op-node/rollup/driver/driver.go	0.00% <0.00%> (ø)
op-node/rollup/driver/metered_engine.go	0.00% <0.00%> (ø)
op-node/rollup/driver/state.go	0.00% <0.00%> (ø)
op-node/metrics/metrics.go	1.37% <9.09%> (+0.24%)	⬆️
op-node/rollup/driver/sequencer.go	83.45% <52.63%> (-11.19%)	⬇️
op-node/heartbeat/service.go	55.26% <0.00%> (-2.64%)	⬇️
.../contracts/libraries/trie/Lib_SecureMerkleTrie.sol
...ontracts-bedrock/contracts/L1/L1StandardBridge.sol
... and 141 more

[protolambda]

@tynes ready for review again