[Execution] New ingestion engine #5593
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stion-new-engine-not-work
zhangchiqing
changed the base branch from
master
to
leo/ingestion-core-use-component-manager
peterargue
approved these changes
Apr 25, 2024
cmd/Dockerfile
Outdated
| @@ -37,7 +37,7 @@ WORKDIR /app | |||
|
|
|||
| ARG GOARCH=amd64 | |||
| # TAGS can be overriden to modify the go build tags (e.g. build without netgo) | |||
| ARG TAGS="netgo" | |||
| ARG TAGS="netgo,osusergo" | |||
There was a problem hiding this comment.
I'm ok with updating the default user, but I think this should go in a different PR since it has implications for node operators and infra
engine/execution/ingestion/core.go
Outdated
| @@ -164,6 +164,8 @@ func (e *Core) launchWorkerToExecuteBlocks(ctx irrecoverable.SignalerContext, re | |||
| } | |||
|
|
|||
| func (e *Core) OnBlock(header *flow.Header, qc *flow.QuorumCertificate) { | |||
| e.log.Debug().Msgf("received block %v (%v)", header.Height, qc.BlockID) | |||
zhangchiqing
commented
Apr 25, 2024
| @@ -120,6 +121,7 @@ func (exeConf *ExecutionConfig) SetupFlags(flags *pflag.FlagSet) { | |||
| flags.BoolVar(&exeConf.onflowOnlyLNs, "temp-onflow-only-lns", false, "do not use unless required. forces node to only request collections from onflow collection nodes") | |||
| flags.BoolVar(&exeConf.enableStorehouse, "enable-storehouse", false, "enable storehouse to store registers on disk, default is false") | |||
| flags.BoolVar(&exeConf.enableChecker, "enable-checker", true, "enable checker to check the correctness of the execution result, default is true") | |||
| flags.BoolVar(&exeConf.enableNewIngestionEngine, "enable-new-ingestion-engine", false, "enable new ingestion engine, default is false") | |||
There was a problem hiding this comment.
By default the new ingestion engine is disabled, and can be enabled by flag.
| @@ -399,11 +399,14 @@ func prepareExecutionService(container testnet.ContainerConfig, i int, n int) Se | |||
| panic(err) | |||
| } | |||
|
|
|||
| enableNewIngestionEngine := true | |||
There was a problem hiding this comment.
the localnet enables new ingestion engine by default
janezpodhostnik
approved these changes
Apr 29, 2024
Base automatically changed from
leo/ingestion-core-use-component-manager
to
master
April 30, 2024 17:26
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please review the following PRs first:
#5248
#5288
#5337
This PR adds a new ingestion engine (Ingestion.NewMachine) that is responsible for fetching data for unexecuted blocks.
The new ingestion engine could resolve the following issues:
#5161
#5217
#5298
The new ingestion engine can be turned on with a new flag
--enable-new-ingestion-engine=true.The new flag allows us to switch back to the old ingestion engine during the testing phase if the new engine ran into any problem, such as the block execution is halted.
It would not cause execution fork, because the ingestion engine is only responsible for determine when a block can be executed, rather than how a block is executed.