Skip to content

Commit

Permalink
Let shuffle always use get_next_message to support holey outboxes
Browse files Browse the repository at this point in the history
This commit changes the shuffle to always use get_next_message to read
the next outbox message. Before we were using get_message which was only
looking at a specific outbox entry. If this outbox entry was empty, then
the shuffle assumed that the outbox is empty. This did not work if the
outbox contained holes. Now with get_next_message, we always scan until
the next outbox message.

The change itself is trivial. In order to ensure that the change works,
this commit added a few unit tests to ensure the desired behaviour.

Note: With replacing get_message with get_next_message, the shuffle will
perform for every read a RocksDB scan operation. This is highly inefficient
and we should replace this logic with a tailing iterator.

This fixes #1639.
  • Loading branch information
tillrohrmann committed Jun 20, 2024
1 parent 20f21c7 commit 98d92f0
Show file tree
Hide file tree
Showing 3 changed files with 446 additions and 35 deletions.
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions crates/worker/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ tracing = { workspace = true }
tracing-opentelemetry = { workspace = true }

[dev-dependencies]
restate-bifrost = { workspace = true, features = ["test-util"] }
restate-core = { workspace = true, features = ["test-util"] }
restate-rocksdb = { workspace = true, features = ["test-util"] }
restate-schema-api = { workspace = true, features = ["test-util"] }
Expand Down
Loading

0 comments on commit 98d92f0

Please sign in to comment.