Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support kafka consumer groups #215

Closed
jfallows opened this issue Apr 20, 2023 · 0 comments · Fixed by #262
Closed

Support kafka consumer groups #215

jfallows opened this issue Apr 20, 2023 · 0 comments · Fixed by #262
Assignees
Labels
enhancement New feature or request

Comments

@jfallows
Copy link
Contributor

This requires creating a new subtype of kafka streams in kafka.idl.

KafkaGroupBeginEx
consumerId
session timeout

DATA
(leader can change assignments)
assignments (payload)

KafkaGroupDataEx
leader
members

FLUSH ⇒ heartbeat now

KafkaGroupClientFactory
configured group instance id (env)
configured client id (env)
handles rebalance internally

KafkaMergedBeginEx
consumerId
session timeout

subscribes only to topic partitions as indicated by assignments
OffsetFetch to bootstrap initial partition offsets
ack triggers OffsetCommit

KafkaOffsetCacheServerFactory
fan-in handles offset commit gaps
perhaps gaps can be handled via tag buffers (newer protocol versions)
or committed_metadata (older protocol versions okay)

KafkaFetchCacheServerFactory
anycast instead of broadcast for consumer group
handle redelivery if client fails, potentially out of order
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants