let kafka support "commit offset after consuming" #206
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…reviewed by Joel Koshy and Jay Kreps.
… by Guozhang Wang
…oup purgatory in coordinator; reviewed by Jun Rao
…; reviewed by Guozhang Wang
… reviewed by Guozhang Wang
mirror-maker producer with close(0) on send error; reviewed by Joel Koshy
consumer connector API; reviewed by Joel Koshy
…mmit to" This reverts commit 23ff851.
….8.2.1; patched by Ismael Juma; reviewed by Jun Rao
…consumer connector API; reviewed by Joel Koshy
default for SNAPSHOTs and allow remote Maven repository configuration from the command line.
… by Yasuhiro Matsuda; reviewed by Onur Karaman, Guozhang Wang and Jun Rao
…; reviewed by Onur Karaman
…rg.apache.kafka.common.network; patched by Gwen Shapira; reviewed by Joel Koshy, Jay Kreps, Jiangjie Qin, Guozhang Wang and Jun Rao
… InvalidMetadataException; reviewed by Ewen Cheslack-Postava and Joel Koshy
…tions list lock; reviewed by Onur Karaman and Jiangjie Qin
…Singh; reviewed by Jun Rao
…ng NetworkClient; patched by Jason Gustafson; reviewed by Jun Rao
…reviewed by Jun Rao
…set; patched by Alexey Ozeritski; reviewed by Jun Rao
…by Tim Brooks; reviewed by Jun Rao
…tractRequest.getRequest; patched by Andrii Biletskyi; reviewed by Jun Rao
…Verma; reviewed by Jun Rao
…tal letter; patched by Sriharsha Chintalapani; reviewed by Jun Rao
…; patched by Manikumar Reddy; reviewed by Jun Rao
guozhangwang This code change properly types ProcessorDef. This also makes KStream.process() typesafe. Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Ismael Juma, Guozhang Wang Closes #289 from ymatsuda/typing_ProcessorDef
Author: Jason Gustafson <jason@confluent.io> Reviewers: Onur Karaman, Guozhang Wang Closes #267 from hachikuji/KAFKA-2596
Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Guozhang Wang Closes #285 from ewencp/kafka-2622-time-logical-type
…terfaces A few of Kafka Stream's interfaces and classes are not as well-aligned with Java 8's functional interfaces. By making these changes, when Kafka moves to Java 8 these classes can extend standard Java 8 functional interfaces while remaining backward compatible. This will make it easier for developers to use Kafka Streams, and may allow us to eventually remove these custom interfaces and just use the standard Java 8 interfaces. The changes include: 1. The 'apply' method of KStream's `Predicate` functional interface was renamed to `test` to match the method name on `java.util.function.BiPredicate`. This will allow KStream's `Predicate` to extend `BiPredicate` when Kafka moves to Java 8, and for the `KStream.filter` and `filterOut` methods to accept `BiPredicate`. 2. Renamed the `ProcessorDef` and `WindowDef` interfaces to `ProcessorSupplier` and `WindowSupplier`, respectively. Also the `SlidingWindowDef` class was renamed to `SlidingWindowSupplier`, and the `MockProcessorDef` test class was renamed to `MockProcessorSupplier`. The `instance()` method in all were renamed to `get()`, so that all of these can extend/implement Java 8's `java.util.function.Supplier<T>` interface in the future with no other changes and while remaining backward compatible. Variable names that used some form of "def" were changed to use "supplier". These two sets of changes were made in separate commits. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes #270 from rhauch/kafka-2600
Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Guozhang Wang Closes #291 from ewencp/fixup-time-logical-type
guozhangwang * added back type safe stateful transform methods (kstream.transform() and kstream.transformValues()) * changed kstream.process() to void Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Guozhang Wang Closes #292 from ymatsuda/transform_method
…ption` threshold (max.connections.per.ip) is reached * Call `ConnectionQuotas.decr` when calling `Selector.close` and when disconnections happen. * Expand `SocketServerTest` to test for this and to close sockets. * Refactor and clean-up `SocketServer` and `Acceptor` to make the code easier to understand. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jun Rao <junrao@gmail.com> Closes #288 from ijuma/kafka-2614-connection-count-not-updated
Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #296 from granthenke/tools-log4j
…uld not return NaN This is a followup ticket from KAFKA-2084 to improve the windowSize calculation in Quotas. I've made the following changes: 1. Added a windowSize function on Rate 2. Calling Rate.windowSize in ClientQuotaManager to return the exact window size to use when computing the delay time. 3. Changed the window size calculation subtly. The current calculation had a bug wherein, it used the number of elapsed seconds from the "lastWindowSeconds" of the most recent Sample object. However, the lastWindowSeconds is the time when the sample is created.. this causes an issue because it implies that the current window elapsed time is always "0" when the sample is created. This is incorrect as demonstrated in a testcase I added in MetricsTest. I've fixed the calculation to count the elapsed time from the "oldest" sample in the set since that gives us an accurate value of the exact amount of time elapsed Author: Aditya Auradkar <aauradkar@linkedin.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com> Closes #213 from auradkar/K-2443
This patch is different than the one attached to the JIRA - I'm applying the new javadoc rules to all subprojects while the one in the JIRA applies only to "clients". We need this since Copycat has the same issues. Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes #147 from gwenshap/KAFKA-2203
Parametrize console consumer sanity test, replication tests and benchmarks tests to run with both PLAINTEXT and SSL. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Geoff Anderson, Ewen Cheslack-Postava, Guozhang Wang Closes #271 from rajinisivaram/KAFKA-2581
…t can be turned down on shared build infrastructure. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma, Sriharsha Chintalapani Closes #293 from ewencp/kafka-2613-user-configurable-max-forks
|
Can we just close this, since we are not planning a new release of 0.8.2? This triggers failed PR builds every time we commit to trunk. |
asfgit
pushed a commit
that referenced
this pull request
Oct 13, 2015
ijuma
added a commit
to ijuma/kafka
that referenced
this pull request
Oct 13, 2015
* apache/trunk: KAFKA-2637; Cipher suite setting should be configurable for SSL Closes apache#206 . *WONT FIX* - no new release planned for 0.8.2 branch closes pr apache#206. *WONT FIX* - no new release planned for 0.8.2 branch KAFKA-2613; Make maxParallelForks configurable via Gradle config so it can be turned down on shared build infrastructure. KAFKA-2581: Run some existing ducktape tests with SSL KAFKA-2203: Getting Java8 to relax about javadoc and let our build pass KAFKA-2443 KAFKA-2567; Expose windowSize on Rate; - Throttle time should not return NaN KAFKA-2633; Default logging from tools to Stderr
soondenana
pushed a commit
to soondenana/kafka
that referenced
this pull request
Jun 26, 2019
…e#206) Remove compile-time ZK dependency in BaseMetricsReporter so that components extending this without ZK configs don't have a dependency on the core kafka jar. Reviewers: Ismael Juma <ismael@juma.me.uk>
omkreddy
pushed a commit
to omkreddy/kafka
that referenced
this pull request
Jul 8, 2019
…e#206) Remove compile-time ZK dependency in BaseMetricsReporter so that components extending this without ZK configs don't have a dependency on the core kafka jar. Reviewers: Ismael Juma <ismael@juma.me.uk>
jsancio
pushed a commit
to jsancio/kafka
that referenced
this pull request
Aug 6, 2019
efeg
pushed a commit
to efeg/kafka
that referenced
this pull request
Jan 29, 2020
xiowu0
pushed a commit
to xiowu0/kafka
that referenced
this pull request
Dec 9, 2021
…he#206) TICKET = LIKAFKA-38536 LI_DESCRIPTION = The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally.
wyuka
pushed a commit
to wyuka/kafka
that referenced
this pull request
Feb 4, 2022
…he#206) TICKET = LIKAFKA-38536 LI_DESCRIPTION = The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally.
wyuka
added a commit
to wyuka/kafka
that referenced
this pull request
Feb 4, 2022
…he#206) (apache#271) TICKET = LIKAFKA-38536 LI_DESCRIPTION = Original hotfix PR linkedin#206 The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally. Co-authored-by: Lucas Wang <luwang@linkedin.com>
wyuka
added a commit
to wyuka/kafka
that referenced
this pull request
Mar 4, 2022
…he#206) (apache#271) TICKET = LIKAFKA-38536 LI_DESCRIPTION = Original hotfix PR linkedin#206 The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally. Co-authored-by: Lucas Wang <luwang@linkedin.com>
wyuka
added a commit
to wyuka/kafka
that referenced
this pull request
Mar 28, 2022
…he#206) (apache#271) TICKET = LIKAFKA-38536 LI_DESCRIPTION = Original hotfix PR linkedin#206 The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally. Co-authored-by: Lucas Wang <luwang@linkedin.com>
wyuka
added a commit
to wyuka/kafka
that referenced
this pull request
Jun 16, 2022
…#206) (apache#271) TICKET = LIKAFKA-38536 LI_DESCRIPTION = Original hotfix PR linkedin#206 The ControllerContext.isReplicaOnline method is frequently called inside a loop, and internally this method relies on several derived fields in the ControllerContext. Repeatedly calculating the derived fields could be expensive, and yet these fields do not change between iterations of the loop. This PR creates a new class ControllerContextSnapshot that tries to cache the derived fields in order to save the repeated computation and speed up the controller. EXIT_CRITERIA = When this change is proposed in upstream kafka and pulled internally. Co-authored-by: Lucas Wang <luwang@linkedin.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kafka Of Original Version do not support "commit offset after consuming",when kafka.consumer.ConsumerIterator[K, V].next() return a MessageAndMetadata,which consumed offset is already being set,and commit thread maybe commit this offset before "consuming process finished",when jvm restart or being down，this msg will not be consumed next time