Plays a file onto a Kafka topic - one line == one message.
Build the project by running sbt assembly
. This produces an uber jar at target/scala-2.12/kafka-player.jar
.
For running against a local Kafka, invoke the player with a command like below:
java -jar $jar_file \
--message-file $message_file \
--num-messages $num_messages \
--messages-per-second $messages_per_second \
--topic $topic \
--broker-string $broker_string
If running against a TLS authenticated Kafka cluster, include additional parameters to specify the keystore location and keystore password:
--keystore-location $keystore_location
--keystore-passphrase $keystore_passphrase
--message-file
- A file containing the lines to play onto Kafka. The player will play one message per line.--num-messages
- The number of messages to play. The default is 100,000 if not provided.--messages-per-second
- The number of messages to play per second. The default is 2 per second if not provided. Float values are supported (e.g. 0.5 to produce one message every two seconds).--topic
- The topic to play the messages onto.--broker-string
- The bootstrap broker string of the target cluster. The default is localhost:9092 if not provided.--keystore-location
- The location of the keystore to use for TLS authentication. Only set this if TLS authentication is desired.--keystore-passphrase
- The passphrase for the keystore. Only set this if TLS authentication is desired and akeystore-location
is also provided.
A docker-compose file is included for local development for spinning up a single broker wurstmeister/kafka cluster (https://hub.docker.com/r/wurstmeister/kafka/) exposed to localhost.
Topics required for specific scenarios may be added to the KAFKA_CREATE_TOPICS
environment variable in docker-compose.yml
.
The required format is described in https://github.com/wurstmeister/kafka-docker#automatically-create-topics.
The example
folder of this repository contains a data file called example.json
and a launch script called play-example.sh
to play the data file onto a local Kafka. Run the example with the following steps:
# start the kafka docker containers in the background - this will also create a topic called `example` on the cluster.
docker-compose up -d
# play the example data file onto the local kafka
./example/play-example.sh
Once the player starts, run kafkacat
in another terminal to verify your messages are coming in.
kafkacat -C -b localhost:9092 -t example
The example plays 10,000 messages at a rate of one message every two seconds.
The example.json
data file is an abridged and slightly transformed (into a line delimited format) version of American movies from https://github.com/jdorfman/awesome-json-datasets#movies.