This demo uses GitHub Actions for developing and valiating Apache Kafka client application, focusing on a Python producer and consumer using Kafka Python Client, plus examples for testing schema compatibility using both Python and Java.
The code defines an GitHub Actions workflow kafka-app with jobs to demonstrate a few capabilities:
- build: Build a Docker image for the Kafka Python client application
- sanity: Sanity test the app with rdkafka mock
- local: Locally test the app using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
- local-avro: Locally test the app with Schema Registry and Avro using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
- ccloud: Test the app to Confluent Cloud
- ccloud-avro: Test the app to Confluent Cloud with Confluent Cloud Schema Registry and Avro
- ccloud-python-schema-compatibility: Use SchemaRegistryClient.test_compatibility() as shown in src/test_schema_compatibility.py to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Python)
- ccloud-java-schema-compatibility: Use kafka-schema-registry-maven-plugin as shown in the pom.xml to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Java)
-
Provision a Kafka cluster, enable Schema Registry and create an API key and secret on Confluent Cloud. Use the quickstart, Confluent Terraform Provider, or ccloud-stack.
-
In your GitHub account, create Encrypted Secrets for your Confluent Cloud cluster and credentials:
# Confluent Cloud Kafka cluster
CONFLUENT_BOOTSTRAP_SERVERS
CONFLUENT_API_KEY
CONFLUENT_API_SECRET
# Confluent Cloud Schema Registry
CONFLUENT_SCHEMA_REGISTRY_URL
CONFLUENT_BASIC_AUTH_USER_INFO- Fork this repository and push a change to trigger the workflow.
- Confluent Cloud: Cloud-native service for Apache Kafka
- GitHub Actions for ksqlDB: demonstrates schema migrations upon a ksqlDB cluster running in Confluent Cloud
- Kafka Python Client
- GitHub Actions