week_6_stream_processing

Week 6: Stream Processing

Code structure

• Java examples
• Python examples
• KSQLD examples

Confluent cloud setup

Confluent cloud provides a free 30 days trial for, you can signup here

Introduction to Stream Processing

• Slides
• 🎥 6.0.1 DE Zoomcamp 6.0.1 - Introduction
• 🎥 6.0.2 DE Zoomcamp 6.0.2 - What is stream processing

Introduction to Kafka

• 🎥 6.3 DE Zoomcamp 6.3 - What is kafka?
• 🎥 6.4 DE Zoomcamp 6.4 - Confluent cloud
• 🎥 6.5 DE Zoomcamp 6.5 - Kafka producer consumer

Kafka Configuration

• 🎥 6.6 DE Zoomcamp 6.6 - Kafka configuration
• Kafka Configuration Reference

Kafka Streams

• Slides
• Streams Concepts
• 🎥 6.7 DE Zoomcamp 6.7 - Kafka streams basics
• 🎥 6.8 DE Zoomcamp 6.8 - Kafka stream join
• 🎥 6.9 DE Zoomcamp 6.9 - Kafka stream testing
• 🎥 6.10 DE Zoomcamp 6.10 - Kafka stream windowing
• 🎥 6.11 DE Zoomcamp 6.11 - Kafka ksqldb & Connect
• 🎥 6.12 DE Zoomcamp 6.12 - Kafka Schema registry

Faust - Python Stream Processing

• Faust Documentation
• Faust vs Kafka Streams

Pyspark - Structured Streaming

Please follow the steps described under pyspark-streaming

• 🎥 6.13 DE Zoomcamp 6.13 - Kafka Streaming with Python
• 🎥 6.14 DE Zoomcamp 6.14 - Pyspark Structured Streaming

Kafka Streams with JVM library

• Confluent Kafka Streams
• Scala Example

KSQL and ksqlDB

• Introducing KSQL: Streaming SQL for Apache Kafka
• ksqlDB

Kafka Connect

• Making Sense of Stream Data

Docker

Starting cluster

Command line for Kafka

Create topic

./bin/kafka-topics.sh --create --topic demo_1 --bootstrap-server localhost:9092 --partitions 2

Homework

Form

The homework is mostly theoretical. In the last question you have to provide working code link, please keep in mind that this question is not scored.

Deadline: 13 March 2023, 22:00 CET

Community notes

Did you take notes? You can share them here.

• Notes by Alvaro Navas
• Marcos Torregrosa's blog (spanish)
• Add your notes here (above this line)