Stock-Market-Real-Time-Data-Engineering-Project

Introduction

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Technology Used

• Programming Language - Python
• Amazon Web Service (AWS)

1. S3 (Simple Storage Service)
2. Athena
3. Glue Crawler
4. Glue Catalog
5. EC2

• Apache Kafka

Dataset Used

You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline)

Here is the dataset used - https://github.com/mihirkudale/Stock-Market-Real-Time-Data-Engineering-Project/blob/main/indexProcessed.csv