A pipeline within AWS to capture schema changes in S3 files and to update them in a DB.

The Project aims to establish a robust data pipeline for tracking and analyzing sales performance using various AWS services. The process involves creating a DynamoDB database, implementing Change Data Capture (CDC), utilizing Kinesis streams, and finally, storing and querying the data in Amazon Athena.

Developed an ETL pipeline for real-time ingestion of stock market data from the stock-market-data-manage.onrender.com API. Engineered the system to store data in Parquet format for optimized query processing and incorporated data quality checks to ensure accuracy prior to visualization.

Collecting the list of songs,album and artists list details from the Spotify Music Application in specific intervals using spotipy API and performing ETL Operations using Amazon Cloud Services

Este projeto tem como objetivo realizar a coleta, catalogo, governança, processamento e visualização de dados.

Unveiling job market trends with Scrapy and AWS

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka. We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Implemented ETL pipeline on AWS for Playstore data using Lambda, Glue Crawlers, and Glue ETL Jobs. Orchestrated workflow with Step Functions and achieved seamless integration, optimal data merging, and enhanced data quality/accessibility.

Working with Glue Data Catalog and running the using S3 Event Notification and creating the entire stack using AWS CloudFormation

AWS Athena, Glue Database, Glue Crawler deployment through CloudFormation stack on already existing S3 buckets on AWS console.

An end-to-end data pipeline built with AWS S3, Glue, Crawler, Athena, Tableau visulization

Open data and cloud computing to answer the question: Are we losing our spring days?

AWS Athena, Glue Database, Glue Crawler deployment on existing S3 bucket through Serverless (sls) Framework.

An end-to-end solution for managing and analyzing YouTube video data from Kaggle, leveraging AWS services and visualized through Quicksight and Tableau

AWS Athena, Glue Database, Glue Crawler and S3 buckets deployment through Serverless (sls) Framework

Smart City Realtime Data Engineering Project

Cloud Development Kit (AWS CDK) using TypeScript, Python and Java

An end-to-end solution for managing and analyzing YouTube video data from Kaggle, leveraging AWS services and visualized through Quicksight and Tableau

An end-to-end data engineering project in which five NYC DOT datasets were modified in an ETL process and analyzed for insights.

In this project I have used the Trending YouTube Video Statistics data from Kaggle to analyze and prepare it for usage.