Skip to content

EndlessBox/Data-Streaming-Twitter

Repository files navigation

                                           # Data_streaming

This side project was made with "learning the data engineering process" in mind. so i made a program that stream data from twitter, following some rules that user can set in "data_sources.json" file. and the same time store all the incoming data into a mysql data.

Further update

  • apply sentiment Analysis on the stored data
  • update my database from another source of data.

technologies

  • python.
  • docker (mysql, phpmyadmin).

usage

  • before executing the "figure_it_out.py" you have to execute "./get_bearer.sh <api_key> <api_secret_key>". it will generate a "bearer token" and store it along with your <api_secret_key> in env variables named :
    • ACCESS_TOKEN_TWT
    • API_KEY_TWT
    • API_SECRET_TWT
  • you can activate and de-activate log output on screen, by changin the "data_sources/log" value in the "data_sources.json" file.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published