The main objective of this project is to scrape data or collect data from websites in a structured format. This data is used to do data analysis and take a decision which puts some good results for the amazon. This project is for learning purposes. I Used Requests. BeautifulSoup modules to make connection with url and fetch data using their tags and attributes. And finally i save all data in CSV file using Pandas Framework.
-
scraper.py : This is our main file which connects the URL by using the requests module and downloading the request page. Then we use BeautifulSoup to fetch Categories and their URLs.
-
page_scraper.py : It contains all the functions which are used to fetch all categories pages.
-
data_scraper : This file also contains functions which used to scrape product data like
Product Description,Product category,Product Price,Product Reviews,Product Reviews, andProduct Urls.