Scrape data from a site that is difficult to use. This is written for a very specific purpose and will not work as a general solution.
- python2.7
- requests
pip install requests
- beautifulsoup
pip install bs4
python scrape_em.py -h
for full options
python scrape_em.py --url 'https://myfavoritesite.com' --max 5 --csv 5_results --load data.json
- give option for config file definition of args. cmd line can get crowded quickly, especially with long urls