Hestia scrapes real estate websites for new rental listings, and broadcasts the results via Telegram. Check out @HestiaPhilipBot on Telegram: https://t.me/HestiaPhilipBot
docker compose -f docker-compose-dev.yml up --build
- postgres admin available under localhost:8080, credentials are
a@a.com
andverysecret
.
The workflow for adding new targets to the scraper is as follows:
- Add a target child class, to the
targets.py
file. This is done by taking another target as a template and modifying it. For example theSpotmakelaardij
class parses the HTML returned by the page, while theVbtverhuurmakelaars
target parses a JSON request. - Add the created class to the
targets
variable at the bottom of thetargets.py
file. - To test the scraper standalone from the whole system, modify the
test.py
file to import your scraper class. Run it withpython3 test.py
. The parsed homes should be printed to the terminal. - Commit and push, the scraper will be automatically active on the live system.