This piece of code will collect hyperlinks by scraping them from a given base url. The output will be generated in form of a csv file. Set desired base url in the shell script.
execute crawler.sh file in linux.
All the links will be saved in file links.csv
PS - By default, the script will scrape 'https://medium.com' website