A robust yet minimal web crawler implemented in Rust, utilizing various libraries and aiming for scalability and extensibility.
🚧 This is a work in progress and therefore you should expect that the application may not have all the features at this moment.
🤦 Another web crawler… but rusty. 🦀
To use this web crawler, simply check out the repository, build, and run with:
cargo b --release && ./target/release/rusty_crawler --help