robots.txt crawl API

Use this API to handle admin operations like add new robots.txt crawl.

Configuration

You can change this properties to change server behavior:

Server

Variable	Description	Default
SERVER_PORT	Default server port	8080
SERVER_ADDRESS	Default server address	localhost
LOGGING_FILE	Server logging file

Redis

You can change database configuration using the values documented here

Build

If you want to build a .jar with all server dependencies, just execute the next command from the project root directory:

$ ./gradlew assemble

The resulting executable will be located inside the build/libs directory.

Also, if you want to execute this project using Docker, from the project root directory execute:

$ make build-crawler

This command will create a new docker image with all project dependencies.