robots

Author:	Antoine Gagné

Contents

• 1   Installation
• 2   Usage
• 3   Development
  • 3.1   Running all the tests and linters

A library that parses and validates rules from robots.txt.

1   Installation

This library is available on hex.pm <https://hex.pm/packages/robots>_. To install this library, simply add the following lines to your rebar.config:

2   Usage

Content = <<"User-Agent: bot\nAllow: /fish">>,
%% This will return an opaque type that contains all the rules and their agents
{ok, RulesIndex} = robots:parse(Content, 200),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/fish/salmon.html">>, RulesIndex),
false = robots:is_allowed(<<"bot/1.0.0">>, <<"/Fish.asp">>, RulesIndex),

3   Development

3.1   Running all the tests and linters

You can run all the tests and linters with the rebar3 alias:

rebar3 check