robots.cr

Simple robots.txt parsing for Crystal

Installation

  1. Add the dependency to your shard.yml:

    dependencies:
  robots:
    github: grkek/robots

  2. Run shards install

Usage

require "robots"

# From reddit
robotstxt = <<-TEXT
# 80legs
User-agent: 008
Disallow: /

User-Agent: *
Disallow: /goto
Disallow: /*after=
Disallow: /*before=
Disallow: /domain/*t=
Disallow: /login
Disallow: /reddits/search
Disallow: /search
Disallow: /r/*/search
Allow: /
TEXT

reddit = Robots.new(robotstxt, "Googlebot")

reddit.allowed?("/login")
# => false

reddit.allowed?("/r/SushiChain")
# => true

reddit.allowed?("/r/SushiChain/search")
# => false

Contributing

  1. Fork it (https://github.com/grkek/robots/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors