class Robots::Parser

• Robots::Parser
• Reference
• Object

Defined in:

robots/parser.cr

Constructors

• .new(base_url : String, user_agent : String)

Instance Method Summary

• #allowed?(uri)

  Given a URI object, or a string representing one, determine whether this robots.txt would allow access to the path.

• #base_url : String
• #body : String
• #rules : Array(Tuple(String, Array(Rule)))
• #sitemaps : Array(String)
• #user_agent : String

Constructor Detail

Instance Method Detail