module LinkFollowability

Overview

Classifies links the way search engines read them: every rel token on every anchor, plus the page-level <meta name="robots"> and X-Robots-Tag header.

Google treats nofollow, ugc and sponsored as hints that a link should not be followed, and more than one of them can sit on the same anchor. noopener and noreferrer are browser-security tokens and do not change followability.

Defined in:

link_followability.cr

Constant Summary

NOT_FOLLOWED_TOKENS = {"nofollow", "ugc", "sponsored"}

rel tokens that mark a link as not followed.

VERSION = "0.1.0"

Class Method Summary

Class Method Detail

def self.anchors(html : String) : Array(Anchor) #

Extracts every <a> start tag with an href, in document order. Handles double-quoted, single-quoted and unquoted attribute values. Anchors without an href are skipped. This is a tokenizer for start tags, not a full HTML parser: comments and script bodies are removed first so that markup inside them is not counted.


[View source]
def self.followed?(anchor : Anchor, page : PageDirectives) : Bool #

Whether a given anchor is followed once page-level directives are applied.


[View source]
def self.merge(*sets : PageDirectives) : PageDirectives #

Merges several directive sets; any noindex/nofollow wins.


[View source]
def self.meta_robots(html : String) : String | Nil #

Returns the content of the first <meta name="robots"> tag, or nil.


[View source]
def self.parse_directives(value : String | Nil) : PageDirectives #

Parses a robots directive list such as "noindex, nofollow" or "none". none is equivalent to noindex, nofollow; all sets neither.


[View source]
def self.parse_x_robots_tag(value : String | Nil, user_agent : String = "googlebot") : PageDirectives #

Parses one X-Robots-Tag header value. A value may be scoped to a crawler ("googlebot: noindex"); scoped values only apply when user_agent matches (case-insensitive). Unscoped values apply to every crawler. Directives that carry their own value after a colon (e.g. "unavailable_after: ...") are not mistaken for a crawler scope.


[View source]
def self.rel_tokens(rel : String | Nil) : Array(String) #

Splits a rel attribute into lower-case tokens. Tokens are separated by ASCII whitespace (HTML spec); duplicates are kept once, order is preserved.


[View source]