Iterates over each token in the HTML input without building a parse tree.

This is the lightest-weight streaming option — it tokenizes the HTML and yields each Token as it's produced. No tree is built, no memory is accumulated beyond the tokenizer's internal buffer. Runs in constant memory regardless of input size.

Tokens are yielded in document order: start tags, text, end tags, comments, doctypes. Note that without tree construction, the token stream reflects the raw markup — it does not include the implicit tags or tree corrections that the full parser would apply.

Example

# Extract all text content
HTML5.each_token(io) do |token|
  print token.data if token.type.text?
end

# Find all image sources
HTML5.each_token(html_string) do |token|
  if token.type.start_tag? && token.data == "img"
    token.attr.each do |a|
      puts a.val if a.key == "src"
    end
  end
end

Iterates over each token in the HTML string without building a parse tree.

parse returns the parse tree for the HTML from the given io into an HTML5::Node.

It implements the HTML5 parsing algorithm https://html.spec.whatwg.org/multipage/syntax.html#tree-construction, which is very complicated. The resultant tree can contain implicitly created nodes that have no explicit listed in passed io's data, and nodes' parents can differ from the nesting implied by a naive processing of start and end s. Conversely, explicit s in passed io's data can be silently dropped, with no corresponding node in the resulting tree.

The input is assumed to be UTF-8 encoded.

parse returns the parse tree for the HTML from the given html string into an HTML5::Node.

parse_fragment parses a fragment of HTML5 and returns the nodes that were found. If the fragment is the InnerHTML for an existing element, pass that element in context.

It has the same intricacies as HTML5.parse.

reparents all of src's child nodes to dst

Parses HTML from an IO and emits SAX-style streaming events to the given handler.

The parser builds the full DOM tree internally (required for correct HTML5 parsing), but the handler receives events incrementally as nodes are constructed. This is useful for processing large documents where you want to react to elements as they appear without waiting for the entire document to be parsed.

Returns the complete document Node tree (same as HTML5.parse).

Example

class LinkExtractor
  include HTML5::StreamingHandler
  getter links = [] of String

  def on_element_open(tag : String, attrs : Array(HTML5::Attribute), namespace : String)
    if tag == "a"
      attrs.each do |attr|
        links << attr.val if attr.key == "href"
      end
    end
  end
end

extractor = LinkExtractor.new
doc = HTML5.stream(io, extractor)
puts extractor.links

Parses HTML from a String and emits SAX-style streaming events to the given handler.

Returns an Iterator over the tokens in the HTML input.

Example

HTML5.token_iterator(io).each do |token|
  puts token.data if token.type.start_tag?
end

Returns an Iterator over the tokens in the HTML string.

escape_string escapes special characters like "<" to become "<". It escapes only five such characters: <, >, &, ' and ". unescape_string(escape_string(s)) == s always holds, but the converse isn't always true.

unescape_string unescapes entities like "<" to become "<". It unescapes a larger range of entities than escape_string escapes. For example, "á" unescapes to "á", as does "á" and "&xE1;". unescape_string(escape_string(s)) == s always holds, but the converse isn't always true.