class Tartrazine::DelegatingTokenizer

• Tartrazine::DelegatingTokenizer
• Tartrazine::BaseTokenizer
• Reference
• Object

Overview

This Tokenizer works with a DelegatingLexer. It first tokenizes using the language lexer, and "Other" tokens are tokenized using the root lexer.

Included Modules

• Iterator({type: String, value: String})

Defined in:

Constructors

• .new(lexer : DelegatingLexer, text : String, secondary = false)

Instance Method Summary

• #next : Iterator::Stop | Token

  Returns the next element in this iterator, or Iterator::Stop::INSTANCE if there are no more elements.

Constructor Detail

Instance Method Detail