class Tartrazine::Tokenizer

• Tartrazine::Tokenizer
• Tartrazine::BaseTokenizer
• Reference
• Object

Included Modules

• Iterator({type: String, value: String})

Defined in:

lexer.cr

Constructors

• .new(lexer : BaseLexer, text : String, secondary = false)

Instance Method Summary

• #lexer : BaseLexer
• #lexer=(lexer : BaseLexer)
• #next : Iterator::Stop | Token

  Returns the next element in this iterator, or Iterator::Stop::INSTANCE if there are no more elements.

• #pos : Int32
• #pos=(pos : Int32)
• #split_tokens(tokens : Array(Token)) : Array(Token)

  If a token contains a newline, split it into two tokens

• #state_stack : Array(String)
• #state_stack=(state_stack : Array(String))
• #text : Bytes
• #text=(text : Bytes)

Constructor Detail

Instance Method Detail