abstract class Tokenizer

• Tokenizer
• Reference
• Object

Direct Known Subclasses

• Tokenizer::Abstract
• Tokenizer::Delimiter
• Tokenizer::Length

Defined in:

tokenizer.cr

Constructors

• .new(token_size : Int32, indicator = nil)
• .new(*delimiter)

  simple tokenizer

• .new(&callback : IO::Memory -> Int32)

  abstract tokenizer

Instance Method Summary

• #buffer : IO::Memory
• #clear

Constructor Detail

Instance Method Detail