class Crodoc::Tokenizer
- Crodoc::Tokenizer
- Reference
- Object
Overview
Represents a Crystal syntaxt tokenizer based on Crystal::Lexer.
Example:
source = Crodoc::Source.new(code, path)
tokenizer = Crodoc::Tokenizer.new(source)
tokenizer.run do |token|
puts token
endDefined in:
crodoc/tokenizer.crConstructors
-
.new(source : Source)
Instantiates a
Tokenizerusingsource. -
.new(lexer : Crystal::Lexer)
Instantiates a
Tokenizerusing aCrystal::Lexer.
Instance Method Summary
-
#run(&block : Crystal::Token -> )
Runs the tokenizer and yields each token as a block argument.
Constructor Detail
Instantiates a Tokenizer using source.
Example:
source = Crodoc::Source.new(code, path)
Crodoc::Tokenizer.new(source)
def self.new(lexer : Crystal::Lexer)
#
Instantiates a Tokenizer using a Crystal::Lexer.
Example:
lexer = Crystal::Lexer.new(code)
Crodoc::Tokenizer.new(lexer)Instance Method Detail
def run(&block : Crystal::Token -> )
#
Runs the tokenizer and yields each token as a block argument.
Example:
Crodoc::Tokenizer.new(source).run do |token|
puts token
end