class CLTK::Lexer

Overview

The Lexer class may be sub-classed to produce new lexers. These lexers have a lot of features, and are described in the main documentation.

Direct Known Subclasses

Defined in:

cltk/lexer.cr
cltk/lexer/environment.cr
cltk/lexer/rule.cr

Class Method Summary

Instance Method Summary

Macro Summary

Instance methods inherited from class Object

in?(collection : Array | Set) in?

Class Method Detail

def self.lex(string, file_name = nil, env = @@env.new(@@start_state)) #

Lex string, using env as the environment. This method will return the array of tokens generated by the lexer with a token of type EOS (End of Stream) appended to the end.

@param [String] string String to be lexed. @param [String] file_name File name used for recording token positions. @param [Environment] env Lexing environment.

@return [Array]


[View source]
def self.lex_file(file_name, env = @@env.new(@start_state)) #

A wrapper function that calls {Lexer.lex} on the contents of a file.

@param [String] file_name File to be lexed. @param [Environment] env Lexing environment.

@return [Array]


[View source]
def self.match_first #

Used to tell a lexer to use the first match found instead of the longest match found.

@return [void]


[View source]
def self.rule(pattern, state = :default, flags = [] of String) #

[View source]
def self.start(state) #

Changes the starting state of the lexer.

@param [Symbol] state Starting state for this lexer.

@return [void]


[View source]
def self.start_state #

[View source]

Instance Method Detail

def env #

@return [Environment] Environment used by an instantiated lexer.


[View source]
def start_state #

@return [Symbol] State in which the lexer starts.


[View source]

Macro Detail

macro rule(pattern, state = :default, flags = [] of Symbol, &action) #

[View source]