Open Sidebar
Close Sidebar
CrystalDoc.info
Source code
Shards.info
merlin
main
Merlin
Context
Error
ASTFault
BadInput
Severe
SyntaxFault
UnexpectedCharacter
MatchedToken
Parser
ParserBuilder
ParserValidator
Position
Token
Tokenizer
Util
Generated using Crystal 1.14.0
2024-12-13 08:47:32 UTC
module
Merlin::
Tokenizer(IdentT)
Direct including types
Merlin::Parser(IdentT, NodeT)
Defined in:
parser/tokenizer.cr
Constant Summary
Log
=
::
Log
.
for
(
"lang.tokenizer"
)
Instance Method Summary
#tokenize
(str : String, filename : String | Nil =
nil
) : Array(MatchedToken(IdentT))
Instance Method Detail
def
tokenize
(str : String, filename : String | Nil =
nil
) : Array(
MatchedToken
(IdentT))
#
[
View source
]