class Mint::SemanticTokenizer

• Mint::SemanticTokenizer
• Reference
• Object

Defined in:

semantic_tokenizer.cr

Constant Summary

TOKEN_MAP = {Ast::TypeVariable => TokenType::TypeParameter, Ast::Comment => TokenType::Comment, Ast::RegexpLiteral => TokenType::Regexp, Ast::NumberLiteral => TokenType::Number, Ast::Id => TokenType::Type}

This represents which token types are used for which node.

TOKEN_TYPES = TokenType.names.map!(&.camelcase(lower: true))

Class Method Summary

• .highlight(path : String, html : Bool = false)
• .tokenize(ast : Ast)

Instance Method Summary

• #add(from : Int64, to : Int64, type : TokenType)
• #add(node : Ast::Node, type : TokenType)
• #cache : Set(Mint::Ast::Node)

  We keep a cache of all tokenized nodes to avoid duplications

• #tokenize(ast : Ast)
• #tokenize(nodes : Array(Ast::Node))
• #tokenize(node : Ast::Variable)
• #tokenize(node : Ast::CssDefinition)
• #tokenize(node : Ast::HtmlElement)
• #tokenize(node : Ast::StringLiteral | Ast::HereDocument)
• #tokenize(node : Ast::HtmlComponent)
• #tokenize(node : Ast::Node | Nil)
• #tokens : Array(Mint::SemanticTokenizer::Token)

  This is where the resulting tokens are stored.

Class Method Detail

Instance Method Detail