class PythonLexer

• PythonLexer
• MiniLexer
• Reference
• Object

Defined in:

Constant Summary

IDENTIFIER = /^[a-zA-Z_]\w*/

Python Keywords Regular Expressions for Tokens

KEYWORDS = {"False" => :FALSE, "await" => :AWAIT, "else" => :ELSE, "import" => :IMPORT, "pass" => :PASS, "None" => :NONE, "break" => :BREAK, "except" => :EXCEPT, "in" => :IN, "raise" => :RAISE, "True" => :TRUE, "class" => :CLASS, "finally" => :FINALLY, "is" => :IS, "return" => :RETURN, "and" => :AND, "continue" => :CONTINUE, "for" => :FOR, "lambda" => :LAMBDA, "try" => :TRY, "as" => :AS, "def" => :DEF, "from" => :FROM, "nonlocal" => :NONLOCAL, "while" => :WHILE, "assert" => :ASSERT, "del" => :DEL, "global" => :GLOBAL, "not" => :NOT, "with" => :WITH, "async" => :ASYNC, "elif" => :ELIF, "if" => :IF, "or" => :OR, "yield" => :YIELD}

https://docs.python.org/3.12/reference/lexical_analysis.html#keywords

OPERATORS = {'+' => :ADD, '-' => :SUB, '*' => :MULT, '/' => :DIV, '%' => :MOD, '=' => :ASSIGN, "==" => :EQUAL, "!=" => :NOTEQUAL, '>' => :RANGLE, '<' => :LANGLE, ">=" => :GE, "<=" => :LE, "&&" => :AND, "||" => :OR, '!' => :BANG, "++" => :INC, "--" => :DEC, "+=" => :ADD_ASSIGN, "-=" => :SUB_ASSIGN, "*=" => :MUL_ASSIGN, "/=" => :DIV_ASSIGN, "%=" => :MOD_ASSIGN, '&' => :BITAND, '|' => :BITOR, '^' => :CARET, '~' => :TILDE, "->" => :ARROW, "=>" => :DOUBLE_ARROW, "?:" => :ELVIS, "<<" => :LEFTSHIFT, ">>" => :RIGHTSHIFT, "**" => :DOUBLESTAR, "&=" => :AMPEREQUAL, "|=" => :VBAREQUAL, "^=" => :CIRCUMFLEXEQUAL, "<<=" => :LEFTSHIFTEQUAL, ">>=" => :RIGHTSHIFTEQUAL, "**=" => :DOUBLESTAREQUAL, "//" => :DOUBLESLASH, "//=" => :DOUBLESLASHEQUAL, "@" => :AT, "@=" => :ATEQUAL, "..." => :ELLIPSIS, ":=" => :COLONEQUAL, "!" => :EXCLAMATION}

https://docs.python.org/3.12/library/token.html#module-token

PUNCTUATION = {'.' => :DOT, ',' => :COMMA, '(' => :LPAREN, ')' => :RPAREN, '{' => :LCURL, '}' => :RCURL, '[' => :LSQUARE, ']' => :RSQUARE, ';' => :SEMI, ':' => :COLON, '?' => :QUESTION}

Token Definitions

Constructors

• .new

Instance Method Summary

• #tokenize_logic(input : String) : Array(Token)

Constructor Detail

Instance Method Detail