Parser
- Checkpoint
- Node
RiSC16
- Assembler
  - AST
    - Immediate
    - Instruction
    - Node
    - Parameter
    - Register
    - Section
    - Statement
    - Text
    - Unit
  - Parser
- CLI
- CustomWindow
- Dce
- Debugger
- Instruction
- ISA
- Lib
- Linker
- Object
  - Section
    - Options
    - Reference
      - Kind
    - Symbol
- Scroll
- Spec
  - Section
  - Segment
    - Default
    - IO
    - Kind
    - Ram
    - Rom
- Table
- UInt16
- VM
  - Addressable
    - Default
    - IO
    - Ram
    - Rom
  - BusError
- Window
Stacklang
- AST
  - Access
  - Binary
  - Call
  - Cast
  - Custom
  - Expression
  - Function
    - Parameter
  - Identifier
  - If
  - Literal
  - Operator
  - Pointer
  - Requirement
  - Return
  - Sizeof
  - Statement
  - Struct
    - Field
  - Table
  - Type
  - Unary
  - Unit
  - Variable
  - While
  - Word
- Compiler
- Exception
- Function
  - Kind
  - Memory
  - Prototype
    - Parameter
  - Registers
  - Variable
- InternalError
- Parser
  - Arity
  - Associativity
  - Category
  - Component
  - Exception
  - Token
- Tokenizer
  - Token
- Type
  - Any
  - Pointer
  - Struct
    - Field
  - Table
  - Word
- Unit
  - Global
UInt16

Generated using Crystal 1.10.1
2024-02-25 07:25:22 UTC

module Stacklang::Tokenizer

Overview

Split a stream of text into tokens

Defined in:

stacklang/tokenizer.cr

Class Method Summary

.tokenize(filename : String) : Array(Token)
.tokenize(io : IO, source = nil) : Array(Token)

Class Method Detail

def self.tokenize(filename : String) : Array(Token) #

[View source]

def self.tokenize(io : IO, source = nil) : Array(Token) #

[View source]