class Cadmium::VisibleCharTokenizer

Defined in:

cadmium/tokenizer/visible_char_tokenizer.cr

Constant Summary

REGEX_PATTERN = /\s+|(?<=[\P{Cc}])(?=[\P{Cc}])/

Constructors

Instance methods inherited from class Cadmium::RegexTokenizer

tokenize(string : String) : Array(String) tokenize

Constructor methods inherited from class Cadmium::RegexTokenizer

new(pattern : Regex, *, gaps = nil, discard_empty = nil) new

Instance methods inherited from class Cadmium::Tokenizer

tokenize(string : String) : Array(String) tokenize, trim(arr) trim

Constructor Detail

def self.new #

[View source]