class Cadmium::Tokenizer::VisibleChar
Defined in:
cadmium/tokenizer/visible_char.crConstant Summary
-
REGEX_PATTERN =
/\s+|(?<=[\P{Cc}])(?=[\P{Cc}])/
/\s+|(?<=[\P{Cc}])(?=[\P{Cc}])/
Cadmium::Tokenizer::Regex
Cadmium::Tokenizer::Regex
Cadmium::Tokenizer::Base
Cadmium::Tokenizer::Diacritics
Cadmium::Tokenizer::StopWords