abstract class Cadmium::Tokenizer::Base

• Cadmium::Tokenizer::Base
• Reference
• Object

Included Modules

• Cadmium::Tokenizer::Diacritics
• Cadmium::Tokenizer::StopWords

Direct Known Subclasses

• Cadmium::Tokenizer::Aggressive
• Cadmium::Tokenizer::Case
• Cadmium::Tokenizer::Pragmatic
• Cadmium::Tokenizer::Regex
• Cadmium::Tokenizer::Sentence
• Cadmium::Tokenizer::TreebankWord

Defined in:

Instance Method Summary

• #tokenize(string : String) : Array(String)
• #trim(arr)

Instance Method Detail