abstract class Cadmium::Tokenizer

• Cadmium::Tokenizer
• Reference
• Object

Direct Known Subclasses

• Cadmium::AggressiveTokenizer
• Cadmium::CaseTokenizer
• Cadmium::PragmaticTokenizer
• Cadmium::RegexTokenizer
• Cadmium::SentenceTokenizer
• Cadmium::TreebankWordTokenizer

Defined in:

cadmium/tokenizer/tokenizer.cr

Instance Method Summary

• #tokenize(string : String) : Array(String)
• #trim(arr)

Instance Method Detail