class Linguist::NGramsTokenizer

• Linguist::NGramsTokenizer
• Cadmium::Tokenizer::Base
• Reference
• Object

Defined in:

linguist/tokenizer.cr

Constant Summary

SYMBOL_RANGES = [0..64, 91..96]

Constructors

• .new(max_size : Int32 = 3)

Instance Method Summary

• #downcase?(string)
• #max_size : Int32
• #max_size=(max_size : Int32)
• #symbol?(char)
• #tokenize(string : String) : Array(String)

Constructor Detail

Instance Method Detail