Open Sidebar
Close Sidebar
CrystalDoc.info
Source code
Shards.info
cadmium_tokenizer
master
Cadmium
StringExtension
Syllable
Guess
Tokenizer
Aggressive
Base
Case
Diacritics
Paragraph
Pragmatic
Languages
Bulgarian
Common
Czech
Deutsch
English
French
Portuguese
Spanish
MentionsOptions
NumbersOptions
PunctuationOptions
Regex
Sentence
StopWords
TreebankWord
VisibleChar
Whitespace
Word
WordPunctuation
Built with Crystal 1.9.2
2023-10-07 10:26:13 UTC
module
Cadmium::StringExtension
Defined in:
cadmium/tokenizer.cr
Instance Method Summary
#tokenize
(tokenizer =
Cadmium
::
Tokenizer
::
WordPunctuation
, *args, **kwargs)
Instance Method Detail
def
tokenize
(tokenizer =
Cadmium
::
Tokenizer
::
WordPunctuation
, *args, **kwargs)
#
[
View source
]