class Cadmium::Tokenizer::Pragmatic::Languages::Common

• Cadmium::Tokenizer::Pragmatic::Languages::Common
• Reference
• Object

Direct Known Subclasses

• Cadmium::Tokenizer::Pragmatic::Languages::Bulgarian
• Cadmium::Tokenizer::Pragmatic::Languages::Czech
• Cadmium::Tokenizer::Pragmatic::Languages::Deutsch
• Cadmium::Tokenizer::Pragmatic::Languages::English
• Cadmium::Tokenizer::Pragmatic::Languages::French
• Cadmium::Tokenizer::Pragmatic::Languages::Portuguese
• Cadmium::Tokenizer::Pragmatic::Languages::Spanish

Defined in:

cadmium/tokenizer/pragmatic/languages/common.cr

Constant Summary

ABBREVIATIONS = Set(String).new

ALNUM_QUOTE = /(\w|\D)'(?!')(?=\W|$)/

Single quotes handling

CONTRACTIONS = {} of String => String

PUNCTUATION_MAP = {"。" => "♳", "．" => "♴", "." => "♵", "！" => "♶", "!" => "♷", "?" => "♸", "？" => "♹", "、" => "♺", "¡" => "⚀", "¿" => "⚁", "„" => "⚂", "“" => "⚃", "[" => "⚄", "]" => "⚅", "\"" => "☇", "#" => "☈", "$" => "☉", "%" => "☊", "&" => "☋", "(" => "☌", ")" => "☍", "*" => "☠", "+" => "☢", "," => "☣", ":" => "☤", ";" => "☥", "<" => "☦", "=" => "☧", ">" => "☀", "@" => "☁", "^" => "☂", "_" => "☃", "`" => "☄", "'" => "☮", "{" => "♔", "|" => "♕", "}" => "♖", "~" => "♗", "-" => "♘", "«" => "♙", "»" => "♚", "”" => "⚘", "‘" => "⚭"}

QUOTE_NOT_TWAS1 = /(\W|^)'(?!twas)/i

QUOTE_NOT_TWAS2 = /(\W|^)‘(?!twas)/i

QUOTE_WORD = /(\W|^)'(?=\w)/

Constructors

• .new

Class Method Summary

• .abbreviations
• .contractions
• .handle_single_quotes(text)

  This 'special treatment' is actually relevant for many other tests.

• .punctuation_map

Constructor Detail

Class Method Detail