struct Hansa::Classifier

• Hansa::Classifier
• Struct
• Value
• Object

Included Modules

• JSON::Serializable

Defined in:

hansa.cr

Constructors

• .new(pull : JSON::PullParser)

Instance Method Summary

• #classify(content : String, candidates : Array(String) = [] of String)

  Scores the content over the candidate languages (go-enry v2 "hints" semantics): with no candidates every known language competes, with candidates only those do.

• #common_extract_and_replace(content : String, re : Regex) : Tuple(Array(String), String)
• #default_token_probability : Float64

  Probability for tokens not present in a language's token table, derived from the corpus like go-enry v2 does

• #extract_and_replace_operator(content : String) : Tuple(Array(String), String)
• #extract_and_replace_punctuation(content : String) : Tuple(Array(String), String)
• #extract_and_replace_regular(content : String) : Tuple(Array(String), String)
• #extract_and_replace_sgml(content : String) : Tuple(Array(String), String)
• #extract_and_replace_shebang(content : String) : Tuple(Array(String), String)
• #extract_remainders(content : String) : Tuple(Array(String), String)
• #get_sgml_attributes(sgml_tag : String) : Array(String)
• #known_languages : Array(String)

  Despite the name this only reports the 100 most common languages in the corpus, to avoid super unilely false positives for obscure languages

• #languages_log_probabilities : Hash(String, Float64)
• #languages_log_probabilities=(languages_log_probabilities : Hash(String, Float64))
• #skip_comments_and_literals(content : String) : Tuple(Array(String), String)
• #tokenize(content : String) : Array(String)
• #tokens_log_probabilities : Hash(String, Hash(String, Float64))
• #tokens_log_probabilities=(tokens_log_probabilities : Hash(String, Hash(String, Float64)))
• #tokens_log_probability(tokens : Array(String), language : String) : Float64
• #tokens_total : Float64
• #tokens_total=(tokens_total : Float64)

Constructor Detail

Instance Method Detail