class SHAInet::BPETokenizer

• SHAInet::BPETokenizer
• Reference
• Object

Overview

Simple byte-pair encoding tokenizer. It can train a vocabulary from text and encode/decode using the learned merges.

Defined in:

shainet/text/bpe_tokenizer.cr

Constructors

• .new

Instance Method Summary

• #decode(ids : Array(Int32)) : String

  Decode an array of token IDs back into a string.

• #encode(text : String) : Array(Int32)

  Encode a string into token IDs.

• #inv_vocab : Array(String)
• #merges : Array(Tuple(String, String))
• #train(text : String, vocab_size : Int32)

  Train the tokenizer vocabulary from the given text using the byte-pair encoding algorithm.

• #vocab : Hash(String, Int32)

Constructor Detail

Instance Method Detail