module Llamero::Tokenizer

Overview

A helpful module for tokenizing text using the Llamero tokenizer.

If using this module outside of the Llamero::BaseModel, then you must define a #model_root_path to the root of the model folder you are using.

Direct including types

• Llamero::BaseModel

Defined in:

tokenization/tokenizer.cr

Instance Method Summary

• #model_name : String
• #model_root_path : Path
• #tokenize(text_to_tokenize : IO | String) : Array(String)

  Counts all of the tokens in the given text, returning an array of the parsed tokens

Instance Method Detail