class Llamero::BaseModel

• Llamero::BaseModel
• Reference
• Object

Overview

The primary client for interacting directly with models that are available on the local computer.

This class allows the app to switch out specific fune-tuning aspects that improve the models accuracy for specific tasks. This includes:

• adding custom grammars for customizing response formats
• adding or switching out LoRA's
• managing the model's context window (token length)

Logs are always disabled with --log-disable when running from within this app

The chat_template_* properties are used to wrap the system and user prompts. This is used to generate the prompts for the LLM. Chat models are the most commonly used types of models and they are the most likely to be used with this class. You can get the symbols you need from the HF repo you got your model from, under the prompt-template section.

Included Modules

• Llamero::Tokenizer

Defined in:

Constructors

• .new(model_name : String, grammar_root_path : Path | Nil = nil, lora_root_path : Path | Nil = nil, model_root_path : Path | Nil = nil, repeat_penalty : Float32 | Nil = nil, top_k_sampling : Int32 | Nil = nil, threads : Int32 | Nil = nil, grammer_file : String | Nil = nil, context_size : Int32 | Nil = nil, temperature : Float32 | Nil = nil, keep : String | Nil = nil, n_predict : Int32 | Nil = nil, chat_template_system_prompt_opening_wrapper : String | Nil = nil, chat_template_system_prompt_closing_wrapper : String | Nil = nil, chat_template_user_prompt_opening_wrapper : String | Nil = nil, chat_template_user_prompt_closing_wrapper : String | Nil = nil, unique_token_at_the_end_of_the_prompt_to_split_on : String | Nil = nil, chat_template_end_of_generation_token : String | Nil = nil)

  Override any of the default values that are set in the child class

Instance Method Summary

• #chat(prompt_chain : Llamero::BasePrompt, grammar_class : Llamero::BaseGrammar, grammar_file : String | Path = Path.new, timeout : Time::Span = Time::Span.new(minutes: 2), max_retries : Int32 = 5, temperature : Float32 | Nil = nil, max_tokens : Int32 | Nil = nil, repeat_penalty : Float32 | Nil = nil, top_k_sampling : Int32 | Nil = nil, n_predict : Int32 | Nil = nil)

  This is the primary method for interacting with the LLM.

• #chat_template_end_of_generation_token : String
• #chat_template_end_of_generation_token=(chat_template_end_of_generation_token : String)
• #chat_template_system_prompt_closing_wrapper : String
• #chat_template_system_prompt_closing_wrapper=(chat_template_system_prompt_closing_wrapper : String)
• #chat_template_system_prompt_opening_wrapper : String

  The chat_template_* properties are used to wrap the system and user prompts.

• #chat_template_system_prompt_opening_wrapper=(chat_template_system_prompt_opening_wrapper : String)

  The chat_template_* properties are used to wrap the system and user prompts.

• #chat_template_user_prompt_closing_wrapper : String
• #chat_template_user_prompt_closing_wrapper=(chat_template_user_prompt_closing_wrapper : String)
• #chat_template_user_prompt_opening_wrapper : String
• #chat_template_user_prompt_opening_wrapper=(chat_template_user_prompt_opening_wrapper : String)
• #context_size : Int32

  Most Llama models use a 2048 context window for their training data.

• #context_size=(context_size : Int32)

  Most Llama models use a 2048 context window for their training data.

• #grammar_file : Path

  This is just the name of the grammer file, relative to the grammar_root_path.

• #grammar_file=(grammar_file : Path)

  This is just the name of the grammer file, relative to the grammar_root_path.

• #grammar_root_path : Path

  The directory where any grammar files will be located

• #grammar_root_path=(grammar_root_path : Path)

  The directory where any grammar files will be located

• #keep : String

  This can be set by using the Llamero::Tokenizer#tokenize method from the Llamero::Tokenizer module.

• #keep=(keep : String)

  This can be set by using the Llamero::Tokenizer#tokenize method from the Llamero::Tokenizer module.

• #lora_root_path : Path

  The directory where any lora filters will be located.

• #lora_root_path=(lora_root_path : Path)

  The directory where any lora filters will be located.

• #model_name : String

  This should be the full filename of the model, including the .gguf file extension.

• #model_name=(model_name : String)

  This should be the full filename of the model, including the .gguf file extension.

• #model_root_path : Path

  The directory where the model files will be located.

• #model_root_path=(model_root_path : Path)

  The directory where the model files will be located.

• #n_predict : Int32

  Setting this changes how many tokens are trying to be predicted at a time.

• #n_predict=(n_predict : Int32)

  Setting this changes how many tokens are trying to be predicted at a time.

• #quick_chat(prompt_chain : Array(NamedTuple(role: String, content: String)), grammar_class : Llamero::BaseGrammar | Nil = nil, grammar_file : String | Path = Path.new, temperature : Float32 | Nil = nil, max_tokens : Int32 | Nil = nil, repeat_penalty : Float32 | Nil = nil, top_k_sampling : Int32 | Nil = nil, n_predict : Int32 | Nil = nil, timeout : Time::Span = Time::Span.new(minutes: 5), max_retries : Int32 = 5) : String

  This is the main method for interacting with the LLM.

• #repeat_penalty : Float32

  Adjust up to punish repetitions more harshly, lower for more monotonous responses.

• #repeat_penalty=(repeat_penalty : Float32)

  Adjust up to punish repetitions more harshly, lower for more monotonous responses.

• #temperature : Float32

  Adjust up or down to play with creativity.

• #temperature=(temperature : Float32)

  Adjust up or down to play with creativity.

• #threads : Int32

  Number of threads.

• #threads=(threads : Int32)

  Number of threads.

• #tmp_grammar_file_path : Path

  Sometimes we'll need to use a temporary file to pass the grammar to the LLM.

• #tmp_grammar_file_path=(tmp_grammar_file_path : Path)

  Sometimes we'll need to use a temporary file to pass the grammar to the LLM.

• #top_k_sampling : Int32

  Adjust up to get more unique responses, adjust down to get more "probable" responses.

• #top_k_sampling=(top_k_sampling : Int32)

  Adjust up to get more unique responses, adjust down to get more "probable" responses.

• #unique_token_at_the_end_of_the_prompt_to_split_on : String

  This is a unique token at the end of the prompt that is used to split off and parse the response from the LLM.

• #unique_token_at_the_end_of_the_prompt_to_split_on=(unique_token_at_the_end_of_the_prompt_to_split_on : String)

  This is a unique token at the end of the prompt that is used to split off and parse the response from the LLM.

Constructor Detail

Instance Method Detail