Initialize service with data directory

Required:

• data_dir: Directory path for database files
• provider: "openai" or "mock"
• api_key: Provider API key (not needed for mock)

Optional:

• store_text: Enable text storage in text.db (default true)
• attach: Hash of alias => path for databases to ATTACH
• model: Embedding model (default depends on provider)
• dimensions: Vector dimensions (auto-detected from model)
• max_tokens: Provider token limit (auto-detected)
• chunking_max_tokens: Max tokens per chunk (default 2000)

Example with ATTACH for unified queries:

memo = Memo::Service.new(
  data_dir: "/var/data/memo",
  attach: {"main" => "data.db"},
  provider: "openai",
  api_key: key
)
# Now can use sql_where: "c.source_id IN (SELECT id FROM main.artifact ...)"

Initialize service with existing database connection

Use this when caller manages the connection lifecycle. Caller is responsible for closing the connection.

Clear completed items from the queue

Removes successfully processed items (status = 0). Returns number of items removed.

Clear all items from the queue

Removes all items regardless of status. Returns number of items removed.

Close database connection

Should be called when done with service to free resources. Safe to call multiple times.

Note: If service was initialized with an existing db connection, close is a no-op (caller owns the connection).

Delete all chunks for a source

Removes all chunks with the given source_id (and optionally source_type). Orphaned embeddings (not referenced by any chunk) are also cleaned up.

Returns number of chunks deleted.

source_type: Optional filter to only delete chunks with matching source_type. If nil, deletes all chunks with the given source_id regardless of type.

TODO Consider adding delete_batch(source_ids : Array(Int64)) if bulk deletion becomes a common use case. Unlike index_batch, there's no API call savings, but it could reduce transaction overhead for large deletions.

Enqueue a document for later embedding

Adds the document to the embed_queue table without embedding it. Use process_queue to embed queued items.

If the source is already in the queue, the text is updated.

Enqueue a document (Document overload)

Enqueue multiple documents for later embedding

More efficient than calling enqueue() multiple times.

Index a document

Enqueues the document and processes it immediately with retry support. Returns number of chunks successfully stored.

Index a document (Document overload)

Convenience method that accepts a Document struct.

Index multiple documents in a batch

Enqueues all documents and processes them with retry support. More efficient than calling index() multiple times.

Returns total number of documents successfully processed.

Mark chunks as read (increment read_count)

Process queued items

Embeds pending items from the queue using the service's batch_size. Returns number of items successfully processed.

Failed items have their status set to the error code and can be retried up to max_retries times.

Process queued items asynchronously

Spawns a fiber to process the queue and returns immediately. Use queue_stats to check progress.

Process a specific queued item

Used by index() for immediate processing with retry support. Returns number of chunks stored.

Get queue statistics

Returns counts of pending and failed items.

Re-index all content of a given source type

Deletes existing embeddings and queues text for re-embedding. Requires text storage to be enabled.

Returns number of items queued for re-indexing.

Re-index all content of a given source type using a block to fetch text

Use this when text storage is disabled. The block receives each source_id and should return the text to embed.

Returns number of items queued for re-indexing.

Example:

memo.reindex("article") do |source_id|
  app.get_article_text(source_id)
end
memo.process_queue

Search for semantically similar chunks

Automatically generates query embedding and searches.

Returns array of search results ranked by similarity.

like: LIKE pattern(s) to filter by text content. Single string or array of strings for AND filtering. Example: like: "%cats%" or like: ["%cats%", "%dogs%"] Only works when text storage is enabled.

match: FTS5 full-text search query. Supports AND, OR, NOT, prefix*, "phrases". Example: match: "cats OR dogs", match: "quick brown*" Only works when text storage is enabled.

sql_where: Raw SQL fragment for filtering chunks. Used with ATTACH to filter by external database tables. Example: "c.source_id IN (SELECT id FROM main.artifact WHERE kind = 'goal')"

include_text: If true, includes text content in search results. Only works when text storage is enabled.

Get statistics about indexed content

Returns counts of embeddings, chunks, and unique sources.

Track whether text storage is enabled