class JoobQ::FailHandler(T)

• JoobQ::FailHandler(T)
• Reference
• Object

Overview

The FailHandler module in JoobQ is responsible for managing jobs that encounter errors during execution. This fail handler provides robust retry logic with exponential backoff, detailed logging, and dead letter queue management to ensure reliability and traceability of job failures.

Key Features

1. Enhanced Logging

The FailHandler includes more detailed logging capabilities:

• Logs error details when a job is dead, including job ID, queue name, start time, error message, and stack trace.
• Logs retry attempts with job ID and retry delay information.
• Logs when a job is moved to the dead letter queue after exhausting retry attempts.

2. Retry Logic with Exponential Backoff

The FailHandler will retry jobs up to a specified maximum number of retries (job.retries). The retry delay is calculated using an exponential backoff strategy to avoid overwhelming the system:

• Base delay starts at 2 seconds.
• Retry delay increases exponentially, with a maximum cap of 5 minutes to avoid excessively long delays.

3. Dead Letter Queue Handling

When a job exhausts its retry attempts, it is moved to a "dead letter" queue for further inspection and manual intervention:

• The mark_as_dead method is called to track jobs that could not be processed successfully.
• This ensures that problematic jobs are not lost and can be debugged later.

Usage

The FailHandler is automatically invoked when a job execution fails:

JoobQ::FailHandler.call(job, start_time, exception, queue)

The method takes four parameters:

• job: The job that failed.
• start_time: The time when the job started.
• exception: The exception that caused the failure.
• queue: The queue that the job belongs to.

Exponential Backoff Calculation

The backoff is calculated using the formula:

(2 ** retry_count) * base_delay

• Base Delay: The initial delay is set to 2 seconds (2000 milliseconds).
• Maximum Delay: Capped at 5 minutes (300,000 milliseconds) to prevent unreasonably long waits between retries.

Logging Examples

• Job Execution Failed:

  [FAIL_HANDLER] Job Execution Failed: job_id=123, queue=critical_jobs, start_time=2024-11-15T08:00:00Z,
  error_message=Some error occurred, stack_trace=...

• Retrying Job:

  [FAIL_HANDLER] Retrying Job: job_id=123, retry_delay=4000ms

• Job Moved to Dead Letter Queue:

  [FAIL_HANDLER] Job Moved to Dead Letter Queue: job_id=123

Configuration

• Maximum Retries: Defined by the job.retries property. Customize this value based on the reliability needs of your system.
• Base and Maximum Delay: Adjust the base_delay and delay values in the exponential_backoff method if you need different retry behavior.

Future Improvements

• Customizable Backoff Strategy: Allow developers to choose different backoff strategies (e.g., linear, exponential with jitter).
• Notification Mechanism: Add support for sending notifications (e.g., email, Slack) when jobs are moved to the dead letter queue.

Defined in:

joobq/fail_handler.cr

Constructors

• .new(queue : BaseQueue, metrics : Metrics)

Instance Method Summary

• #handle_failure(job : T, ex : Exception)

Constructor Detail

Instance Method Detail