JoobQ

JoobQ Logo

Crystal CI

Overview

Welcome to JoobQ – is a great solution for fast, efficient, and reliable asynchronous job queue scheduling! Whether you're building a small application or a large-scale system, JoobQ is designed to handle your job processing needs with ease and precision.

Why JoobQ?

JoobQ is more than just a job queue scheduler; it's a robust framework that ensures your jobs are managed and executed seamlessly. Here's why developers love JoobQ:

Key Features

Get Started!

Ready to dive in? Follow our Getting Started guide to set up JoobQ in your project and start processing jobs like a pro!

Table of Contents

Getting Started

This section will help you get started with JoobQ. Follow the instructions below to set up and run the project.

Prerequisites

Setting Up

  1. Clone the repository:

    git clone https://github.com/azutoolkit/joobq.git
cd joobq

  2. Install dependencies:

    shards install

  3. Start Redis with TimeSeries module:

    docker-compose up -d

Installation

Add the following to your shard.yml:

dependencies:
  joobq:
    github: azutoolkit/joobq

Then run:

shards install

Usage

require "joobq"

Environment Variables

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_POOL_SIZE=50
REDIS_PASS=somepass
REDIS_TIMEOUT=0.2

Configuration

JoobQ can be configured using the JoobQ.configure method. Here is an example configuration:

JoobQ.configure do
  queue "default", 10, EmailJob

  scheduler do
    cron("*/1 * * * *") { # Do Something }
    every 1.hour, EmailJob, email_address: "[email protected]"
  end
end

Configuration Properties

The JoobQ::Configure struct provides several properties that can be customized to configure the JoobQ system:

Example Configuration with All Properties

JoobQ.configure do |config|
  config.default_queue = "default"
  config.retries = 5
  config.expires = 2.days
  config.timeout = 5.seconds
  config.failed_ttl = 1.minute
  config.dead_letter_ttl = 14.days

  queue "default", 10, EmailJob
  queue "priority", 5, PriorityJob, throttle: { limit: 100, period: 1.minute }

  scheduler do
    cron("*/1 * * * *") { # Do Something }
    every 1.hour, EmailJob, email_address: "[email protected]"
  end
end

Defining Queues

Queues are of type Hash(String, Queue(T)) where the name of the key matches the name of the Queue.

Example:

JoobQ.configure do
  queue name: "single", workers: 10, job: Job1, throttle: { limit: 20, period: 1.minute }
  queue "example", 10, ExampleJob | FailJob

  # Scheduling Recurring Jobs
  scheduler do
    cron("*/1 * * * *") { # Do Something }
    cron("*/5 20-23 * * *") { # Do Something }
    every 1.hour, ExampleJob, x: 1
  end
end

Queue Throttling

The worker throttle mechanism in JoobQ works in conjunction with the Queue Throttle Limit property to manage the rate at which jobs are processed. Here's how it works:

Queue Throttle Limit Property

The Queue Throttle limit property sets a maximum number of jobs that can be processed within a specific time frame. This helps to control the load on the system and ensures that the job processing rate does not exceed a certain threshold.

How They Work Together

Example:

Here is an example of how you might configure the Queue Throttle limit property:

JoobQ.configure do
  queue "default", 10, EmailJob, throttle: { limit: 100, period: 60.seconds }
end

In this example, the throttle limit is set to 100 jobs per 60 seconds. This means that the worker will process up to 100 jobs every 60 seconds. If the limit is reached, the worker will wait until the next period to continue processing jobs.

Summary

The worker throttle mechanism, combined with the Queue Throttle limit property, ensures that job processing is controlled and efficient. By managing job availability, expiration, and processing rate, JoobQ provides a robust system for handling job queues without overwhelming the system.

Defining Jobs

To define Jobs, include the JoobQ::Job module, and implement the perform method.

struct EmailJob
  include JoobQ::Job

  # Name of the queue to be processed by
  @queue   = "default"
  # Number Of Retries for this job
  @retries = 0
  # Job Expiration
  @expires = 1.days.total_seconds.to_i

  # Initialize as normal with or without named tuple arguments
  def initialize(@email_address : String)
  end

  def perform
    # Logic to handle job execution
  end
end

Executing Job:

# Enqueue the job (Async)
EmailJob.enqueue(email_address: "[email protected]")

# Batch enqueue jobs
EmailJob.batch_enqueue([job1, job2, job3])

# Perform Immediately
EmailJob.new(email_address: "[email protected]").perform

# Delayed
EmailJob.delay(for: 1.hour, email_address: "[email protected]")
EmailJob.enqueue_at(time: 3.minutes.from_now, email_address: "[email protected]")

# Recurring at given interval
EmailJob.schedule(every: 1.second, email_address: "[email protected]")

Best Practices for Defining Jobs

When defining jobs in JoobQ, it's important to follow certain best practices to ensure reliability and maintainability. Here are some key recommendations:

Idempotency

Jobs must be idempotent. This means that running the same job multiple times should produce the same result. Idempotency is crucial for ensuring that jobs can be retried safely without causing unintended side effects. To achieve idempotency:

Simple Primitive Types for Arguments

Job arguments must be simple primitive types such as integers, strings, and booleans. This ensures that the job data can be easily serialized and deserialized, and reduces the risk of errors during job execution. Complex objects or data structures should be avoided as job arguments.

Number of Arguments

Keep the number of arguments for jobs to a minimum. Having too many arguments can make the job definition complex and harder to maintain. As a best practice:

By following these best practices, you can ensure that your jobs are reliable, maintainable, and easy to work with in the JoobQ framework.

JoobQ HTTP Server

The APIServer class provides a REST API to interact with the JoobQ job queue system. It listens for HTTP requests and handles job enqueuing, job registry retrieval, and queue metrics.

To start the API server.

APIServer.start

Rest API

JoobQ provides a REST API to interact with the job queue. Below are the available endpoints:

GET /joobq/jobs/registry

This endpoint returns all available registered jobs and their JSON schemas that can be enqueued via the REST API.

Request:

GET /joobq/jobs/registry HTTP/1.1
Host: localhost:8080

Response:

{
  "EmailJob": {
    "type": "object",
    "properties": {
      "email_address": {
        "type": "string"
      }
    },
    "required": ["email_address"]
  }
}

POST /joobq/jobs

This endpoint allows users to enqueue jobs.

Request:

POST /joobq/jobs HTTP/1.1
Host: localhost:8080
Content-Type: application/json
Content-Length: 175

{
    "jid": "a13324f4-bdd8-4cf5-b566-c0c9c312f68b",
    "queue": "queue:test",
    "retries": 3,
    "expires": {{timestamp_plus_30}},
    "status": "enqueued",
    // Job initialization arguments
    "x": 10
}

Response:

{
  "status": "Job enqueued",
  "queue": "default",
  "job_id": "some-unique-job-id"
}

GET /joob/metrics

This endpoint returns metrics about the queue

Rquest:

GET /joobq/metrics HTTP/1.1
Host: localhost:8080

Response:

[
  {
    "queue:test": {
      "total_workers": 5,
      "status": "Running",
      "metrics": {
        "enqueued": 394775,
        "completed": 171446,
        "retried": 1757,
        "dead": 0,
        "processing": 3,
        "running_workers": 5,
        "jobs_per_second": 24624.079804538018,
        "errors_per_second": 252.17920194481889,
        "enqueued_per_second": 56652.61565975511,
        "jobs_latency": "00:00:00.000040613",
        "elapsed_time": "00:00:06.970125250"
      }
    }
  }
]

Performance

JoobQ is designed for high performance and scalability. In our benchmarks, JoobQ has easily achieved processing rates of 35,000 jobs per second. This performance is made possible by leveraging Crystal's concurrency model and efficient job handling mechanisms.

Performance Comparison

To provide a clearer picture of JoobQ's performance, we have compared it with other popular job queue processing tools in various programming languages, including Sidekiq (Ruby), Celery (Python), Laravel Queue (PHP), and Quartz (Java). The results are summarized in the table below:

| Job Queue Tool | Language | Jobs per Second | | -------------- | -------- | --------------- | | JoobQ | Crystal | 35,000 | | Sidekiq | Ruby | 23,500 | | Celery | Python | 15,000 | | Laravel Queue | PHP | 10,000 | | Quartz | Java | 25,000 |

JoobQ Hardware benchmarks performed

Hardware Overview:
  Model Name:             MacBook Pro
  Model Identifier:       Mac15,10
  Model Number:           MRX53LL/A
  Chip:                   Apple M3 Max
  Total Number of Cores: 14 (10 performance and 4 efficiency)
  Memory:                 36 GB

As shown in the table, JoobQ outperforms many of the popular job queue processing tools, making it an excellent choice for high-throughput job processing needs.

Disclaimer

JoobQ out of the box provides great performance, achieving processing rates of up to 35,000 jobs per second in our benchmarks. However, it's important to note that with the right environment and settings, any job scheduler can be performant. Factors such as hardware specifications, network conditions, and job complexity can significantly impact the performance of job queue processing tools. Therefore, it's essential to optimize your environment and configurations to achieve the best possible performance for your specific use case.

Contributing

  1. Fork it (https://github.com/azutoolkit/joobq/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Testing

To run the tests, use the following command:

crystal spec

Deployment

To deploy JoobQ, ensure that you have a running Redis instance. You can use the provided docker-compose.yml to set up Redis.

docker-compose up -d

Roadmap

License

The MIT License (MIT). Please see License File for more information.

Acknowledgments

Elias J. Perez - creator and maintainer Crystal Language Redis