JSON Lines

Extensions to the Crystal standard library for processing JSON lines format.

Installation

  1. Add the dependency to your shard.yml:

    dependencies:
  jsonl:
    gitlab: arctic-fox/jsonl

  2. Run shards install

Usage

Require jsonl to include all functionality.

The .from_jsonl class method is added to types that include JSON::Serializable. This method takes a string or IO object as an argument and processes multiple lines of JSON objects of the same type. That is, all processed lines are expected to be the same type of object. A block can be provided that yields each object as it is read. If no block is provided, all lines are read and an array of objects is returned.

The #to_jsonl method is added to Enumerable. This method iterates through each object in the collection and writes their JSON format to individual lines. It accepts an IO object to write to. If none is provided, it will build a string and return it.

require "jsonl"

class LogEntry
  include JSON::Serializable

  getter time : Time
  getter level : String
  getter message : String

  def initialize(@time, @level, @message)
  end

  def to_s(io : IO) : Nil
    io << time << ' ' << level << " - " << message
  end
end

entries = [
  LogEntry.new(Time.utc, "INFO", "Message 1"),
  LogEntry.new(Time.utc, "INFO", "Message 2"),
  LogEntry.new(Time.utc, "INFO", "Message 3")
]

# Write JSON lines to a file.
File.open("log.jsonl", "a") do |io|
  entries.to_jsonl(io)
end

# Stream JSON log lines from a file.
File.open("log.jsonl") do |io|
  LogEntry.from_jsonl(io) do |entry|
    puts entry
  end
end

An Iterator can be used to lazily process JSON lines. The .from_jsonl class method creates an iterator that processes lines instead array entries.

entries = Iterator(LogEntry).from_jsonl(io)
puts entries.map(&.message).skip(5).to_a

Contributing

  1. Fork it (https://gitlab.com/arctic-fox/jsonl/-/forks/new)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors