class App::Downloader::HTTPClient

Overview

HTML downloader implemented using Crystal's HTTP::Client.

Supports basic functions like HTTP Keep-Alive, redirections, configurable number of connections per domain, and configurable timeouts.

It is limited to HTTP protocol version which is supported by Crystal.

Defined in:

app/downloader/httpclient.cr

Constructors

Instance Method Summary

Instance methods inherited from class App::Downloader

run run, worker worker

Constructor methods inherited from class App::Downloader

new(processor : App::Processor, download_tasks : Channel(Tuple(Int32, String, URI)), parse_tasks : Channel(NamedTuple(idx: Int32, url: String, title: String, gt: Time::Span, response: HTTP::Client::Response)), capacity) new

Constructor Detail

def self.new(processor, download_tasks, parse_tasks, capacity) #

[View source]

Instance Method Detail

def finalize #

[View source]
def worker #

[View source]