abstract class App::Downloader

Overview

Base class for all download clients.

URLs do not necessarily have to be retrieved from the web using a HTTP client. Different types of downloaders could load URLs from local cache for reparsing, or they could download page data via service-specific APIs (e.g. Wikipedia API).

Direct Known Subclasses

Defined in:

app/downloader.cr

Constructors

Instance Method Summary

Constructor Detail

def self.new(processor : App::Processor, download_tasks : Channel(Tuple(Int32, String, URI)), parse_tasks : Channel(NamedTuple(idx: Int32, url: String, title: String, gt: Time::Span, response: HTTP::Client::Response)), capacity) #

[View source]

Instance Method Detail

def run #

Implements download worker. Usually spawned in dedicated Fibers.


[View source]
abstract def worker #

[View source]