module MediaFilter

Defined in:

utils/media_filter.cr

Constant Summary

MAX_FILE_SIZE = MediaFilter.env_size("NOIR_MAX_FILE_SIZE", (10 * 1024) * 1024)

Maximum file size for processing (default 10MB). Specification documents have their own budget — see MAX_SPEC_FILE_SIZE below. Can be overridden with the environment variable NOIR_MAX_FILE_SIZE. Supported formats for NOIR_MAX_FILE_SIZE:

• Plain bytes integer (e.g., 5242880)
• Human-readable with unit suffix (K, KB, M, MB, G, GB) e.g., 5MB, 500K, 1G Invalid / unparsable values fall back to the default (10MB).

MAX_SPEC_FILE_SIZE = {MediaFilter.env_size("NOIR_MAX_SPEC_FILE_SIZE", (64 * 1024) * 1024), MAX_FILE_SIZE}.max

Size budget for specification documents, overridable with NOIR_MAX_SPEC_FILE_SIZE (same value syntax as NOIR_MAX_FILE_SIZE).

MAX_FILE_SIZE exists to keep images, archives and compiled blobs out of the scan, and 10MB is a sensible ceiling for that. Applying it to API descriptions was collateral damage: those are plain text, they are the highest-value input noir has, and generated ones routinely pass 10MB. NetBox ships a 12.35MB contrib/openapi.json describing 308 paths, and every one of them was dropped — not because the document was unreadable, but because a filter aimed at image files measured it.

The budget still has to be bounded, because the document is read whole, held in the content cache, and parsed into a JSON/YAML tree that peaks at roughly 5x its size in memory. Measured with a release build: NetBox's 12.35MB document costs 0.18s and 81MB peak RSS for the whole scan; a 76MB variant of it (same document, component schemas multiplied, forced through with the override) costs 0.90s and 392MB. 64MB is the ceiling because it keeps one document under a second and a few hundred MB while leaving 5x headroom over the largest generated spec observed in the wild. Past that a .json is a data dump, not an API description.

Floored at MAX_FILE_SIZE so that raising the general cap past 64MB lifts specification documents with it rather than capping them below everything else. Lowering the general cap does not lower this budget; use NOIR_MAX_SPEC_FILE_SIZE for that.

MEDIA_EXTENSION_SET = MEDIA_EXTENSIONS.to_set

O(1) lookup set materialized from MEDIA_EXTENSIONS. Used on the hot path — every file in the project is checked once.

MEDIA_EXTENSIONS = [".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp", ".tiff", ".svg", ".ico", ".psd", ".raw", ".cr2", ".nef", ".orf", ".sr2", ".heic", ".heif", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".m4v", ".mpg", ".mpeg", ".3gp", ".vob", ".rm", ".rmvb", ".asf", ".ogv", ".mp3", ".wav", ".flac", ".aac", ".ogg", ".wma", ".m4a", ".ape", ".ac3", ".dts", ".opus", ".amr", ".au", ".ra", ".aiff", ".zip", ".rar", ".7z", ".tar", ".gz", ".bz2", ".xz", ".dmg", ".iso", ".pdf", ".doc", ".docx", ".ppt", ".pptx", ".xls", ".xlsx", ".exe", ".dll", ".so", ".dylib", ".bin", ".app", ".deb", ".rpm", ".db", ".sqlite", ".sqlite3", ".mdb", ".accdb", ".ttf", ".otf", ".woff", ".woff2", ".eot"]

Common media file extensions that should be skipped

SPEC_DOCUMENT_EXTENSION_SET = SPEC_DOCUMENT_EXTENSIONS.to_set

SPEC_DOCUMENT_EXTENSIONS = [".json", ".yaml", ".yml", ".xml", ".toml", ".tf", ".har", ".raml", ".wsdl", ".smithy", ".tsp", ".proto", ".bru", ".graphql", ".gql", ".graphqls", ".http", ".rest"]

Extensions that carry an API description or a structured document noir reads as one. These get MAX_SPEC_FILE_SIZE instead of the media cap.

The list mirrors the file gates of src/detector/detectors/specification/* (grep -h "detector_for" src/detector/detectors/specification/*.cr) with two deliberate omissions. Binary container formats get no relief: mitmproxy's .flow dumps are NUL-bearing blobs the walk drops on content regardless of size. Neither do the source extensions the AWS CDK detector reads (.ts, .js, .py, …) — a 12MB .js or .py is a bundled or generated artifact, where a 12MB .json is usually exactly the document the scan is looking for.

The list is by extension rather than by content because a document's own marker ("openapi", "swagger") can sit anywhere in it, and a prefix sniff that misses one is a silent loss again. The cost of being wrong the other way is small: a 32MB package-lock.json matches no specification marker, so the walk reads it and drops every spec detector without parsing — 0.08s more than skipping it outright.

Class Method Summary

• .binary_content_signature?(file_path : String) : Bool

  Cheap binary-content sniff.

• .env_size(name : String, default : Int32) : Int32

  Byte budget read from name, falling back to default when the variable is unset, unparsable or non-positive.

• .file_too_large?(file_path : String, max_size : Int32 | Nil = nil) : Bool

  Check if a file is too large to process.

• .max_size_for(file_path : String) : Int32

  The size budget that applies to file_path.

• .media_file?(file_path : String) : Bool

  Check if a file should be skipped based on extension

• .parse_size(str : String) : Int32

  Parse size strings like "10MB", "500K", "1G" or raw bytes ("1048576")

• .should_skip_file?(file_path : String, max_size : Int32 | Nil = nil, info : File::Info | Nil = nil, sniff_binary : Bool = true) : Bool

  Combined check - returns true if file should be skipped.

• .skip_check(file_path : String, max_size : Int32 | Nil = nil, info : File::Info | Nil = nil, sniff_binary : Bool = true) : String | Nil

  Decide whether a file should be skipped and, if so, return the human readable reason in a single pass — avoids re-stat'ing the file just to compose the log message.

• .skip_reason(file_path : String, max_size : Int32 | Nil = nil, info : File::Info | Nil = nil, sniff_binary : Bool = true) : String | Nil

  Get a human-readable reason why a file was skipped.

• .spec_document?(file_path : String) : Bool

  True when the path names a specification document (see SPEC_DOCUMENT_EXTENSIONS), which is read under the larger budget.

Class Method Detail