class Analyzer

Included Modules

Direct Known Subclasses

Defined in:

models/analyzer.cr

Constant Summary

DEFAULT_CHANNEL_CAPACITY = 128
DEFAULT_CONTENT_CHANNEL_CAPACITY = 16

Detector reader → worker content channel. Each buffered entry holds a full file's content (up to MAX_FILE_SIZE), so this bounds worst-case unreclaimable memory as capacity * MAX_FILE_SIZE.

Was raised 16 -> 64 under the theory that the single reader fiber was blocking on send and stalling disk I/O. Re-measured (#2362) on a heavy-load corpus (multi-detector + passive-scan + -T + --ai-context, ~80MB of large files): 16 and 64 perform identically (~14.7s avg either way over 9 interleaved runs). There is no preview_mt build, so reader and workers are cooperative fibers on one thread with no blocking-I/O yield point to recover — a deeper buffer just lets the reader run further ahead without unblocking anything. Kept at 16.

MAX_ANALYZER_WORKERS = 64

Constructors

Macro Summary

Instance Method Summary

Instance methods inherited from module FileHelper

all_files : Array(String) all_files, get_files_by_basename(basename : String) : Array(String) get_files_by_basename, get_files_by_extension(extension : String) : Array(String) get_files_by_extension, get_files_by_extensions(extensions : Array(String)) : Array(String) get_files_by_extensions, get_files_by_prefix(prefix : String) : Array(String) get_files_by_prefix, get_files_by_prefix_and_extension(prefix : String, extension : String) : Array(String) get_files_by_prefix_and_extension, get_files_by_relative_path(relative_path : String, root : String = "") : Array(String) get_files_by_relative_path, get_public_dir_files(base_path : String, folder : String) : Array(String) get_public_dir_files, get_public_files(base_path : String, anchors : Array(String) = ["shard.yml", "Gemfile"]) : Array(String) get_public_files, walked_path(expanded : String) : String walked_path

Constructor Detail

def self.new(options : Hash(String, YAML::Any)) #

[View source]

Macro Detail

macro analyzer_for(tech) #

Declares which tech this analyzer implements.

class Gin < GoEngine
  analyzer_for "go_gin"
end

initialize_analyzers reads the registry off the classes themselves, so this is the only place the name is written. Before it, the mapping lived in a hand-maintained define_analyzers list — a second place to remember, where forgetting produced no error and no failing spec, just an analyzer that never ran. Mirrors Detector.detector_for.

Every concrete class under the Analyzer:: namespace must declare one: the derivation calls tech_name on each, so omitting it is a compile error rather than a silent no-op. Shared base classes (GoEngine, SpecificationEngine, …) are abstract and sit outside that set.


[View source]

Instance Method Detail

def analyze #

[View source]
def base_path : String #

[View source]
def base_paths : Array(String) #

[View source]
def base_relative_path(path : String) : String #

URL for a file-path-routed stack — plain PHP, JSP, CFML, Classic ASP, WebForms — where the file's location under the web root is the route.

The root is the configured scan base unless one of markers names a document root deeper in the tree. The last occurrence wins: for app/wwwroot/x/htdocs/y.asp the deepest marker is the real root, whereas testing markers in list order would pick whichever name happened to be listed first.

Keep markers unambiguous. Generic names like public/ or www/ collide with build output (docs/public/) — the false positive FileHelper#get_public_files documents. The file's location relative to the configured scan base, /-separated and rooted with a leading /.

Convention filters — "is this file under includes/?", "is this a test fixture?" — must be applied to this, never to the absolute path. A checkout that merely lives under a directory with a matching name otherwise matches every file in it: scanning the same Classic ASP site from /srv/inc/site instead of /srv/site dropped 33 of its 35 endpoints, because every page looked like an #include fragment.


[View source]
def callees_needed? : Bool #

Callees feed --include-callee (direct output) and --ai-context (aggregated review context). Analyzers should consult this before running their callee extractor so the work is skipped on default scans where neither flag is set.


[View source]
def content_matches?(content : String, markers : Regex) : Bool #

Whether content matches markers, skipping PCRE2's per-call UTF-8 revalidation. Analyzer content always comes from #read_file_content (or the detector cache it reads through), so the subject is known-valid UTF-8 — see Noir::TextFile::MATCH_OPTIONS.

Use this in place of a chain of String#includes? over whole file content: String#includes? runs Rabin-Karp per marker, while one precompiled Regex.union of the same literals is the same predicate in a single JIT-compiled pass. A single includes? on a short string (one source line, a path segment) is not worth converting — the win scales with the size of the subject.


[View source]
def http_header_name(name : String) : String | Nil #

HTTP_X_FORWARDED_FOR -> x-forwarded-for.

CGI-style server-variable collections (Request.ServerVariables in ASP/WebForms, cgi.* in CFML) expose inbound headers under an HTTP_-prefixed, underscore-separated name. Everything without that prefix (REMOTE_ADDR, SCRIPT_NAME, ...) is server state, not a request parameter, and returns nil.


[View source]
def line_number_for_index(content : String, char_index : Int32) : Int32 #

1-based line number for a character offset into content.

Counts newlines over the raw byte buffer rather than slicing content[0...index], which would be an O(n) copy per call. The scan is correct because \n is ASCII and never appears inside a UTF-8 multi-byte sequence.

PhpEngine#php_line_number_for_index predates this and is identical; it should fold into this helper when PHP is next touched.


[View source]
def logger : NoirLogger #

[View source]
def parallel_analyze(files : Array(String), &block : String -> Nil) #

Preferred overload: accepts a file list and creates both the producer and worker fibers inside a single WaitGroup so every fiber is tracked. The bare-spawn producer in the channel-based overload below was an orphan that could trigger "can't resume a running fiber" under Crystal ≥1.20's M:N scheduler when multiple analyzers ran concurrently.


[View source]
def read_file_content(path : String) : String #

Prefer the detector-populated cache over a fresh disk read. On cache miss (budget exhausted, cache cleared between runs, path not registered via register_file) falls back to a disk read.

Analyzers migrating from direct File.read(path, ...) calls should use this helper so the second read of files the detector already loaded is free.


[View source]
def result : Array(Endpoint) #

[View source]
def unique_params(params : Array(Param)) : Array(Param) #

Order-preserving dedup of params by (name, param_type). Analyzers that accumulate every reference they see (rather than every route) collect the same key many times over.


[View source]
def url : String #

[View source]
def web_root_path(path : String, markers : Array(String)) : String #

[View source]