module
Noir::TopLevelSplit
Overview
Splits a delimited list at the top nesting level, respecting quoted runs and (optionally) backslash escapes.
This replaces ~44 hand-rolled copies of the same loop spread across
src/analyzer/analyzers/, src/analyzer/engines/ and src/miniparsers/,
19 of which were byte-identical to a sibling. A quote- or escape-handling
bug used to need 44 separate fixes; the copies had already drifted along
seven independent axes (see Rules), so "just pick one and inline it"
would have changed detection for whichever analyzers lost their variant.
Implementation note — why a single forward pass into a String::Builder
and not text[i] / text[start...index]:
String#[](Int) and char-range slicing are O(index) the moment a string
contains one multi-byte UTF-8 codepoint, because Crystal has to walk the
bytes to find the char boundary. Several of the copies being replaced
here indexed per char inside a while loop, making them O(n^2) on any
source file with a Korean comment or an emoji in a string literal, and at
least one carried a comment about materializing .chars up front to work
around exactly that. A splitter only ever moves forward, so each_char
plus one String::Builder per part is O(n) with no random access at all,
no .chars array, and no ASCII-vs-UTF-8 dispatch.
This is deliberately NOT the single_byte? / Bytes-vs-Array(Char)
dispatch used by Noir::JSLiteralScanner. That pattern exists there for
find_matching_*, which must random-access from an arbitrary index and so
genuinely needs an indexable source. A forward-only splitter does not, and
paying for the dispatch (plus a full .chars materialization on non-ASCII
input) would be strictly slower than just iterating.
Extended Modules
Defined in:
utils/top_level_split.crInstance Method Summary
-
#split(text : String, delimiter : Char, rules : Rules) : Array(String)
Splits
texton every occurrence ofdelimiterthat sits at depth 0 and outside a quoted run. -
#split(text : String, delimiter : String, rules : Rules) : Array(String)
Multi-character separator variant.
-
#split_spans(text : String, delimiter : Char, rules : Rules, start_pos : Int32 = 0, end_pos : Int32 = -1) : Array(Tuple(String, Int32))
Splits like
#split, but also reports where each part starts.
Instance Method Detail
Splits text on every occurrence of delimiter that sits at depth 0 and
outside a quoted run.
Multi-character separator variant. Only a separator run that sits
ENTIRELY at depth 0 and outside quotes splits, so ":>" does not fire on
a bare ":" and "||" inside f(a || b) is invisible.
LIMITATION — a delimiter with a state-changing proper prefix: matching a
multi-character separator means buffering characters while they are still
a prefix of it, and a mismatch releases only the first buffered character
before retrying the rest. That release is what changes cursor state, so if
a proper prefix of the delimiter contains a quote character or an opener
of an enabled Nest kind, releasing it opens a quoted run or raises the
depth — and from that point the separator can no longer match at depth 0.
#split("(((xy", "((x", nest: Paren) therefore yields ["(((xy"]: the
only place "((x" could match starts at index 1, and index 0's ( has
already taken the depth to 1, so the depth-0 rule forbids the split. Same
for quotes: #split("\"\"\"xy", "\"\"x", quotes: "\"") yields
["\"\"\"xy"], the first " having opened a run that swallows the rest.
This suppresses SPLITS only. The characters themselves are never lost or reordered: the moment a release ends the top-level/unquoted state, the rest of the buffer is flushed immediately, so every part holds the input characters of its window in source order.
Every separator actually used in the tree is punctuation outside both sets
("," and ":>" / ":<|>" in haskell/servant.cr, "||" in
specification/traefik.cr), so no caller sees the limitation at all. The
note in Cursor#consume about a quote-bearing delimiter is this same
situation seen from the other end.
Splits like #split, but also reports where each part starts.
Returns {part, offset} tuples where offset is an ABSOLUTE CHAR index
into text (not a byte index, and not relative to start_pos) pointing
at the first non-whitespace character of that part. Callers use it to map
an argument back onto the file it was sliced out of, so they need a
position in the same coordinate system they passed the window in.
start_pos and end_pos bound the scan, again as char indices;
end_pos < 0 means "to the end of text". Both are clamped into
0..text.size, and an inverted window is treated as empty.
The window is applied by testing the index while iterating, NOT by
slicing text first: char-range slicing is O(n) on any string holding a
multi-byte codepoint, which is the very cost this module exists to avoid.
Offset of an empty or whitespace-only part: the first non-whitespace
character at or after where the part began, looked for in the WHOLE of
text and not just inside the part — so a part that is only whitespace
reports the index of the delimiter that ended it, and a whitespace-only
final part can report an index past end_pos. That is what the four
hand-rolled copies this replaces did (their skip_whitespace helper was
bounded by content.size, never by the part or the window), and callers
feed the offset straight back into text, so the position must stay a
valid index into text rather than being clamped to the part.
Char delimiter only, deliberately: every caller that needs offsets splits
on a single character, and the multi-character path buffers a pending
prefix that is only committed to a part once the match fails (see the
String overload's limitation note) — so a part could begin with
characters read well before the scan decided they belonged to it, which
would make its offset ambiguous.