TODO docs

Returns the Location at the beginning of the syntax.

The given location is after skipping characters.

Captures all characters parsed inside the block.

Returns a String

NOTE skip characters are still captured, use partial_capture instead.

capture do
  repeat(/ +/) do
    parse(/\w+/)
  end
end

"a   bc d e"  # ~> "a   bc d e"
"Hello World" # ~> "Hello World"

TODO docs

Returns the current precedence.

Initially zero. Changes inside an infix or when with_precedence is used.

Allows to match a word only if a non-alphanumeric follows

parse("foo") { end_word }

Equivalent to: (but faster)
parse(/foo\b/) { nil }

"foo bar" # ~> "foo"
"foobar"  # ~> Unexpected character 'b', expected 'EOF'

NOTE end_word returns nil.

Returns the given result, or break the current sequence if result is a Fail.

This macro is the recommended ways to handle Fail.

result = Array.new(3) do
  parse("a")
end

typeof(result) # => Array(String) | TopDown::Parser::Fail

result = forward_fail(result)
typeof(result) # => Array(String)

TODO docs

In the context of an infix member, returns the left result of the current union.

Returns nil in a context other than inside an infix.

Each time a union member is successfully parsed, left value is updated.

As a union continue parsing infix until possible, left allows to retrieve the subsequent results.

syntax(:exp) do
  union do
    parse('a')
    infix(1, :comma)
  end
end

syntax(:comma) do
  l = left() # left returns either
  r = parse(:exp)
  "#{l}, #{r}"
end

Parses the sequence inside the block, returns nil if it fails.

x = parse('1').to_i
y =
  maybe do
    parse('+')
    parse!('1').to_i
  end
parse!(';')
x + (y || 0)

"1;"   # ~> 1
"1+1;" # ~> 2
"1+*;" # ~> "Unexpected character '*', expected '1'"
"1*;"  # ~> "Unexpected character '*', expected ';'"

Equivalent to maybe { union { ... } }

Prevents the skip rules to be triggered inside the given block.

Parses the given parselet (ParseletLiteral), and yield/return the according result.

failure:

If the given parselet fails to parse, it break the current sequence with a Fail. Failure is catch by the surrounding context.

• inside an union, tell the union that member have fail. The union tries to parse the next member.
• inside a maybe, the maybe will return nil.
• inside a repeat, make the repeat to stop.
• inside a syntax, the syntax is considered to fail, it will in turn break or raises.

block:

A block could be given to let return the value of the block.

precedence:

with_precedence (NumberLiteral) changes the current_precedence, the given parselet will be parsed as if the contained syntax have this precedence. Allow to handle multi-precedence for ternary-or-more operator.

Similar to Parser.parse, but raises SyntaxError if the parsing fail.

error: allows to customize error message, it can be:

• a StringLiteral: Following percent interpolations are available:
  • %{got}: The character or token causing the failure.
  • %{expected}: The expected Char, String, Regex, Token or syntax name (Symbol).
• a ProcLiteral taking two arguments: 'got' and 'expected', with types explained above.

at: indicates where the error is raised, it can be a Location, Range(Location, Location) or Range(Location, Nil).

failure:

Because it raises, this shouldn't be used as first parselet inside a union, maybe, repeat and syntax. This would raises before the surrounding context could try an other solution.

However, this should generally be used everywhere else, to allow more localized errors.

Captures chosen characters parsed inside the block.

Yields a String::Builder in which characters to capture can be added.

Returns a String.

partial_capture do |io|
  repeat(/ +/) do
    io << parse(/\w+/)
  end
end

"a   bc d e"  # ~> "abcde"
"Hello World" # ~> "HelloWorld"

TODO docs parse parselet if succeed, don't move the cursor and return parselet result else, don't move the cursor and break a failure

Equivalent to parse(parselet, with_precedence: precedence)

Repeatedly parses the sequence inside the block until it fails.

Returns nil. Results should be either collected by capture or stored inside a variable or array.

x = parse('1').to_i
repeat do
  parse('+')
  x += parse!('1').to_i
end
parse!(';')
x

"1;"         # ~> 1
"1+1+1+1+1;" # ~> 5
"1+1+1+*;"   # ~> "Unexpected character '*', expected '1'"
"1*;"        # ~> "Unexpected character '*', expected ';'"

Repeatedly parses the sequence inside the block until it fails, with a separator parselet between each iteration.

Returns nil. Results should be either collected by capture or stored inside a variable or array.

x = 0
parse('(')
repeat(',') do
  x += parse('1').to_i
end
parse!(')')
x

"()"          # ~> 0
"(1,1,1,1,1)" # ~> 5
"(1,1,)"      # ~> "Unexpected character ',', expected ')'"
"(11)"        # ~> "Unexpected character '1', expected ')'"

TODO docs

TODO docs

TODO docs

TODO docs

Equivalent to repeat { union { ... } }

Defines the main syntax to parse.

parselet (ParseletLiteral): the syntax name or parselet to be the root.

Allows to group a sequence of parsing (inside a union for example).

union do
  sequence do
    parse("a")
    parse!("b")
  end
  parse("2")

Defines what is ignored during the parsing.

It is triggered every time prior a parselet is parsed.

It works similarly to the members of a Parser.union, in which each members are consumed while possible.

class Parser < TopDown::Parser
  skip do
    # Skips spaces:
    parse ' '
    parse '\n'
    parse '\t'
    parse '\r'

    # Line comments:
    parse "//" { repeat { parse not('\n') } }

    # Block comments:
    parse "/*" do
      repeat { parse not("*/") }
      parse "*/"
    end
  end
end

Defines a syntax that can be called with parse(SymbolLiteral).

syntax_name (SymbolLiteral): The identifier of the syntax.

*prefix (ParseletLiteral): A list of parselet, to parse at the beginning of the syntax, results are yielded in the given block.

block: The body of the syntax, work like a usual body function.

class Parser < TopDown::Parser
  syntax :expression do
    a = parse('a')
    parse(/( )+/)
    foo = parse!("foo")

    {a, foo}
  end
end

Defines a token to parse. Can be used only inside Parser.tokens.

Each token have a token_name (StringLiteral), and is parsed according to parselet. The block indicate the value of the resulting Token.

• If no parselet provided, it deduced from token_name.
• If no block provided, token value is nil. (use &.itself to actually keep the parsed string)

Example:

class Parser < TopDown::Parser
  tokens do
    token("+")                            # Parses '+', produces Token[+]
    token("hey")                          # Parses "hey", produces Token[hey]
    token("new_line", '\n')               # Parses '\n', produces Token[new_line]
    token("int", /\d+/) { |v| v.to_i }    # Parses /\d+/, produces Token[int:<int_value>]
    token("string", :tk_string, &.itself) # Parses syntax :tk_string, produces Token[string:<string_value>]
  end
end

This macro allows to define how tokens are parsed.

Each line of the block correspond a token.

class Parser < TopDown::Parser
  tokens do
    token("+")
    token("-")
    token("**")
    token("*")
    token("/")
    token("new_line", '\n')
  end
end

Use Parser.token to define how a token is parsed

The order of token matter, because there are parsed in this order. Hence, if * is moved before **, the result will be two token * parsed instead of one **.

In addition, characters are never skipped while parsing tokens (and all inner syntax calls), see no_skip.

Tries to parse each member of the union, returns the result of the first that succeed.

union do
  parse('1')
  parse('2')
  sequence do
    parse('a')
    parse('b')
    parse!('c')
  end
  parse('a')
end

"1"   # ~> '1'
"abc" # ~> 'c'
"ab*" # ~> "Unexpected character '*', expected 'c'"
"a"   # ~> 'a'
"*"   # ~> "Could not parse syntax ':main'

members:

Members are delimited by each expression in the Expressions of the given block.

NOTE a block begin/end doesn't group a member since it is inlined by the crystal parser. Use sequence instead.

failure:

If all members of the union fail, the union is considered to fail, and will break the current sequence

infix:

infix members could be added to a union.

infix members are always treated after each normal members, in the order there are defined. A union act as follow:

1. Tries to parse each normal member.
2. When one succeed, store the result. left allows to retrieve this result.
3. Tries to parse infix members whose precedence is greater than current precedence (initial is zero).
4. Inner of infix is executed, it potentially triggers parsing of this union recursively, but current precedence is sets to the infix precedence.
5. When one fully succeed, store the result. left is updated.
6. Repeats step 3-5) until there no more infix.
7. Returns last result.

This is mainly the top-down operator precedence algorithm, also known as precedence climbing.

Similar to Parser.union, but raises SyntaxError if the parsing fail.

Yields successively parsed tokens.

Override this method to modify the default error message when any token can be parsed.

Override this method to modify the default error message when parse!(not('a')) fail.

Override this method to modify the default error message when parse!(not('a'..'z')) fail.

Override this method to modify the default error message when parse!('a'..'z') fail.

Override this method to modify the default error message when parse!(not(["TOKEN"])) fail.

Override this method to modify the default error message when parse!('a') fail.

Override this method to modify the default error message when parse!(/regex/) fail.

Override this method to modify the default error message when parse!(:syntax) fail.

Override this method to modify the default error message when parse!(["TOKEN"]) fail.

Override this method to modify the default error message when parse!("string") fail.

Returns the current cursor location.

The location can be used later to raise an error at that point.

loc = self.location
parse('(')
exp = parse!(:expression)
parse!(')', error: "Unterminated parenthesis expression", at: loc)
exp

Move the cursor to the new location.

The location should be well formed, otherwise error display won't be right. It is recommended to always use a location got by self.location.

This methods is used to backtrack the parser. However, prefer using Parser.union, Parser.maybe, and Parser.repeat over manual backtracks.

Parses the source contained in this parser.

Returns the result of the root syntax. Expects eof after parsing root syntax. Raises SyntaxError if fail to parse.

Raises a SyntaxError at current #location and current source.

location could be a range between two Location. A endless range mean a location up to self.location.

Raises a SyntaxError at current #location and current source.

location could be a range between two Location. A endless range mean a location up to self.location.

Raises a SyntaxError at current #location and current source.

location could be a range between two Location. A endless range mean a location up to self.location.

Modifies the source and reset the cursor #location to zero.

Returns the array of parsed tokens.