lark/docs/tree_construction.md

Automatic Tree Construction - Reference

Lark builds a tree automatically based on the structure of the grammar, where each rule that is matched becomes a branch (node) in the tree, and its children are its matches, in the order of matching.

For example, the rule node: child1 child2 will create a tree node with two children. If it is matched as part of another rule (i.e. if it isn't the root), the new rule's tree node will become its parent.

Using item+ or item* will result in a list of items, equivalent to writing item item item ...

Terminals

Terminals are always values in the tree, never branches.

Lark filters out certain types of terminals by default, considering them punctuation:

• Terminals that won't appear in the tree are:

  • Unnamed literals (like "keyword" or "+")
  • Terminals whose name starts with an underscore (like _DIGIT)

• Terminals that will appear in the tree are:

  • Unnamed regular expressions (like /[0-9]/)
  • Named terminals whose name starts with a letter (like DIGIT)

Rules prefixed with ! will retain all their literals regardless.

Example:

    expr: "(" expr ")"
        | NAME+

    NAME: /\w+/

    %ignore " "

Lark will parse "((hello world))" as:

expr
    expr
        expr
            "hello"
            "world"

The brackets do not appear in the tree by design. The words appear because they are matched by a named terminal.

Shaping the tree

Users can alter the automatic construction of the tree using a collection of grammar features.

• Rules whose name begins with an underscore will be inlined into their containing rule.

Example:

    start: "(" _greet ")"
    _greet: /\w+/ /\w+/

Lark will parse "(hello world)" as:

start
    "hello"
    "world"

• Rules that receive a question mark (?) at the beginning of their definition, will be inlined if they have a single child, after filtering.

Example:

    start: greet greet
    ?greet: "(" /\w+/ ")"
          | /\w+/ /\w+/

Lark will parse "hello world (planet)" as:

start
    greet
        "hello"
        "world"
    "planet"

• Rules that begin with an exclamation mark will keep all their terminals (they won't get filtered).

    !expr: "(" expr ")"
         | NAME+
    NAME: /\w+/
    %ignore " "

Will parse "((hello world))" as:

expr
  (
  expr
    (
    expr
      hello
      world
    )
  )

Using the ! prefix is usually a "code smell", and may point to a flaw in your grammar design.

• Aliases - options in a rule can receive an alias. It will be then used as the branch name for the option, instead of the rule name.

Example:

    start: greet greet
    greet: "hello"
         | "world" -> planet

Lark will parse "hello world" as:

start
    greet
    planet