From ca36404257691d45d41b4585eb50cbf7d25a756d Mon Sep 17 00:00:00 2001 From: Erez Sh Date: Mon, 4 Nov 2019 11:37:12 +0200 Subject: [PATCH] Some improvements to visitor documentation --- docs/classes.md | 24 ++++++------ ...{transfromer_and_vistor.md => visitors.md} | 39 +++++++++++-------- mkdocs.yml | 1 + 3 files changed, 36 insertions(+), 28 deletions(-) rename docs/{transfromer_and_vistor.md => visitors.md} (78%) diff --git a/docs/classes.md b/docs/classes.md index b63f8f1..1555a1f 100644 --- a/docs/classes.md +++ b/docs/classes.md @@ -1,15 +1,13 @@ -# Classes - Reference +# Classes Reference This page details the important classes in Lark. ---- -## Lark +## lark.Lark The Lark class is the main interface for the library. It's mostly a thin wrapper for the many different parsers, and for the tree constructor. -### Methods - #### \_\_init\_\_(self, grammar, **options) The Lark class accepts a grammar string or file object, and keyword options: @@ -50,14 +48,10 @@ If a transformer is supplied to `__init__`, returns whatever is the result of th The main tree class -### Properties - * `data` - The name of the rule or alias * `children` - List of matched sub-rules and terminals * `meta` - Line & Column numbers, if using `propagate_positions` -### Methods - #### \_\_init\_\_(self, data, children) Creates a new tree, and stores "data" and "children" in attributes of the same name. @@ -92,8 +86,6 @@ Trees can be hashed and compared. ---- -[Guide](transfromer_and_vistor.md) - ## Token When using a lexer, the resulting tokens in the trees will be of the Token class, which inherits from Python's string. So, normal string comparisons and operations will work as expected. Tokens also have other useful attributes: @@ -105,17 +97,25 @@ When using a lexer, the resulting tokens in the trees will be of the Token class * `end_line` - The line where the token ends * `end_column` - The next column after the end of the token. For example, if the token is a single character with a `column` value of 4, `end_column` will be 5. +## Transformer +## Visitor +## Interpreter + +See the [visitors page](visitors.md) + ## UnexpectedInput +## UnexpectedToken + +## UnexpectedException + - `UnexpectedInput` - `UnexpectedToken` - The parser recieved an unexpected token - `UnexpectedCharacters` - The lexer encountered an unexpected string After catching one of these exceptions, you may call the following helper methods to create a nicer error message: -### Methods - #### get_context(text, span) Returns a pretty string pinpointing the error in the text, with `span` amount of context characters around it. diff --git a/docs/transfromer_and_vistor.md b/docs/visitors.md similarity index 78% rename from docs/transfromer_and_vistor.md rename to docs/visitors.md index 463c8d1..c60c1dc 100644 --- a/docs/transfromer_and_vistor.md +++ b/docs/visitors.md @@ -2,9 +2,9 @@ Transformers & Visitors provide a convenient interface to process the parse-trees that Lark returns. -They are used by inheriting from the correct class (visitor or transformer), and implementing methods corresponding to the rule you wish to process. Each methods accepts the children as an argument. That can be modified using the `v_args` decorator, which allows to inline the arguments (akin to `*args`), or add the tree `meta` property as an argument. +They are used by inheriting from the correct class (visitor or transformer), and implementing methods corresponding to the rule you wish to process. Each method accepts the children as an argument. That can be modified using the `v_args` decorator, which allows to inline the arguments (akin to `*args`), or add the tree `meta` property as an argument. -See: https://github.com/lark-parser/lark/blob/master/lark/visitors.py +See: visitors.py ### Visitors @@ -40,6 +40,8 @@ Because nodes are reduced from leaf to root, at any point the callbacks may assu Transformers can be chained into a new transformer by using multiplication. +`Transformer` can do anything `Visitor` can do, but because it reconstructs the tree, it is slightly less efficient. + **Example:** ```python @@ -55,25 +57,30 @@ print(EvalExpressions().transform( t )) # Prints: Tree(a, [3]) ``` -By default, transformer works only on rules, `visit_tokens=True` will make transfomer process tokens. This is handy in parsing simple token, such as turn `INT` to `int`, `NUMBER` to `float`. etc. - -```python -class T(Transformer): - INT = int # same with def INT(self, tok): int(tok) - NUMBER = float # same with def INT(self, tok): int(tok) - def NAME(self, name): - return lookup_dict.get(name, name) - - -T(visit_tokens=True).transform(tree) -``` - -Here are the classes that implement the transformer interface: +All these classes implement the transformer interface: - Transformer - Recursively transforms the tree. This is the one you probably want. - Transformer_InPlace - Non-recursive. Changes the tree in-place instead of returning new instances - Transformer_InPlaceRecursive - Recursive. Changes the tree in-place instead of returning new instances +### visit_tokens + +By default, transformers only visit rules. `visit_tokens=True` will tell Transformer to visit tokens as well. This is a slightly slower alternative to `lexer_callbacks`, but it's easier to maintain and works for all algorithms (even when there isn't a lexer). + +Example: + +```python +class T(Transformer): + INT = int + NUMBER = float + def NAME(self, name): + return lookup_dict.get(name, name) + + +T(visit_tokens=True).transform(tree) +``` + + ### v_args `v_args` is a decorator. diff --git a/mkdocs.yml b/mkdocs.yml index 63bdd61..f5b0d1d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,5 +9,6 @@ pages: - How To Develop (Guide): how_to_develop.md - Grammar Reference: grammar.md - Tree Construction Reference: tree_construction.md + - Visitors and Transformers: visitors.md - Classes Reference: classes.md - Recipes: recipes.md