Rooba
/
lark
mirror of https://github.com/lark-parser/lark.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
602 Commits 59 Branches 52 Tags 7 MiB
Commit Graph

107 Commits

Author SHA1 Message Date
Rogdham 2f6b97cfd3 Allow renaming relative import rule 
%import .local.foo -> bar
 2019-03-03 17:10:36 +01:00
Rogdham 9b22d41e49 Change namespace naming and aliasing for imports 
The `.` character is invalid as a group name in regex, replacing by `__`.
The bug arose in `lexer._build_mres`.
 2019-03-03 17:10:36 +01:00
Rogdham 0485a8a5b1 Tests for import rename 
%import common.NUMBER -> N
%import .local.NUMBER -> N
 2019-03-03 17:10:26 +01:00
Erez Shinan 532c955962 Fixed tests to work with pytest, improved docs 2019-03-02 15:00:21 +02:00
Kyungdahm Yun d5c9006f01 Fix literal range escape assertion 2019-02-23 15:32:31 -08:00
Erez Shinan 6b2df208c2 Fixed support for hex encoding (\xAA) 2019-02-13 13:35:32 +02:00
Kyungdahm Yun 70d724732d Support hex escape (\xhh) in string literal 2019-02-12 12:49:03 -08:00
Erez Shinan d2f55fe3ba BUGFIX: Fixed common.ESCAPED_STRING (Issue #309) 2019-02-01 00:03:45 +02:00
Erez Shinan f8486a02b0 Minor cleanup 2019-01-20 15:13:58 +02:00
Erez Shinan 899f69a94b Temporarily silence test_earley3 and test_earley4 (priority in terminals) 2019-01-20 14:47:52 +02:00
Erez Shinan 688c453456 Corrections 2019-01-12 16:56:58 +02:00
Erez Shinan 989dd9c498 Merge branch 'master' into 0.7d 2019-01-12 16:39:06 +02:00
night199uk 80a09697fe Rebuild the way Earley prioritizes ambiguities 
- Makes rule ordering the default ambiguity tie breaker.
    E.g.
    start: a | b
    a: "A"
    b: "A"

    will return:
    start
        a

    start: b | a
    a: "A"
    b: "A"

    will return
    start
        b

- Replaces the ambiguity='resolve__antiscore_sum' with a separate option: 'priority'.
    The priority option has 4 values: 'auto', 'none', 'normal', 'invert'.
    'Auto' maps to 'Normal' for CYK and Earley and 'None' for LALR.
    'None' filters your priorities and ignores them. This saves some extra tree walking on Earley.
    'Normal' uses your priorities untouched, mimicing the old behaviour.
    'Invert' negates your priorities, emulating the old 'resolve__antiscore_sum' behaviour.

    This allows you to use priority logic even when ambiguity=='explicit', to get a better idea
    of the shape of your tree; and to easily disable priorities without removing them from the
    grammar for testing (or performance).

- ambiguity='explicit' now correctly returns an ambiguous tree again, as 0.6 did.
 2018-12-22 20:18:16 +01:00
night199uk 72118eaf8c Disable some debug that crept in 2018-12-21 19:15:42 +01:00
Erez Shinan 085eba3b8e Merge branch 'master' into 0.7c 
Update from latest 0.6
 2018-12-21 18:22:23 +02:00
evandrocoan 48351c3d3e Trimmed whitespaces and ensured a new line on the end of the file 2018-12-20 21:41:35 -02:00
Erez Shinan 467483553b Fixed placeholder code, should work as expected (Issue #285) 2018-12-20 01:24:23 +02:00
Erez Shinan adb30180aa maybe_placeholder now doesn't apply to anonymous terminals 2018-12-19 18:52:24 +02:00
Erez Shinan 222df5bab4 New Feature: Added maybe_placeholders option (Issue #285) 2018-12-19 15:37:20 +02:00
Erez Shinan 76e185a36c Added the Forest interface for explicit ambiguity 2018-12-18 15:06:19 +02:00
Erez Shinan b2489e13e2 A few fixes (tests + interface) 2018-11-14 00:44:50 +02:00
Erez Shinan 0077366255 Earley now ignores infinite recursion 2018-11-13 14:57:20 +02:00
Erez Shinan 63e1ea0226 Merge branch 'earley_sppf' (Oct 18) 2018-10-24 16:29:38 +03:00
Erez Shinan 1798971455 Adjustments 2018-10-24 16:28:54 +03:00
Erez Shinan 6c8ba76b79 Fixed a deep bug in grammar analysis involving empty rules (Issue #250) 2018-10-15 21:52:27 +03:00
night199uk 1d5fd7301a Heavy modifications to the Earley parser to try and make it handle more 
types of ambiguity.
 - Rewritten along the lines of Elizabeth Scott's parser.
https://www.sciencedirect.com/science/article/pii/S1571066108001497

 - Implement SPPF trees per Elizabeth Scott and Bram van der Sanden's work.
http://www.bramvandersanden.com/post/2014/06/shared-packed-parse-forest/
 2018-10-14 23:15:03 +02:00
Erez Shinan 242ac24ea6 Fixed the propagate_positions implementation, and added start_pos/end_pos attributes to Tree.Meta 
Related to issue #216
 2018-08-28 11:39:20 +03:00
Erez Shinan 181f061091 BUGFIX - Fixed 2 issues with line counting 
1) Failed to detect newlines in regexps of the form [^...]

2) Last token didn't get end_line & end_column
 2018-08-14 17:35:47 +03:00
Erez Shinan 32b78b8ee5 BUGFIX: Repeated use of optional rules tripped up the simplifier, manifesting when aliases were used (Issue #197) 2018-07-30 17:54:12 +03:00
Erez Shinan 454c88b58a Refactoring and fixes for merge 2fd0087 2018-07-22 22:47:12 +03:00
Julien Malard a03e01bc12 Fixed test. 2018-07-22 23:12:09 +05:30
Julien Malard 12004b3c65 Reimplemented relative and multiple imports. 2018-07-22 23:09:35 +05:30
Julien Malard 405f6a399d From and relative type imports seem to work. 2018-07-08 02:30:02 +05:30
Erez Shinan 6ea4588bcf Dynamic lexer is now returns the maximum match only. Complete lexing beahvior moved to "dynamic_complete" 
Changed dynamic lexer behavior to only match terminals to their maximum length (i.e. greedy match), emulating the standard lexer.

The original dynamic lexer behavior, that attempts to match all appearances of a terminal, has been moved to the "dynamic_complete" lexer.

For example, when applying a terminal "a"+ to the text "aaa":

- dynamic: ["aaa"]
- dynamic_complete: ["a", "aa", "aaa"]
 2018-06-27 16:31:02 +03:00
Erez Shinan 5c6df8e825 Moved and restructured exceptions 
* All exceptions are now under exceptions.py
* UnexpectedInput is now superclass of UnexpectedToken and UnexpectedCharacters,
  all of which support the get_context() and match_examples() methods.
 2018-06-18 15:14:31 +03:00
Erez Shinan 6bfc27c11d New transformers near completion 
Nearley tool still needs fixing
 2018-05-18 15:14:59 +03:00
Erez Shinan 5e546f38a9 args decorators actually work now 2018-05-13 00:54:06 +03:00
Erez Shinan 9daacb9082 Refactored transformers, better code 2018-05-13 00:42:50 +03:00
Erez Shinan 80848946b2 Merge branch '0.6.0' into new_transformers 
Small collisions
 2018-05-12 23:27:11 +03:00
Erez Shinan 2b4ef11ebf Columns now start at 1 2018-05-10 12:34:19 +03:00
Erez Shinan 7b32ffd83a Fixed token visibility rules (Issue #109) 
Anonymous tokens would become visible if they had the same value as named tokens.
That's because they are merged for the lexer. But after this change, the rules for
visibility are based on their use in the rule, and not their name or identity.
 2018-05-08 12:05:16 +03:00
Erez Shinan 33caa391d5 Breaking backwards compatibility: 
* Removed the scanless parsing feature (dynamic lexing is king)
* Default LALR lexer is now contextual
 2018-05-05 17:03:52 +03:00
Erez Shinan 0f0776c0fa BUGIX in lexer: Embedding strings overwrote priority (Issue #121) 2018-04-24 15:37:11 +03:00
Erez Shinan c3bce19dc2 More steps towards a good solution 2018-04-13 13:41:30 +03:00
Erez Shinan f69bceb335 Snap more things into place 2018-04-13 12:02:01 +03:00
Erez Shinan f960c1b8ac Initial: Added transformers.py, and Meta to tree 2018-04-13 00:40:28 +03:00
Erez Shinan 4f2330fc9b Fixed bug in Earley prioritization 2018-04-05 16:09:42 +03:00
Erez Shinan 25c3c51b1c Fixed bug in Earley: A tree builder optimization clashed with explicit ambiguity 2018-04-05 15:41:36 +03:00
Erez Shinan 255ef0d973 Added error message for the alias syntax in terminals (Issue #97) 2018-04-01 00:16:08 +03:00
Erez Shinan 7d11dfa5cd FEATURE: Added support for ranged-repeat for rules and terminals (Issues #75, #19) 
Syntax: symbol~number
      | symbol~min..max

Example:

  HEXCOLOR: "#" (HEXDIGIT~3 | HEXDIGIT~6)
  short_sentence: word~4..20

Added range for tokens
 2018-02-10 20:00:17 +02:00