spaCy/spacy/tests/tokenizer
Adriane Boyd 0e7f94b247
Update Tokenizer.explain with special matches (#7749)
* Update Tokenizer.explain with special matches

Update `Tokenizer.explain` and the pseudo-code in the docs to include
the processing of special cases that contain affixes or whitespace.

* Handle optional settings in explain

* Add test for special matches in explain

Add test for `Tokenizer.explain` for special cases containing affixes.
2021-04-19 19:08:20 +10:00
..
__init__.py Revert #4334 2019-09-29 17:32:12 +02:00
sun.txt Revert #4334 2019-09-29 17:32:12 +02:00
test_exceptions.py Tidy up and auto-format 2020-02-18 15:38:18 +01:00
test_explain.py Update Tokenizer.explain with special matches (#7749) 2021-04-19 19:08:20 +10:00
test_naughty_strings.py Merge branch 'develop' into master-tmp 2020-09-04 13:15:36 +02:00
test_tokenizer.py Use special matcher for exceptions with spaces (#6668) 2021-01-06 12:05:10 +08:00
test_urls.py Merge branch 'develop' into master-tmp 2020-06-20 15:52:00 +02:00
test_whitespace.py Merge branch 'develop' into master-tmp 2020-09-04 13:15:36 +02:00