spaCy/tests/tokenizer/test_only_punct.py

from __future__ import unicode_literals
import pytest

from spacy.en import English


def test_only_pre1():
    EN = English()
    assert len(EN("(")) == 1


def test_only_pre2():
    EN = English()
    assert len(EN("((")) == 2
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 16:01:46 +00:00			`from __future__ import unicode_literals`
			`import pytest`

* Tests passing except for morphology/lemmatization stuff 2014-12-23 00:40:32 +00:00			`from spacy.en import English`
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 16:01:46 +00:00
Tweak line spacing 2015-04-19 19:39:18 +00:00
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 16:01:46 +00:00			`def test_only_pre1():`
* Upd tests 2014-12-30 10:34:09 +00:00			`EN = English()`
* Tests passing except for morphology/lemmatization stuff 2014-12-23 00:40:32 +00:00			`assert len(EN("(")) == 1`
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 16:01:46 +00:00

			`def test_only_pre2():`
* Upd tests 2014-12-30 10:34:09 +00:00			`EN = English()`
* Tests passing except for morphology/lemmatization stuff 2014-12-23 00:40:32 +00:00			`assert len(EN("((")) == 2`