spaCy/tests/tokenizer/test_infix.py

from __future__ import unicode_literals

import pytest

from spacy.en import English

EN = English()

def test_hyphen():
    tokens = EN.tokenizer('best-known')
    assert len(tokens) == 3


def test_period():
    tokens = EN.tokenizer('best.Known')
    assert len(tokens) == 3
    tokens = EN.tokenizer('zombo.com')
    assert len(tokens) == 1
* Test hyphenation etc 2014-10-14 09:26:16 +00:00			`from __future__ import unicode_literals`

			`import pytest`

* Tests passing except for morphology/lemmatization stuff 2014-12-23 00:40:32 +00:00			`from spacy.en import English`
* Test hyphenation etc 2014-10-14 09:26:16 +00:00
* Restore hyphenation test to test_infix 2015-06-06 03:57:36 +00:00			`EN = English()`
* Test hyphenation etc 2014-10-14 09:26:16 +00:00
* Restore hyphenation test to test_infix 2015-06-06 03:57:36 +00:00			`def test_hyphen():`
			`tokens = EN.tokenizer('best-known')`
			`assert len(tokens) == 3`
* Test hyphenation etc 2014-10-14 09:26:16 +00:00

			`def test_period():`
* Update tests, preventing the parser from being loaded if possible 2015-05-24 23:02:03 +00:00			`tokens = EN.tokenizer('best.Known')`
* Test hyphenation etc 2014-10-14 09:26:16 +00:00			`assert len(tokens) == 3`
* Restore hyphenation test to test_infix 2015-06-06 03:57:36 +00:00			`tokens = EN.tokenizer('zombo.com')`
* Test hyphenation etc 2014-10-14 09:26:16 +00:00			`assert len(tokens) == 1`