Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
spaCy/spacy/tests/test_matcher.py

112 lines
3.6 KiB
Python
Raw Normal View History

* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
from __future__ import unicode_literals
import pytest
* Reimplement matching in Cython, instead of Python.
2015-08-04 23:05:54 +00:00
from spacy.strings import StringStore
* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
from spacy.matcher import *
* Upd tests
2015-09-06 03:40:10 +00:00
from spacy.attrs import LOWER
* Reimplement matching in Cython, instead of Python.
2015-08-04 23:05:54 +00:00
from spacy.tokens.doc import Doc
from spacy.vocab import Vocab
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
from spacy.en import English
* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
@pytest.fixture
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def matcher():
* Update matcher tests
2015-08-06 12:33:35 +00:00
patterns = {
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
'JS': ['PRODUCT', {}, [[{'ORTH': 'JavaScript'}]]],
* Update matcher tests
2015-08-06 14:09:28 +00:00
'GoogleNow': ['PRODUCT', {}, [[{'ORTH': 'Google'}, {'ORTH': 'Now'}]]],
* Upd tests
2015-09-06 03:40:10 +00:00
'Java': ['PRODUCT', {}, [[{'LOWER': 'java'}]]],
* Update matcher tests
2015-08-06 12:33:35 +00:00
}
Refactor so that the tokenizer data is read from Python data, rather than from disk
2016-09-25 12:49:53 +00:00
return Matcher(Vocab(lex_attr_getters=English.Defaults.lex_attr_getters), patterns)
* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
def test_compile(matcher):
* Reimplement matching in Cython, instead of Python.
2015-08-04 23:05:54 +00:00
assert matcher.n_patterns == 3
* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
* Update matcher tests
2015-08-06 12:33:35 +00:00
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def test_no_match(matcher):
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words=['I', 'like', 'cheese', '.'])
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert matcher(doc) == []
def test_match_start(matcher):
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words=['JavaScript', 'is', 'good'])
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert matcher(doc) == [(matcher.vocab.strings['JS'],
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
matcher.vocab.strings['PRODUCT'], 0, len('JavaScript'))]
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def test_match_end(matcher):
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words=['I', 'like', 'java'])
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert matcher(doc) == [(doc.vocab.strings['Java'],
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
doc.vocab.strings['PRODUCT'],
len('I like '), len('I like java'))]
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def test_match_middle(matcher):
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words=['I', 'like', 'Google', 'Now', 'best'])
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert matcher(doc) == [(doc.vocab.strings['GoogleNow'],
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
doc.vocab.strings['PRODUCT'], len('I like '),
len('I like Google Now'))]
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def test_match_multi(matcher):
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words='I like Google Now and java best'.split())
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert matcher(doc) == [(doc.vocab.strings['GoogleNow'],
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
doc.vocab.strings['PRODUCT'], len('I like '),
len('I like Google Now')),
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
(doc.vocab.strings['Java'],
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
doc.vocab.strings['PRODUCT'],
len('I like Google Now and '),
len('I like Google Now and java'))]
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
def test_match_zero(matcher):
matcher.add('Quote', '', {}, [
[
{'ORTH': '"'},
{'OP': '!', 'IS_PUNCT': True},
{'OP': '!', 'IS_PUNCT': True},
{'ORTH': '"'}
]])
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words='He said , " some words " ...'.split())
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert len(matcher(doc)) == 1
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words='He said , " some three words " ...'.split())
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert len(matcher(doc)) == 0
matcher.add('Quote', '', {}, [
[
{'ORTH': '"'},
{'IS_PUNCT': True},
{'IS_PUNCT': True},
{'IS_PUNCT': True},
{'ORTH': '"'}
]])
assert len(matcher(doc)) == 0
def test_match_zero_plus(matcher):
matcher.add('Quote', '', {}, [
[
{'ORTH': '"'},
{'OP': '*', 'IS_PUNCT': False},
{'ORTH': '"'}
]])
Update matcher tests
2016-10-16 23:53:24 +00:00
doc = Doc(matcher.vocab, words='He said , " some words " ...'.split())
Initial, limited support for quantified patterns in Matcher, and tracking of ent_id attribute in Token and Span. The quantifiers need a lot more testing, and there are some known problems. The main known problem is that the zero-plus and one-plus quantifiers won't work if a token can match both the quantified pattern expression AND the tail of the match.
2016-09-21 12:54:55 +00:00
assert len(matcher(doc)) == 1
* Update matcher tests
2015-08-06 12:33:35 +00:00
* Add draft dfa matcher, in Python. Passing tests.
2015-08-04 13:55:28 +00:00
Update matcher test, to reflect character offset return instead of token offset.
2016-10-17 14:00:10 +00:00
@pytest.mark.models
def test_match_preserved(EN):
patterns = {
'JS': ['PRODUCT', {}, [[{'ORTH': 'JavaScript'}]]],
'GoogleNow': ['PRODUCT', {}, [[{'ORTH': 'Google'}, {'ORTH': 'Now'}]]],
'Java': ['PRODUCT', {}, [[{'LOWER': 'java'}]]],
}
matcher = Matcher(EN.vocab, patterns)
doc = EN.tokenizer('I like java.')
EN.tagger(doc)
assert len(doc.ents) == 0
doc = EN.tokenizer('I like java.')
doc.ents += tuple(matcher(doc))
assert len(doc.ents) == 1
EN.tagger(doc)
EN.entity(doc)
assert len(doc.ents) == 1