2016-10-31 18:04:15 +00:00
|
|
|
|
//- 💫 DOCS > API > ANNOTATION SPECS
|
|
|
|
|
|
|
|
|
|
include ../../_includes/_mixins
|
|
|
|
|
|
|
|
|
|
p This document describes the target annotations spaCy is trained to predict.
|
|
|
|
|
|
|
|
|
|
+h(2, "tokenization") Tokenization
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| Tokenization standards are based on the
|
|
|
|
|
| #[+a("https://catalog.ldc.upenn.edu/LDC2013T19") OntoNotes 5] corpus.
|
|
|
|
|
| The tokenizer differs from most by including tokens for significant
|
|
|
|
|
| whitespace. Any sequence of whitespace characters beyond a single space
|
|
|
|
|
| (#[code ' ']) is included as a token.
|
|
|
|
|
|
|
|
|
|
+aside-code("Example").
|
|
|
|
|
from spacy.en import English
|
|
|
|
|
nlp = English(parser=False)
|
|
|
|
|
tokens = nlp('Some\nspaces and\ttab characters')
|
|
|
|
|
print([t.orth_ for t in tokens])
|
|
|
|
|
# ['Some', '\n', 'spaces', ' ', 'and', '\t', 'tab', 'characters']
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| The whitespace tokens are useful for much the same reason punctuation is
|
|
|
|
|
| – it's often an important delimiter in the text. By preserving it in the
|
|
|
|
|
| token output, we are able to maintain a simple alignment between the
|
|
|
|
|
| tokens and the original string, and we ensure that no information is
|
|
|
|
|
| lost during processing.
|
|
|
|
|
|
|
|
|
|
+h(2, "sentence-boundary") Sentence boundary detection
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| Sentence boundaries are calculated from the syntactic parse tree, so
|
|
|
|
|
| features such as punctuation and capitalisation play an important but
|
|
|
|
|
| non-decisive role in determining the sentence boundaries. Usually this
|
|
|
|
|
| means that the sentence boundaries will at least coincide with clause
|
|
|
|
|
| boundaries, even given poorly punctuated text.
|
|
|
|
|
|
|
|
|
|
+h(2, "pos-tagging") Part-of-speech Tagging
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| The part-of-speech tagger uses the
|
|
|
|
|
| #[+a("https://catalog.ldc.upenn.edu/LDC2013T19") OntoNotes 5] version of
|
|
|
|
|
| the Penn Treebank tag set. We also map the tags to the simpler Google
|
|
|
|
|
| Universal POS Tag set. See
|
|
|
|
|
| #[+src(gh("spaCy", "spacy/tagger.pyx")) tagger.pyx] for details.
|
|
|
|
|
|
|
|
|
|
+h(2, "lemmatization") Lemmatization
|
|
|
|
|
|
|
|
|
|
p A "lemma" is the uninflected form of a word. In English, this means:
|
|
|
|
|
|
|
|
|
|
+list
|
|
|
|
|
+item #[strong Adjectives]: The form like "happy", not "happier" or "happiest"
|
|
|
|
|
+item #[strong Adverbs]: The form like "badly", not "worse" or "worst"
|
|
|
|
|
+item #[strong Nouns]: The form like "dog", not "dogs"; like "child", not "children"
|
|
|
|
|
+item #[strong Verbs]: The form like "write", not "writes", "writing", "wrote" or "written"
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| The lemmatization data is taken from
|
|
|
|
|
| #[+a("https://wordnet.princeton.edu") WordNet]. However, we also add a
|
|
|
|
|
| special case for pronouns: all pronouns are lemmatized to the special
|
|
|
|
|
| token #[code -PRON-].
|
|
|
|
|
|
|
|
|
|
+h(2, "dependency-parsing") Syntactic Dependency Parsing
|
|
|
|
|
|
|
|
|
|
p
|
|
|
|
|
| The parser is trained on data produced by the
|
|
|
|
|
| #[+a("http://www.clearnlp.com") ClearNLP] converter. Details of the
|
|
|
|
|
| annotation scheme can be found
|
|
|
|
|
| #[+a("http://www.mathcs.emory.edu/~choi/doc/clear-dependency-2012.pdf") here].
|
|
|
|
|
|
|
|
|
|
+h(2, "named-entities") Named Entity Recognition
|
|
|
|
|
|
2016-11-25 14:48:48 +00:00
|
|
|
|
+table([ "Type", "Description" ])
|
2016-10-31 18:04:15 +00:00
|
|
|
|
+row
|
|
|
|
|
+cell #[code PERSON]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell People, including fictional.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code NORP]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Nationalities or religious or political groups.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
2016-11-25 14:48:48 +00:00
|
|
|
|
+cell #[code FACILITY]
|
|
|
|
|
+cell Buildings, airports, highways, bridges, etc.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code ORG]
|
|
|
|
|
+cell Companies, agencies, institutions, etc.
|
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code GPE]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Countries, cities, states.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code LOC]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Non-GPE locations, mountain ranges, bodies of water.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code PRODUCT]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Objects, vehicles, foods, etc. (Not services.)
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code EVENT]
|
|
|
|
|
+cell Named hurricanes, battles, wars, sports events, etc.
|
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code WORK_OF_ART]
|
|
|
|
|
+cell Titles of books, songs, etc.
|
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code LANGUAGE]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Any named language.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
p The following values are also annotated in a style similar to names:
|
|
|
|
|
|
2016-11-25 14:48:48 +00:00
|
|
|
|
+table([ "Type", "Description" ])
|
2016-10-31 18:04:15 +00:00
|
|
|
|
+row
|
|
|
|
|
+cell #[code DATE]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Absolute or relative dates or periods.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code TIME]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Times smaller than a day.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code PERCENT]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Percentage, including "%".
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code MONEY]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Monetary values, including unit.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code QUANTITY]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Measurements, as of weight or distance.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code ORDINAL]
|
2016-11-25 14:48:48 +00:00
|
|
|
|
+cell "first", "second", etc.
|
2016-10-31 18:04:15 +00:00
|
|
|
|
|
|
|
|
|
+row
|
|
|
|
|
+cell #[code CARDINAL]
|
2016-11-25 14:53:40 +00:00
|
|
|
|
+cell Numerals that do not fall under another type.
|