9.7 KiB
The central data structures in spaCy are the Language
class,
the Vocab
and the Doc
object. The Language
class
is used to process a text and turn it into a Doc
object. It's typically stored
as a variable called nlp
. The Doc
object owns the sequence of tokens and
all their annotations. By centralizing strings, word vectors and lexical
attributes in the Vocab
, we avoid storing multiple copies of this data. This
saves memory, and ensures there's a single source of truth.
Text annotations are also designed to allow a single source of truth: the Doc
object owns the data, and Span
and Token
are
views that point into it. The Doc
object is constructed by the
Tokenizer
, and then modified in place by the components
of the pipeline. The Language
object coordinates these components. It takes
raw text and sends it through the pipeline, returning an annotated document.
It also orchestrates training and serialization.
Container objects
Name | Description |
---|---|
Doc |
A container for accessing linguistic annotations. |
DocBin |
A collection of Doc objects for efficient binary serialization. Also used for training data. |
Example |
A collection of training annotations, containing two Doc objects: the reference data and the predictions. |
Language |
Processing class that turns text into Doc objects. Different languages implement their own subclasses of it. The variable is typically called nlp . |
Lexeme |
An entry in the vocabulary. It's a word type with no context, as opposed to a word token. It therefore has no part-of-speech tag, dependency parse etc. |
Span |
A slice from a Doc object. |
SpanGroup |
A named collection of spans belonging to a Doc . |
Token |
An individual token — i.e. a word, punctuation symbol, whitespace, etc. |
Processing pipeline
The processing pipeline consists of one or more pipeline components that are
called on the Doc
in order. The tokenizer runs before the components. Pipeline
components can be added using Language.add_pipe
.
They can contain a statistical model and trained weights, or only make
rule-based modifications to the Doc
. spaCy provides a range of built-in
components for different language processing tasks and also allows adding
custom components.
Name | Description |
---|---|
AttributeRuler |
Set token attributes using matcher rules. |
DependencyParser |
Predict syntactic dependencies. |
EditTreeLemmatizer |
Predict base forms of words. |
EntityLinker |
Disambiguate named entities to nodes in a knowledge base. |
EntityRecognizer |
Predict named entities, e.g. persons or products. |
EntityRuler |
Add entity spans to the Doc using token-based rules or exact phrase matches. |
Lemmatizer |
Determine the base forms of words using rules and lookups. |
Morphologizer |
Predict morphological features and coarse-grained part-of-speech tags. |
SentenceRecognizer |
Predict sentence boundaries. |
Sentencizer |
Implement rule-based sentence boundary detection that doesn't require the dependency parse. |
Tagger |
Predict part-of-speech tags. |
TextCategorizer |
Predict categories or labels over the whole document. |
Tok2Vec |
Apply a "token-to-vector" model and set its outputs. |
Tokenizer |
Segment raw text and create Doc objects from the words. |
TrainablePipe |
Class that all trainable pipeline components inherit from. |
Transformer |
Use a transformer model and set its outputs. |
Other functions | Automatically apply something to the Doc , e.g. to merge spans of tokens. |
Matchers
Matchers help you find and extract information from Doc
objects
based on match patterns describing the sequences you're looking for. A matcher
operates on a Doc
and gives you access to the matched tokens in context.
Name | Description |
---|---|
DependencyMatcher |
Match sequences of tokens based on dependency trees using Semgrex operators. |
Matcher |
Match sequences of tokens, based on pattern rules, similar to regular expressions. |
PhraseMatcher |
Match sequences of tokens based on phrases. |
Other classes
Name | Description |
---|---|
Corpus |
Class for managing annotated corpora for training and evaluation data. |
KnowledgeBase |
Abstract base class for storage and retrieval of data for entity linking. |
InMemoryLookupKB |
Implementation of KnowledgeBase storing all data in memory. |
Candidate |
Object associating a textual mention with a specific entity contained in a KnowledgeBase . |
Lookups |
Container for convenient access to large lookup tables and dictionaries. |
MorphAnalysis |
A morphological analysis. |
Morphology |
Store morphological analyses and map them to and from hash values. |
Scorer |
Compute evaluation scores. |
StringStore |
Map strings to and from hash values. |
Vectors |
Container class for vector data keyed by string. |
Vocab |
The shared vocabulary that stores strings and gives you access to Lexeme objects. |