29 KiB
title | teaser | tag | source | new | api_base_class | api_string_name |
---|---|---|---|---|---|---|
Transformer | Pipeline component for multi-task learning with transformer models | class | github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py | 3 | /api/pipe | transformer |
Installation
$ pip install spacy-transformers
This component is available via the extension package
spacy-transformers
. It
exposes the component via entry points, so if you have the package installed,
using factory = "transformer"
in your
training config or nlp.add_pipe("transformer")
will
work out-of-the-box.
This pipeline component lets you use transformer models in your pipeline.
Supports all models that are available via the
HuggingFace transformers
library.
Usually you will connect subsequent components to the shared transformer using
the TransformerListener layer. This
works similarly to spaCy's Tok2Vec component and
Tok2VecListener sublayer.
The component assigns the output of the transformer to the Doc
's extension
attributes. We also calculate an alignment between the word-piece tokens and the
spaCy tokenization, so that we can use the last hidden states to set the
Doc.tensor
attribute. When multiple word-piece tokens align to the same spaCy
token, the spaCy token receives the sum of their values. To access the values,
you can use the custom Doc._.trf_data
attribute. The
package also adds the function registries @span_getters
and
@annotation_setters
with several built-in registered
functions. For more details, see the usage documentation.
Config and implementation
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
config
argument on nlp.add_pipe
or in your
config.cfg
for training. See the
model architectures documentation for details on the
architectures and their arguments and hyperparameters.
Example
from spacy_transformers import Transformer, DEFAULT_CONFIG nlp.add_pipe("transformer", config=DEFAULT_CONFIG)
Setting | Type | Description | Default |
---|---|---|---|
max_batch_items |
int | Maximum size of a padded batch. | 4096 |
annotation_setter |
Callable | Function that takes a batch of Doc objects and a FullTransformerBatch and can set additional annotations on the Doc . The Doc._.transformer_data attribute is set prior to calling the callback. By default, no additional annotations are set. |
null_annotation_setter |
model |
Model |
Input: List[Doc] . Output: FullTransformerBatch . The Thinc Model wrapping the transformer. |
TransformerModel |
https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py
Transformer.__init__
Example
# Construction via add_pipe with default model trf = nlp.add_pipe("transformer") # Construction via add_pipe with custom config config = { "model": { "@architectures": "spacy-transformers.TransformerModel.v1", "name": "bert-base-uncased", "tokenizer_config": {"use_fast": True} } } trf = nlp.add_pipe("transformer", config=config) # Construction from class from spacy_transformers import Transformer trf = Transformer(nlp.vocab, model)
Construct a Transformer
component. One or more subsequent spaCy components can
use the transformer outputs as features in its model, with gradients
backpropagated to the single shared weights. The activations from the
transformer are saved in the Doc._.trf_data
extension
attribute. You can also provide a callback to set additional annotations. In
your application, you would normally use a shortcut for this and instantiate the
component using its string name and nlp.add_pipe
.
Name | Type | Description |
---|---|---|
vocab |
Vocab |
The shared vocabulary. |
model |
Model |
Input: List[Doc] . Output: FullTransformerBatch . The Thinc Model wrapping the transformer. Usually you will want to use the TransformerModel layer for this. |
annotation_setter |
Callable |
Function that takes a batch of Doc objects and a FullTransformerBatch and can set additional annotations on the Doc . The Doc._.transformer_data attribute is set prior to calling the callback. By default, no additional annotations are set. |
keyword-only | ||
name |
str | String name of the component instance. Used to add entries to the losses during training. |
max_batch_items |
int | Maximum size of a padded batch. Defaults to 128*32 . |
Transformer.__call__
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the nlp
object is called on a text
and all pipeline components are applied to the Doc
in order. Both
__call__
and pipe
delegate
to the predict
and
set_annotations
methods.
Example
doc = nlp("This is a sentence.") trf = nlp.add_pipe("transformer") # This usually happens under the hood processed = transformer(doc)
Name | Type | Description |
---|---|---|
doc |
Doc |
The document to process. |
RETURNS | Doc |
The processed document. |
Transformer.pipe
Apply the pipe to a stream of documents. This usually happens under the hood
when the nlp
object is called on a text and all pipeline components are
applied to the Doc
in order. Both __call__
and
pipe
delegate to the
predict
and
set_annotations
methods.
Example
trf = nlp.add_pipe("transformer") for doc in trf.pipe(docs, batch_size=50): pass
Name | Type | Description |
---|---|---|
stream |
Iterable[Doc] |
A stream of documents. |
keyword-only | ||
batch_size |
int | The number of documents to buffer. Defaults to 128 . |
YIELDS | Doc |
The processed documents in order. |
Transformer.begin_training
Initialize the pipe for training, using data examples if available. Returns an
Optimizer
object.
Example
trf = nlp.add_pipe("transformer") optimizer = trf.begin_training(pipeline=nlp.pipeline)
Name | Type | Description |
---|---|---|
get_examples |
Callable[[], Iterable[Example]] |
Optional function that returns gold-standard annotations in the form of Example objects. |
keyword-only | ||
pipeline |
List[Tuple[str, Callable]] |
Optional list of pipeline components that this component is part of. |
sgd |
Optimizer |
An optional optimizer. Will be created via create_optimizer if not set. |
RETURNS | Optimizer |
The optimizer. |
Transformer.predict
Apply the pipeline's model to a batch of docs, without modifying them.
Example
trf = nlp.add_pipe("transformer") scores = trf.predict([doc1, doc2])
Name | Type | Description |
---|---|---|
docs |
Iterable[Doc] |
The documents to predict. |
RETURNS | - | The model's prediction for each document. |
Transformer.set_annotations
Assign the extracted features to the Doc objects. By default, the
TransformerData
object is written to the
Doc._.trf_data
attribute. Your annotation_setter
callback is then called, if provided.
Example
trf = nlp.add_pipe("transformer") scores = trf.predict(docs) trf.set_annotations(docs, scores)
Name | Type | Description |
---|---|---|
docs |
Iterable[Doc] |
The documents to modify. |
scores |
- | The scores to set, produced by Transformer.predict . |
Transformer.update
Prepare for an update to the transformer. Like the Tok2Vec
component, the Transformer
component is unusual in that it does not receive
"gold standard" annotations to calculate a weight update. The optimal output of
the transformer data is unknown – it's a hidden layer inside the network that is
updated by backpropagating from output layers.
The Transformer
component therefore does not perform a weight update
during its own update
method. Instead, it runs its transformer model and
communicates the output and the backpropagation callback to any downstream
components that have been connected to it via the
TransformerListener sublayer. If there
are multiple listeners, the last layer will actually backprop to the transformer
and call the optimizer, while the others simply increment the gradients.
Example
trf = nlp.add_pipe("transformer") optimizer = nlp.begin_training() losses = trf.update(examples, sgd=optimizer)
Name | Type | Description |
---|---|---|
examples |
Iterable[Example] |
A batch of Example objects. Only the Example.predicted Doc object is used, the reference Doc is ignored. |
keyword-only | ||
drop |
float | The dropout rate. |
set_annotations |
bool | Whether or not to update the Example objects with the predictions, delegating to set_annotations . |
sgd |
Optimizer |
The optimizer. |
losses |
Dict[str, float] |
Optional record of the loss during training. Updated using the component name as the key. |
RETURNS | Dict[str, float] |
The updated losses dictionary. |
Transformer.create_optimizer
Create an optimizer for the pipeline component.
Example
trf = nlp.add_pipe("transformer") optimizer = trf.create_optimizer()
Name | Type | Description |
---|---|---|
RETURNS | Optimizer |
The optimizer. |
Transformer.use_params
Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored.
Example
trf = nlp.add_pipe("transformer") with trf.use_params(optimizer.averages): trf.to_disk("/best_model")
Name | Type | Description |
---|---|---|
params |
dict | The parameter values to use in the model. |
Transformer.to_disk
Serialize the pipe to disk.
Example
trf = nlp.add_pipe("transformer") trf.to_disk("/path/to/transformer")
Name | Type | Description |
---|---|---|
path |
str / Path |
A path to a directory, which will be created if it doesn't exist. Paths may be either strings or Path -like objects. |
keyword-only | ||
exclude |
Iterable[str] |
String names of serialization fields to exclude. |
Transformer.from_disk
Load the pipe from disk. Modifies the object in place and returns it.
Example
trf = nlp.add_pipe("transformer") trf.from_disk("/path/to/transformer")
Name | Type | Description |
---|---|---|
path |
str / Path |
A path to a directory. Paths may be either strings or Path -like objects. |
keyword-only | ||
exclude |
Iterable[str] |
String names of serialization fields to exclude. |
RETURNS | Tok2Vec |
The modified Tok2Vec object. |
Transformer.to_bytes
Example
trf = nlp.add_pipe("transformer") trf_bytes = trf.to_bytes()
Serialize the pipe to a bytestring.
Name | Type | Description |
---|---|---|
keyword-only | ||
exclude |
Iterable[str] |
String names of serialization fields to exclude. |
RETURNS | bytes | The serialized form of the Tok2Vec object. |
Transformer.from_bytes
Load the pipe from a bytestring. Modifies the object in place and returns it.
Example
trf_bytes = trf.to_bytes() trf = nlp.add_pipe("transformer") trf.from_bytes(trf_bytes)
Name | Type | Description |
---|---|---|
bytes_data |
bytes | The data to load from. |
keyword-only | ||
exclude |
Iterable[str] |
String names of serialization fields to exclude. |
RETURNS | Tok2Vec |
The Tok2Vec object. |
Serialization fields
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from
serialization by passing in the string names via the exclude
argument.
Example
data = trf.to_disk("/path", exclude=["vocab"])
Name | Description |
---|---|
vocab |
The shared Vocab . |
cfg |
The config file. You usually don't want to exclude this. |
model |
The binary model data. You usually don't want to exclude this. |
TransformerData
Transformer tokens and outputs for one Doc
object.
Name | Type | Description |
---|---|---|
tokens |
Dict |
|
tensors |
List[FloatsXd] |
|
align |
Ragged |
|
width |
int |
TransformerData.empty
Name | Type | Description |
---|---|---|
RETURNS | TransformerData |
FullTransformerBatch
Name | Type | Description |
---|---|---|
spans |
List[List[Span]] |
|
tokens |
transformers.BatchEncoding |
|
tensors |
List[torch.Tensor] |
|
align |
Ragged |
|
doc_data |
List[TransformerData] |
FullTransformerBatch.unsplit_by_doc
Name | Type | Description |
---|---|---|
arrays |
List[List[Floats3d]] |
|
RETURNS | FullTransformerBatch |
FullTransformerBatch.split_by_doc
Split a TransformerData
object that represents a batch into a list with one
TransformerData
per Doc
.
Name | Type | Description |
---|---|---|
RETURNS | List[TransformerData] |
Span getters
Span getters are functions that take a batch of Doc
objects and
return a lists of Span
objects for each doc, to be processed by
the transformer. This is used to manage long documents, by cutting them into
smaller sequences before running the transformer. The spans are allowed to
overlap, and you can also omit sections of the Doc if they are not relevant.
Span getters can be referenced in the [components.transformer.model.get_spans]
block of the config to customize the sequences processed by the transformer. You
can also register custom span getters using the @spacy.registry.span_getters
decorator.
Example
@spacy.registry.span_getters("sent_spans.v1") def configure_get_sent_spans() -> Callable: def get_sent_spans(docs: Iterable[Doc]) -> List[List[Span]]: return [list(doc.sents) for doc in docs] return get_sent_spans
Name | Type | Description |
---|---|---|
docs |
Iterable[Doc] |
A batch of Doc objects. |
RETURNS | List[List[Span]] |
The spans to process by the transformer. |
doc_spans.v1
Example config
[transformer.model.get_spans] @span_getters = "doc_spans.v1"
Create a span getter that uses the whole document as its spans. This is the best
approach if your Doc
objects already refer to relatively short
texts.
sent_spans.v1
Example config
[transformer.model.get_spans] @span_getters = "sent_spans.v1"
Create a span getter that uses sentence boundary markers to extract the spans.
This requires sentence boundaries to be set (e.g. by the
Sentencizer
), and may result in somewhat uneven batches,
depending on the sentence lengths. However, it does provide the transformer with
more meaningful windows to attend over.
strided_spans.v1
Example config
[transformer.model.get_spans] @span_getters = "strided_spans.v1" window = 128 stride = 96
Create a span getter for strided spans. If you set the window
and stride
to
the same value, the spans will cover each token once. Setting stride
lower
than window
will allow for an overlap, so that some tokens are counted twice.
This can be desirable, because it allows all tokens to have both a left and
right context.
Name | Type | Description |
---|---|---|
window |
int | The window size. |
stride |
int | The stride size. |
Annotation setters
Annotation setters are functions that that take a batch of Doc
objects and a
FullTransformerBatch
and can set
additional annotations on the Doc
, e.g. to set custom or built-in attributes.
You can register custom annotation setters using the
@registry.annotation_setters
decorator.
Example
@registry.annotation_setters("spacy-transformer.null_annotation_setter.v1") def configure_null_annotation_setter() -> Callable: def setter(docs: List[Doc], trf_data: FullTransformerBatch) -> None: pass return setter
Name | Type | Description |
---|---|---|
docs |
List[Doc] |
A batch of Doc objects. |
trf_data |
FullTransformerBatch |
The transformers data for the batch. |
The following built-in functions are available:
Name | Description |
---|---|
spacy-transformer.null_annotation_setter.v1 |
Don't set any additional annotations. |
Custom attributes
The component sets the following custom extension attributes:
Name | Type | Description |
---|---|---|
Doc.trf_data |
TransformerData |
Transformer tokens and outputs for the Doc object. |