Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
# cython: infer_types=True
|
|
|
|
# cython: profile=True
|
2017-04-15 10:05:47 +00:00
|
|
|
# coding: utf8
|
|
|
|
from __future__ import unicode_literals
|
|
|
|
|
2017-05-06 12:22:20 +00:00
|
|
|
from thinc.api import chain, layerize, with_getitem
|
|
|
|
from thinc.neural import Model, Softmax
|
2017-05-07 16:04:24 +00:00
|
|
|
import numpy
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
cimport numpy as np
|
2017-05-16 14:17:30 +00:00
|
|
|
import cytoolz
|
2017-05-18 09:29:51 +00:00
|
|
|
import util
|
2017-05-29 13:40:45 +00:00
|
|
|
from collections import OrderedDict
|
2017-06-01 17:18:36 +00:00
|
|
|
import ujson
|
2017-06-02 15:18:37 +00:00
|
|
|
import msgpack
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-05-20 18:23:05 +00:00
|
|
|
from thinc.api import add, layerize, chain, clone, concatenate, with_flatten
|
2017-05-16 14:17:30 +00:00
|
|
|
from thinc.neural import Model, Maxout, Softmax, Affine
|
|
|
|
from thinc.neural._classes.hash_embed import HashEmbed
|
|
|
|
from thinc.neural.util import to_categorical
|
|
|
|
|
2017-06-05 13:40:03 +00:00
|
|
|
from thinc.neural.pooling import Pooling, max_pool, mean_pool
|
|
|
|
from thinc.neural._classes.difference import Siamese, CauchySimilarity
|
|
|
|
|
2017-05-16 14:17:30 +00:00
|
|
|
from thinc.neural._classes.convolution import ExtractWindow
|
|
|
|
from thinc.neural._classes.resnet import Residual
|
|
|
|
from thinc.neural._classes.batchnorm import BatchNorm as BN
|
2017-05-06 12:22:20 +00:00
|
|
|
|
2017-05-08 12:53:45 +00:00
|
|
|
from .tokens.doc cimport Doc
|
2017-05-16 09:21:59 +00:00
|
|
|
from .syntax.parser cimport Parser as LinearParser
|
|
|
|
from .syntax.nn_parser cimport Parser as NeuralParser
|
2017-05-13 22:55:01 +00:00
|
|
|
from .syntax.parser import get_templates as get_feature_templates
|
2017-03-11 13:00:20 +00:00
|
|
|
from .syntax.beam_parser cimport BeamParser
|
2016-10-15 23:47:12 +00:00
|
|
|
from .syntax.ner cimport BiluoPushDown
|
|
|
|
from .syntax.arc_eager cimport ArcEager
|
2016-10-16 19:34:57 +00:00
|
|
|
from .tagger import Tagger
|
2017-05-17 10:04:50 +00:00
|
|
|
from .syntax.stateclass cimport StateClass
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
from .gold cimport GoldParse
|
2017-05-17 10:04:50 +00:00
|
|
|
from .morphology cimport Morphology
|
|
|
|
from .vocab cimport Vocab
|
2017-05-22 10:17:44 +00:00
|
|
|
from .syntax import nonproj
|
2017-06-01 17:18:36 +00:00
|
|
|
from .compat import json_dumps
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-17 10:04:50 +00:00
|
|
|
from .attrs import ID, LOWER, PREFIX, SUFFIX, SHAPE, TAG, DEP, POS
|
2017-05-19 18:26:36 +00:00
|
|
|
from ._ml import rebatch, Tok2Vec, flatten, get_col, doc2feats
|
2017-07-19 22:18:15 +00:00
|
|
|
from ._ml import build_text_classifier
|
2017-05-17 10:04:50 +00:00
|
|
|
from .parts_of_speech import X
|
2016-10-15 23:47:12 +00:00
|
|
|
|
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
class BaseThincComponent(object):
|
|
|
|
name = None
|
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, *shape, **kwargs):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
|
|
|
scores = self.predict([doc])
|
|
|
|
self.set_annotations([doc], scores)
|
|
|
|
return doc
|
|
|
|
|
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
|
|
|
docs = list(docs)
|
|
|
|
scores = self.predict(docs)
|
|
|
|
self.set_annotations(docs, scores)
|
|
|
|
yield from docs
|
|
|
|
|
|
|
|
def predict(self, docs):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def set_annotations(self, docs, scores):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def update(self, docs_tensors, golds, state=None, drop=0., sgd=None, losses=None):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
|
|
|
raise NotImplementedError
|
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, gold_tuples=tuple(), pipeline=None):
|
2017-07-19 22:18:15 +00:00
|
|
|
token_vector_width = pipeline[0].model.nO
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(1, token_vector_width)
|
|
|
|
|
|
|
|
def use_params(self, params):
|
|
|
|
with self.model.use_params(params):
|
|
|
|
yield
|
|
|
|
|
|
|
|
def to_bytes(self, **exclude):
|
|
|
|
serialize = OrderedDict((
|
|
|
|
('model', lambda: self.model.to_bytes()),
|
|
|
|
('vocab', lambda: self.vocab.to_bytes())
|
|
|
|
))
|
|
|
|
return util.to_bytes(serialize, exclude)
|
|
|
|
|
|
|
|
def from_bytes(self, bytes_data, **exclude):
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model()
|
|
|
|
deserialize = OrderedDict((
|
|
|
|
('model', lambda b: self.model.from_bytes(b)),
|
|
|
|
('vocab', lambda b: self.vocab.from_bytes(b))
|
|
|
|
))
|
|
|
|
util.from_bytes(bytes_data, deserialize, exclude)
|
|
|
|
return self
|
|
|
|
|
|
|
|
def to_disk(self, path, **exclude):
|
|
|
|
serialize = OrderedDict((
|
|
|
|
('model', lambda p: p.open('wb').write(self.model.to_bytes())),
|
2017-07-22 22:33:43 +00:00
|
|
|
('vocab', lambda p: self.vocab.to_disk(p)),
|
|
|
|
('cfg', lambda p: p.open('w').write(json_dumps(self.cfg)))
|
2017-07-19 22:18:15 +00:00
|
|
|
))
|
|
|
|
util.to_disk(path, serialize, exclude)
|
|
|
|
|
|
|
|
def from_disk(self, path, **exclude):
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model()
|
|
|
|
deserialize = OrderedDict((
|
|
|
|
('model', lambda p: self.model.from_bytes(p.open('rb').read())),
|
2017-07-22 22:33:43 +00:00
|
|
|
('vocab', lambda p: self.vocab.from_disk(p)),
|
2017-07-23 12:11:07 +00:00
|
|
|
('cfg', lambda p: self.cfg.update(_load_cfg(p)))
|
2017-07-19 22:18:15 +00:00
|
|
|
))
|
|
|
|
util.from_disk(path, deserialize, exclude)
|
|
|
|
return self
|
|
|
|
|
|
|
|
|
2017-07-23 12:11:07 +00:00
|
|
|
def _load_cfg(path):
|
|
|
|
if path.exists():
|
|
|
|
return ujson.load(path.open())
|
|
|
|
else:
|
|
|
|
return {}
|
|
|
|
|
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
class TokenVectorEncoder(BaseThincComponent):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Assign position-sensitive vectors to tokens, using a CNN or RNN."""
|
2017-05-31 11:42:39 +00:00
|
|
|
name = 'tensorizer'
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-15 19:46:08 +00:00
|
|
|
@classmethod
|
2017-05-23 20:20:16 +00:00
|
|
|
def Model(cls, width=128, embed_size=7500, **cfg):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Create a new statistical model for the class.
|
|
|
|
|
|
|
|
width (int): Output size of the model.
|
|
|
|
embed_size (int): Number of vectors in the embedding table.
|
|
|
|
**cfg: Config parameters.
|
|
|
|
RETURNS (Model): A `thinc.neural.Model` or similar instance.
|
|
|
|
"""
|
2017-05-18 09:29:51 +00:00
|
|
|
width = util.env_opt('token_vector_width', width)
|
|
|
|
embed_size = util.env_opt('embed_size', embed_size)
|
2017-05-16 14:17:30 +00:00
|
|
|
return Tok2Vec(width, embed_size, preprocess=None)
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-15 19:46:08 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Construct a new statistical model. Weights are not allocated on
|
|
|
|
initialisation.
|
|
|
|
|
|
|
|
vocab (Vocab): A `Vocab` instance. The model must share the same `Vocab`
|
|
|
|
instance with the `Doc` objects it will process.
|
|
|
|
model (Model): A `Model` instance or `True` allocate one later.
|
|
|
|
**cfg: Config parameters.
|
|
|
|
|
|
|
|
EXAMPLE:
|
|
|
|
>>> from spacy.pipeline import TokenVectorEncoder
|
|
|
|
>>> tok2vec = TokenVectorEncoder(nlp.vocab)
|
|
|
|
>>> tok2vec.model = tok2vec.Model(128, 5000)
|
|
|
|
"""
|
2017-05-15 19:46:08 +00:00
|
|
|
self.vocab = vocab
|
|
|
|
self.doc2feats = doc2feats()
|
2017-05-18 09:29:51 +00:00
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-05-17 11:13:14 +00:00
|
|
|
|
2017-05-28 13:11:58 +00:00
|
|
|
def __call__(self, doc):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Add context-sensitive vectors to a `Doc`, e.g. from a CNN or LSTM
|
|
|
|
model. Vectors are set to the `Doc.tensor` attribute.
|
|
|
|
|
|
|
|
docs (Doc or iterable): One or more documents to add vectors to.
|
|
|
|
RETURNS (dict or None): Intermediate computations.
|
|
|
|
"""
|
2017-05-28 13:11:58 +00:00
|
|
|
tokvecses = self.predict([doc])
|
|
|
|
self.set_annotations([doc], tokvecses)
|
|
|
|
return doc
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-05-18 13:30:59 +00:00
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Process `Doc` objects as a stream.
|
|
|
|
|
|
|
|
stream (iterator): A sequence of `Doc` objects to process.
|
|
|
|
batch_size (int): Number of `Doc` objects to group.
|
|
|
|
n_threads (int): Number of threads.
|
2017-05-21 18:46:23 +00:00
|
|
|
YIELDS (iterator): A sequence of `Doc` objects, in order of input.
|
2017-05-18 22:00:02 +00:00
|
|
|
"""
|
2017-05-19 18:26:36 +00:00
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
2017-05-21 22:52:01 +00:00
|
|
|
docs = list(docs)
|
2017-05-20 18:23:05 +00:00
|
|
|
tokvecses = self.predict(docs)
|
|
|
|
self.set_annotations(docs, tokvecses)
|
2017-05-19 18:26:36 +00:00
|
|
|
yield from docs
|
2017-05-18 09:29:51 +00:00
|
|
|
|
2017-05-16 14:17:30 +00:00
|
|
|
def predict(self, docs):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Return a single tensor for a batch of documents.
|
|
|
|
|
|
|
|
docs (iterable): A sequence of `Doc` objects.
|
|
|
|
RETURNS (object): Vector representations for each token in the documents.
|
|
|
|
"""
|
2017-05-16 14:17:30 +00:00
|
|
|
feats = self.doc2feats(docs)
|
|
|
|
tokvecs = self.model(feats)
|
|
|
|
return tokvecs
|
|
|
|
|
2017-05-20 18:23:05 +00:00
|
|
|
def set_annotations(self, docs, tokvecses):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Set the tensor attribute for a batch of documents.
|
|
|
|
|
|
|
|
docs (iterable): A sequence of `Doc` objects.
|
|
|
|
tokvecs (object): Vector representation for each token in the documents.
|
|
|
|
"""
|
2017-05-21 22:52:01 +00:00
|
|
|
for doc, tokvecs in zip(docs, tokvecses):
|
|
|
|
assert tokvecs.shape[0] == len(doc)
|
|
|
|
doc.tensor = tokvecs
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2017-05-25 01:09:51 +00:00
|
|
|
def update(self, docs, golds, state=None, drop=0., sgd=None, losses=None):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Update the model.
|
|
|
|
|
|
|
|
docs (iterable): A batch of `Doc` objects.
|
|
|
|
golds (iterable): A batch of `GoldParse` objects.
|
|
|
|
drop (float): The droput rate.
|
2017-05-21 11:17:40 +00:00
|
|
|
sgd (callable): An optimizer.
|
2017-05-18 22:00:02 +00:00
|
|
|
RETURNS (dict): Results from the update.
|
|
|
|
"""
|
2017-05-16 14:17:30 +00:00
|
|
|
if isinstance(docs, Doc):
|
|
|
|
docs = [docs]
|
|
|
|
feats = self.doc2feats(docs)
|
|
|
|
tokvecs, bp_tokvecs = self.model.begin_update(feats, drop=drop)
|
2017-05-19 18:26:36 +00:00
|
|
|
return tokvecs, bp_tokvecs
|
2017-05-06 12:22:20 +00:00
|
|
|
|
2017-05-16 14:17:30 +00:00
|
|
|
def get_loss(self, docs, golds, scores):
|
2017-05-18 22:00:02 +00:00
|
|
|
# TODO: implement
|
2017-05-16 14:17:30 +00:00
|
|
|
raise NotImplementedError
|
2017-05-06 12:22:20 +00:00
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, gold_tuples=tuple(), pipeline=None):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Allocate models, pre-process training data and acquire a trainer and
|
|
|
|
optimizer.
|
|
|
|
|
|
|
|
gold_tuples (iterable): Gold-standard training data.
|
|
|
|
pipeline (list): The pipeline the model is part of.
|
|
|
|
"""
|
2017-05-18 09:29:51 +00:00
|
|
|
self.doc2feats = doc2feats()
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model()
|
|
|
|
|
2017-05-28 23:37:57 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
class NeuralTagger(BaseThincComponent):
|
2017-06-01 15:37:53 +00:00
|
|
|
name = 'tagger'
|
2017-07-22 22:52:47 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-05-16 14:17:30 +00:00
|
|
|
self.vocab = vocab
|
2017-05-17 10:04:50 +00:00
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-05-19 18:26:36 +00:00
|
|
|
def __call__(self, doc):
|
2017-05-23 08:06:53 +00:00
|
|
|
tags = self.predict([doc.tensor])
|
2017-05-16 14:17:30 +00:00
|
|
|
self.set_annotations([doc], tags)
|
2017-05-28 13:11:58 +00:00
|
|
|
return doc
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
2017-05-19 18:26:36 +00:00
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
2017-05-21 14:05:34 +00:00
|
|
|
tokvecs = [d.tensor for d in docs]
|
2017-05-19 18:26:36 +00:00
|
|
|
tag_ids = self.predict(tokvecs)
|
2017-05-16 14:17:30 +00:00
|
|
|
self.set_annotations(docs, tag_ids)
|
2017-05-19 18:26:36 +00:00
|
|
|
yield from docs
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
def predict(self, tokvecs):
|
|
|
|
scores = self.model(tokvecs)
|
2017-05-21 14:05:34 +00:00
|
|
|
scores = self.model.ops.flatten(scores)
|
2017-05-13 22:20:23 +00:00
|
|
|
guesses = scores.argmax(axis=1)
|
|
|
|
if not isinstance(guesses, numpy.ndarray):
|
|
|
|
guesses = guesses.get()
|
2017-05-21 14:05:34 +00:00
|
|
|
guesses = self.model.ops.unflatten(guesses,
|
|
|
|
[tv.shape[0] for tv in tokvecs])
|
2017-05-16 14:17:30 +00:00
|
|
|
return guesses
|
|
|
|
|
2017-05-18 09:29:51 +00:00
|
|
|
def set_annotations(self, docs, batch_tag_ids):
|
2017-05-16 14:17:30 +00:00
|
|
|
if isinstance(docs, Doc):
|
|
|
|
docs = [docs]
|
|
|
|
cdef Doc doc
|
|
|
|
cdef int idx = 0
|
2017-05-18 09:29:51 +00:00
|
|
|
cdef Vocab vocab = self.vocab
|
2017-05-08 12:53:45 +00:00
|
|
|
for i, doc in enumerate(docs):
|
2017-05-21 14:05:34 +00:00
|
|
|
doc_tag_ids = batch_tag_ids[i]
|
2017-05-18 09:29:51 +00:00
|
|
|
for j, tag_id in enumerate(doc_tag_ids):
|
2017-06-04 20:52:42 +00:00
|
|
|
# Don't clobber preset POS tags
|
|
|
|
if doc.c[j].tag == 0 and doc.c[j].pos == 0:
|
|
|
|
vocab.morphology.assign_tag_id(&doc.c[j], tag_id)
|
2017-05-08 12:53:45 +00:00
|
|
|
idx += 1
|
2017-06-04 23:35:07 +00:00
|
|
|
doc.is_tagged = True
|
2017-05-08 12:53:45 +00:00
|
|
|
|
2017-05-25 01:09:51 +00:00
|
|
|
def update(self, docs_tokvecs, golds, drop=0., sgd=None, losses=None):
|
2017-05-19 18:26:36 +00:00
|
|
|
docs, tokvecs = docs_tokvecs
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
if self.model.nI is None:
|
2017-05-20 18:23:05 +00:00
|
|
|
self.model.nI = tokvecs[0].shape[1]
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2017-05-16 14:17:30 +00:00
|
|
|
tag_scores, bp_tag_scores = self.model.begin_update(tokvecs, drop=drop)
|
|
|
|
loss, d_tag_scores = self.get_loss(docs, golds, tag_scores)
|
2017-05-18 09:29:51 +00:00
|
|
|
|
|
|
|
d_tokvecs = bp_tag_scores(d_tag_scores, sgd=sgd)
|
2017-05-19 18:26:36 +00:00
|
|
|
return d_tokvecs
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
2017-05-20 18:23:05 +00:00
|
|
|
scores = self.model.ops.flatten(scores)
|
2017-05-18 09:29:51 +00:00
|
|
|
tag_index = {tag: i for i, tag in enumerate(self.vocab.morphology.tag_names)}
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-18 09:29:51 +00:00
|
|
|
cdef int idx = 0
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
correct = numpy.zeros((scores.shape[0],), dtype='i')
|
2017-05-19 18:26:36 +00:00
|
|
|
guesses = scores.argmax(axis=1)
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
for gold in golds:
|
|
|
|
for tag in gold.tags:
|
2017-05-19 18:26:36 +00:00
|
|
|
if tag is None:
|
|
|
|
correct[idx] = guesses[idx]
|
|
|
|
else:
|
|
|
|
correct[idx] = tag_index[tag]
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
idx += 1
|
2017-05-18 13:30:59 +00:00
|
|
|
correct = self.model.ops.xp.array(correct, dtype='i')
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
|
2017-05-27 23:32:46 +00:00
|
|
|
d_scores /= d_scores.shape[0]
|
2017-05-18 09:29:51 +00:00
|
|
|
loss = (d_scores**2).sum()
|
2017-05-20 18:23:05 +00:00
|
|
|
d_scores = self.model.ops.unflatten(d_scores, [len(d) for d in docs])
|
2017-05-18 13:30:59 +00:00
|
|
|
return float(loss), d_scores
|
2016-10-15 23:47:12 +00:00
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, gold_tuples=tuple(), pipeline=None):
|
2017-05-18 13:30:59 +00:00
|
|
|
orig_tag_map = dict(self.vocab.morphology.tag_map)
|
|
|
|
new_tag_map = {}
|
2017-05-17 10:04:50 +00:00
|
|
|
for raw_text, annots_brackets in gold_tuples:
|
|
|
|
for annots, brackets in annots_brackets:
|
|
|
|
ids, words, tags, heads, deps, ents = annots
|
|
|
|
for tag in tags:
|
2017-05-18 13:30:59 +00:00
|
|
|
if tag in orig_tag_map:
|
|
|
|
new_tag_map[tag] = orig_tag_map[tag]
|
|
|
|
else:
|
|
|
|
new_tag_map[tag] = {POS: X}
|
2017-06-05 01:16:30 +00:00
|
|
|
if 'SP' not in new_tag_map:
|
|
|
|
new_tag_map['SP'] = orig_tag_map.get('SP', {POS: X})
|
2017-05-17 10:04:50 +00:00
|
|
|
cdef Vocab vocab = self.vocab
|
2017-06-01 08:04:36 +00:00
|
|
|
if new_tag_map:
|
|
|
|
vocab.morphology = Morphology(vocab.strings, new_tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
vocab.morphology.lemmatizer,
|
|
|
|
exc=vocab.morphology.exc)
|
2017-05-19 18:26:36 +00:00
|
|
|
token_vector_width = pipeline[0].model.nO
|
2017-05-29 18:23:47 +00:00
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags, token_vector_width)
|
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, n_tags, token_vector_width):
|
|
|
|
return with_flatten(
|
2017-05-25 01:09:51 +00:00
|
|
|
chain(Maxout(token_vector_width, token_vector_width),
|
2017-05-29 18:23:47 +00:00
|
|
|
Softmax(n_tags, token_vector_width)))
|
2017-05-18 13:30:59 +00:00
|
|
|
|
|
|
|
def use_params(self, params):
|
|
|
|
with self.model.use_params(params):
|
|
|
|
yield
|
|
|
|
|
2017-05-29 08:14:20 +00:00
|
|
|
def to_bytes(self, **exclude):
|
2017-05-29 22:53:06 +00:00
|
|
|
serialize = OrderedDict((
|
2017-06-01 09:55:49 +00:00
|
|
|
('model', lambda: self.model.to_bytes()),
|
2017-06-02 15:18:37 +00:00
|
|
|
('vocab', lambda: self.vocab.to_bytes()),
|
2017-06-02 15:29:21 +00:00
|
|
|
('tag_map', lambda: msgpack.dumps(self.vocab.morphology.tag_map,
|
|
|
|
use_bin_type=True,
|
|
|
|
encoding='utf8'))
|
2017-05-29 22:53:06 +00:00
|
|
|
))
|
2017-05-29 08:14:20 +00:00
|
|
|
return util.to_bytes(serialize, exclude)
|
|
|
|
|
|
|
|
def from_bytes(self, bytes_data, **exclude):
|
2017-05-29 18:23:47 +00:00
|
|
|
def load_model(b):
|
|
|
|
if self.model is True:
|
|
|
|
token_vector_width = util.env_opt('token_vector_width', 128)
|
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags, token_vector_width)
|
2017-06-01 17:18:36 +00:00
|
|
|
self.model.from_bytes(b)
|
2017-06-02 15:18:37 +00:00
|
|
|
|
|
|
|
def load_tag_map(b):
|
2017-06-02 15:29:21 +00:00
|
|
|
tag_map = msgpack.loads(b, encoding='utf8')
|
2017-06-02 15:18:37 +00:00
|
|
|
self.vocab.morphology = Morphology(
|
|
|
|
self.vocab.strings, tag_map=tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
lemmatizer=self.vocab.morphology.lemmatizer,
|
|
|
|
exc=self.vocab.morphology.exc)
|
|
|
|
|
2017-05-29 22:53:06 +00:00
|
|
|
deserialize = OrderedDict((
|
|
|
|
('vocab', lambda b: self.vocab.from_bytes(b)),
|
2017-06-02 15:18:37 +00:00
|
|
|
('tag_map', load_tag_map),
|
2017-05-29 22:53:06 +00:00
|
|
|
('model', lambda b: load_model(b)),
|
|
|
|
))
|
2017-05-29 18:23:47 +00:00
|
|
|
util.from_bytes(bytes_data, deserialize, exclude)
|
2017-05-29 08:14:20 +00:00
|
|
|
return self
|
|
|
|
|
2017-05-29 09:45:45 +00:00
|
|
|
def to_disk(self, path, **exclude):
|
2017-06-01 17:18:36 +00:00
|
|
|
serialize = OrderedDict((
|
|
|
|
('vocab', lambda p: self.vocab.to_disk(p)),
|
2017-06-02 15:29:21 +00:00
|
|
|
('tag_map', lambda p: p.open('wb').write(msgpack.dumps(
|
|
|
|
self.vocab.morphology.tag_map,
|
|
|
|
use_bin_type=True,
|
|
|
|
encoding='utf8'))),
|
2017-06-01 17:18:36 +00:00
|
|
|
('model', lambda p: p.open('wb').write(self.model.to_bytes())),
|
2017-07-22 22:33:43 +00:00
|
|
|
('cfg', lambda p: p.open('w').write(json_dumps(self.cfg)))
|
2017-06-01 17:18:36 +00:00
|
|
|
))
|
2017-05-29 09:45:45 +00:00
|
|
|
util.to_disk(path, serialize, exclude)
|
|
|
|
|
|
|
|
def from_disk(self, path, **exclude):
|
2017-06-01 17:18:36 +00:00
|
|
|
def load_model(p):
|
|
|
|
if self.model is True:
|
|
|
|
token_vector_width = util.env_opt('token_vector_width', 128)
|
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags, token_vector_width)
|
|
|
|
self.model.from_bytes(p.open('rb').read())
|
|
|
|
|
|
|
|
def load_tag_map(p):
|
2017-06-02 15:29:21 +00:00
|
|
|
with p.open('rb') as file_:
|
|
|
|
tag_map = msgpack.loads(file_.read(), encoding='utf8')
|
2017-06-01 17:18:36 +00:00
|
|
|
self.vocab.morphology = Morphology(
|
|
|
|
self.vocab.strings, tag_map=tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
lemmatizer=self.vocab.morphology.lemmatizer,
|
|
|
|
exc=self.vocab.morphology.exc)
|
2017-06-01 17:18:36 +00:00
|
|
|
|
|
|
|
deserialize = OrderedDict((
|
|
|
|
('vocab', lambda p: self.vocab.from_disk(p)),
|
|
|
|
('tag_map', load_tag_map),
|
|
|
|
('model', load_model),
|
2017-07-22 22:33:43 +00:00
|
|
|
('cfg', lambda p: self.cfg.update(ujson.load(p.open()))),
|
2017-06-01 17:18:36 +00:00
|
|
|
))
|
2017-05-29 09:45:45 +00:00
|
|
|
util.from_disk(path, deserialize, exclude)
|
|
|
|
return self
|
2017-05-29 08:14:20 +00:00
|
|
|
|
|
|
|
|
2017-05-21 22:52:30 +00:00
|
|
|
class NeuralLabeller(NeuralTagger):
|
|
|
|
name = 'nn_labeller'
|
2017-07-22 22:52:47 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-05-21 22:52:30 +00:00
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
|
|
|
|
|
|
|
@property
|
|
|
|
def labels(self):
|
|
|
|
return self.cfg.get('labels', {})
|
|
|
|
|
|
|
|
@labels.setter
|
|
|
|
def labels(self, value):
|
|
|
|
self.cfg['labels'] = value
|
2017-05-21 22:52:30 +00:00
|
|
|
|
|
|
|
def set_annotations(self, docs, dep_ids):
|
|
|
|
pass
|
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, gold_tuples=tuple(), pipeline=None):
|
2017-05-22 10:17:44 +00:00
|
|
|
gold_tuples = nonproj.preprocess_training_data(gold_tuples)
|
2017-05-21 22:52:30 +00:00
|
|
|
for raw_text, annots_brackets in gold_tuples:
|
|
|
|
for annots, brackets in annots_brackets:
|
|
|
|
ids, words, tags, heads, deps, ents = annots
|
|
|
|
for dep in deps:
|
|
|
|
if dep not in self.labels:
|
|
|
|
self.labels[dep] = len(self.labels)
|
|
|
|
token_vector_width = pipeline[0].model.nO
|
2017-05-29 18:23:47 +00:00
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(len(self.labels), token_vector_width)
|
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, n_tags, token_vector_width):
|
|
|
|
return with_flatten(
|
2017-05-25 01:09:51 +00:00
|
|
|
chain(Maxout(token_vector_width, token_vector_width),
|
2017-05-29 18:23:47 +00:00
|
|
|
Softmax(n_tags, token_vector_width)))
|
2017-05-21 22:52:30 +00:00
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
|
|
|
scores = self.model.ops.flatten(scores)
|
|
|
|
cdef int idx = 0
|
|
|
|
correct = numpy.zeros((scores.shape[0],), dtype='i')
|
|
|
|
guesses = scores.argmax(axis=1)
|
|
|
|
for gold in golds:
|
|
|
|
for tag in gold.labels:
|
2017-05-22 15:41:20 +00:00
|
|
|
if tag is None or tag not in self.labels:
|
2017-05-21 22:52:30 +00:00
|
|
|
correct[idx] = guesses[idx]
|
|
|
|
else:
|
|
|
|
correct[idx] = self.labels[tag]
|
|
|
|
idx += 1
|
|
|
|
correct = self.model.ops.xp.array(correct, dtype='i')
|
|
|
|
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
|
2017-05-27 23:32:46 +00:00
|
|
|
d_scores /= d_scores.shape[0]
|
2017-05-21 22:52:30 +00:00
|
|
|
loss = (d_scores**2).sum()
|
|
|
|
d_scores = self.model.ops.unflatten(d_scores, [len(d) for d in docs])
|
|
|
|
return float(loss), d_scores
|
|
|
|
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
class SimilarityHook(BaseThincComponent):
|
2017-06-05 13:40:03 +00:00
|
|
|
"""
|
|
|
|
Experimental
|
|
|
|
|
|
|
|
A pipeline component to install a hook for supervised similarity into
|
|
|
|
Doc objects. Requires a Tensorizer to pre-process documents. The similarity
|
|
|
|
model can be any object obeying the Thinc Model interface. By default,
|
|
|
|
the model concatenates the elementwise mean and elementwise max of the two
|
|
|
|
tensors, and compares them using the Cauchy-like similarity function
|
|
|
|
from Chen (2013):
|
|
|
|
|
|
|
|
similarity = 1. / (1. + (W * (vec1-vec2)**2).sum())
|
|
|
|
|
|
|
|
Where W is a vector of dimension weights, initialized to 1.
|
|
|
|
"""
|
|
|
|
name = 'similarity'
|
2017-07-22 22:52:47 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-06-05 13:40:03 +00:00
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, length):
|
|
|
|
return Siamese(Pooling(max_pool, mean_pool), CauchySimilarity(length))
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
|
|
|
'''Install similarity hook'''
|
|
|
|
doc.user_hooks['similarity'] = self.predict
|
|
|
|
return doc
|
|
|
|
|
|
|
|
def pipe(self, docs, **kwargs):
|
|
|
|
for doc in docs:
|
|
|
|
yield self(doc)
|
|
|
|
|
|
|
|
def predict(self, doc1, doc2):
|
|
|
|
return self.model.predict([(doc1.tensor, doc2.tensor)])
|
|
|
|
|
|
|
|
def update(self, doc1_tensor1_doc2_tensor2, golds, sgd=None, drop=0.):
|
|
|
|
doc1s, tensor1s, doc2s, tensor2s = doc1_tensor1_doc2_tensor2
|
|
|
|
sims, bp_sims = self.model.begin_update(zip(tensor1s, tensor2s),
|
|
|
|
drop=drop)
|
|
|
|
d_tensor1s, d_tensor2s = bp_sims(golds, sgd=sgd)
|
|
|
|
|
|
|
|
return d_tensor1s, d_tensor2s
|
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, _=tuple(), pipeline=None):
|
2017-06-05 13:40:03 +00:00
|
|
|
"""
|
|
|
|
Allocate model, using width from tensorizer in pipeline.
|
|
|
|
|
|
|
|
gold_tuples (iterable): Gold-standard training data.
|
|
|
|
pipeline (list): The pipeline the model is part of.
|
|
|
|
"""
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(pipeline[0].model.nO)
|
|
|
|
|
|
|
|
|
2017-07-21 23:14:07 +00:00
|
|
|
class TextCategorizer(BaseThincComponent):
|
|
|
|
name = 'textcat'
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
@classmethod
|
2017-07-22 18:04:43 +00:00
|
|
|
def Model(cls, nr_class=1, width=64, **cfg):
|
2017-07-19 22:18:15 +00:00
|
|
|
return build_text_classifier(nr_class, width, **cfg)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-07-22 22:33:43 +00:00
|
|
|
|
|
|
|
@property
|
|
|
|
def labels(self):
|
|
|
|
return self.cfg.get('labels', ['LABEL'])
|
|
|
|
|
|
|
|
@labels.setter
|
|
|
|
def labels(self, value):
|
|
|
|
self.cfg['labels'] = value
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def __call__(self, doc):
|
|
|
|
scores = self.predict([doc])
|
|
|
|
self.set_annotations([doc], scores)
|
|
|
|
return doc
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
|
|
|
docs = list(docs)
|
|
|
|
scores = self.predict(docs)
|
|
|
|
self.set_annotations(docs, scores)
|
|
|
|
yield from docs
|
|
|
|
|
|
|
|
def predict(self, docs):
|
|
|
|
scores = self.model(docs)
|
|
|
|
scores = self.model.ops.asarray(scores)
|
|
|
|
return scores
|
|
|
|
|
|
|
|
def set_annotations(self, docs, scores):
|
|
|
|
for i, doc in enumerate(docs):
|
2017-07-22 18:04:43 +00:00
|
|
|
for j, label in enumerate(self.labels):
|
2017-07-19 22:18:15 +00:00
|
|
|
doc.cats[label] = float(scores[i, j])
|
|
|
|
|
|
|
|
def update(self, docs_tensors, golds, state=None, drop=0., sgd=None, losses=None):
|
|
|
|
docs, tensors = docs_tensors
|
|
|
|
scores, bp_scores = self.model.begin_update(docs, drop=drop)
|
|
|
|
loss, d_scores = self.get_loss(docs, golds, scores)
|
|
|
|
d_tensors = bp_scores(d_scores, sgd=sgd)
|
|
|
|
if losses is not None:
|
|
|
|
losses.setdefault(self.name, 0.0)
|
|
|
|
losses[self.name] += loss
|
|
|
|
return d_tensors
|
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
|
|
|
truths = numpy.zeros((len(golds), len(self.labels)), dtype='f')
|
|
|
|
for i, gold in enumerate(golds):
|
|
|
|
for j, label in enumerate(self.labels):
|
|
|
|
truths[i, j] = label in gold.cats
|
|
|
|
truths = self.model.ops.asarray(truths)
|
|
|
|
d_scores = (scores-truths) / scores.shape[0]
|
|
|
|
mean_square_error = ((scores-truths)**2).sum(axis=1).mean()
|
|
|
|
return mean_square_error, d_scores
|
|
|
|
|
2017-07-22 18:04:43 +00:00
|
|
|
def begin_training(self, gold_tuples=tuple(), pipeline=None):
|
|
|
|
if pipeline:
|
|
|
|
token_vector_width = pipeline[0].model.nO
|
|
|
|
else:
|
|
|
|
token_vector_width = 64
|
2017-06-05 13:40:03 +00:00
|
|
|
if self.model is True:
|
2017-07-19 22:18:15 +00:00
|
|
|
self.model = self.Model(len(self.labels), token_vector_width)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
|
2017-05-16 09:21:59 +00:00
|
|
|
cdef class EntityRecognizer(LinearParser):
|
2017-05-21 11:32:15 +00:00
|
|
|
"""Annotate named entities on Doc objects."""
|
2016-10-16 19:34:57 +00:00
|
|
|
TransitionSystem = BiluoPushDown
|
2017-03-11 13:00:20 +00:00
|
|
|
|
2016-10-16 19:34:57 +00:00
|
|
|
feature_templates = get_feature_templates('ner')
|
2016-10-15 23:47:12 +00:00
|
|
|
|
2016-10-23 15:45:44 +00:00
|
|
|
def add_label(self, label):
|
2017-05-16 09:21:59 +00:00
|
|
|
LinearParser.add_label(self, label)
|
2016-10-23 15:45:44 +00:00
|
|
|
if isinstance(label, basestring):
|
|
|
|
label = self.vocab.strings[label]
|
|
|
|
|
2016-10-15 23:47:12 +00:00
|
|
|
|
2017-03-15 14:27:41 +00:00
|
|
|
cdef class BeamEntityRecognizer(BeamParser):
|
2017-05-21 11:32:15 +00:00
|
|
|
"""Annotate named entities on Doc objects."""
|
2017-03-15 14:27:41 +00:00
|
|
|
TransitionSystem = BiluoPushDown
|
|
|
|
|
|
|
|
feature_templates = get_feature_templates('ner')
|
2017-04-15 10:05:47 +00:00
|
|
|
|
2017-03-15 14:27:41 +00:00
|
|
|
def add_label(self, label):
|
2017-05-16 09:21:59 +00:00
|
|
|
LinearParser.add_label(self, label)
|
2017-03-15 14:27:41 +00:00
|
|
|
if isinstance(label, basestring):
|
|
|
|
label = self.vocab.strings[label]
|
|
|
|
|
|
|
|
|
2017-05-16 09:21:59 +00:00
|
|
|
cdef class DependencyParser(LinearParser):
|
2016-10-16 19:34:57 +00:00
|
|
|
TransitionSystem = ArcEager
|
|
|
|
feature_templates = get_feature_templates('basic')
|
2016-10-23 15:45:44 +00:00
|
|
|
|
|
|
|
def add_label(self, label):
|
2017-05-16 09:21:59 +00:00
|
|
|
LinearParser.add_label(self, label)
|
2016-10-23 15:45:44 +00:00
|
|
|
if isinstance(label, basestring):
|
|
|
|
label = self.vocab.strings[label]
|
|
|
|
|
2016-10-15 23:47:12 +00:00
|
|
|
|
2017-05-16 09:21:59 +00:00
|
|
|
cdef class NeuralDependencyParser(NeuralParser):
|
|
|
|
name = 'parser'
|
|
|
|
TransitionSystem = ArcEager
|
|
|
|
|
2017-05-27 20:46:06 +00:00
|
|
|
def __reduce__(self):
|
|
|
|
return (NeuralDependencyParser, (self.vocab, self.moves, self.model), None, None)
|
|
|
|
|
2017-05-16 09:21:59 +00:00
|
|
|
|
|
|
|
cdef class NeuralEntityRecognizer(NeuralParser):
|
2017-05-31 11:42:39 +00:00
|
|
|
name = 'ner'
|
2017-05-16 09:21:59 +00:00
|
|
|
TransitionSystem = BiluoPushDown
|
|
|
|
|
2017-05-17 10:04:50 +00:00
|
|
|
nr_feature = 6
|
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def predict_confidences(self, docs):
|
|
|
|
tensors = [d.tensor for d in docs]
|
|
|
|
samples = []
|
|
|
|
for i in range(10):
|
|
|
|
states = self.parse_batch(docs, tensors, drop=0.3)
|
|
|
|
for state in states:
|
|
|
|
samples.append(self._get_entities(state))
|
|
|
|
|
2017-05-27 20:46:06 +00:00
|
|
|
def __reduce__(self):
|
|
|
|
return (NeuralEntityRecognizer, (self.vocab, self.moves, self.model), None, None)
|
|
|
|
|
|
|
|
|
2017-03-15 14:27:41 +00:00
|
|
|
cdef class BeamDependencyParser(BeamParser):
|
|
|
|
TransitionSystem = ArcEager
|
|
|
|
|
|
|
|
feature_templates = get_feature_templates('basic')
|
|
|
|
|
|
|
|
def add_label(self, label):
|
2017-04-14 21:52:17 +00:00
|
|
|
Parser.add_label(self, label)
|
2017-03-15 14:27:41 +00:00
|
|
|
if isinstance(label, basestring):
|
|
|
|
label = self.vocab.strings[label]
|
|
|
|
|
|
|
|
|
2017-05-13 23:10:23 +00:00
|
|
|
__all__ = ['Tagger', 'DependencyParser', 'EntityRecognizer', 'BeamDependencyParser',
|
|
|
|
'BeamEntityRecognizer', 'TokenVectorEnoder']
|