Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
# cython: infer_types=True
|
|
|
|
# cython: profile=True
|
2017-04-15 10:05:47 +00:00
|
|
|
# coding: utf8
|
|
|
|
from __future__ import unicode_literals
|
|
|
|
|
2017-05-07 16:04:24 +00:00
|
|
|
import numpy
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
cimport numpy as np
|
2017-05-16 14:17:30 +00:00
|
|
|
import cytoolz
|
2018-07-18 17:43:16 +00:00
|
|
|
from collections import OrderedDict, defaultdict
|
2017-06-01 17:18:36 +00:00
|
|
|
import ujson
|
2018-03-28 22:14:55 +00:00
|
|
|
|
|
|
|
from .util import msgpack
|
|
|
|
from .util import msgpack_numpy
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-10-27 18:29:08 +00:00
|
|
|
from thinc.api import chain
|
2017-11-03 19:20:26 +00:00
|
|
|
from thinc.v2v import Affine, SELU, Softmax
|
2017-10-27 18:29:08 +00:00
|
|
|
from thinc.t2v import Pooling, max_pool, mean_pool
|
2017-11-01 15:32:44 +00:00
|
|
|
from thinc.neural.util import to_categorical, copy_array
|
2017-06-05 13:40:03 +00:00
|
|
|
from thinc.neural._classes.difference import Siamese, CauchySimilarity
|
|
|
|
|
2017-05-08 12:53:45 +00:00
|
|
|
from .tokens.doc cimport Doc
|
2017-10-26 10:38:23 +00:00
|
|
|
from .syntax.nn_parser cimport Parser
|
2017-10-07 00:00:47 +00:00
|
|
|
from .syntax import nonproj
|
2016-10-15 23:47:12 +00:00
|
|
|
from .syntax.ner cimport BiluoPushDown
|
|
|
|
from .syntax.arc_eager cimport ArcEager
|
2017-05-17 10:04:50 +00:00
|
|
|
from .morphology cimport Morphology
|
|
|
|
from .vocab cimport Vocab
|
2017-05-22 10:17:44 +00:00
|
|
|
from .syntax import nonproj
|
2017-06-01 17:18:36 +00:00
|
|
|
from .compat import json_dumps
|
2018-03-27 17:23:02 +00:00
|
|
|
from .matcher import Matcher
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2018-07-18 17:43:16 +00:00
|
|
|
from .matcher import Matcher, PhraseMatcher
|
|
|
|
from .tokens.span import Span
|
2017-10-27 18:29:08 +00:00
|
|
|
from .attrs import POS
|
2017-05-17 10:04:50 +00:00
|
|
|
from .parts_of_speech import X
|
2017-10-27 18:29:08 +00:00
|
|
|
from ._ml import Tok2Vec, build_text_classifier, build_tagger_model
|
2017-11-03 19:20:26 +00:00
|
|
|
from ._ml import link_vectors_to_models, zero_init, flatten
|
2017-11-06 13:26:26 +00:00
|
|
|
from ._ml import create_default_optimizer
|
2018-04-03 13:50:31 +00:00
|
|
|
from .errors import Errors, TempErrors
|
2018-07-18 17:43:16 +00:00
|
|
|
from .compat import json_dumps, basestring_
|
2017-10-27 18:29:08 +00:00
|
|
|
from . import util
|
2016-10-15 23:47:12 +00:00
|
|
|
|
|
|
|
|
2017-09-02 10:53:38 +00:00
|
|
|
class SentenceSegmenter(object):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""A simple spaCy hook, to allow custom sentence boundary detection logic
|
2017-10-27 18:29:08 +00:00
|
|
|
(that doesn't require the dependency parse). To change the sentence
|
|
|
|
boundary detection strategy, pass a generator function `strategy` on
|
|
|
|
initialization, or assign a new strategy to the .strategy attribute.
|
2017-09-02 10:53:38 +00:00
|
|
|
Sentence detection strategies should be generators that take `Doc` objects
|
|
|
|
and yield `Span` objects for each sentence.
|
2017-09-25 16:37:13 +00:00
|
|
|
"""
|
2017-09-02 10:53:38 +00:00
|
|
|
name = 'sbd'
|
|
|
|
|
|
|
|
def __init__(self, vocab, strategy=None):
|
|
|
|
self.vocab = vocab
|
|
|
|
if strategy is None or strategy == 'on_punct':
|
|
|
|
strategy = self.split_on_punct
|
|
|
|
self.strategy = strategy
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
|
|
|
doc.user_hooks['sents'] = self.strategy
|
2017-10-17 13:32:56 +00:00
|
|
|
return doc
|
2017-09-02 10:53:38 +00:00
|
|
|
|
|
|
|
@staticmethod
|
|
|
|
def split_on_punct(doc):
|
|
|
|
start = 0
|
|
|
|
seen_period = False
|
|
|
|
for i, word in enumerate(doc):
|
|
|
|
if seen_period and not word.is_punct:
|
2017-10-27 18:29:08 +00:00
|
|
|
yield doc[start:word.i]
|
2017-09-02 10:53:38 +00:00
|
|
|
start = word.i
|
|
|
|
seen_period = False
|
|
|
|
elif word.text in ['.', '!', '?']:
|
|
|
|
seen_period = True
|
|
|
|
if start < len(doc):
|
2017-10-27 18:29:08 +00:00
|
|
|
yield doc[start:len(doc)]
|
2017-09-02 10:53:38 +00:00
|
|
|
|
|
|
|
|
2018-03-14 23:18:51 +00:00
|
|
|
def merge_noun_chunks(doc):
|
|
|
|
"""Merge noun chunks into a single token.
|
|
|
|
|
|
|
|
doc (Doc): The Doc object.
|
|
|
|
RETURNS (Doc): The Doc object with merged noun chunks.
|
|
|
|
"""
|
|
|
|
if not doc.is_parsed:
|
2018-04-09 12:51:02 +00:00
|
|
|
return doc
|
2018-03-14 23:18:51 +00:00
|
|
|
spans = [(np.start_char, np.end_char, np.root.tag, np.root.dep)
|
|
|
|
for np in doc.noun_chunks]
|
|
|
|
for start, end, tag, dep in spans:
|
|
|
|
doc.merge(start, end, tag=tag, dep=dep)
|
|
|
|
return doc
|
|
|
|
|
|
|
|
|
|
|
|
def merge_entities(doc):
|
|
|
|
"""Merge entities into a single token.
|
|
|
|
|
|
|
|
doc (Doc): The Doc object.
|
|
|
|
RETURNS (Doc): The Doc object with merged noun entities.
|
|
|
|
"""
|
|
|
|
spans = [(e.start_char, e.end_char, e.root.tag, e.root.dep, e.label)
|
|
|
|
for e in doc.ents]
|
|
|
|
for start, end, tag, dep, ent_type in spans:
|
|
|
|
doc.merge(start, end, tag=tag, dep=dep, ent_type=ent_type)
|
|
|
|
return doc
|
|
|
|
|
|
|
|
|
2018-03-27 17:23:02 +00:00
|
|
|
def merge_subtokens(doc, label='subtok'):
|
|
|
|
merger = Matcher(doc.vocab)
|
|
|
|
merger.add('SUBTOK', None, [{'DEP': label, 'op': '+'}])
|
|
|
|
matches = merger(doc)
|
|
|
|
spans = [doc[start:end+1] for _, start, end in matches]
|
|
|
|
offsets = [(span.start_char, span.end_char) for span in spans]
|
|
|
|
for start_char, end_char in offsets:
|
|
|
|
doc.merge(start_char, end_char)
|
|
|
|
return doc
|
2018-07-18 17:43:16 +00:00
|
|
|
|
|
|
|
|
|
|
|
class EntityRuler(object):
|
|
|
|
name = 'entity_ruler'
|
|
|
|
|
|
|
|
def __init__(self, nlp, **cfg):
|
|
|
|
"""Initialise the entitiy ruler. If patterns are supplied here, they
|
|
|
|
need to be a list of dictionaries with a `"label"` and `"pattern"`
|
|
|
|
key. A pattern can either be a token pattern (list) or a phrase pattern
|
|
|
|
(string). For example: `{'label': 'ORG', 'pattern': 'Apple'}`.
|
|
|
|
|
|
|
|
nlp (Language): The shared nlp object to pass the vocab to the matchers
|
|
|
|
and process phrase patterns.
|
|
|
|
patterns (iterable): Optional patterns to load in.
|
|
|
|
overwrite_ents (bool): If existing entities are present, e.g. entities
|
|
|
|
added by the model, overwrite them by matches if necessary.
|
|
|
|
**cfg: Other config parameters. If pipeline component is loaded as part
|
|
|
|
of a model pipeline, this will include all keyword arguments passed
|
|
|
|
to `spacy.load`.
|
|
|
|
RETURNS (EntityRuler): The newly constructed object.
|
|
|
|
"""
|
|
|
|
self.nlp = nlp
|
|
|
|
self.overwrite = cfg.get('overwrite_ents', False)
|
|
|
|
self.token_patterns = defaultdict(list)
|
|
|
|
self.phrase_patterns = defaultdict(list)
|
|
|
|
self.matcher = Matcher(nlp.vocab)
|
|
|
|
self.phrase_matcher = PhraseMatcher(nlp.vocab)
|
|
|
|
patterns = cfg.get('patterns')
|
|
|
|
if patterns is not None:
|
|
|
|
self.add_patterns(patterns)
|
|
|
|
|
|
|
|
def __len__(self):
|
|
|
|
"""The number of all patterns added to the entity ruler."""
|
|
|
|
n_token_patterns = sum(len(p) for p in self.token_patterns.values())
|
|
|
|
n_phrase_patterns = sum(len(p) for p in self.phrase_patterns.values())
|
|
|
|
return n_token_patterns + n_phrase_patterns
|
|
|
|
|
|
|
|
def __contains__(self, label):
|
|
|
|
"""Whether a label is present in the patterns."""
|
|
|
|
return label in self.token_patterns or label in self.phrase_patterns
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
|
|
|
"""Find matches in document and add them as entities.
|
|
|
|
|
|
|
|
doc (Doc): The Doc object in the pipeline.
|
|
|
|
RETURNS (Doc): The Doc with added entities, if available.
|
|
|
|
"""
|
|
|
|
matches = list(self.matcher(doc)) + list(self.phrase_matcher(doc))
|
|
|
|
matches = set([(m_id, start, end) for m_id, start, end in matches
|
|
|
|
if start != end])
|
|
|
|
get_sort_key = lambda m: (m[2] - m[1], m[1])
|
|
|
|
matches = sorted(matches, key=get_sort_key, reverse=True)
|
|
|
|
entities = list(doc.ents)
|
|
|
|
new_entities = []
|
|
|
|
seen_tokens = set()
|
|
|
|
for match_id, start, end in matches:
|
|
|
|
if any(t.ent_type for t in doc[start:end]) and not self.overwrite:
|
|
|
|
continue
|
|
|
|
# check for end - 1 here because boundaries are inclusive
|
|
|
|
if start not in seen_tokens and end - 1 not in seen_tokens:
|
|
|
|
new_entities.append(Span(doc, start, end, label=match_id))
|
|
|
|
entities = [e for e in entities
|
|
|
|
if not (e.start < end and e.end > start)]
|
|
|
|
seen_tokens.update(range(start, end))
|
|
|
|
doc.ents = entities + new_entities
|
|
|
|
return doc
|
|
|
|
|
|
|
|
@property
|
|
|
|
def labels(self):
|
|
|
|
"""All labels present in the match patterns.
|
|
|
|
|
|
|
|
RETURNS (set): The string labels.
|
|
|
|
"""
|
|
|
|
all_labels = set(self.token_patterns.keys())
|
|
|
|
all_labels.update(self.phrase_patterns.keys())
|
|
|
|
return all_labels
|
|
|
|
|
|
|
|
@property
|
|
|
|
def patterns(self):
|
|
|
|
"""Get all patterns that were added to the entity ruler.
|
|
|
|
|
|
|
|
RETURNS (list): The original patterns, one dictionary per pattern.
|
|
|
|
"""
|
|
|
|
all_patterns = []
|
|
|
|
for label, patterns in self.token_patterns.items():
|
|
|
|
for pattern in patterns:
|
|
|
|
all_patterns.append({'label': label, 'pattern': pattern})
|
|
|
|
for label, patterns in self.phrase_patterns.items():
|
|
|
|
for pattern in patterns:
|
|
|
|
all_patterns.append({'label': label, 'pattern': pattern.text})
|
|
|
|
return all_patterns
|
|
|
|
|
|
|
|
def add_patterns(self, patterns):
|
|
|
|
"""Add patterns to the entitiy ruler. A pattern can either be a token
|
|
|
|
pattern (list of dicts) or a phrase pattern (string). For example:
|
|
|
|
{'label': 'ORG', 'pattern': 'Apple'}
|
|
|
|
{'label': 'GPE', 'pattern': [{'lower': 'san'}, {'lower': 'francisco'}]}
|
|
|
|
|
|
|
|
patterns (list): The patterns to add.
|
|
|
|
"""
|
|
|
|
for entry in patterns:
|
|
|
|
label = entry['label']
|
|
|
|
pattern = entry['pattern']
|
|
|
|
if isinstance(pattern, basestring_):
|
|
|
|
self.phrase_patterns[label].append(self.nlp(pattern))
|
|
|
|
elif isinstance(pattern, list):
|
|
|
|
self.token_patterns[label].append(pattern)
|
|
|
|
else:
|
|
|
|
raise ValueError(Errors.E097.format(pattern=pattern))
|
|
|
|
for label, patterns in self.token_patterns.items():
|
|
|
|
self.matcher.add(label, None, *patterns)
|
|
|
|
for label, patterns in self.phrase_patterns.items():
|
|
|
|
self.phrase_matcher.add(label, None, *patterns)
|
|
|
|
|
|
|
|
def from_bytes(self, patterns_bytes, **kwargs):
|
|
|
|
"""Load the entity ruler from a bytestring.
|
|
|
|
|
|
|
|
patterns_bytes (bytes): The bytestring to load.
|
|
|
|
**kwargs: Other config paramters, mostly for consistency.
|
|
|
|
RETURNS (EntityRuler): The loaded entity ruler.
|
|
|
|
"""
|
|
|
|
patterns = msgpack.loads(patterns_bytes)
|
|
|
|
self.add_patterns(patterns)
|
|
|
|
return self
|
|
|
|
|
|
|
|
def to_bytes(self, **kwargs):
|
|
|
|
"""Serialize the entity ruler patterns to a bytestring.
|
|
|
|
|
|
|
|
RETURNS (bytes): The serialized patterns.
|
|
|
|
"""
|
|
|
|
return msgpack.dumps(self.patterns)
|
|
|
|
|
|
|
|
def from_disk(self, path, **kwargs):
|
|
|
|
"""Load the entity ruler from a file. Expects a file containing
|
|
|
|
newline-delimited JSON (JSONL) with one entry per line.
|
|
|
|
|
|
|
|
path (unicode / Path): The JSONL file to load.
|
|
|
|
**kwargs: Other config paramters, mostly for consistency.
|
|
|
|
RETURNS (EntityRuler): The loaded entity ruler.
|
|
|
|
"""
|
|
|
|
path = util.ensure_path(path)
|
|
|
|
path = path.with_suffix('.jsonl')
|
|
|
|
patterns = util.read_jsonl(path)
|
|
|
|
self.add_patterns(patterns)
|
|
|
|
return self
|
|
|
|
|
|
|
|
def to_disk(self, path, **kwargs):
|
|
|
|
"""Save the entity ruler patterns to a directory. The patterns will be
|
|
|
|
saved as newline-delimited JSON (JSONL).
|
|
|
|
|
|
|
|
path (unicode / Path): The JSONL file to load.
|
|
|
|
**kwargs: Other config paramters, mostly for consistency.
|
|
|
|
RETURNS (EntityRuler): The loaded entity ruler.
|
|
|
|
"""
|
|
|
|
path = util.ensure_path(path)
|
|
|
|
path = path.with_suffix('.jsonl')
|
|
|
|
data = [json_dumps(line, indent=0) for line in self.patterns]
|
|
|
|
path.open('w').write('\n'.join(data))
|
|
|
|
|
2018-03-27 17:23:02 +00:00
|
|
|
|
2017-10-26 10:40:40 +00:00
|
|
|
class Pipe(object):
|
2017-10-27 18:29:08 +00:00
|
|
|
"""This class is not instantiated directly. Components inherit from it, and
|
|
|
|
it defines the interface that components should follow to function as
|
|
|
|
components in a spaCy analysis pipeline.
|
|
|
|
"""
|
2017-07-19 22:18:15 +00:00
|
|
|
name = None
|
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, *shape, **kwargs):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Initialize a model for the pipe."""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Create a new pipe instance."""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Apply the pipe to one document. The document is
|
2017-09-25 14:20:49 +00:00
|
|
|
modified in-place, and returned.
|
2017-09-25 16:37:13 +00:00
|
|
|
|
2017-09-25 14:20:49 +00:00
|
|
|
Both __call__ and pipe should delegate to the `predict()`
|
|
|
|
and `set_annotations()` methods.
|
2017-09-25 16:37:13 +00:00
|
|
|
"""
|
2017-11-03 10:20:05 +00:00
|
|
|
scores, tensors = self.predict([doc])
|
|
|
|
self.set_annotations([doc], scores, tensors=tensors)
|
2017-07-19 22:18:15 +00:00
|
|
|
return doc
|
|
|
|
|
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Apply the pipe to a stream of documents.
|
2017-09-25 14:20:49 +00:00
|
|
|
|
|
|
|
Both __call__ and pipe should delegate to the `predict()`
|
|
|
|
and `set_annotations()` methods.
|
2017-09-25 16:37:13 +00:00
|
|
|
"""
|
2017-07-19 22:18:15 +00:00
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
|
|
|
docs = list(docs)
|
2017-11-03 10:20:05 +00:00
|
|
|
scores, tensors = self.predict(docs)
|
|
|
|
self.set_annotations(docs, scores, tensor=tensors)
|
2017-07-19 22:18:15 +00:00
|
|
|
yield from docs
|
|
|
|
|
|
|
|
def predict(self, docs):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Apply the pipeline's model to a batch of docs, without
|
2017-09-25 14:20:49 +00:00
|
|
|
modifying them.
|
2017-09-25 16:37:13 +00:00
|
|
|
"""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
2017-11-03 10:20:05 +00:00
|
|
|
def set_annotations(self, docs, scores, tensors=None):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Modify a batch of documents, using pre-computed scores."""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
2017-09-25 14:20:49 +00:00
|
|
|
def update(self, docs, golds, drop=0., sgd=None, losses=None):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Learn from a batch of documents and gold-standard information,
|
2017-09-25 14:20:49 +00:00
|
|
|
updating the pipe's model.
|
|
|
|
|
|
|
|
Delegates to predict() and get_loss().
|
2017-09-25 16:37:13 +00:00
|
|
|
"""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Find the loss and gradient of loss for the batch of
|
|
|
|
documents and their predicted scores."""
|
2017-07-19 22:18:15 +00:00
|
|
|
raise NotImplementedError
|
|
|
|
|
2017-11-01 15:32:44 +00:00
|
|
|
def add_label(self, label):
|
|
|
|
"""Add an output label, to be predicted by the model.
|
|
|
|
|
|
|
|
It's possible to extend pre-trained models with new labels,
|
|
|
|
but care should be taken to avoid the "catastrophic forgetting"
|
|
|
|
problem.
|
|
|
|
"""
|
|
|
|
raise NotImplementedError
|
2018-03-27 17:23:02 +00:00
|
|
|
|
2017-11-06 13:26:26 +00:00
|
|
|
def create_optimizer(self):
|
|
|
|
return create_default_optimizer(self.model.ops,
|
|
|
|
**self.cfg.get('optimizer', {}))
|
2017-11-01 15:32:44 +00:00
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def begin_training(self, get_gold_tuples=lambda: [], pipeline=None, sgd=None,
|
2018-02-12 09:18:39 +00:00
|
|
|
**kwargs):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Initialize the pipe for training, using data exampes if available.
|
|
|
|
If no model has been initialized yet, the model is added."""
|
2017-07-19 22:18:15 +00:00
|
|
|
if self.model is True:
|
2017-09-25 14:20:49 +00:00
|
|
|
self.model = self.Model(**self.cfg)
|
2017-09-25 14:22:07 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-07-19 22:18:15 +00:00
|
|
|
|
|
|
|
def use_params(self, params):
|
2017-10-27 18:29:08 +00:00
|
|
|
"""Modify the pipe's model, to use the given parameter values."""
|
2017-07-19 22:18:15 +00:00
|
|
|
with self.model.use_params(params):
|
|
|
|
yield
|
|
|
|
|
|
|
|
def to_bytes(self, **exclude):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Serialize the pipe to a bytestring."""
|
2017-10-10 01:58:12 +00:00
|
|
|
serialize = OrderedDict()
|
|
|
|
serialize['cfg'] = lambda: json_dumps(self.cfg)
|
|
|
|
if self.model in (True, False, None):
|
|
|
|
serialize['model'] = lambda: self.model
|
|
|
|
else:
|
|
|
|
serialize['model'] = self.model.to_bytes
|
|
|
|
serialize['vocab'] = self.vocab.to_bytes
|
2017-07-19 22:18:15 +00:00
|
|
|
return util.to_bytes(serialize, exclude)
|
|
|
|
|
|
|
|
def from_bytes(self, bytes_data, **exclude):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Load the pipe from a bytestring."""
|
2017-09-02 13:17:20 +00:00
|
|
|
def load_model(b):
|
2018-03-28 14:02:59 +00:00
|
|
|
# TODO: Remove this once we don't have to handle previous models
|
2018-04-10 20:19:05 +00:00
|
|
|
if self.cfg.get('pretrained_dims') and 'pretrained_vectors' not in self.cfg:
|
2018-04-03 19:40:29 +00:00
|
|
|
self.cfg['pretrained_vectors'] = self.vocab.vectors.name
|
2017-09-02 13:17:20 +00:00
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(**self.cfg)
|
|
|
|
self.model.from_bytes(b)
|
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
deserialize = OrderedDict((
|
2017-09-02 13:17:20 +00:00
|
|
|
('cfg', lambda b: self.cfg.update(ujson.loads(b))),
|
2017-09-26 11:45:14 +00:00
|
|
|
('vocab', lambda b: self.vocab.from_bytes(b)),
|
2017-09-22 20:33:27 +00:00
|
|
|
('model', load_model),
|
2017-07-19 22:18:15 +00:00
|
|
|
))
|
|
|
|
util.from_bytes(bytes_data, deserialize, exclude)
|
|
|
|
return self
|
|
|
|
|
|
|
|
def to_disk(self, path, **exclude):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Serialize the pipe to disk."""
|
2017-10-10 01:58:12 +00:00
|
|
|
serialize = OrderedDict()
|
|
|
|
serialize['cfg'] = lambda p: p.open('w').write(json_dumps(self.cfg))
|
|
|
|
serialize['vocab'] = lambda p: self.vocab.to_disk(p)
|
|
|
|
if self.model not in (None, True, False):
|
|
|
|
serialize['model'] = lambda p: p.open('wb').write(self.model.to_bytes())
|
2017-07-19 22:18:15 +00:00
|
|
|
util.to_disk(path, serialize, exclude)
|
|
|
|
|
|
|
|
def from_disk(self, path, **exclude):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Load the pipe from disk."""
|
2017-09-02 13:17:20 +00:00
|
|
|
def load_model(p):
|
2018-03-28 14:02:59 +00:00
|
|
|
# TODO: Remove this once we don't have to handle previous models
|
2018-04-10 20:19:05 +00:00
|
|
|
if self.cfg.get('pretrained_dims') and 'pretrained_vectors' not in self.cfg:
|
2018-04-03 19:40:29 +00:00
|
|
|
self.cfg['pretrained_vectors'] = self.vocab.vectors.name
|
2017-09-02 13:17:20 +00:00
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(**self.cfg)
|
|
|
|
self.model.from_bytes(p.open('rb').read())
|
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
deserialize = OrderedDict((
|
2017-09-02 13:17:20 +00:00
|
|
|
('cfg', lambda p: self.cfg.update(_load_cfg(p))),
|
2017-07-22 22:33:43 +00:00
|
|
|
('vocab', lambda p: self.vocab.from_disk(p)),
|
2017-09-22 20:33:27 +00:00
|
|
|
('model', load_model),
|
2017-07-19 22:18:15 +00:00
|
|
|
))
|
|
|
|
util.from_disk(path, deserialize, exclude)
|
|
|
|
return self
|
|
|
|
|
|
|
|
|
2017-07-23 12:11:07 +00:00
|
|
|
def _load_cfg(path):
|
|
|
|
if path.exists():
|
2018-02-13 19:44:33 +00:00
|
|
|
with path.open() as file_:
|
|
|
|
return ujson.load(file_)
|
2017-07-23 12:11:07 +00:00
|
|
|
else:
|
|
|
|
return {}
|
|
|
|
|
|
|
|
|
2017-10-26 10:40:40 +00:00
|
|
|
class Tensorizer(Pipe):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Assign position-sensitive vectors to tokens, using a CNN or RNN."""
|
2017-05-31 11:42:39 +00:00
|
|
|
name = 'tensorizer'
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-15 19:46:08 +00:00
|
|
|
@classmethod
|
2017-11-03 19:20:26 +00:00
|
|
|
def Model(cls, output_size=300, input_size=384, **cfg):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Create a new statistical model for the class.
|
|
|
|
|
|
|
|
width (int): Output size of the model.
|
|
|
|
embed_size (int): Number of vectors in the embedding table.
|
|
|
|
**cfg: Config parameters.
|
|
|
|
RETURNS (Model): A `thinc.neural.Model` or similar instance.
|
|
|
|
"""
|
2017-11-03 19:20:26 +00:00
|
|
|
model = chain(
|
|
|
|
SELU(output_size, input_size),
|
|
|
|
SELU(output_size, output_size),
|
|
|
|
zero_init(Affine(output_size, output_size)))
|
|
|
|
return model
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
|
2017-05-15 19:46:08 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Construct a new statistical model. Weights are not allocated on
|
|
|
|
initialisation.
|
|
|
|
|
2017-10-27 18:29:08 +00:00
|
|
|
vocab (Vocab): A `Vocab` instance. The model must share the same
|
|
|
|
`Vocab` instance with the `Doc` objects it will process.
|
2017-05-18 22:00:02 +00:00
|
|
|
model (Model): A `Model` instance or `True` allocate one later.
|
|
|
|
**cfg: Config parameters.
|
|
|
|
|
|
|
|
EXAMPLE:
|
|
|
|
>>> from spacy.pipeline import TokenVectorEncoder
|
|
|
|
>>> tok2vec = TokenVectorEncoder(nlp.vocab)
|
|
|
|
>>> tok2vec.model = tok2vec.Model(128, 5000)
|
|
|
|
"""
|
2017-05-15 19:46:08 +00:00
|
|
|
self.vocab = vocab
|
2017-05-18 09:29:51 +00:00
|
|
|
self.model = model
|
2017-11-03 19:20:26 +00:00
|
|
|
self.input_models = []
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-09-22 14:38:22 +00:00
|
|
|
self.cfg.setdefault('cnn_maxout_pieces', 3)
|
2017-05-17 11:13:14 +00:00
|
|
|
|
2017-05-28 13:11:58 +00:00
|
|
|
def __call__(self, doc):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Add context-sensitive vectors to a `Doc`, e.g. from a CNN or LSTM
|
|
|
|
model. Vectors are set to the `Doc.tensor` attribute.
|
|
|
|
|
|
|
|
docs (Doc or iterable): One or more documents to add vectors to.
|
|
|
|
RETURNS (dict or None): Intermediate computations.
|
|
|
|
"""
|
2017-05-28 13:11:58 +00:00
|
|
|
tokvecses = self.predict([doc])
|
|
|
|
self.set_annotations([doc], tokvecses)
|
|
|
|
return doc
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-05-18 13:30:59 +00:00
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Process `Doc` objects as a stream.
|
|
|
|
|
|
|
|
stream (iterator): A sequence of `Doc` objects to process.
|
|
|
|
batch_size (int): Number of `Doc` objects to group.
|
|
|
|
n_threads (int): Number of threads.
|
2017-05-21 18:46:23 +00:00
|
|
|
YIELDS (iterator): A sequence of `Doc` objects, in order of input.
|
2017-05-18 22:00:02 +00:00
|
|
|
"""
|
2017-05-19 18:26:36 +00:00
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
2017-05-21 22:52:01 +00:00
|
|
|
docs = list(docs)
|
2017-11-03 19:20:26 +00:00
|
|
|
tensors = self.predict(docs)
|
|
|
|
self.set_annotations(docs, tensors)
|
2017-05-19 18:26:36 +00:00
|
|
|
yield from docs
|
2017-05-18 09:29:51 +00:00
|
|
|
|
2017-05-16 14:17:30 +00:00
|
|
|
def predict(self, docs):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Return a single tensor for a batch of documents.
|
|
|
|
|
|
|
|
docs (iterable): A sequence of `Doc` objects.
|
2017-10-27 18:29:08 +00:00
|
|
|
RETURNS (object): Vector representations for each token in the docs.
|
2017-05-18 22:00:02 +00:00
|
|
|
"""
|
2017-11-03 19:20:26 +00:00
|
|
|
inputs = self.model.ops.flatten([doc.tensor for doc in docs])
|
|
|
|
outputs = self.model(inputs)
|
|
|
|
return self.model.ops.unflatten(outputs, [len(d) for d in docs])
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-11-03 19:20:26 +00:00
|
|
|
def set_annotations(self, docs, tensors):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Set the tensor attribute for a batch of documents.
|
|
|
|
|
|
|
|
docs (iterable): A sequence of `Doc` objects.
|
2017-11-03 19:20:26 +00:00
|
|
|
tensors (object): Vector representation for each token in the docs.
|
2017-05-18 22:00:02 +00:00
|
|
|
"""
|
2017-11-03 19:20:26 +00:00
|
|
|
for doc, tensor in zip(docs, tensors):
|
2018-04-03 13:50:31 +00:00
|
|
|
if tensor.shape[0] != len(doc):
|
|
|
|
raise ValueError(Errors.E076.format(rows=tensor.shape[0], words=len(doc)))
|
2017-11-03 19:20:26 +00:00
|
|
|
doc.tensor = tensor
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2017-05-25 01:09:51 +00:00
|
|
|
def update(self, docs, golds, state=None, drop=0., sgd=None, losses=None):
|
2017-05-18 22:00:02 +00:00
|
|
|
"""Update the model.
|
|
|
|
|
|
|
|
docs (iterable): A batch of `Doc` objects.
|
|
|
|
golds (iterable): A batch of `GoldParse` objects.
|
|
|
|
drop (float): The droput rate.
|
2017-05-21 11:17:40 +00:00
|
|
|
sgd (callable): An optimizer.
|
2017-05-18 22:00:02 +00:00
|
|
|
RETURNS (dict): Results from the update.
|
|
|
|
"""
|
2017-05-16 14:17:30 +00:00
|
|
|
if isinstance(docs, Doc):
|
|
|
|
docs = [docs]
|
2017-11-03 19:20:26 +00:00
|
|
|
inputs = []
|
|
|
|
bp_inputs = []
|
|
|
|
for tok2vec in self.input_models:
|
|
|
|
tensor, bp_tensor = tok2vec.begin_update(docs, drop=drop)
|
|
|
|
inputs.append(tensor)
|
|
|
|
bp_inputs.append(bp_tensor)
|
|
|
|
inputs = self.model.ops.xp.hstack(inputs)
|
|
|
|
scores, bp_scores = self.model.begin_update(inputs, drop=drop)
|
|
|
|
loss, d_scores = self.get_loss(docs, golds, scores)
|
|
|
|
d_inputs = bp_scores(d_scores, sgd=sgd)
|
|
|
|
d_inputs = self.model.ops.xp.split(d_inputs, len(self.input_models), axis=1)
|
2017-11-05 11:25:10 +00:00
|
|
|
for d_input, bp_input in zip(d_inputs, bp_inputs):
|
2017-11-03 19:20:26 +00:00
|
|
|
bp_input(d_input, sgd=sgd)
|
|
|
|
if losses is not None:
|
|
|
|
losses.setdefault(self.name, 0.)
|
|
|
|
losses[self.name] += loss
|
|
|
|
return loss
|
|
|
|
|
|
|
|
def get_loss(self, docs, golds, prediction):
|
|
|
|
target = []
|
|
|
|
i = 0
|
|
|
|
for doc in docs:
|
|
|
|
vectors = self.model.ops.xp.vstack([w.vector for w in doc])
|
|
|
|
target.append(vectors)
|
|
|
|
target = self.model.ops.xp.vstack(target)
|
2018-07-18 17:43:16 +00:00
|
|
|
d_scores = (prediction - target)
|
2017-11-03 19:20:26 +00:00
|
|
|
loss = (d_scores**2).sum()
|
|
|
|
return loss, d_scores
|
2017-05-06 12:22:20 +00:00
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def begin_training(self, gold_tuples=lambda: [], pipeline=None, sgd=None,
|
2018-02-12 09:18:39 +00:00
|
|
|
**kwargs):
|
2017-11-06 13:26:26 +00:00
|
|
|
"""Allocate models, pre-process training data and acquire an
|
2017-05-18 22:00:02 +00:00
|
|
|
optimizer.
|
|
|
|
|
|
|
|
gold_tuples (iterable): Gold-standard training data.
|
|
|
|
pipeline (list): The pipeline the model is part of.
|
|
|
|
"""
|
2017-11-03 19:20:26 +00:00
|
|
|
for name, model in pipeline:
|
|
|
|
if getattr(model, 'tok2vec', None):
|
|
|
|
self.input_models.append(model.tok2vec)
|
2017-05-18 09:29:51 +00:00
|
|
|
if self.model is True:
|
2017-11-03 19:20:26 +00:00
|
|
|
self.cfg['input_size'] = 384
|
|
|
|
self.cfg['output_size'] = 300
|
2017-09-21 00:15:49 +00:00
|
|
|
self.model = self.Model(**self.cfg)
|
2017-09-26 10:42:52 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-05-18 09:29:51 +00:00
|
|
|
|
2017-05-28 23:37:57 +00:00
|
|
|
|
2017-10-26 10:40:40 +00:00
|
|
|
class Tagger(Pipe):
|
2017-06-01 15:37:53 +00:00
|
|
|
name = 'tagger'
|
2017-10-27 18:29:08 +00:00
|
|
|
|
2017-07-22 22:52:47 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-05-16 14:17:30 +00:00
|
|
|
self.vocab = vocab
|
2017-05-17 10:04:50 +00:00
|
|
|
self.model = model
|
2017-11-08 11:10:49 +00:00
|
|
|
self.cfg = OrderedDict(sorted(cfg.items()))
|
2017-09-21 00:15:49 +00:00
|
|
|
self.cfg.setdefault('cnn_maxout_pieces', 2)
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-11-01 15:32:44 +00:00
|
|
|
@property
|
|
|
|
def labels(self):
|
2017-11-01 20:10:45 +00:00
|
|
|
return self.vocab.morphology.tag_names
|
2017-11-01 15:32:44 +00:00
|
|
|
|
2017-11-03 19:20:26 +00:00
|
|
|
@property
|
|
|
|
def tok2vec(self):
|
|
|
|
if self.model in (None, True, False):
|
|
|
|
return None
|
|
|
|
else:
|
|
|
|
return chain(self.model.tok2vec, flatten)
|
|
|
|
|
2017-05-19 18:26:36 +00:00
|
|
|
def __call__(self, doc):
|
2017-11-03 10:20:05 +00:00
|
|
|
tags, tokvecs = self.predict([doc])
|
|
|
|
self.set_annotations([doc], tags, tensors=tokvecs)
|
2017-05-28 13:11:58 +00:00
|
|
|
return doc
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
2017-05-19 18:26:36 +00:00
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
2017-08-18 20:02:35 +00:00
|
|
|
docs = list(docs)
|
2017-11-03 10:20:05 +00:00
|
|
|
tag_ids, tokvecs = self.predict(docs)
|
|
|
|
self.set_annotations(docs, tag_ids, tensors=tokvecs)
|
2017-05-19 18:26:36 +00:00
|
|
|
yield from docs
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-09-21 12:59:48 +00:00
|
|
|
def predict(self, docs):
|
2018-06-29 11:44:25 +00:00
|
|
|
if not any(len(doc) for doc in docs):
|
|
|
|
# Handle case where there are no tokens in any docs.
|
2018-06-29 13:13:45 +00:00
|
|
|
n_labels = len(self.labels)
|
2018-06-29 14:05:40 +00:00
|
|
|
guesses = [self.model.ops.allocate((0, n_labels)) for doc in docs]
|
|
|
|
tokvecs = self.model.ops.allocate((0, self.model.tok2vec.nO))
|
|
|
|
return guesses, tokvecs
|
2017-11-03 10:20:05 +00:00
|
|
|
tokvecs = self.model.tok2vec(docs)
|
|
|
|
scores = self.model.softmax(tokvecs)
|
2017-11-03 12:29:36 +00:00
|
|
|
guesses = []
|
|
|
|
for doc_scores in scores:
|
|
|
|
doc_guesses = doc_scores.argmax(axis=1)
|
|
|
|
if not isinstance(doc_guesses, numpy.ndarray):
|
|
|
|
doc_guesses = doc_guesses.get()
|
|
|
|
guesses.append(doc_guesses)
|
2017-11-03 10:20:05 +00:00
|
|
|
return guesses, tokvecs
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-11-03 10:20:05 +00:00
|
|
|
def set_annotations(self, docs, batch_tag_ids, tensors=None):
|
2017-05-16 14:17:30 +00:00
|
|
|
if isinstance(docs, Doc):
|
|
|
|
docs = [docs]
|
|
|
|
cdef Doc doc
|
|
|
|
cdef int idx = 0
|
2017-05-18 09:29:51 +00:00
|
|
|
cdef Vocab vocab = self.vocab
|
2017-05-08 12:53:45 +00:00
|
|
|
for i, doc in enumerate(docs):
|
2017-05-21 14:05:34 +00:00
|
|
|
doc_tag_ids = batch_tag_ids[i]
|
2017-08-18 20:02:35 +00:00
|
|
|
if hasattr(doc_tag_ids, 'get'):
|
|
|
|
doc_tag_ids = doc_tag_ids.get()
|
2017-05-18 09:29:51 +00:00
|
|
|
for j, tag_id in enumerate(doc_tag_ids):
|
2017-06-04 20:52:42 +00:00
|
|
|
# Don't clobber preset POS tags
|
|
|
|
if doc.c[j].tag == 0 and doc.c[j].pos == 0:
|
2017-11-06 11:36:05 +00:00
|
|
|
# Don't clobber preset lemmas
|
|
|
|
lemma = doc.c[j].lemma
|
2017-11-01 20:10:45 +00:00
|
|
|
vocab.morphology.assign_tag_id(&doc.c[j], tag_id)
|
2017-11-06 15:56:19 +00:00
|
|
|
if lemma != 0 and lemma != doc.c[j].lex.orth:
|
2017-11-06 11:36:05 +00:00
|
|
|
doc.c[j].lemma = lemma
|
2017-05-08 12:53:45 +00:00
|
|
|
idx += 1
|
2018-06-29 17:21:38 +00:00
|
|
|
if tensors is not None and len(tensors):
|
2017-11-05 14:34:40 +00:00
|
|
|
if isinstance(doc.tensor, numpy.ndarray) \
|
|
|
|
and not isinstance(tensors[i], numpy.ndarray):
|
|
|
|
doc.extend_tensor(tensors[i].get())
|
|
|
|
else:
|
|
|
|
doc.extend_tensor(tensors[i])
|
2018-04-10 14:14:52 +00:00
|
|
|
doc.is_tagged = True
|
2017-05-08 12:53:45 +00:00
|
|
|
|
2017-09-21 12:59:48 +00:00
|
|
|
def update(self, docs, golds, drop=0., sgd=None, losses=None):
|
2017-08-20 12:42:23 +00:00
|
|
|
if losses is not None and self.name not in losses:
|
|
|
|
losses[self.name] = 0.
|
2017-05-16 14:17:30 +00:00
|
|
|
|
2017-09-21 12:59:48 +00:00
|
|
|
tag_scores, bp_tag_scores = self.model.begin_update(docs, drop=drop)
|
2017-05-16 14:17:30 +00:00
|
|
|
loss, d_tag_scores = self.get_loss(docs, golds, tag_scores)
|
2017-09-23 00:58:06 +00:00
|
|
|
bp_tag_scores(d_tag_scores, sgd=sgd)
|
2017-05-18 09:29:51 +00:00
|
|
|
|
2017-08-20 12:42:23 +00:00
|
|
|
if losses is not None:
|
|
|
|
losses[self.name] += loss
|
2017-05-16 14:17:30 +00:00
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
2017-05-20 18:23:05 +00:00
|
|
|
scores = self.model.ops.flatten(scores)
|
2017-11-01 18:27:49 +00:00
|
|
|
tag_index = {tag: i for i, tag in enumerate(self.labels)}
|
2017-05-18 09:29:51 +00:00
|
|
|
cdef int idx = 0
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
correct = numpy.zeros((scores.shape[0],), dtype='i')
|
2017-05-19 18:26:36 +00:00
|
|
|
guesses = scores.argmax(axis=1)
|
2018-06-25 20:28:59 +00:00
|
|
|
known_labels = numpy.ones((scores.shape[0], 1), dtype='f')
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
for gold in golds:
|
|
|
|
for tag in gold.tags:
|
2017-05-19 18:26:36 +00:00
|
|
|
if tag is None:
|
|
|
|
correct[idx] = guesses[idx]
|
2018-06-25 20:00:51 +00:00
|
|
|
elif tag in tag_index:
|
2017-05-19 18:26:36 +00:00
|
|
|
correct[idx] = tag_index[tag]
|
2018-06-25 20:00:51 +00:00
|
|
|
else:
|
2018-06-25 20:24:54 +00:00
|
|
|
correct[idx] = 0
|
|
|
|
known_labels[idx] = 0.
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
idx += 1
|
2017-05-18 13:30:59 +00:00
|
|
|
correct = self.model.ops.xp.array(correct, dtype='i')
|
Update draft of parser neural network model
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU.
Outline of the model:
We first predict context-sensitive vectors for each word in the input:
(embed_lower | embed_prefix | embed_suffix | embed_shape)
>> Maxout(token_width)
>> convolution ** 4
This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features.
To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this
by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a
representation that's one affine transform from this informative lexical information. This is obviously good for the
parser (which backprops to the convolutions too).
The parser model makes a state vector by concatenating the vector representations for its context tokens. Current
results suggest few context tokens works well. Maybe this is a bug.
The current context tokens:
* S0, S1, S2: Top three words on the stack
* B0, B1: First two words of the buffer
* S0L1, S0L2: Leftmost and second leftmost children of S0
* S0R1, S0R2: Rightmost and second rightmost children of S0
* S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0
This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately,
there's a way to structure the computation to save some expense (and make it more GPU friendly).
The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks
with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications
for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden
weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN
-- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model
is so big.)
This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity.
The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved
to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier.
We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle
in CUDA to train.
Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to
be 0 cost. This is defined as:
(exp(score) / Z) - (exp(score) / gZ)
Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly,
but so far this isn't working well.
Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit
greatly from the pre-computation trick.
2017-05-12 21:09:15 +00:00
|
|
|
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
|
2018-09-13 12:05:37 +00:00
|
|
|
d_scores *= self.ops.asarray(known_labels)
|
2017-05-18 09:29:51 +00:00
|
|
|
loss = (d_scores**2).sum()
|
2017-05-20 18:23:05 +00:00
|
|
|
d_scores = self.model.ops.unflatten(d_scores, [len(d) for d in docs])
|
2017-05-18 13:30:59 +00:00
|
|
|
return float(loss), d_scores
|
2016-10-15 23:47:12 +00:00
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def begin_training(self, get_gold_tuples=lambda: [], pipeline=None, sgd=None,
|
2018-02-12 09:18:39 +00:00
|
|
|
**kwargs):
|
2017-05-18 13:30:59 +00:00
|
|
|
orig_tag_map = dict(self.vocab.morphology.tag_map)
|
2017-11-08 11:10:49 +00:00
|
|
|
new_tag_map = OrderedDict()
|
2018-03-27 09:39:59 +00:00
|
|
|
for raw_text, annots_brackets in get_gold_tuples():
|
2017-05-17 10:04:50 +00:00
|
|
|
for annots, brackets in annots_brackets:
|
|
|
|
ids, words, tags, heads, deps, ents = annots
|
|
|
|
for tag in tags:
|
2017-05-18 13:30:59 +00:00
|
|
|
if tag in orig_tag_map:
|
|
|
|
new_tag_map[tag] = orig_tag_map[tag]
|
|
|
|
else:
|
|
|
|
new_tag_map[tag] = {POS: X}
|
2017-05-17 10:04:50 +00:00
|
|
|
cdef Vocab vocab = self.vocab
|
2017-06-01 08:04:36 +00:00
|
|
|
if new_tag_map:
|
|
|
|
vocab.morphology = Morphology(vocab.strings, new_tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
vocab.morphology.lemmatizer,
|
|
|
|
exc=vocab.morphology.exc)
|
2018-03-28 14:32:41 +00:00
|
|
|
self.cfg['pretrained_vectors'] = kwargs.get('pretrained_vectors')
|
2017-05-29 18:23:47 +00:00
|
|
|
if self.model is True:
|
2017-09-21 18:07:26 +00:00
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags, **self.cfg)
|
2017-09-26 10:42:52 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-05-29 18:23:47 +00:00
|
|
|
|
|
|
|
@classmethod
|
2017-09-23 00:58:06 +00:00
|
|
|
def Model(cls, n_tags, **cfg):
|
2018-03-28 14:02:59 +00:00
|
|
|
if cfg.get('pretrained_dims') and not cfg.get('pretrained_vectors'):
|
2018-04-03 19:40:29 +00:00
|
|
|
raise ValueError(TempErrors.T008)
|
2017-09-23 00:58:06 +00:00
|
|
|
return build_tagger_model(n_tags, **cfg)
|
2017-09-16 17:46:02 +00:00
|
|
|
|
2017-11-01 20:49:24 +00:00
|
|
|
def add_label(self, label, values=None):
|
2017-11-01 15:32:44 +00:00
|
|
|
if label in self.labels:
|
|
|
|
return 0
|
2017-11-01 20:49:24 +00:00
|
|
|
if self.model not in (True, False, None):
|
|
|
|
# Here's how the model resizing will work, once the
|
|
|
|
# neuron-to-tag mapping is no longer controlled by
|
|
|
|
# the Morphology class, which sorts the tag names.
|
|
|
|
# The sorting makes adding labels difficult.
|
|
|
|
# smaller = self.model._layers[-1]
|
|
|
|
# larger = Softmax(len(self.labels)+1, smaller.nI)
|
|
|
|
# copy_array(larger.W[:smaller.nO], smaller.W)
|
|
|
|
# copy_array(larger.b[:smaller.nO], smaller.b)
|
|
|
|
# self.model._layers[-1] = larger
|
2018-04-03 13:50:31 +00:00
|
|
|
raise ValueError(TempErrors.T003)
|
2017-11-01 20:49:24 +00:00
|
|
|
tag_map = dict(self.vocab.morphology.tag_map)
|
|
|
|
if values is None:
|
|
|
|
values = {POS: "X"}
|
|
|
|
tag_map[label] = values
|
|
|
|
self.vocab.morphology = Morphology(
|
|
|
|
self.vocab.strings, tag_map=tag_map,
|
|
|
|
lemmatizer=self.vocab.morphology.lemmatizer,
|
|
|
|
exc=self.vocab.morphology.exc)
|
|
|
|
return 1
|
2017-11-01 15:32:44 +00:00
|
|
|
|
2017-05-18 13:30:59 +00:00
|
|
|
def use_params(self, params):
|
|
|
|
with self.model.use_params(params):
|
|
|
|
yield
|
|
|
|
|
2017-05-29 08:14:20 +00:00
|
|
|
def to_bytes(self, **exclude):
|
2017-10-10 01:58:12 +00:00
|
|
|
serialize = OrderedDict()
|
|
|
|
if self.model in (None, True, False):
|
|
|
|
serialize['model'] = lambda: self.model
|
|
|
|
else:
|
|
|
|
serialize['model'] = self.model.to_bytes
|
|
|
|
serialize['vocab'] = self.vocab.to_bytes
|
2018-01-23 18:10:49 +00:00
|
|
|
serialize['cfg'] = lambda: ujson.dumps(self.cfg)
|
2017-11-08 12:08:48 +00:00
|
|
|
tag_map = OrderedDict(sorted(self.vocab.morphology.tag_map.items()))
|
2017-10-27 18:29:08 +00:00
|
|
|
serialize['tag_map'] = lambda: msgpack.dumps(
|
2017-11-08 11:10:49 +00:00
|
|
|
tag_map, use_bin_type=True, encoding='utf8')
|
2017-05-29 08:14:20 +00:00
|
|
|
return util.to_bytes(serialize, exclude)
|
|
|
|
|
|
|
|
def from_bytes(self, bytes_data, **exclude):
|
2017-05-29 18:23:47 +00:00
|
|
|
def load_model(b):
|
2018-03-28 14:02:59 +00:00
|
|
|
# TODO: Remove this once we don't have to handle previous models
|
2018-04-10 20:19:05 +00:00
|
|
|
if self.cfg.get('pretrained_dims') and 'pretrained_vectors' not in self.cfg:
|
2018-04-03 19:40:29 +00:00
|
|
|
self.cfg['pretrained_vectors'] = self.vocab.vectors.name
|
|
|
|
|
2017-05-29 18:23:47 +00:00
|
|
|
if self.model is True:
|
2017-10-27 18:29:08 +00:00
|
|
|
token_vector_width = util.env_opt(
|
|
|
|
'token_vector_width',
|
|
|
|
self.cfg.get('token_vector_width', 128))
|
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags,
|
|
|
|
**self.cfg)
|
2017-06-01 17:18:36 +00:00
|
|
|
self.model.from_bytes(b)
|
2017-06-02 15:18:37 +00:00
|
|
|
|
|
|
|
def load_tag_map(b):
|
2017-06-02 15:29:21 +00:00
|
|
|
tag_map = msgpack.loads(b, encoding='utf8')
|
2017-06-02 15:18:37 +00:00
|
|
|
self.vocab.morphology = Morphology(
|
|
|
|
self.vocab.strings, tag_map=tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
lemmatizer=self.vocab.morphology.lemmatizer,
|
|
|
|
exc=self.vocab.morphology.exc)
|
2017-09-16 17:46:02 +00:00
|
|
|
|
2017-05-29 22:53:06 +00:00
|
|
|
deserialize = OrderedDict((
|
|
|
|
('vocab', lambda b: self.vocab.from_bytes(b)),
|
2017-06-02 15:18:37 +00:00
|
|
|
('tag_map', load_tag_map),
|
2017-11-01 20:10:45 +00:00
|
|
|
('cfg', lambda b: self.cfg.update(ujson.loads(b))),
|
2017-05-29 22:53:06 +00:00
|
|
|
('model', lambda b: load_model(b)),
|
|
|
|
))
|
2017-05-29 18:23:47 +00:00
|
|
|
util.from_bytes(bytes_data, deserialize, exclude)
|
2017-05-29 08:14:20 +00:00
|
|
|
return self
|
|
|
|
|
2017-05-29 09:45:45 +00:00
|
|
|
def to_disk(self, path, **exclude):
|
2017-11-08 12:08:48 +00:00
|
|
|
tag_map = OrderedDict(sorted(self.vocab.morphology.tag_map.items()))
|
2017-06-01 17:18:36 +00:00
|
|
|
serialize = OrderedDict((
|
|
|
|
('vocab', lambda p: self.vocab.to_disk(p)),
|
2017-06-02 15:29:21 +00:00
|
|
|
('tag_map', lambda p: p.open('wb').write(msgpack.dumps(
|
2017-11-08 11:10:49 +00:00
|
|
|
tag_map, use_bin_type=True, encoding='utf8'))),
|
2017-06-01 17:18:36 +00:00
|
|
|
('model', lambda p: p.open('wb').write(self.model.to_bytes())),
|
2017-07-22 22:33:43 +00:00
|
|
|
('cfg', lambda p: p.open('w').write(json_dumps(self.cfg)))
|
2017-06-01 17:18:36 +00:00
|
|
|
))
|
2017-05-29 09:45:45 +00:00
|
|
|
util.to_disk(path, serialize, exclude)
|
|
|
|
|
|
|
|
def from_disk(self, path, **exclude):
|
2017-06-01 17:18:36 +00:00
|
|
|
def load_model(p):
|
2018-03-28 14:02:59 +00:00
|
|
|
# TODO: Remove this once we don't have to handle previous models
|
2018-04-10 20:19:05 +00:00
|
|
|
if self.cfg.get('pretrained_dims') and 'pretrained_vectors' not in self.cfg:
|
2018-03-28 14:02:59 +00:00
|
|
|
self.cfg['pretrained_vectors'] = self.vocab.vectors.name
|
2017-06-01 17:18:36 +00:00
|
|
|
if self.model is True:
|
2017-09-21 18:07:26 +00:00
|
|
|
self.model = self.Model(self.vocab.morphology.n_tags, **self.cfg)
|
2018-02-13 19:44:33 +00:00
|
|
|
with p.open('rb') as file_:
|
|
|
|
self.model.from_bytes(file_.read())
|
2017-06-01 17:18:36 +00:00
|
|
|
|
|
|
|
def load_tag_map(p):
|
2017-06-02 15:29:21 +00:00
|
|
|
with p.open('rb') as file_:
|
|
|
|
tag_map = msgpack.loads(file_.read(), encoding='utf8')
|
2017-06-01 17:18:36 +00:00
|
|
|
self.vocab.morphology = Morphology(
|
|
|
|
self.vocab.strings, tag_map=tag_map,
|
2017-06-04 21:34:32 +00:00
|
|
|
lemmatizer=self.vocab.morphology.lemmatizer,
|
|
|
|
exc=self.vocab.morphology.exc)
|
2017-06-01 17:18:36 +00:00
|
|
|
|
|
|
|
deserialize = OrderedDict((
|
2017-09-19 21:42:27 +00:00
|
|
|
('cfg', lambda p: self.cfg.update(_load_cfg(p))),
|
2017-06-01 17:18:36 +00:00
|
|
|
('vocab', lambda p: self.vocab.from_disk(p)),
|
|
|
|
('tag_map', load_tag_map),
|
|
|
|
('model', load_model),
|
|
|
|
))
|
2017-05-29 09:45:45 +00:00
|
|
|
util.from_disk(path, deserialize, exclude)
|
|
|
|
return self
|
2017-05-29 08:14:20 +00:00
|
|
|
|
|
|
|
|
2017-10-26 10:38:23 +00:00
|
|
|
class MultitaskObjective(Tagger):
|
2017-10-27 18:29:08 +00:00
|
|
|
"""Experimental: Assist training of a parser or tagger, by training a
|
|
|
|
side-objective.
|
|
|
|
"""
|
2017-05-21 22:52:30 +00:00
|
|
|
name = 'nn_labeller'
|
2017-10-27 18:29:08 +00:00
|
|
|
|
2017-09-26 10:42:52 +00:00
|
|
|
def __init__(self, vocab, model=True, target='dep_tag_offset', **cfg):
|
2017-05-21 22:52:30 +00:00
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-09-26 10:42:52 +00:00
|
|
|
if target == 'dep':
|
|
|
|
self.make_label = self.make_dep
|
|
|
|
elif target == 'tag':
|
|
|
|
self.make_label = self.make_tag
|
|
|
|
elif target == 'ent':
|
|
|
|
self.make_label = self.make_ent
|
|
|
|
elif target == 'dep_tag_offset':
|
|
|
|
self.make_label = self.make_dep_tag_offset
|
|
|
|
elif target == 'ent_tag':
|
|
|
|
self.make_label = self.make_ent_tag
|
2018-03-27 17:23:02 +00:00
|
|
|
elif target == 'sent_start':
|
|
|
|
self.make_label = self.make_sent_start
|
2017-09-26 10:42:52 +00:00
|
|
|
elif hasattr(target, '__call__'):
|
|
|
|
self.make_label = target
|
|
|
|
else:
|
2018-04-03 13:50:31 +00:00
|
|
|
raise ValueError(Errors.E016)
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-09-21 00:15:49 +00:00
|
|
|
self.cfg.setdefault('cnn_maxout_pieces', 2)
|
2017-07-22 22:52:47 +00:00
|
|
|
|
|
|
|
@property
|
|
|
|
def labels(self):
|
2017-08-18 20:02:35 +00:00
|
|
|
return self.cfg.setdefault('labels', {})
|
2017-07-22 22:52:47 +00:00
|
|
|
|
|
|
|
@labels.setter
|
|
|
|
def labels(self, value):
|
|
|
|
self.cfg['labels'] = value
|
2017-05-21 22:52:30 +00:00
|
|
|
|
2017-11-03 10:20:05 +00:00
|
|
|
def set_annotations(self, docs, dep_ids, tensors=None):
|
2017-05-21 22:52:30 +00:00
|
|
|
pass
|
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def begin_training(self, get_gold_tuples=lambda: [], pipeline=None, tok2vec=None,
|
2018-02-12 09:18:39 +00:00
|
|
|
sgd=None, **kwargs):
|
2018-03-27 09:39:59 +00:00
|
|
|
gold_tuples = nonproj.preprocess_training_data(get_gold_tuples())
|
2017-05-21 22:52:30 +00:00
|
|
|
for raw_text, annots_brackets in gold_tuples:
|
|
|
|
for annots, brackets in annots_brackets:
|
|
|
|
ids, words, tags, heads, deps, ents = annots
|
2017-09-26 10:42:52 +00:00
|
|
|
for i in range(len(ids)):
|
|
|
|
label = self.make_label(i, words, tags, heads, deps, ents)
|
|
|
|
if label is not None and label not in self.labels:
|
|
|
|
self.labels[label] = len(self.labels)
|
2017-05-29 18:23:47 +00:00
|
|
|
if self.model is True:
|
2017-09-27 16:43:58 +00:00
|
|
|
token_vector_width = util.env_opt('token_vector_width')
|
2018-01-21 18:21:34 +00:00
|
|
|
self.model = self.Model(len(self.labels), tok2vec=tok2vec)
|
2017-09-26 10:42:52 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-05-29 18:23:47 +00:00
|
|
|
|
|
|
|
@classmethod
|
2017-09-26 10:42:52 +00:00
|
|
|
def Model(cls, n_tags, tok2vec=None, **cfg):
|
2018-01-21 18:21:34 +00:00
|
|
|
token_vector_width = util.env_opt('token_vector_width', 128)
|
|
|
|
softmax = Softmax(n_tags, token_vector_width)
|
|
|
|
model = chain(
|
|
|
|
tok2vec,
|
|
|
|
softmax
|
|
|
|
)
|
|
|
|
model.tok2vec = tok2vec
|
|
|
|
model.softmax = softmax
|
|
|
|
return model
|
|
|
|
|
|
|
|
def predict(self, docs):
|
|
|
|
tokvecs = self.model.tok2vec(docs)
|
|
|
|
scores = self.model.softmax(tokvecs)
|
|
|
|
return tokvecs, scores
|
2017-09-16 17:46:02 +00:00
|
|
|
|
2017-05-21 22:52:30 +00:00
|
|
|
def get_loss(self, docs, golds, scores):
|
2018-04-03 13:50:31 +00:00
|
|
|
if len(docs) != len(golds):
|
|
|
|
raise ValueError(Errors.E077.format(value='loss', n_docs=len(docs),
|
|
|
|
n_golds=len(golds)))
|
2017-05-21 22:52:30 +00:00
|
|
|
cdef int idx = 0
|
|
|
|
correct = numpy.zeros((scores.shape[0],), dtype='i')
|
|
|
|
guesses = scores.argmax(axis=1)
|
2018-02-17 17:41:18 +00:00
|
|
|
for i, gold in enumerate(golds):
|
|
|
|
for j in range(len(docs[i])):
|
|
|
|
# Handes alignment for tokenization differences
|
2018-03-27 17:23:02 +00:00
|
|
|
label = self.make_label(j, gold.words, gold.tags,
|
2018-02-17 17:41:18 +00:00
|
|
|
gold.heads, gold.labels, gold.ents)
|
2017-09-26 10:42:52 +00:00
|
|
|
if label is None or label not in self.labels:
|
2017-05-21 22:52:30 +00:00
|
|
|
correct[idx] = guesses[idx]
|
|
|
|
else:
|
2017-09-26 10:42:52 +00:00
|
|
|
correct[idx] = self.labels[label]
|
2017-05-21 22:52:30 +00:00
|
|
|
idx += 1
|
|
|
|
correct = self.model.ops.xp.array(correct, dtype='i')
|
|
|
|
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
|
|
|
|
loss = (d_scores**2).sum()
|
|
|
|
return float(loss), d_scores
|
|
|
|
|
2017-09-26 10:42:52 +00:00
|
|
|
@staticmethod
|
|
|
|
def make_dep(i, words, tags, heads, deps, ents):
|
|
|
|
if deps[i] is None or heads[i] is None:
|
|
|
|
return None
|
|
|
|
return deps[i]
|
|
|
|
|
|
|
|
@staticmethod
|
|
|
|
def make_tag(i, words, tags, heads, deps, ents):
|
|
|
|
return tags[i]
|
|
|
|
|
|
|
|
@staticmethod
|
|
|
|
def make_ent(i, words, tags, heads, deps, ents):
|
|
|
|
if ents is None:
|
|
|
|
return None
|
|
|
|
return ents[i]
|
|
|
|
|
|
|
|
@staticmethod
|
|
|
|
def make_dep_tag_offset(i, words, tags, heads, deps, ents):
|
|
|
|
if deps[i] is None or heads[i] is None:
|
|
|
|
return None
|
|
|
|
offset = heads[i] - i
|
|
|
|
offset = min(offset, 2)
|
|
|
|
offset = max(offset, -2)
|
|
|
|
return '%s-%s:%d' % (deps[i], tags[i], offset)
|
|
|
|
|
|
|
|
@staticmethod
|
|
|
|
def make_ent_tag(i, words, tags, heads, deps, ents):
|
|
|
|
if ents is None or ents[i] is None:
|
|
|
|
return None
|
|
|
|
else:
|
|
|
|
return '%s-%s' % (tags[i], ents[i])
|
|
|
|
|
2018-03-27 17:23:02 +00:00
|
|
|
@staticmethod
|
|
|
|
def make_sent_start(target, words, tags, heads, deps, ents, cache=True, _cache={}):
|
|
|
|
'''A multi-task objective for representing sentence boundaries,
|
|
|
|
using BILU scheme. (O is impossible)
|
|
|
|
|
|
|
|
The implementation of this method uses an internal cache that relies
|
|
|
|
on the identity of the heads array, to avoid requiring a new piece
|
|
|
|
of gold data. You can pass cache=False if you know the cache will
|
|
|
|
do the wrong thing.
|
|
|
|
'''
|
|
|
|
assert len(words) == len(heads)
|
|
|
|
assert target < len(words), (target, len(words))
|
|
|
|
if cache:
|
|
|
|
if id(heads) in _cache:
|
|
|
|
return _cache[id(heads)][target]
|
|
|
|
else:
|
|
|
|
for key in list(_cache.keys()):
|
|
|
|
_cache.pop(key)
|
|
|
|
sent_tags = ['I-SENT'] * len(words)
|
|
|
|
_cache[id(heads)] = sent_tags
|
|
|
|
else:
|
|
|
|
sent_tags = ['I-SENT'] * len(words)
|
|
|
|
|
|
|
|
def _find_root(child):
|
|
|
|
seen = set([child])
|
|
|
|
while child is not None and heads[child] != child:
|
|
|
|
seen.add(child)
|
|
|
|
child = heads[child]
|
|
|
|
return child
|
|
|
|
|
|
|
|
sentences = {}
|
|
|
|
for i in range(len(words)):
|
|
|
|
root = _find_root(i)
|
|
|
|
if root is None:
|
|
|
|
sent_tags[i] = None
|
|
|
|
else:
|
|
|
|
sentences.setdefault(root, []).append(i)
|
|
|
|
for root, span in sorted(sentences.items()):
|
|
|
|
if len(span) == 1:
|
|
|
|
sent_tags[span[0]] = 'U-SENT'
|
|
|
|
else:
|
|
|
|
sent_tags[span[0]] = 'B-SENT'
|
|
|
|
sent_tags[span[-1]] = 'L-SENT'
|
|
|
|
return sent_tags[target]
|
|
|
|
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2017-10-26 10:40:40 +00:00
|
|
|
class SimilarityHook(Pipe):
|
2017-06-05 13:40:03 +00:00
|
|
|
"""
|
2017-10-27 18:29:08 +00:00
|
|
|
Experimental: A pipeline component to install a hook for supervised
|
|
|
|
similarity into `Doc` objects. Requires a `Tensorizer` to pre-process
|
|
|
|
documents. The similarity model can be any object obeying the Thinc `Model`
|
|
|
|
interface. By default, the model concatenates the elementwise mean and
|
|
|
|
elementwise max of the two tensors, and compares them using the
|
|
|
|
Cauchy-like similarity function from Chen (2013):
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-10-27 18:29:08 +00:00
|
|
|
>>> similarity = 1. / (1. + (W * (vec1-vec2)**2).sum())
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
Where W is a vector of dimension weights, initialized to 1.
|
|
|
|
"""
|
|
|
|
name = 'similarity'
|
2017-10-27 18:29:08 +00:00
|
|
|
|
2017-07-22 22:52:47 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
2017-06-05 13:40:03 +00:00
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
@classmethod
|
|
|
|
def Model(cls, length):
|
|
|
|
return Siamese(Pooling(max_pool, mean_pool), CauchySimilarity(length))
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
2017-09-25 16:37:13 +00:00
|
|
|
"""Install similarity hook"""
|
2017-06-05 13:40:03 +00:00
|
|
|
doc.user_hooks['similarity'] = self.predict
|
|
|
|
return doc
|
|
|
|
|
|
|
|
def pipe(self, docs, **kwargs):
|
|
|
|
for doc in docs:
|
|
|
|
yield self(doc)
|
|
|
|
|
|
|
|
def predict(self, doc1, doc2):
|
2017-09-21 12:59:48 +00:00
|
|
|
return self.model.predict([(doc1, doc2)])
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-09-21 12:59:48 +00:00
|
|
|
def update(self, doc1_doc2, golds, sgd=None, drop=0.):
|
|
|
|
sims, bp_sims = self.model.begin_update(doc1_doc2, drop=drop)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2018-02-12 09:18:39 +00:00
|
|
|
def begin_training(self, _=tuple(), pipeline=None, sgd=None, **kwargs):
|
2017-10-27 18:29:08 +00:00
|
|
|
"""Allocate model, using width from tensorizer in pipeline.
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
gold_tuples (iterable): Gold-standard training data.
|
|
|
|
pipeline (list): The pipeline the model is part of.
|
|
|
|
"""
|
|
|
|
if self.model is True:
|
|
|
|
self.model = self.Model(pipeline[0].model.nO)
|
2017-09-22 14:38:22 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
|
2017-10-26 10:40:40 +00:00
|
|
|
class TextCategorizer(Pipe):
|
2017-07-21 23:14:07 +00:00
|
|
|
name = 'textcat'
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
@classmethod
|
2018-04-29 13:48:53 +00:00
|
|
|
def Model(cls, nr_class, **cfg):
|
|
|
|
return build_text_classifier(nr_class, **cfg)
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def __init__(self, vocab, model=True, **cfg):
|
|
|
|
self.vocab = vocab
|
|
|
|
self.model = model
|
2017-07-22 22:52:47 +00:00
|
|
|
self.cfg = dict(cfg)
|
2017-07-22 22:33:43 +00:00
|
|
|
|
|
|
|
@property
|
|
|
|
def labels(self):
|
2017-11-06 21:09:02 +00:00
|
|
|
return self.cfg.setdefault('labels', [])
|
2017-07-22 22:33:43 +00:00
|
|
|
|
|
|
|
@labels.setter
|
|
|
|
def labels(self, value):
|
|
|
|
self.cfg['labels'] = value
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def __call__(self, doc):
|
2017-11-03 10:20:05 +00:00
|
|
|
scores, tensors = self.predict([doc])
|
|
|
|
self.set_annotations([doc], scores, tensors=tensors)
|
2017-07-19 22:18:15 +00:00
|
|
|
return doc
|
2017-06-05 13:40:03 +00:00
|
|
|
|
2017-07-19 22:18:15 +00:00
|
|
|
def pipe(self, stream, batch_size=128, n_threads=-1):
|
|
|
|
for docs in cytoolz.partition_all(batch_size, stream):
|
|
|
|
docs = list(docs)
|
2017-11-03 10:20:05 +00:00
|
|
|
scores, tensors = self.predict(docs)
|
|
|
|
self.set_annotations(docs, scores, tensors=tensors)
|
2017-07-19 22:18:15 +00:00
|
|
|
yield from docs
|
|
|
|
|
|
|
|
def predict(self, docs):
|
|
|
|
scores = self.model(docs)
|
|
|
|
scores = self.model.ops.asarray(scores)
|
2017-11-05 11:25:10 +00:00
|
|
|
tensors = [doc.tensor for doc in docs]
|
|
|
|
return scores, tensors
|
2017-07-19 22:18:15 +00:00
|
|
|
|
2017-11-03 10:20:05 +00:00
|
|
|
def set_annotations(self, docs, scores, tensors=None):
|
2017-07-19 22:18:15 +00:00
|
|
|
for i, doc in enumerate(docs):
|
2017-07-22 18:04:43 +00:00
|
|
|
for j, label in enumerate(self.labels):
|
2017-07-19 22:18:15 +00:00
|
|
|
doc.cats[label] = float(scores[i, j])
|
|
|
|
|
2017-09-21 12:59:48 +00:00
|
|
|
def update(self, docs, golds, state=None, drop=0., sgd=None, losses=None):
|
2017-07-19 22:18:15 +00:00
|
|
|
scores, bp_scores = self.model.begin_update(docs, drop=drop)
|
|
|
|
loss, d_scores = self.get_loss(docs, golds, scores)
|
2017-09-21 12:59:48 +00:00
|
|
|
bp_scores(d_scores, sgd=sgd)
|
2017-07-19 22:18:15 +00:00
|
|
|
if losses is not None:
|
|
|
|
losses.setdefault(self.name, 0.0)
|
|
|
|
losses[self.name] += loss
|
|
|
|
|
|
|
|
def get_loss(self, docs, golds, scores):
|
|
|
|
truths = numpy.zeros((len(golds), len(self.labels)), dtype='f')
|
2017-10-05 23:43:02 +00:00
|
|
|
not_missing = numpy.ones((len(golds), len(self.labels)), dtype='f')
|
2017-07-19 22:18:15 +00:00
|
|
|
for i, gold in enumerate(golds):
|
|
|
|
for j, label in enumerate(self.labels):
|
2017-10-05 23:43:02 +00:00
|
|
|
if label in gold.cats:
|
|
|
|
truths[i, j] = gold.cats[label]
|
|
|
|
else:
|
|
|
|
not_missing[i, j] = 0.
|
2017-07-19 22:18:15 +00:00
|
|
|
truths = self.model.ops.asarray(truths)
|
2017-10-05 23:43:02 +00:00
|
|
|
not_missing = self.model.ops.asarray(not_missing)
|
2017-07-19 22:18:15 +00:00
|
|
|
d_scores = (scores-truths) / scores.shape[0]
|
2017-10-05 23:43:02 +00:00
|
|
|
d_scores *= not_missing
|
2017-07-19 22:18:15 +00:00
|
|
|
mean_square_error = ((scores-truths)**2).sum(axis=1).mean()
|
|
|
|
return mean_square_error, d_scores
|
|
|
|
|
2017-11-01 15:32:44 +00:00
|
|
|
def add_label(self, label):
|
|
|
|
if label in self.labels:
|
|
|
|
return 0
|
2017-11-01 16:06:43 +00:00
|
|
|
if self.model not in (None, True, False):
|
2018-03-27 17:23:02 +00:00
|
|
|
# This functionality was available previously, but was broken.
|
|
|
|
# The problem is that we resize the last layer, but the last layer
|
|
|
|
# is actually just an ensemble. We're not resizing the child layers
|
|
|
|
# -- a huge problem.
|
|
|
|
raise ValueError(
|
|
|
|
"Cannot currently add labels to pre-trained text classifier. "
|
|
|
|
"Add labels before training begins. This functionality was "
|
|
|
|
"available in previous versions, but had significant bugs that "
|
|
|
|
"let to poor performance")
|
2018-07-06 09:31:22 +00:00
|
|
|
#smaller = self.model._layers[-1]
|
|
|
|
#larger = Affine(len(self.labels)+1, smaller.nI)
|
|
|
|
#copy_array(larger.W[:smaller.nO], smaller.W)
|
|
|
|
#copy_array(larger.b[:smaller.nO], smaller.b)
|
|
|
|
#self.model._layers[-1] = larger
|
2017-11-01 15:32:44 +00:00
|
|
|
self.labels.append(label)
|
|
|
|
return 1
|
|
|
|
|
2018-04-29 12:49:26 +00:00
|
|
|
def begin_training(self, get_gold_tuples=lambda: [], pipeline=None, sgd=None,
|
2018-03-28 14:32:41 +00:00
|
|
|
**kwargs):
|
2017-09-02 12:56:30 +00:00
|
|
|
if pipeline and getattr(pipeline[0], 'name', None) == 'tensorizer':
|
2017-07-22 18:04:43 +00:00
|
|
|
token_vector_width = pipeline[0].model.nO
|
|
|
|
else:
|
|
|
|
token_vector_width = 64
|
2018-03-28 14:32:41 +00:00
|
|
|
|
2017-06-05 13:40:03 +00:00
|
|
|
if self.model is True:
|
2018-03-28 14:32:41 +00:00
|
|
|
self.cfg['pretrained_vectors'] = kwargs.get('pretrained_vectors')
|
2018-04-29 13:48:53 +00:00
|
|
|
self.model = self.Model(len(self.labels), **self.cfg)
|
2017-09-22 14:38:22 +00:00
|
|
|
link_vectors_to_models(self.vocab)
|
2017-11-06 13:26:26 +00:00
|
|
|
if sgd is None:
|
|
|
|
sgd = self.create_optimizer()
|
|
|
|
return sgd
|
2017-06-05 13:40:03 +00:00
|
|
|
|
|
|
|
|
2017-10-26 10:38:23 +00:00
|
|
|
cdef class DependencyParser(Parser):
|
2017-05-16 09:21:59 +00:00
|
|
|
name = 'parser'
|
|
|
|
TransitionSystem = ArcEager
|
|
|
|
|
2017-10-07 00:00:47 +00:00
|
|
|
@property
|
|
|
|
def postprocesses(self):
|
|
|
|
return [nonproj.deprojectivize]
|
2018-03-27 17:23:02 +00:00
|
|
|
|
2018-01-21 18:37:02 +00:00
|
|
|
def add_multitask_objective(self, target):
|
|
|
|
labeller = MultitaskObjective(self.vocab, target=target)
|
|
|
|
self._multitasks.append(labeller)
|
2017-10-07 00:00:47 +00:00
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def init_multitask_objectives(self, get_gold_tuples, pipeline, sgd=None, **cfg):
|
2018-01-21 18:37:02 +00:00
|
|
|
for labeller in self._multitasks:
|
2018-09-13 12:08:55 +00:00
|
|
|
tok2vec = self.model.tok2vec
|
2018-03-27 09:39:59 +00:00
|
|
|
labeller.begin_training(get_gold_tuples, pipeline=pipeline,
|
2017-11-06 13:26:26 +00:00
|
|
|
tok2vec=tok2vec, sgd=sgd)
|
2017-09-26 10:42:52 +00:00
|
|
|
|
2017-05-27 20:46:06 +00:00
|
|
|
def __reduce__(self):
|
2017-10-27 18:29:08 +00:00
|
|
|
return (DependencyParser, (self.vocab, self.moves, self.model),
|
|
|
|
None, None)
|
2017-05-27 20:46:06 +00:00
|
|
|
|
2017-05-16 09:21:59 +00:00
|
|
|
|
2017-10-26 10:38:23 +00:00
|
|
|
cdef class EntityRecognizer(Parser):
|
2017-05-31 11:42:39 +00:00
|
|
|
name = 'ner'
|
2017-05-16 09:21:59 +00:00
|
|
|
TransitionSystem = BiluoPushDown
|
|
|
|
|
2017-05-17 10:04:50 +00:00
|
|
|
nr_feature = 6
|
2018-03-27 17:23:02 +00:00
|
|
|
|
2018-01-21 18:37:02 +00:00
|
|
|
def add_multitask_objective(self, target):
|
|
|
|
labeller = MultitaskObjective(self.vocab, target=target)
|
|
|
|
self._multitasks.append(labeller)
|
2017-05-17 10:04:50 +00:00
|
|
|
|
2018-03-27 09:39:59 +00:00
|
|
|
def init_multitask_objectives(self, get_gold_tuples, pipeline, sgd=None, **cfg):
|
2018-01-21 18:37:02 +00:00
|
|
|
for labeller in self._multitasks:
|
2018-09-13 12:08:55 +00:00
|
|
|
tok2vec = self.model.tok2vec
|
2018-03-27 09:39:59 +00:00
|
|
|
labeller.begin_training(get_gold_tuples, pipeline=pipeline,
|
2017-10-27 18:29:08 +00:00
|
|
|
tok2vec=tok2vec)
|
2017-08-18 20:02:35 +00:00
|
|
|
|
2017-05-27 20:46:06 +00:00
|
|
|
def __reduce__(self):
|
2017-10-27 18:29:08 +00:00
|
|
|
return (EntityRecognizer, (self.vocab, self.moves, self.model),
|
|
|
|
None, None)
|
2017-10-07 00:00:47 +00:00
|
|
|
|
2017-03-15 14:27:41 +00:00
|
|
|
|
2017-10-26 10:38:23 +00:00
|
|
|
__all__ = ['Tagger', 'DependencyParser', 'EntityRecognizer', 'Tensorizer']
|