2020-07-22 11:42:59 +00:00
|
|
|
|
# cython: infer_types=True, profile=True, binding=True
|
2023-06-14 15:48:41 +00:00
|
|
|
|
from typing import Callable, List, Optional
|
|
|
|
|
|
2021-01-29 00:51:21 +00:00
|
|
|
|
import srsly
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
|
|
|
|
from ..tokens.doc cimport Doc
|
2021-08-10 13:13:39 +00:00
|
|
|
|
|
2023-06-14 15:48:41 +00:00
|
|
|
|
from .. import util
|
2020-07-22 11:42:59 +00:00
|
|
|
|
from ..language import Language
|
Refactor the Scorer to improve flexibility (#5731)
* Refactor the Scorer to improve flexibility
Refactor the `Scorer` to improve flexibility for arbitrary pipeline
components.
* Individual pipeline components provide their own `evaluate` methods
that score a list of `Example`s and return a dictionary of scores
* `Scorer` is initialized either:
* with a provided pipeline containing components to be scored
* with a default pipeline containing the built-in statistical
components (senter, tagger, morphologizer, parser, ner)
* `Scorer.score` evaluates a list of `Example`s and returns a dictionary
of scores referring to the scores provided by the components in the
pipeline
Significant differences:
* `tags_acc` is renamed to `tag_acc` to be consistent with `token_acc`
and the new `morph_acc`, `pos_acc`, and `lemma_acc`
* Scoring is no longer cumulative: `Scorer.score` scores a list of
examples rather than a single example and does not retain any state
about previously scored examples
* PRF values in the returned scores are no longer multiplied by 100
* Add kwargs to Morphologizer.evaluate
* Create generalized scoring methods in Scorer
* Generalized static scoring methods are added to `Scorer`
* Methods require an attribute (either on Token or Doc) that is
used to key the returned scores
Naming differences:
* `uas`, `las`, and `las_per_type` in the scores dict are renamed to
`dep_uas`, `dep_las`, and `dep_las_per_type`
Scoring differences:
* `Doc.sents` is now scored as spans rather than on sentence-initial
token positions so that `Doc.sents` and `Doc.ents` can be scored with
the same method (this lowers scores since a single incorrect sentence
start results in two incorrect spans)
* Simplify / extend hasattr check for eval method
* Add hasattr check to tokenizer scoring
* Simplify to hasattr check for component scoring
* Reset Example alignment if docs are set
Reset the Example alignment if either doc is set in case the
tokenization has changed.
* Add PRF tokenization scoring for tokens as spans
Add PRF scores for tokens as character spans. The scores are:
* token_acc: # correct tokens / # gold tokens
* token_p/r/f: PRF for (token.idx, token.idx + len(token))
* Add docstring to Scorer.score_tokenization
* Rename component.evaluate() to component.score()
* Update Scorer API docs
* Update scoring for positive_label in textcat
* Fix TextCategorizer.score kwargs
* Update Language.evaluate docs
* Update score names in default config
2020-07-25 10:53:02 +00:00
|
|
|
|
from ..scorer import Scorer
|
2023-06-14 15:48:41 +00:00
|
|
|
|
from .pipe import Pipe
|
|
|
|
|
from .senter import senter_score
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
# see #9050
|
|
|
|
|
BACKWARD_OVERWRITE = False
|
2021-08-10 13:13:39 +00:00
|
|
|
|
|
2020-07-22 11:42:59 +00:00
|
|
|
|
@Language.factory(
|
|
|
|
|
"sentencizer",
|
|
|
|
|
assigns=["token.is_sent_start", "doc.sents"],
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
default_config={"punct_chars": None, "overwrite": False, "scorer": {"@scorers": "spacy.senter_scorer.v1"}},
|
2020-07-27 10:27:40 +00:00
|
|
|
|
default_score_weights={"sents_f": 1.0, "sents_p": 0.0, "sents_r": 0.0},
|
2020-07-22 11:42:59 +00:00
|
|
|
|
)
|
|
|
|
|
def make_sentencizer(
|
|
|
|
|
nlp: Language,
|
|
|
|
|
name: str,
|
2021-08-10 13:13:39 +00:00
|
|
|
|
punct_chars: Optional[List[str]],
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
overwrite: bool,
|
2021-08-10 13:13:39 +00:00
|
|
|
|
scorer: Optional[Callable],
|
2020-07-22 11:42:59 +00:00
|
|
|
|
):
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
return Sentencizer(name, punct_chars=punct_chars, overwrite=overwrite, scorer=scorer)
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class Sentencizer(Pipe):
|
|
|
|
|
"""Segment the Doc into sentences using a rule-based strategy.
|
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
default_punct_chars = ['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹',
|
|
|
|
|
'।', '॥', '၊', '။', '።', '፧', '፨', '᙮', '᜵', '᜶', '᠃', '᠉', '᥄',
|
|
|
|
|
'᥅', '᪨', '᪩', '᪪', '᪫', '᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿',
|
|
|
|
|
'‼', '‽', '⁇', '⁈', '⁉', '⸮', '⸼', '꓿', '꘎', '꘏', '꛳', '꛷', '꡶',
|
|
|
|
|
'꡷', '꣎', '꣏', '꤯', '꧈', '꧉', '꩝', '꩞', '꩟', '꫰', '꫱', '꯫', '﹒',
|
|
|
|
|
'﹖', '﹗', '!', '.', '?', '𐩖', '𐩗', '𑁇', '𑁈', '𑂾', '𑂿', '𑃀',
|
|
|
|
|
'𑃁', '𑅁', '𑅂', '𑅃', '𑇅', '𑇆', '𑇍', '𑇞', '𑇟', '𑈸', '𑈹', '𑈻', '𑈼',
|
|
|
|
|
'𑊩', '𑑋', '𑑌', '𑗂', '𑗃', '𑗉', '𑗊', '𑗋', '𑗌', '𑗍', '𑗎', '𑗏', '𑗐',
|
|
|
|
|
'𑗑', '𑗒', '𑗓', '𑗔', '𑗕', '𑗖', '𑗗', '𑙁', '𑙂', '𑜼', '𑜽', '𑜾', '𑩂',
|
|
|
|
|
'𑩃', '𑪛', '𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈',
|
|
|
|
|
'。', '。']
|
|
|
|
|
|
2021-08-10 13:13:39 +00:00
|
|
|
|
def __init__(
|
|
|
|
|
self,
|
|
|
|
|
name="sentencizer",
|
|
|
|
|
*,
|
|
|
|
|
punct_chars=None,
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
overwrite=BACKWARD_OVERWRITE,
|
2021-08-10 13:13:39 +00:00
|
|
|
|
scorer=senter_score,
|
|
|
|
|
):
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""Initialize the sentencizer.
|
|
|
|
|
|
|
|
|
|
punct_chars (list): Punctuation characters to split on. Will be
|
|
|
|
|
serialized with the nlp object.
|
2021-08-12 10:50:03 +00:00
|
|
|
|
scorer (Optional[Callable]): The scoring method. Defaults to
|
|
|
|
|
Scorer.score_spans for the attribute "sents".
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#init
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
self.name = name
|
|
|
|
|
if punct_chars:
|
|
|
|
|
self.punct_chars = set(punct_chars)
|
|
|
|
|
else:
|
|
|
|
|
self.punct_chars = set(self.default_punct_chars)
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
self.overwrite = overwrite
|
2021-08-10 13:13:39 +00:00
|
|
|
|
self.scorer = scorer
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
|
|
|
|
def __call__(self, doc):
|
|
|
|
|
"""Apply the sentencizer to a Doc and set Token.is_sent_start.
|
|
|
|
|
|
2020-07-27 16:11:45 +00:00
|
|
|
|
doc (Doc): The document to process.
|
|
|
|
|
RETURNS (Doc): The processed Doc.
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#call
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
2021-01-29 00:51:21 +00:00
|
|
|
|
error_handler = self.get_error_handler()
|
|
|
|
|
try:
|
2021-02-26 08:48:14 +00:00
|
|
|
|
tags = self.predict([doc])
|
|
|
|
|
self.set_annotations([doc], tags)
|
2021-01-29 00:51:21 +00:00
|
|
|
|
return doc
|
|
|
|
|
except Exception as e:
|
|
|
|
|
error_handler(self.name, self, [doc], e)
|
|
|
|
|
|
2020-07-22 11:42:59 +00:00
|
|
|
|
def predict(self, docs):
|
2020-07-27 16:11:45 +00:00
|
|
|
|
"""Apply the pipe to a batch of docs, without modifying them.
|
|
|
|
|
|
|
|
|
|
docs (Iterable[Doc]): The documents to predict.
|
|
|
|
|
RETURNS: The predictions for each document.
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
if not any(len(doc) for doc in docs):
|
|
|
|
|
# Handle cases where there are no tokens in any docs.
|
|
|
|
|
guesses = [[] for doc in docs]
|
|
|
|
|
return guesses
|
|
|
|
|
guesses = []
|
|
|
|
|
for doc in docs:
|
|
|
|
|
doc_guesses = [False] * len(doc)
|
|
|
|
|
if len(doc) > 0:
|
|
|
|
|
start = 0
|
|
|
|
|
seen_period = False
|
|
|
|
|
doc_guesses[0] = True
|
|
|
|
|
for i, token in enumerate(doc):
|
|
|
|
|
is_in_punct_chars = token.text in self.punct_chars
|
|
|
|
|
if seen_period and not token.is_punct and not is_in_punct_chars:
|
|
|
|
|
doc_guesses[start] = True
|
|
|
|
|
start = token.i
|
|
|
|
|
seen_period = False
|
|
|
|
|
elif is_in_punct_chars:
|
|
|
|
|
seen_period = True
|
|
|
|
|
if start < len(doc):
|
|
|
|
|
doc_guesses[start] = True
|
|
|
|
|
guesses.append(doc_guesses)
|
|
|
|
|
return guesses
|
|
|
|
|
|
|
|
|
|
def set_annotations(self, docs, batch_tag_ids):
|
2020-07-27 16:11:45 +00:00
|
|
|
|
"""Modify a batch of documents, using pre-computed scores.
|
|
|
|
|
|
|
|
|
|
docs (Iterable[Doc]): The documents to modify.
|
|
|
|
|
scores: The tag IDs produced by Sentencizer.predict.
|
|
|
|
|
"""
|
2020-07-22 11:42:59 +00:00
|
|
|
|
if isinstance(docs, Doc):
|
|
|
|
|
docs = [docs]
|
|
|
|
|
cdef Doc doc
|
|
|
|
|
cdef int idx = 0
|
|
|
|
|
for i, doc in enumerate(docs):
|
|
|
|
|
doc_tag_ids = batch_tag_ids[i]
|
|
|
|
|
for j, tag_id in enumerate(doc_tag_ids):
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
if doc.c[j].sent_start == 0 or self.overwrite:
|
2020-07-22 11:42:59 +00:00
|
|
|
|
if tag_id:
|
|
|
|
|
doc.c[j].sent_start = 1
|
|
|
|
|
else:
|
|
|
|
|
doc.c[j].sent_start = -1
|
|
|
|
|
|
2020-07-29 13:14:07 +00:00
|
|
|
|
def to_bytes(self, *, exclude=tuple()):
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""Serialize the sentencizer to a bytestring.
|
|
|
|
|
|
|
|
|
|
RETURNS (bytes): The serialized object.
|
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#to_bytes
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
return srsly.msgpack_dumps({"punct_chars": list(self.punct_chars), "overwrite": self.overwrite})
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
2020-07-29 13:14:07 +00:00
|
|
|
|
def from_bytes(self, bytes_data, *, exclude=tuple()):
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""Load the sentencizer from a bytestring.
|
|
|
|
|
|
|
|
|
|
bytes_data (bytes): The data to load.
|
|
|
|
|
returns (Sentencizer): The loaded object.
|
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#from_bytes
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
cfg = srsly.msgpack_loads(bytes_data)
|
|
|
|
|
self.punct_chars = set(cfg.get("punct_chars", self.default_punct_chars))
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
self.overwrite = cfg.get("overwrite", self.overwrite)
|
2020-07-22 11:42:59 +00:00
|
|
|
|
return self
|
|
|
|
|
|
2020-07-29 13:14:07 +00:00
|
|
|
|
def to_disk(self, path, *, exclude=tuple()):
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""Serialize the sentencizer to disk.
|
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#to_disk
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
path = util.ensure_path(path)
|
|
|
|
|
path = path.with_suffix(".json")
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
srsly.write_json(path, {"punct_chars": list(self.punct_chars), "overwrite": self.overwrite})
|
2020-07-22 11:42:59 +00:00
|
|
|
|
|
|
|
|
|
|
2020-07-29 13:14:07 +00:00
|
|
|
|
def from_disk(self, path, *, exclude=tuple()):
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""Load the sentencizer from disk.
|
|
|
|
|
|
2021-01-30 09:09:38 +00:00
|
|
|
|
DOCS: https://spacy.io/api/sentencizer#from_disk
|
2020-07-22 11:42:59 +00:00
|
|
|
|
"""
|
|
|
|
|
path = util.ensure_path(path)
|
|
|
|
|
path = path.with_suffix(".json")
|
|
|
|
|
cfg = srsly.read_json(path)
|
|
|
|
|
self.punct_chars = set(cfg.get("punct_chars", self.default_punct_chars))
|
Add overwrite settings for more components (#9050)
* Add overwrite settings for more components
For pipeline components where it's relevant and not already implemented,
add an explicit `overwrite` setting that controls whether
`set_annotations` overwrites existing annotation.
For the `morphologizer`, add an additional setting `extend`, which
controls whether the existing features are preserved.
* +overwrite, +extend: overwrite values of existing features, add any new
features
* +overwrite, -extend: overwrite completely, removing any existing
features
* -overwrite, +extend: keep values of existing features, add any new
features
* -overwrite, -extend: do not modify the existing value if set
In all cases an unset value will be set by `set_annotations`.
Preserve current overwrite defaults:
* True: morphologizer, entity linker
* False: tagger, sentencizer, senter
* Add backwards compat overwrite settings
* Put empty line back
Removed by accident in last commit
* Set backwards-compatible defaults in __init__
Because the `TrainablePipe` serialization methods update `cfg`, there's
no straightforward way to detect whether models serialized with a
previous version are missing the overwrite settings.
It would be possible in the sentencizer due to its separate
serialization methods, however to keep the changes parallel, this also
sets the default in `__init__`.
* Remove traces
Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com>
2021-09-30 13:35:55 +00:00
|
|
|
|
self.overwrite = cfg.get("overwrite", self.overwrite)
|
2020-07-22 11:42:59 +00:00
|
|
|
|
return self
|