2019-03-23 14:45:02 +00:00
|
|
|
|
---
|
|
|
|
|
title: Sentencizer
|
|
|
|
|
tag: class
|
|
|
|
|
source: spacy/pipeline/pipes.pyx
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
A simple pipeline component, to allow custom sentence boundary detection logic
|
|
|
|
|
that doesn't require the dependency parse. By default, sentence segmentation is
|
|
|
|
|
performed by the [`DependencyParser`](/api/dependencyparser), so the
|
|
|
|
|
`Sentencizer` lets you implement a simpler, rule-based strategy that doesn't
|
|
|
|
|
require a statistical model to be loaded. The component is also available via
|
2020-07-26 22:29:45 +00:00
|
|
|
|
the string name `"sentencizer"`.
|
2019-03-23 14:45:02 +00:00
|
|
|
|
|
|
|
|
|
## Sentencizer.\_\_init\_\_ {#init tag="method"}
|
|
|
|
|
|
|
|
|
|
Initialize the sentencizer.
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
2020-07-26 22:29:45 +00:00
|
|
|
|
> # Construction via add_pipe
|
|
|
|
|
> sentencizer = nlp.add_pipe("sentencizer")
|
2019-03-23 14:45:02 +00:00
|
|
|
|
> ```
|
|
|
|
|
|
2020-07-04 12:23:10 +00:00
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ------------- | ------------- | ----------------------------------------------------------------------------------------------- |
|
|
|
|
|
| `punct_chars` | list | Optional custom list of punctuation characters that mark sentence ends. See below for defaults. |
|
|
|
|
|
| **RETURNS** | `Sentencizer` | The newly constructed object. |
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
### punct_chars defaults
|
|
|
|
|
['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።',
|
|
|
|
|
'፧', '፨', '᙮', '᜵', '᜶', '᠃', '᠉', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫',
|
|
|
|
|
'᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉',
|
|
|
|
|
'⸮', '⸼', '꓿', '꘎', '꘏', '꛳', '꛷', '꡶', '꡷', '꣎', '꣏', '꤯', '꧈',
|
|
|
|
|
'꧉', '꩝', '꩞', '꩟', '꫰', '꫱', '꯫', '﹒', '﹖', '﹗', '!', '.', '?',
|
|
|
|
|
'𐩖', '𐩗', '𑁇', '𑁈', '𑂾', '𑂿', '𑃀', '𑃁', '𑅁', '𑅂', '𑅃', '𑇅',
|
|
|
|
|
'𑇆', '𑇍', '𑇞', '𑇟', '𑈸', '𑈹', '𑈻', '𑈼', '𑊩', '𑑋', '𑑌', '𑗂',
|
|
|
|
|
'𑗃', '𑗉', '𑗊', '𑗋', '𑗌', '𑗍', '𑗎', '𑗏', '𑗐', '𑗑', '𑗒', '𑗓',
|
|
|
|
|
'𑗔', '𑗕', '𑗖', '𑗗', '𑙁', '𑙂', '𑜼', '𑜽', '𑜾', '𑩂', '𑩃', '𑪛',
|
|
|
|
|
'𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '。', '。']
|
|
|
|
|
```
|
2019-03-23 14:45:02 +00:00
|
|
|
|
|
|
|
|
|
## Sentencizer.\_\_call\_\_ {#call tag="method"}
|
|
|
|
|
|
|
|
|
|
Apply the sentencizer on a `Doc`. Typically, this happens automatically after
|
|
|
|
|
the component has been added to the pipeline using
|
|
|
|
|
[`nlp.add_pipe`](/api/language#add_pipe).
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
|
|
|
|
> from spacy.lang.en import English
|
|
|
|
|
>
|
|
|
|
|
> nlp = English()
|
2020-07-26 22:29:45 +00:00
|
|
|
|
> nlp.add_pipe("sentencizer")
|
2019-09-12 14:11:15 +00:00
|
|
|
|
> doc = nlp("This is a sentence. This is another sentence.")
|
2019-11-13 14:24:14 +00:00
|
|
|
|
> assert len(list(doc.sents)) == 2
|
2019-03-23 14:45:02 +00:00
|
|
|
|
> ```
|
|
|
|
|
|
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ----------- | ----- | ------------------------------------------------------------ |
|
|
|
|
|
| `doc` | `Doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. |
|
|
|
|
|
| **RETURNS** | `Doc` | The modified `Doc` with added sentence boundaries. |
|
|
|
|
|
|
|
|
|
|
## Sentencizer.to_disk {#to_disk tag="method"}
|
|
|
|
|
|
|
|
|
|
Save the sentencizer settings (punctuation characters) a directory. Will create
|
|
|
|
|
a file `sentencizer.json`. This also happens automatically when you save an
|
|
|
|
|
`nlp` object with a sentencizer added to its pipeline.
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
|
|
|
|
> sentencizer = Sentencizer(punct_chars=[".", "?", "!", "。"])
|
|
|
|
|
> sentencizer.to_disk("/path/to/sentencizer.jsonl")
|
|
|
|
|
> ```
|
|
|
|
|
|
2020-05-24 15:23:00 +00:00
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ------ | ------------ | ---------------------------------------------------------------------------------------------------------------- |
|
|
|
|
|
| `path` | str / `Path` | A path to a file, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. |
|
2019-03-23 14:45:02 +00:00
|
|
|
|
|
|
|
|
|
## Sentencizer.from_disk {#from_disk tag="method"}
|
|
|
|
|
|
|
|
|
|
Load the sentencizer settings from a file. Expects a JSON file. This also
|
|
|
|
|
happens automatically when you load an `nlp` object or model with a sentencizer
|
|
|
|
|
added to its pipeline.
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
|
|
|
|
> sentencizer = Sentencizer()
|
|
|
|
|
> sentencizer.from_disk("/path/to/sentencizer.json")
|
|
|
|
|
> ```
|
|
|
|
|
|
2020-05-24 15:23:00 +00:00
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ----------- | ------------- | -------------------------------------------------------------------------- |
|
|
|
|
|
| `path` | str / `Path` | A path to a JSON file. Paths may be either strings or `Path`-like objects. |
|
|
|
|
|
| **RETURNS** | `Sentencizer` | The modified `Sentencizer` object. |
|
2019-03-23 14:45:02 +00:00
|
|
|
|
|
|
|
|
|
## Sentencizer.to_bytes {#to_bytes tag="method"}
|
|
|
|
|
|
|
|
|
|
Serialize the sentencizer settings to a bytestring.
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
|
|
|
|
> sentencizer = Sentencizer(punct_chars=[".", "?", "!", "。"])
|
|
|
|
|
> sentencizer_bytes = sentencizer.to_bytes()
|
|
|
|
|
> ```
|
|
|
|
|
|
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ----------- | ----- | -------------------- |
|
|
|
|
|
| **RETURNS** | bytes | The serialized data. |
|
|
|
|
|
|
|
|
|
|
## Sentencizer.from_bytes {#from_bytes tag="method"}
|
|
|
|
|
|
|
|
|
|
Load the pipe from a bytestring. Modifies the object in place and returns it.
|
|
|
|
|
|
|
|
|
|
> #### Example
|
|
|
|
|
>
|
|
|
|
|
> ```python
|
|
|
|
|
> sentencizer_bytes = sentencizer.to_bytes()
|
|
|
|
|
> sentencizer = Sentencizer()
|
|
|
|
|
> sentencizer.from_bytes(sentencizer_bytes)
|
|
|
|
|
> ```
|
|
|
|
|
|
|
|
|
|
| Name | Type | Description |
|
|
|
|
|
| ------------ | ------------- | ---------------------------------- |
|
|
|
|
|
| `bytes_data` | bytes | The bytestring to load. |
|
|
|
|
|
| **RETURNS** | `Sentencizer` | The modified `Sentencizer` object. |
|