2020-07-04 12:23:10 +00:00
---
title: Corpus
teaser: An annotated corpus
tag: class
source: spacy/gold/corpus.py
new: 3
---
This class manages annotated corpora and can read training and development
datasets in the [DocBin ](/api/docbin ) (`.spacy`) format.
## Corpus.\_\_init\_\_ {#init tag="method"}
Create a `Corpus` . The input data can be a file or a directory of files.
2020-07-29 09:36:42 +00:00
> #### Example
>
> ```python
> from spacy.gold import Corpus
>
> corpus = Corpus("./train.spacy", "./dev.spacy")
> ```
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
| Name | Type | Description |
| ------- | ------------ | ---------------------------------------------------------------- |
| `train` | str / `Path` | Training data (`.spacy` file or directory of `.spacy` files). |
| `dev` | str / `Path` | Development data (`.spacy` file or directory of `.spacy` files). |
| `limit` | int | Maximum number of examples returned. `0` for no limit (default). |
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
## Corpus.train_dataset {#train_dataset tag="method"}
Yield examples from the training data.
> #### Example
>
> ```python
> from spacy.gold import Corpus
> import spacy
>
> corpus = Corpus("./train.spacy", "./dev.spacy")
> nlp = spacy.blank("en")
> train_data = corpus.train_dataset(nlp)
> ```
| Name | Type | Description |
| -------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `nlp` | `Language` | The current `nlp` object. |
| _keyword-only_ | | |
| `shuffle` | bool | Whether to shuffle the examples. Defaults to `True` . |
| `gold_preproc` | bool | Whether to train on gold-standard sentences and tokens. Defaults to `False` . |
| `max_length` | int | Maximum document length. Longer documents will be split into sentences, if sentence boundaries are available. `0` for no limit (default).
| **YIELDS** | `Example` | The examples. |
## Corpus.dev_dataset {#dev_dataset tag="method"}
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
Yield examples from the development data.
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
> #### Example
>
> ```python
> from spacy.gold import Corpus
> import spacy
>
> corpus = Corpus("./train.spacy", "./dev.spacy")
> nlp = spacy.blank("en")
> dev_data = corpus.dev_dataset(nlp)
> ```
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
| Name | Type | Description |
| -------------- | ---------- | ---------------------------------------------------------------------------- |
| `nlp` | `Language` | The current `nlp` object. |
| _keyword-only_ | | |
| `gold_preproc` | bool | Whether to train on gold-standard sentences and tokens. Defaults to `False` . |
| **YIELDS** | `Example` | The examples. |
2020-07-04 12:23:10 +00:00
## Corpus.count_train {#count_train tag="method"}
2020-07-29 09:36:42 +00:00
Get the word count of all training examples.
2020-07-04 12:23:10 +00:00
2020-07-29 09:36:42 +00:00
> #### Example
>
> ```python
> from spacy.gold import Corpus
> import spacy
>
> corpus = Corpus("./train.spacy", "./dev.spacy")
> nlp = spacy.blank("en")
> word_count = corpus.count_train(nlp)
> ```
| Name | Type | Description |
| ----------- | ---------- | ------------------------- |
| `nlp` | `Language` | The current `nlp` object. |
| **RETURNS** | int | The word count. |
<!-- TODO: document remaining methods? / decide which to document -->
