mirror of https://github.com/explosion/spaCy.git
284 lines
21 KiB
Markdown
284 lines
21 KiB
Markdown
<a href="https://explosion.ai"><img src="https://explosion.ai/assets/img/logo.svg" width="125" height="125" align="right" /></a>
|
|
|
|
# spaCy: Industrial-strength NLP
|
|
|
|
spaCy is a library for **advanced Natural Language Processing** in Python and
|
|
Cython. It's built on the very latest research, and was designed from day one to
|
|
be used in real products.
|
|
|
|
spaCy comes with [pretrained pipelines](https://spacy.io/models) and currently
|
|
supports tokenization and training for **70+ languages**. It features
|
|
state-of-the-art speed and **neural network models** for tagging, parsing,
|
|
**named entity recognition**, **text classification** and more, multi-task
|
|
learning with pretrained **transformers** like BERT, as well as a
|
|
production-ready [**training system**](https://spacy.io/usage/training) and easy
|
|
model packaging, deployment and workflow management. spaCy is commercial
|
|
open-source software, released under the
|
|
[MIT license](https://github.com/explosion/spaCy/blob/master/LICENSE).
|
|
|
|
💫 **Version 3.6 out now!**
|
|
[Check out the release notes here.](https://github.com/explosion/spaCy/releases)
|
|
|
|
[](https://github.com/explosion/spaCy/actions/workflows/tests.yml)
|
|
[](https://github.com/explosion/spaCy/releases)
|
|
[](https://pypi.org/project/spacy/)
|
|
[](https://anaconda.org/conda-forge/spacy)
|
|
[](https://github.com/explosion/wheelwright/releases)
|
|
[](https://github.com/ambv/black)
|
|
<br />
|
|
[](https://pypi.org/project/spacy/)
|
|
[](https://anaconda.org/conda-forge/spacy)
|
|
[](https://twitter.com/spacy_io)
|
|
|
|
## 📖 Documentation
|
|
|
|
| Documentation | |
|
|
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| ⭐️ **[spaCy 101]** | New to spaCy? Here's everything you need to know! |
|
|
| 📚 **[Usage Guides]** | How to use spaCy and its features. |
|
|
| 🚀 **[New in v3.0]** | New features, backwards incompatibilities and migration guide. |
|
|
| 🪐 **[Project Templates]** | End-to-end workflows you can clone, modify and run. |
|
|
| 🎛 **[API Reference]** | The detailed reference for spaCy's API. |
|
|
| 📦 **[Models]** | Download trained pipelines for spaCy. |
|
|
| 🌌 **[Universe]** | Plugins, extensions, demos and books from the spaCy ecosystem. |
|
|
| ⚙️ **[spaCy VS Code Extension]** | Additional tooling and features for working with spaCy's config files. |
|
|
| 👩🏫 **[Online Course]** | Learn spaCy in this free and interactive online course. |
|
|
| 📺 **[Videos]** | Our YouTube channel with video tutorials, talks and more. |
|
|
| 🛠 **[Changelog]** | Changes and version history. |
|
|
| 💝 **[Contribute]** | How to contribute to the spaCy project and code base. |
|
|
| <a href="https://explosion.ai/spacy-tailored-pipelines"><img src="https://user-images.githubusercontent.com/13643239/152853098-1c761611-ccb0-4ec6-9066-b234552831fe.png" width="125" alt="spaCy Tailored Pipelines"/></a> | Get a custom spaCy pipeline, tailor-made for your NLP problem by spaCy's core developers. Streamlined, production-ready, predictable and maintainable. Start by completing our 5-minute questionnaire to tell us what you need and we'll be in touch! **[Learn more →](https://explosion.ai/spacy-tailored-pipelines)** |
|
|
| <a href="https://explosion.ai/spacy-tailored-analysis"><img src="https://user-images.githubusercontent.com/1019791/206151300-b00cd189-e503-4797-aa1e-1bb6344062c5.png" width="125" alt="spaCy Tailored Pipelines"/></a> | Bespoke advice for problem solving, strategy and analysis for applied NLP projects. Services include data strategy, code reviews, pipeline design and annotation coaching. Curious? Fill in our 5-minute questionnaire to tell us what you need and we'll be in touch! **[Learn more →](https://explosion.ai/spacy-tailored-analysis)** |
|
|
|
|
[spacy 101]: https://spacy.io/usage/spacy-101
|
|
[new in v3.0]: https://spacy.io/usage/v3
|
|
[usage guides]: https://spacy.io/usage/
|
|
[api reference]: https://spacy.io/api/
|
|
[models]: https://spacy.io/models
|
|
[universe]: https://spacy.io/universe
|
|
[spacy vs code extension]: https://github.com/explosion/spacy-vscode
|
|
[videos]: https://www.youtube.com/c/ExplosionAI
|
|
[online course]: https://course.spacy.io
|
|
[project templates]: https://github.com/explosion/projects
|
|
[changelog]: https://spacy.io/usage#changelog
|
|
[contribute]: https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md
|
|
|
|
## 💬 Where to ask questions
|
|
|
|
The spaCy project is maintained by the [spaCy team](https://explosion.ai/about).
|
|
Please understand that we won't be able to provide individual support via email.
|
|
We also believe that help is much more valuable if it's shared publicly, so that
|
|
more people can benefit from it.
|
|
|
|
| Type | Platforms |
|
|
| ------------------------------- | --------------------------------------- |
|
|
| 🚨 **Bug Reports** | [GitHub Issue Tracker] |
|
|
| 🎁 **Feature Requests & Ideas** | [GitHub Discussions] |
|
|
| 👩💻 **Usage Questions** | [GitHub Discussions] · [Stack Overflow] |
|
|
| 🗯 **General Discussion** | [GitHub Discussions] |
|
|
|
|
[github issue tracker]: https://github.com/explosion/spaCy/issues
|
|
[github discussions]: https://github.com/explosion/spaCy/discussions
|
|
[stack overflow]: https://stackoverflow.com/questions/tagged/spacy
|
|
|
|
## Features
|
|
|
|
- Support for **70+ languages**
|
|
- **Trained pipelines** for different languages and tasks
|
|
- Multi-task learning with pretrained **transformers** like BERT
|
|
- Support for pretrained **word vectors** and embeddings
|
|
- State-of-the-art speed
|
|
- Production-ready **training system**
|
|
- Linguistically-motivated **tokenization**
|
|
- Components for named **entity recognition**, part-of-speech-tagging,
|
|
dependency parsing, sentence segmentation, **text classification**,
|
|
lemmatization, morphological analysis, entity linking and more
|
|
- Easily extensible with **custom components** and attributes
|
|
- Support for custom models in **PyTorch**, **TensorFlow** and other frameworks
|
|
- Built in **visualizers** for syntax and NER
|
|
- Easy **model packaging**, deployment and workflow management
|
|
- Robust, rigorously evaluated accuracy
|
|
|
|
📖 **For more details, see the
|
|
[facts, figures and benchmarks](https://spacy.io/usage/facts-figures).**
|
|
|
|
## ⏳ Install spaCy
|
|
|
|
For detailed installation instructions, see the
|
|
[documentation](https://spacy.io/usage).
|
|
|
|
- **Operating system**: macOS / OS X · Linux · Windows (Cygwin, MinGW, Visual
|
|
Studio)
|
|
- **Python version**: Python 3.6+ (only 64 bit)
|
|
- **Package managers**: [pip] · [conda] (via `conda-forge`)
|
|
|
|
[pip]: https://pypi.org/project/spacy/
|
|
[conda]: https://anaconda.org/conda-forge/spacy
|
|
|
|
### pip
|
|
|
|
Using pip, spaCy releases are available as source packages and binary wheels.
|
|
Before you install spaCy and its dependencies, make sure that your `pip`,
|
|
`setuptools` and `wheel` are up to date.
|
|
|
|
```bash
|
|
pip install -U pip setuptools wheel
|
|
pip install spacy
|
|
```
|
|
|
|
To install additional data tables for lemmatization and normalization you can
|
|
run `pip install spacy[lookups]` or install
|
|
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data)
|
|
separately. The lookups package is needed to create blank models with
|
|
lemmatization data, and to lemmatize in languages that don't yet come with
|
|
pretrained models and aren't powered by third-party libraries.
|
|
|
|
When using pip it is generally recommended to install packages in a virtual
|
|
environment to avoid modifying system state:
|
|
|
|
```bash
|
|
python -m venv .env
|
|
source .env/bin/activate
|
|
pip install -U pip setuptools wheel
|
|
pip install spacy
|
|
```
|
|
|
|
### conda
|
|
|
|
You can also install spaCy from `conda` via the `conda-forge` channel. For the
|
|
feedstock including the build recipe and configuration, check out
|
|
[this repository](https://github.com/conda-forge/spacy-feedstock).
|
|
|
|
```bash
|
|
conda install -c conda-forge spacy
|
|
```
|
|
|
|
### Updating spaCy
|
|
|
|
Some updates to spaCy may require downloading new statistical models. If you're
|
|
running spaCy v2.0 or higher, you can use the `validate` command to check if
|
|
your installed models are compatible and if not, print details on how to update
|
|
them:
|
|
|
|
```bash
|
|
pip install -U spacy
|
|
python -m spacy validate
|
|
```
|
|
|
|
If you've trained your own models, keep in mind that your training and runtime
|
|
inputs must match. After updating spaCy, we recommend **retraining your models**
|
|
with the new version.
|
|
|
|
📖 **For details on upgrading from spaCy 2.x to spaCy 3.x, see the
|
|
[migration guide](https://spacy.io/usage/v3#migrating).**
|
|
|
|
## 📦 Download model packages
|
|
|
|
Trained pipelines for spaCy can be installed as **Python packages**. This means
|
|
that they're a component of your application, just like any other module. Models
|
|
can be installed using spaCy's [`download`](https://spacy.io/api/cli#download)
|
|
command, or manually by pointing pip to a path or URL.
|
|
|
|
| Documentation | |
|
|
| -------------------------- | ---------------------------------------------------------------- |
|
|
| **[Available Pipelines]** | Detailed pipeline descriptions, accuracy figures and benchmarks. |
|
|
| **[Models Documentation]** | Detailed usage and installation instructions. |
|
|
| **[Training]** | How to train your own pipelines on your data. |
|
|
|
|
[available pipelines]: https://spacy.io/models
|
|
[models documentation]: https://spacy.io/usage/models
|
|
[training]: https://spacy.io/usage/training
|
|
|
|
```bash
|
|
# Download best-matching version of specific model for your spaCy installation
|
|
python -m spacy download en_core_web_sm
|
|
|
|
# pip install .tar.gz archive or .whl from path or URL
|
|
pip install /Users/you/en_core_web_sm-3.0.0.tar.gz
|
|
pip install /Users/you/en_core_web_sm-3.0.0-py3-none-any.whl
|
|
pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz
|
|
```
|
|
|
|
### Loading and using models
|
|
|
|
To load a model, use [`spacy.load()`](https://spacy.io/api/top-level#spacy.load)
|
|
with the model name or a path to the model data directory.
|
|
|
|
```python
|
|
import spacy
|
|
nlp = spacy.load("en_core_web_sm")
|
|
doc = nlp("This is a sentence.")
|
|
```
|
|
|
|
You can also `import` a model directly via its full name and then call its
|
|
`load()` method with no arguments.
|
|
|
|
```python
|
|
import spacy
|
|
import en_core_web_sm
|
|
|
|
nlp = en_core_web_sm.load()
|
|
doc = nlp("This is a sentence.")
|
|
```
|
|
|
|
📖 **For more info and examples, check out the
|
|
[models documentation](https://spacy.io/docs/usage/models).**
|
|
|
|
## ⚒ Compile from source
|
|
|
|
The other way to install spaCy is to clone its
|
|
[GitHub repository](https://github.com/explosion/spaCy) and build it from
|
|
source. That is the common way if you want to make changes to the code base.
|
|
You'll need to make sure that you have a development environment consisting of a
|
|
Python distribution including header files, a compiler,
|
|
[pip](https://pip.pypa.io/en/latest/installing/),
|
|
[virtualenv](https://virtualenv.pypa.io/en/latest/) and
|
|
[git](https://git-scm.com) installed. The compiler part is the trickiest. How to
|
|
do that depends on your system.
|
|
|
|
| Platform | |
|
|
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| **Ubuntu** | Install system-level dependencies via `apt-get`: `sudo apt-get install build-essential python-dev git` . |
|
|
| **Mac** | Install a recent version of [XCode](https://developer.apple.com/xcode/), including the so-called "Command Line Tools". macOS and OS X ship with Python and git preinstalled. |
|
|
| **Windows** | Install a version of the [Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or [Visual Studio Express](https://visualstudio.microsoft.com/vs/express/) that matches the version that was used to compile your Python interpreter. |
|
|
|
|
For more details and instructions, see the documentation on
|
|
[compiling spaCy from source](https://spacy.io/usage#source) and the
|
|
[quickstart widget](https://spacy.io/usage#section-quickstart) to get the right
|
|
commands for your platform and Python version.
|
|
|
|
```bash
|
|
git clone https://github.com/explosion/spaCy
|
|
cd spaCy
|
|
|
|
python -m venv .env
|
|
source .env/bin/activate
|
|
|
|
# make sure you are using the latest pip
|
|
python -m pip install -U pip setuptools wheel
|
|
|
|
pip install -r requirements.txt
|
|
pip install --no-build-isolation --editable .
|
|
```
|
|
|
|
To install with extras:
|
|
|
|
```bash
|
|
pip install --no-build-isolation --editable .[lookups,cuda102]
|
|
```
|
|
|
|
## 🚦 Run tests
|
|
|
|
spaCy comes with an [extensive test suite](spacy/tests). In order to run the
|
|
tests, you'll usually want to clone the repository and build spaCy from source.
|
|
This will also install the required development dependencies and test utilities
|
|
defined in the [`requirements.txt`](requirements.txt).
|
|
|
|
Alternatively, you can run `pytest` on the tests from within the installed
|
|
`spacy` package. Don't forget to also install the test utilities via spaCy's
|
|
[`requirements.txt`](requirements.txt):
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
python -m pytest --pyargs spacy
|
|
```
|