20 KiB
title | next | menu | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Install spaCy | /usage/models |
|
Quickstart
📖 Looking for the old docs?
To help you make the transition from v2.x to v3.0, we've uploaded the old website to v2.spacy.io. To see what's changed and how to migrate, see the v3.0 guide.
import QuickstartInstall from 'widgets/quickstart-install.js'
Installation instructions
spaCy is compatible with 64-bit CPython 3.6+ and runs on Unix/Linux, macOS/OS X and Windows. The latest spaCy releases are available over pip and conda.
pip
Using pip, spaCy releases are available as source packages and binary wheels.
Before you install spaCy and its dependencies, make sure that your pip
,
setuptools
and wheel
are up to date.
Download pipelines
After installation you typically want to download a trained pipeline. For more info and available packages, see the models directory.
$ python -m spacy download en_core_web_sm >>> import spacy >>> nlp = spacy.load("en_core_web_sm")
$ pip install -U pip setuptools wheel
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
When using pip it is generally recommended to install packages in a virtual environment to avoid modifying system state:
$ python -m venv .env
$ source .env/bin/activate
$ pip install -U pip setuptools wheel
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
spaCy also lets you install extra dependencies by specifying the following
keywords in brackets, e.g. spacy[ja]
or spacy[lookups,transformers]
(with
multiple comma-separated extras). See the [options.extras_require]
section in
spaCy's setup.cfg
for details on what's included.
Example
$ pip install %%SPACY_PKG_NAME[lookups,transformers]%%SPACY_PKG_FLAGS
Name | Description |
---|---|
lookups |
Install spacy-lookups-data for data tables for lemmatization and lexeme normalization. The data is serialized with trained pipelines, so you only need this package if you want to train your own models. |
transformers |
Install spacy-transformers . The package will be installed automatically when you install a transformer-based pipeline. |
ray |
Install spacy-ray to add CLI commands for parallel training. |
cuda , ... |
Install spaCy with GPU support provided by CuPy for your given CUDA version. See the GPU installation instructions for details and options. |
ja , ko , th , zh |
Install additional dependencies required for tokenization for the languages. |
conda
Thanks to our great community, we've been able to re-add conda support. You can
also install spaCy via conda-forge
:
$ conda install -c conda-forge spacy
For the feedstock including the build recipe and configuration, check out this repository. Note that we currently don't publish any pre-releases on conda.
Upgrading spaCy
Upgrading from v2 to v3
Although we've tried to keep breaking changes to a minimum, upgrading from spaCy v2.x to v3.x may still require some changes to your code base. For details see the sections on backwards incompatibilities and migrating. Also remember to download the new trained pipelines, and retrain your own pipelines.
When updating to a newer version of spaCy, it's generally recommended to start with a clean virtual environment. If you're upgrading to a new major version, make sure you have the latest compatible trained pipelines installed, and that there are no old and incompatible packages left over in your environment, as this can often lead to unexpected results and errors. If you've trained your own models, keep in mind that your train and runtime inputs must match. This means you'll have to retrain your pipelines with the new version.
spaCy also provides a validate
command, which lets you
verify that all installed pipeline packages are compatible with your spaCy
version. If incompatible packages are found, tips and installation instructions
are printed. It's recommended to run the command with python -m
to make sure
you're executing the correct version of spaCy.
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
$ python -m spacy validate
Run spaCy with GPU
As of v2.0, spaCy comes with neural network models that are implemented in our machine learning library, Thinc. For GPU support, we've been grateful to use the work of Chainer's CuPy module, which provides a numpy-compatible interface for GPU arrays.
spaCy can be installed on GPU by specifying spacy[cuda]
, spacy[cuda90]
,
spacy[cuda91]
, spacy[cuda92]
, spacy[cuda100]
, spacy[cuda101]
,
spacy[cuda102]
, spacy[cuda110]
or spacy[cuda111]
. If you know your cuda
version, using the more explicit specifier allows cupy to be installed via
wheel, saving some compilation time. The specifiers should install
cupy
.
$ pip install -U %%SPACY_PKG_NAME[cuda92]%%SPACY_PKG_FLAGS
Once you have a GPU-enabled installation, the best way to activate it is to call
spacy.prefer_gpu
or
spacy.require_gpu()
somewhere in your
script before any pipelines have been loaded. require_gpu
will raise an error
if no GPU is available.
import spacy
spacy.prefer_gpu()
nlp = spacy.load("en_core_web_sm")
Compile from source
The other way to install spaCy is to clone its GitHub repository and build it from source. That is the common way if you want to make changes to the code base. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip and git installed. The compiler part is the trickiest. How to do that depends on your system. See notes on Ubuntu, macOS / OS X and Windows for details.
$ python -m pip install -U pip setuptools wheel # install/update build tools
$ git clone https://github.com/explosion/spaCy # clone spaCy
$ cd spaCy # navigate into dir
$ python -m venv .env # create environment in .env
$ source .env/bin/activate # activate virtual env
$ pip install . # compile and install spaCy
To install with extras:
$ pip install .[lookups,cuda102] # install spaCy with extras
To install all dependencies required for development:
$ pip install -r requirements.txt
Compared to a regular install via pip, the
requirements.txt
additionally includes
developer dependencies such as Cython and the libraries required to run the test
suite. See the quickstart widget to get the right commands for
your platform and Python version.
- Ubuntu: Install system-level dependencies via
apt-get
:sudo apt-get install build-essential python-dev git
- macOS / OS X: Install a recent version of XCode, including the so-called "Command Line Tools". macOS and OS X ship with Python and Git preinstalled.
- Windows: Install a version of the Visual C++ Build Tools or Visual Studio Express that matches the version that was used to compile your Python interpreter.
Additional options for developers
Some additional options may be useful for spaCy developers who are editing the source code and recompiling frequently.
-
Install in editable mode. Changes to
.py
files will be reflected as soon as the files are saved, but edits to Cython files (.pxd
,.pyx
) will require thepip install
orpython setup.py build_ext
command below to be run again. Before installing in editable mode, be sure you have removed any previous installs withpip uninstall spacy
, which you may need to run multiple times to remove all traces of earlier installs.$ pip install -r requirements.txt $ pip install --no-build-isolation --editable .
-
Build in parallel using
N
CPUs to speed up compilation and then install in editable mode:$ pip install -r requirements.txt $ python setup.py build_ext --inplace -j N $ pip install --no-build-isolation --editable .
Building an executable
The spaCy repository includes a Makefile
that
builds an executable zip file using pex
(Python Executable). The executable includes spaCy and all its package
dependencies and only requires the system Python at runtime. Building an
executable .pex
file is often the most convenient way to deploy spaCy, as it
lets you separate the build from the deployment process.
Usage
To use a
.pex
file, just replacepython
with the path to the file when you execute your code or CLI commands. This is equivalent to running Python in a virtual environment with spaCy installed.$ ./spacy.pex my_script.py $ ./spacy.pex -m spacy info
$ git clone https://github.com/explosion/spaCy
$ cd spaCy
$ make
You can configure the build process with the following environment variables:
Variable | Description |
---|---|
SPACY_EXTRAS |
Additional Python packages to install alongside spaCy with optional version specifications. Should be a string that can be passed to pip install . See Makefile for defaults. |
PYVER |
The Python version to build against. This version needs to be available on your build and runtime machines. Defaults to 3.6 . |
WHEELHOUSE |
Directory to store the wheel files during compilation. Defaults to ./wheelhouse . |
Run tests
spaCy comes with an extensive test suite. In order
to run the tests, you'll usually want to clone the repository
and build spaCy from source. This will also install the required
development dependencies and test utilities defined in the requirements.txt
.
Alternatively, you can find out where spaCy is installed and run pytest
on
that directory. Don't forget to also install the test utilities via spaCy's
requirements.txt
:
$ python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))"
$ pip install -r path/to/requirements.txt
$ python -m pytest --pyargs %%SPACY_PKG_NAME
Calling pytest
on the spaCy directory will run only the basic tests. The flag
--slow
is optional and enables additional tests that take longer.
$ python -m pip install -U pytest # update pytest
$ python -m pytest --pyargs %%SPACY_PKG_NAME # basic tests
$ python -m pytest --pyargs %%SPACY_PKG_NAME --slow # basic and slow tests
Troubleshooting guide
This section collects some of the most common errors you may come across when installing, loading and using spaCy, as well as their solutions.
Help us improve this guide
Did you come across a problem like the ones listed here and want to share the solution? You can find the "Suggest edits" button at the bottom of this page that points you to the source. We always appreciate pull requests!
No compatible package found for [lang] (spaCy vX.X.X).
This usually means that the trained pipeline you're trying to download does not
exist, or isn't available for your version of spaCy. Check the
compatibility table
to see which packages are available for your spaCy version. If you're using an
old version, consider upgrading to the latest release. Note that while spaCy
supports tokenization for a variety of languages, not
all of them come with trained pipelines. To only use the tokenizer, import the
language's Language
class instead, for example
from spacy.lang.fr import French
.
no such option: --no-cache-dir
The download
command uses pip to install the pipeline packages and sets the
--no-cache-dir
flag to prevent it from requiring too much memory.
This setting
requires pip v6.0 or newer. Run pip install -U pip
to upgrade to the latest
version of pip. To see which version you have installed, run pip --version
.
sre_constants.error: bad character range
In v2.1, spaCy changed its implementation of regular expressions for tokenization to make it up to 2-3 times faster. But this also means that it's very important now that you run spaCy with a wide unicode build of Python. This means that the build has 1114111 unicode characters available, instead of only 65535 in a narrow unicode build. You can check this by running the following command:
$ python -c "import sys; print(sys.maxunicode)"
If you're running a narrow unicode build, reinstall Python and use a wide
unicode build instead. You can also rebuild Python and set the
--enable-unicode=ucs4
flag.
ValueError: unknown locale: UTF-8
This error can sometimes occur on OSX and is likely related to a still
unresolved Python bug. However, it's easy
to fix: just add the following to your ~/.bash_profile
or ~/.zshrc
and then
run source ~/.bash_profile
or source ~/.zshrc
. Make sure to add both
lines for LC_ALL
and LANG
.
$ export LC_ALL=en_US.UTF-8
$ export LANG=en_US.UTF-8
Import Error: No module named spacy
This error means that the spaCy module can't be located on your system, or in
your environment. Make sure you have spaCy installed. If you're using a virtual
environment, make sure it's activated and check that spaCy is installed in that
environment – otherwise, you're trying to load a system installation. You can
also run which python
to find out where your Python executable is located.
ImportError: No module named 'en_core_web_sm'
As of spaCy v1.7, all trained pipelines can be installed as Python packages.
This means that they'll become importable modules of your application. If this
fails, it's usually a sign that the package is not installed in the current
environment. Run pip list
or pip freeze
to check which pipeline packages you
have installed, and install the correct package if necessary. If
you're importing a package manually at the top of a file, make sure to use the
full name of the package.
command not found: spacy
This error may occur when running the spacy
command from the command line.
spaCy does not currently add an entry to your PATH
environment variable, as
this can lead to unexpected results, especially when using a virtual
environment. Instead, spaCy adds an auto-alias that maps spacy
to
python -m spacy
. If this is not working as expected, run the command with
python -m
, yourself – for example python -m spacy download en_core_web_sm
.
For more info on this, see the download
command.
AttributeError: 'module' object has no attribute 'load'
While this could technically have many causes, including spaCy being broken, the
most likely one is that your script's file or directory name is "shadowing" the
module – e.g. your file is called spacy.py
, or a directory you're importing
from is called spacy
. So, when using spaCy, never call anything else spacy
.
If your training data only contained new entities and you didn't mix in any examples the model previously recognized, it can cause the model to "forget" what it had previously learned. This is also referred to as the "catastrophic forgetting problem". A solution is to pre-label some text, and mix it with the new text in your updates. You can also do this by running spaCy over some text, extracting a bunch of entities the model previously recognized correctly, and adding them to your training examples.
TypeError: unhashable type: 'list'
If you're training models, writing them to disk, and versioning them with git,
you might encounter this error when trying to load them in a Windows
environment. This happens because a default install of Git for Windows is
configured to automatically convert Unix-style end-of-line characters (LF) to
Windows-style ones (CRLF) during file checkout (and the reverse when
committing). While that's mostly fine for text files, a trained model written to
disk has some binary files that should not go through this conversion. When they
do, you get the error above. You can fix it by either changing your
core.autocrlf
setting to "false"
, or by committing a
.gitattributes
file to your
repository to tell Git on which files or folders it shouldn't do LF-to-CRLF
conversion, with an entry like path/to/spacy/model/** -text
. After you've done
either of these, clone your repository again.
Changelog
import Changelog from 'widgets/changelog.js'