mirror of https://github.com/explosion/spaCy.git
StringStore/Vocab dev docs (#9142)
* First take at StringStore/Vocab docs Things to check: 1. The mysterious vocab members 2. How to make table of contents? Is it autogenerated? 3. Anything I missed / needs more detail? * Update docs * Apply suggestions from code review Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Updates based on review feedback * Minor fix * Move example code down Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
This commit is contained in:
Paul O'Leary McCann
GitHub
parent
20f63e7154
commit
9ceb8f413c
|
|
@ -0,0 +1,216 @@
|
|||
# StringStore & Vocab
|
||||
|
||||
> Reference: `spacy/strings.pyx`
|
||||
> Reference: `spacy/vocab.pyx`
|
||||
|
||||
## Overview
|
||||
|
||||
spaCy represents mosts strings internally using a `uint64` in Cython which
|
||||
corresponds to a hash. The magic required to make this largely transparent is
|
||||
handled by the `StringStore`, and is integrated into the pipelines using the
|
||||
`Vocab`, which also connects it to some other information.
|
||||
|
||||
These are mostly internal details that average library users should never have
|
||||
to think about. On the other hand, when developing a component it's normal to
|
||||
interact with the Vocab for lexeme data or word vectors, and it's not unusual
|
||||
to add labels to the `StringStore`.
|
||||
|
||||
## StringStore
|
||||
|
||||
### Overview
|
||||
|
||||
The `StringStore` is a `cdef class` that looks a bit like a two-way dictionary,
|
||||
though it is not a subclass of anything in particular.
|
||||
|
||||
The main functionality of the `StringStore` is that `__getitem__` converts
|
||||
hashes into strings or strings into hashes.
|
||||
|
||||
The full details of the conversion are complicated. Normally you shouldn't have
|
||||
to worry about them, but the first applicable case here is used to get the
|
||||
return value:
|
||||
|
||||
1. 0 and the empty string are special cased to each other
|
||||
2. internal symbols use a lookup table (`SYMBOLS_BY_STR`)
|
||||
3. normal strings or bytes are hashed
|
||||
4. internal symbol IDs in `SYMBOLS_BY_INT` are handled
|
||||
5. anything not yet handled is used as a hash to lookup a string
|
||||
|
||||
For the symbol enums, see [`symbols.pxd`](https://github.com/explosion/spaCy/blob/master/spacy/symbols.pxd).
|
||||
|
||||
Almost all strings in spaCy are stored in the `StringStore`. This naturally
|
||||
includes tokens, but also includes things like labels (not just NER/POS/dep,
|
||||
but also categories etc.), lemmas, lowercase forms, word shapes, and so on. One
|
||||
of the main results of this is that tokens can be represented by a compact C
|
||||
struct ([`LexemeC`](https://spacy.io/api/cython-structs#lexemec)/[`TokenC`](https://github.com/explosion/spaCy/issues/4854)) that mostly consists of string hashes. This also means that converting
|
||||
input for the models is straightforward, and there's not a token mapping step
|
||||
like in many machine learning frameworks. Additionally, because the token IDs
|
||||
in spaCy are based on hashes, they are consistent across environments or
|
||||
models.
|
||||
|
||||
One pattern you'll see a lot in spaCy APIs is that `something.value` returns an
|
||||
`int` and `something.value_` returns a string. That's implemented using the
|
||||
`StringStore`. Typically the `int` is stored in a C struct and the string is
|
||||
generated via a property that calls into the `StringStore` with the `int`.
|
||||
|
||||
Besides `__getitem__`, the `StringStore` has functions to return specifically a
|
||||
string or specifically a hash, regardless of whether the input was a string or
|
||||
hash to begin with, though these are only used occasionally.
|
||||
|
||||
### Implementation Details: Hashes and Allocations
|
||||
|
||||
Hashes are 64-bit and are computed using [murmurhash][] on UTF-8 bytes. There is no
|
||||
mechanism for detecting and avoiding collisions. To date there has never been a
|
||||
reproducible collision or user report about any related issues.
|
||||
|
||||
[murmurhash]: https://github.com/explosion/murmurhash
|
||||
|
||||
The empty string is not hashed, it's just converted to/from 0.
|
||||
|
||||
A small number of strings use indices into a lookup table (so low integers)
|
||||
rather than hashes. This is mostly Universal Dependencies labels or other
|
||||
strings considered "core" in spaCy. This was critical in v1, which hadn't
|
||||
introduced hashing yet. Since v2 it's important for items in `spacy.attrs`,
|
||||
especially lexeme flags, but is otherwise only maintained for backwards
|
||||
compatibility.
|
||||
|
||||
You can call `strings["mystring"]` with a string the `StringStore` has never seen
|
||||
before and it will return a hash. But in order to do the reverse operation, you
|
||||
need to call `strings.add("mystring")` first. Without a call to `add` the
|
||||
string will not be interned.
|
||||
|
||||
Example:
|
||||
|
||||
```
|
||||
from spacy.strings import StringStore
|
||||
|
||||
ss = StringStore()
|
||||
hashval = ss["spacy"] # 10639093010105930009
|
||||
try:
|
||||
# this won't work
|
||||
ss[hashval]
|
||||
except KeyError:
|
||||
print(f"key {hashval} unknown in the StringStore.")
|
||||
|
||||
ss.add("spacy")
|
||||
assert ss[hashval] == "spacy" # it works now
|
||||
|
||||
# There is no `.keys` property, but you can iterate over keys
|
||||
# The empty string will never be in the list of keys
|
||||
for key in ss:
|
||||
print(key)
|
||||
```
|
||||
|
||||
In normal use nothing is ever removed from the `StringStore`. In theory this
|
||||
means that if you do something like iterate through all hex values of a certain
|
||||
length you can have explosive memory usage. In practice this has never been an
|
||||
issue. (Note that this is also different from using `sys.intern` to intern
|
||||
Python strings, which does not guarantee they won't be garbage collected later.)
|
||||
|
||||
Strings are stored in the `StringStore` in a peculiar way: each string uses a
|
||||
union that is either an eight-byte `char[]` or a `char*`. Short strings are
|
||||
stored directly in the `char[]`, while longer strings are stored in allocated
|
||||
memory and prefixed with their length. This is a strategy to reduce indirection
|
||||
and memory fragmentation. See `decode_Utf8Str` and `_allocate` in
|
||||
`strings.pyx` for the implementation.
|
||||
|
||||
### When to Use the StringStore?
|
||||
|
||||
While you can ignore the `StringStore` in many cases, there are situations where
|
||||
you should make use of it to avoid errors.
|
||||
|
||||
Any time you introduce a string that may be set on a `Doc` field that has a hash,
|
||||
you should add the string to the `StringStore`. This mainly happens when adding
|
||||
labels in components, but there are some other cases:
|
||||
|
||||
- syntax iterators, mainly `get_noun_chunks`
|
||||
- external data used in components, like the `KnowledgeBase` in the `entity_linker`
|
||||
- labels used in tests
|
||||
|
||||
## Vocab
|
||||
|
||||
The `Vocab` is a core component of a `Language` pipeline. Its main function is
|
||||
to manage `Lexeme`s, which are structs that contain information about a token
|
||||
that depends only on its surface form, without context. `Lexeme`s store much of
|
||||
the data associated with `Token`s. As a side effect of this the `Vocab` also
|
||||
manages the `StringStore` for a pipeline and a grab-bag of other data.
|
||||
|
||||
These are things stored in the vocab:
|
||||
|
||||
- `Lexeme`s
|
||||
- `StringStore`
|
||||
- `Morphology`: manages info used in `MorphAnalysis` objects
|
||||
- `vectors`: basically a dict for word vectors
|
||||
- `lookups`: language specific data like lemmas
|
||||
- `writing_system`: language specific metadata
|
||||
- `get_noun_chunks`: a syntax iterator
|
||||
- lex attribute getters: functions like `is_punct`, set in language defaults
|
||||
- `cfg`: **not** the pipeline config, this is mostly unused
|
||||
- `_unused_object`: Formerly an unused object, kept around until v4 for compatability
|
||||
|
||||
Some of these, like the Morphology and Vectors, are complex enough that they
|
||||
need their own explanations. Here we'll just look at Vocab-specific items.
|
||||
|
||||
### Lexemes
|
||||
|
||||
A `Lexeme` is a type that mainly wraps a `LexemeC`, a struct consisting of ints
|
||||
that identify various context-free token attributes. Lexemes are the core data
|
||||
of the `Vocab`, and can be accessed using `__getitem__` on the `Vocab`. The memory
|
||||
for storing `LexemeC` objects is managed by a pool that belongs to the `Vocab`.
|
||||
|
||||
Note that `__getitem__` on the `Vocab` works much like the `StringStore`, in
|
||||
that it accepts a hash or id, with one important difference: if you do a lookup
|
||||
using a string, that value is added to the `StringStore` automatically.
|
||||
|
||||
The attributes stored in a `LexemeC` are:
|
||||
|
||||
- orth (the raw text)
|
||||
- lower
|
||||
- norm
|
||||
- shape
|
||||
- prefix
|
||||
- suffix
|
||||
|
||||
Most of these are straightforward. All of them can be customized, and (except
|
||||
`orth`) probably should be since the defaults are based on English, but in
|
||||
practice this is rarely done at present.
|
||||
|
||||
### Lookups
|
||||
|
||||
This is basically a dict of dicts, implemented using a `Table` for each
|
||||
sub-dict, that stores lemmas and other language-specific lookup data.
|
||||
|
||||
A `Table` is a subclass of `OrderedDict` used for string-to-string data. It uses
|
||||
Bloom filters to speed up misses and has some extra serialization features.
|
||||
Tables are not used outside of the lookups.
|
||||
|
||||
### Lex Attribute Getters
|
||||
|
||||
Lexical Attribute Getters like `is_punct` are defined on a per-language basis,
|
||||
much like lookups, but take the form of functions rather than string-to-string
|
||||
dicts, so they're stored separately.
|
||||
|
||||
### Writing System
|
||||
|
||||
This is a dict with three attributes:
|
||||
|
||||
- `direction`: ltr or rtl (default ltr)
|
||||
- `has_case`: bool (default `True`)
|
||||
- `has_letters`: bool (default `True`, `False` only for CJK for now)
|
||||
|
||||
Currently these are not used much - the main use is that `direction` is used in
|
||||
visualizers, though `rtl` doesn't quite work (see
|
||||
[#4854](https://github.com/explosion/spaCy/issues/4854)). In the future they
|
||||
could be used when choosing hyperparameters for subwords, controlling word
|
||||
shape generation, and similar tasks.
|
||||
|
||||
### Other Vocab Members
|
||||
|
||||
The Vocab is kind of the default place to store things from `Language.defaults`
|
||||
that don't belong to the Tokenizer. The following properties are in the Vocab
|
||||
just because they don't have anywhere else to go.
|
||||
|
||||
- `get_noun_chunks`
|
||||
- `cfg`: This is a dict that just stores `oov_prob` (hardcoded to `-20`)
|
||||
- `_unused_object`: Leftover C member, should be removed in next major version
|
||||
|
||||
|
||||
Loading…
Reference in New Issue