mirror of https://github.com/explosion/spaCy.git
4448680750
* When checking for token alignments, check not only that the tokens are identical but that the character positions are both at the start of a token. It's possible for the tokens to be identical even though the two tokens aren't aligned one-to-one in a case like `["a'", "''"]` vs. `["a", "''", "'"]`, where the middle tokens are identical but should not be aligned on the token level at character position 2 since it's the start of one token but the middle of another. * Use the lowercased version of the token texts to create the character-to-token alignment because lowercasing can change the string length (e.g., for `İ`, see the not-a-bug bug report: https://bugs.python.org/issue34723) |
||
|---|---|---|
| .. | ||
| converters | ||
| __init__.pxd | ||
| __init__.py | ||
| align.pyx | ||
| alignment.py | ||
| augment.py | ||
| batchers.py | ||
| corpus.py | ||
| example.pxd | ||
| example.pyx | ||
| gold_io.pyx | ||
| initialize.py | ||
| iob_utils.py | ||
| loggers.py | ||
| loop.py | ||
| pretrain.py | ||